wiki:UserDocumentation

User Documentation

Submission scripts for clusters

Real life simulations are usually launched on HPC clusters through a job scheduler. Each cluster has it's own set of constraints, resources, specificities etc.. that we cannot cover in this documentation.

But what we can do is to provide a set of "copy paste" script that you might want to use as a template to write your own.

In the remainder, we assume that we are using Open MPI, and we will use explicit path location of the distribution. This is probably a safe decision if you have more than one distribution and/or if it is not in a "default" location. To know where you default MPI distribution is installed, you can run:

$ which mpiexec
/softs/openmpi-1.6.3-intel-13/bin/mpiexec
$ 

and use you own MPI distribution location (here /softs/openmpi-1.6.3-intel-13/).

We assume a Unix-like distribution and we use the bash shell.

We assume that our default environment (which defines required environment variables like LD_LIBRARY_PATH) is set in /softs/env_default.sh. In the following scripts, you should replace source /softs/env_default.sh with the setting of your own environment.

OAR

Naive version

To launch a cubby job with 64 MPI processes through OAR, one could use the following cubby.oar script:

#!/usr/bin/env bash 
#OAR -n cubby64 
#OAR -l core=64,walltime=1:0:0   
 
source /softs/env_default.sh
                        
/softs/openmpi-1.6.3-intel-13/bin/mpiexec -x LD_LIBRARY_PATH -machinefile $OAR_FILE_NODES ./cubby --cube-dim 128

The script must be executable:

$ ls -l ./cubby.oar
-rwxr-xr-x 1 alainm sit 212 Dec  5 14:34 ./cubby.oar
$

It can be submitted in the default queue with:

$ oarsub -S ./cubby.oar
[ADMISSION RULE] Modify resource description with type constraints
OAR_JOB_ID=3493
$ 

Job placement

With the previous script, the job will be dispatched on 64 core that will be arbitrarily choosen on the cluster. This is not what you usually want for real simulations.

Here, we assume that our cluster is built of nodes with 2 cpu, each cpu providing 4 cores.

*To regroup our MPI processes into 4 processes per cpu (one per core):

#!/usr/bin/env bash 
#OAR -n cubby64 
#OAR -l /cpu=16/core=4,walltime=1:0:0   
 
source /softs/env_default.sh
                        
/softs/openmpi-1.6.3-intel-13/bin/mpiexec -x LD_LIBRARY_PATH -machinefile $OAR_FILE_NODES ./cubby --cube-dim 128

that is: allocate 16 cpus, and allocate 4 cores per cpu.

*Same thing, but we want to compact the cpu on the nodes:

#!/usr/bin/env bash 
#OAR -n cubby64 
#OAR -l /nodes=2/cpu=8/core=4,walltime=1:0:0   
 
source /softs/env_default.sh
                        
/softs/openmpi-1.6.3-intel-13/bin/mpiexec -x LD_LIBRARY_PATH -machinefile $OAR_FILE_NODES ./cubby --cube-dim 128
Last modified 8 years ago Last modified on Dec 5, 2012 2:48:15 PM