= User Documentation =

== Submission scripts for clusters ==

Real life simulations are usually launched on HPC clusters through a [http://en.wikipedia.org/wiki/Job_scheduler job scheduler].
Each cluster has it's own set of constraints, resources, specificities etc.. that we cannot cover in this documentation. 

But what we can do is to provide a set of "copy paste" script that you might want to use as a template to write your own.

In the remainder, we assume that we are using [http://www.open-mpi.org/ Open MPI], and we will use explicit path location of the distribution. This is probably a safe decision if you have more than one distribution and/or if it is not in a "default" location.
To know where you default MPI distribution is installed, you can run:

{{{
$ which mpiexec
/softs/openmpi-1.6.3-intel-13/bin/mpiexec
$ 
}}}
and use you own MPI distribution location (here '''/softs/openmpi-1.6.3-intel-13/''').

We assume a Unix-like distribution and we use the [http://en.wikipedia.org/wiki/Bash_%28Unix_shell%29 bash shell].

We assume that our default environment (which defines required environment variables like '''LD_LIBRARY_PATH''') is set in '''/softs/env_default.sh'''. In the following scripts, you should replace '''source /softs/env_default.sh''' with the setting of your own environment.

 
=== OAR ===

==== Naive version ====

To launch a '''cubby''' job with 64 MPI processes through [http://oar.imag.fr/ OAR], one could use the following ''cubby.oar'' script:

{{{
#!/usr/bin/env bash 
#OAR -n cubby64 
#OAR -l core=64,walltime=1:0:0   
 
source /softs/env_default.sh
                        
/softs/openmpi-1.6.3-intel-13/bin/mpiexec -x LD_LIBRARY_PATH -machinefile $OAR_FILE_NODES ./cubby --cube-dim 128
}}}

The script must be executable:
{{{
$ ls -l ./cubby.oar
-rwxr-xr-x 1 alainm sit 212 Dec  5 14:34 ./cubby.oar
$
}}}

It can be submitted in the default queue with:
{{{
$ oarsub -S ./cubby.oar
[ADMISSION RULE] Modify resource description with type constraints
OAR_JOB_ID=3493
$ 
}}}

==== Job placement ====

With the previous script, the job will be dispatched on 64 core that will be arbitrarily choosen on the cluster. This is not what you usually want for real simulations.

Here, we assume that our cluster is built of nodes with 2 cpu, each cpu providing 4 cores.

 *To regroup our MPI processes into 4 processes per cpu (one per core):
{{{
#!/usr/bin/env bash 
#OAR -n cubby64 
#OAR -l /cpu=16/core=4,walltime=1:0:0   
 
source /softs/env_default.sh
                        
/softs/openmpi-1.6.3-intel-13/bin/mpiexec -x LD_LIBRARY_PATH -machinefile $OAR_FILE_NODES ./cubby --cube-dim 128
}}}
that is: allocate 16 cpus, and allocate 4 cores per cpu.
 *Same thing, but we want to compact the cpu on the nodes:
{{{
#!/usr/bin/env bash 
#OAR -n cubby64 
#OAR -l /nodes=2/cpu=8/core=4,walltime=1:0:0   
 
source /softs/env_default.sh
                        
/softs/openmpi-1.6.3-intel-13/bin/mpiexec -x LD_LIBRARY_PATH -machinefile $OAR_FILE_NODES ./cubby --cube-dim 128
}}}