wiki:DevBuildInfrastructure

Requirements

Source code
We assume you have a copy of our source code, either as a tarball or directly from our SubVersioN server. We assume that the code has been deployed in the <srcdir> directory, which should contain the preconfigure.sh file:
cubby$ ls -l <srcdir>/preconfigure.sh 
-rwxr-xr-x 1 alainm sit 3115 2010-03-27 02:09 <srcdir>/preconfigure.sh
cubby$ 
MPI
You need a MPI environment on your machine. The MPI C++ wrapper (named mpic++, mpicxx or mpiCC) should be accessible through your path. OpenMPI, from 1.3.2 to 1.4.0, have been tested.
FFTW
We strongly recommend that you use the FFTW implementation for the Fast Fourier Transform. Although we provide a default implementation that might work. Right now we are using the 3.2 version but, as far as we now, there are no dependencies on the library version. We assume that either your library is in a default location or in the <fftwdir> directory.
Boost
We use various components from the boost library. We advise using a post 1.40.0 version and support up to 1.42.0 included. Note that we use the MPI component from boost and that this component must be explicitly activated we building Boost. Also, don't forget to edit the boost/mpi/config.hpp directory, and specially to set the BOOST_MPI_HOMOGENEOUS value that match you configuration. Finally, use the same MPI environment as above (See this page for more details). As for compiled libraries (most of Boost is in header files only) we only use the followin components: serialization, date_time, mpi and program_options. So if something else does not build, that's probably not an issue. We assume that your Boost distribution is either in a default location or in the <boostdir> directory.
CMake
The use CMake as a build environment, we have successfully built with versions 2.6 and above.

Generating the build system

Before you can build, you need to generate the build system with CMake. We support both in-source and out-of-source (preferred) build.

Most common situations

Here, we provide the one-liner for generating the build system from the command line. This can be done through a fancy GUI application, for sissies people who like this kind of things. I don't do GUI.

All the dependencies are in a default location

Assuming that all the components are in a place that you compiler can find with no extra option (on unix/linux, that's typically under /usr/ or /usr/local, the only thing you need to take care of is the build mode, which is controlled by the CMAKE_BUILD_TYPE variable:

For a release build:

$ cd <srcdir>
$ cmake -DCMAKE_BUILD_TYPE:STRING=Release
-- The CXX compiler identification is ....
      ...
- Configuring done
-- Generating done
-- Build files have been written to: <srcdir>
$

All dependencies are NOT in a default location.…

Assuming that the various needed components are not in a default location, and considering the above notation for those components, the build system can be generated that way:

$cmake -DBOOST_ROOT:String='<boostdir>' -DFFTW_ROOT:String='<fftwdir>' -DCMAKE_BUILD_TYPE:STRING=Release
-- The CXX compiler identification is ....
      ...
- Configuring done
-- Generating done
-- Build files have been written to: <srcdir>
$

... nor is the MPI environement

You need to first setup your MPI environment so that the mpi wrapper can be found before you genrate the build environement as before:

$ export PATH=<mpidir>/bin:$PATH
$ which mpicxx
<mpidir>/mpicxx
$cmake -DBOOST_ROOT:String='<boostdir>' -DFFTW_ROOT:String='<fftwdir>' -DCMAKE_BUILD_TYPE:STRING=Release
-- The CXX compiler identification is ....
      ...
- Configuring done
-- Generating done
-- Build files have been written to: <srcdir>
$

Remember that this must be the same MPI as the one used when building ''Boost''.

More on build system configuration

The CMakeCache.txt file

Once the build system has been generated, all the configuration variables are stored in a file named CMakeCache.txt. You can edit that file to update all the values that you would otherwise set through the cmake call. You will usually find some documentation in there (like the list of possible values for each variable).

The cmake-gui interface

Also, you can update all that stuff throught the cmake-gui application to edit and set those values.

Special compilation flags

The following is far from exhaustive.

Compilation flags
Additional compilation flags can be specified through the CMAKE_CXX_FLAGS and CMAKE_C_FLAGS variables.
Special compilers
You can explicitly select a specific compiler through the CMAKE_CXX_COMPILER and CMAKE_C_COMPILER variables.

The following example select specific compilers and flags:

$cmake -DBOOST_ROOT:String='/opt/boost/1.42.0/intel' -DFFTW_ROOT:String='/opt/fftw-3.2/gnu/' -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_CXX_COMPILER:FILEPATH=/opt/intel/Compiler/11.1/069/bin/ia32/icpc -DCMAKE_C_COMPILER:FILEPATH=/opt/intel/Compiler/11.1/069/bin/ia32/icc -DCMAKE_CXX_FLAGS:STRING='-gcc-name=gcc-4.2 -gxx-name=g++-4.2 -Wno-unknown-pragmas -wd279 -wd654' -DCMAKE_C_FLAGS:STRING='-gcc-name=gcc-4.
2 -gxx-name=g++-4.2 -Wno-unknown-pragmas '

Troubleshooting

  • Intel 11.1 compiler does not play nice with some gcc installation. The problem show up like that:
    [ 42%] Building CXX object libcubby/CMakeFiles/cubby_field.dir/cubby/field/field_map.cpp.o
    /usr/include/c++/4.4.1/cmath(500): error: identifier "__builtin_fpclassify" is undefined
            return __builtin_fpclassify(FP_NAN, FP_INFINITE, FP_NORMAL,
                   ^
              detected during:
    ...
    
    to get rid of that problem, you need to explicitly select another gcc version (which need to be installed) with the following compilation options -gcc-name=gcc-4.2 -gxx-name=g++-4.2. We use the folloing configuration wich also get rid of some annoying and, in IMHO, unjustified warnings:
    $cmake ...... -DCMAKE_CXX_COMPILER:FILEPATH=/opt/intel/Compiler/11.1/069/bin/ia32/icpc -DCMAKE_C_COMPILER:FILEPATH=/opt/intel/Compiler/11.1/069/bin/ia32/icc -DCMAKE_CXX_FLAGS:STRING='-gcc-name=gcc-4.2 -gxx-name=g++-4.2 -Wno-unknown-pragmas -wd279 -wd654' -DCMAKE_C_FLAGS:STRING='-gcc-name=gcc-4.
    2 -gxx-name=g++-4.2 -Wno-unknown-pragmas '
    

Building

Just enter the make command, depending on you cmake version, it should produce something like:

$make
[  1%] Building CXX object libcubby/CMakeFiles/oca_util.dir/oca/util/timer.cpp.o
Linking CXX static library liboca_util.a
[  1%] Built target oca_util
[  2%] Generating oca/array/op.hpp
[  2%] Building CXX object libcubby/CMakeFiles/oca_array.dir/oca/array/array.cpp.o
Linking CXX static library liboca_array.a
[  2%] Built target oca_array
...
Linking C static library libjmfftc.a
...
Linking CXX static library libcubby_field.a
...
Linking CXX static library libcubby_physic.a
...
Linking CXX executable cubby
...
Linking CXX executable mulass_lp_C1_div_V1_rp_minus_A2_4d
[100%] Built target mulass_lp_C1_div_V1_rp_minus_A2_4d
$

Note that parallel build is supported, excepted for the first run (some problem with generated files).

Testing

MPI tests

Some (most) tests are mpi applications, if you machine has more than one core, you can use that to speed up the testing by setting the NBCORES environment variable. Otherwise, the MPI test will be run with 2 processes (the number of processes does not need to be equal to the numer of cores though).

The number of available cores should be detected by the [preconfigure.sh script. If that's not the case (the detection being platform specific) or if the detected number does not suists you, you can use the -n <number of processes> option of the script:

alainm@vai:~/views/cubby/trunk/rel$ ../preconfigure.sh -n 4
...
-- NBCORES provided: 4
...
alainm@vai:~/views/cubby/trunk/rel$ 

Otherwise, this variable can be set manually:

alainm@satch:~/views/codes/cubby/branches/work$ export NBCORES=4
alainm@satch:~/views/codes/cubby/branches/work$ ctest 
Start processing tests
Test project /home/alainm/views/codes/cubby/branches/work
  1/ 47 Testing bug20                            Passed
  2/ 47 Testing bug32                            Passed
  .....
 47/ 47 Testing hydro_penalization               Passed

100% tests passed, 0 tests failed out of 47
alainm@satch:~/views/codes/cubby/branches/work$ 

Note that due to some compiler (mostly Intel) limitations, some platform might have a smaller number of test.

Running the tests

The test are run through the ctest command which provided by the cmake distribution and has its own manual.

running test in sequential mode

On all platforms, test can be launched simply by calling ctest:

alainm@satch:~/views/codes/cubby/branches/work$ ctest 
Start processing tests
Test project /home/alainm/views/codes/cubby/branches/work
  1/ 47 Testing bug20                            Passed
  2/ 47 Testing bug32                            Passed
  .....
 47/ 47 Testing hydro_penalization               Passed

100% tests passed, 0 tests failed out of 47
alainm@satch:~/views/codes/cubby/branches/work$ 

running test in parallel

For some clusters, we developed parallel script to launch the tests in parallel which can be a lot faster (if resources are available). In the following examples, we run from a build directory called rel.

Coverage

Test coverage allows us to know how much of the code is actually tested (it says nothing about the quality of the test nor code).

On intel platform

First, we need to enable the necessary code instrumentation, the easiest way to do that is to pre-configure with the -t option. You probably want to do that in a dedicated build directory:

[alainm@login01 trunk]$ rm -rf cov
[alainm@login01 trunk]$ mkdir cov
[alainm@login01 trunk]$ cd cov/
[alainm@login01 cov]$ ../preconfigure.sh -h licallo.mpi143 -n 8  -c -t
-- Build files have been written to: /scratch/alainm/view/cubby/trunk/cov
[alainm@login01 cov]$  ls profiled/
4f1039ae_14892.dyn  pgopti.spi  pgopti.spl
[alainm@login01 cov]$ 

Note that the files in profiled can be safely deleted before the build and testing phase.

then build and test as usual:

[alainm@login01 cov]$ make -j24
...
[100%] Built target cubby
[alainm@login01 cov]$  ../utils/bin/licallotest.sh
...
>>>>> 59 passed, 0 failed, 2 waiting
    ****************  
>>>>> 61 passed, 0 failed, 0 waiting
started at Fri Jan 13 15:09:07 CET 2012, finished at Fri Jan 13 15:11:28 CET 2012
Passed 61 and failed 0
[alainm@login01 cov]$ 

This should populate the ./profiled directory, which contains the coverage annotation (among other things)

 ls ./profiled/ | wc -l
350
[alainm@login01 cov]$ 

That contains informations for all the runs, we need to merge them:

[alainm@login01 cov]$ cd profiled/
[alainm@login01 profiled]$ profmerge 
...
profmerge: looking at dynamic file:	./4f1039ae_14892.dyn
[alainm@login01 profiled]$ 

Then we need to generated the coverage report, that's the job of codecov (thos manpage is available, but you need to look for man codecov_c):

[alainm@login01 profiled]$  codecov -prj cubby -spi pgopti.spi -dpi pgopti.dpi
Intel(R) C++/Fortran Compiler code-coverage tool for applications running on Intel(R) 64, Version 11.1    Build 20101201 Package ID: l_cprof_p_11.1.075
Copyright (C) 1985-2010 Intel Corporation.  All rights reserved.
Warning - File /scratch/alainm/view/cubby/trunk/cov/CMakeFiles/CMakeTmp/testCXXCompiler.cxx not found!
Progress: 10% .. 20% .. 30% .. 40% .. 50% .. 60% .. 70% .. 80% .. 90% .. 100%
[alainm@login01 profiled]$

But this might cover too many file (test files are one examples) that we might not want to deal with. There is a way to provide codecov with a listing of covered files.

First, we need to generate the file, one possibility is:

[alainm@login01 profiled]$ (cd  ../../src ; find `pwd` -name "*.?pp" -print ; cd ../; find `pwd` -maxdepth 1 -name "*.?pp") | grep -v version | grep -v test_features > covered.txt 
[alainm@login01 profiled]$ 

and then, call codecov:

[alainm@login01 profiled]$ codecov -prj cubby -spi pgopti.spi -dpi pgopti.dpi -comp ./covered.txt Intel(R) C++/Fortran Compiler code-coverage tool for applications running on Intel(R) 64, Version 11.1    Build 20101201 Package ID: l_cprof_p_11.1.075
Copyright (C) 1985-2010 Intel Corporation.  All rights reserved.

Progress: 10% .. 20% .. 30% .. 40% .. 50% .. 60% .. 70% .. 80% .. 90% .. 100%
[alainm@login01 profiled]$ 

The previous step should produce a directory, CodeCoverage?, containing a bunch of web pages, and an html index called CODE_COVERAGE.HTML,

[alainm@login01 profiled]$  ls | grep -i code
CodeCoverage
CODE_COVERAGE.HTML
[alainm@login01 profiled]$ 

Just open CODE_COVERAGE.HTML with you web browser.

Other platforms

Not supported yet.

Last modified 9 years ago Last modified on Jan 13, 2012 5:22:34 PM