User Scientific Software
What is a software area ?
When computing on a cluster, you need a central place where to store scientific software required by compute jobs. This must be a directory shared between the workernodes of the cluster and the user interfaces, because users need to have the same software environment on the user interfaces, where they write and test their codes, as on the workernodes, where they will run their codes in production. Such a central shared software directory is what we call a software area.
BEgrid software area
BEgrid maintains a virtual organization called beapps that is open to all Belgian researchers. In the shell, the environment variable $VO_BEAPPS_SW_DIR gives you the actual path of this software area :
[vsc10003@ui01bgrid ~]$ echo $VO_BEAPPS_SW_DIR /sandbox/beappssgm
However, you don't really need to know this path, you can always use the environment variable instead in your job script or when working in the shell. For example, if you want to check the content of the beapps software area :
[vsc10003@ui01bgrid ~]$ ls -al $VO_BEAPPS_SW_DIR total 5335586 drwxrwxr-x 20 22000 22000 33 Apr 30 17:22 . drwxr-xr-x 22 root root 24 Mar 3 2017 .. drwxr-xr-x 3 22000 22050 3 Dec 17 2015 blast+ drwxr-xr-x 14 22000 22050 34 Feb 25 2013 gamess ...
This kind of environment variables is not specific to BEgrid, it is actually a convention adopted by all EGI sites for all VOs. For any VO and any site, you don't need to know what's the actual path of the software area, instead you can always use the environment variable VO_<VO_NAME>_SW_DIR where <VO_NAME> is the name of the VO.
Ordinary users don't have write access to the VO software area. If you need some software to be installed in it, please send us a mail and we will do it for you.
CVMFS software areas
CVMFS (or CernVM-File System) is a software deployment technology that has been created by CERN to distribute on sites the software needed by High-Energy Physics (HEP) researchers. Although it is mainly intended to HEP communities, it contains generic software (like Python, GCC, BLAS, LAPACK, FFT,...) that can be of interest for other disciplines than physics.
Instead of trying to explain in details how the /cvmfs sub-directories are organized, we will limit ourselves in this wiki to give some relevant links about the /cvmfs/sft.cern.ch (or SFT) software area :
- High-level description of the software packages and how they are grouped in LCG Configurations
- Website of the SFT group managing what's in /cvmfs/sft.cern.ch
The following sections will illustrate how to use the SFT software area.
Basic commands to use the software in SFT
The software packages are grouped in LCG Configurations. In each of these configurations, you will normally find set of software that are compatible with each others. To get the list of LCG Configurations :
$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh
Now, let's say you want to know more about the "LCG_93c" configuration :
$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh LCG_93c Available <arch-os-complier> for LCG_93c : x86_64-centos7-gcc62-opt x86_64-centos7-gcc7-dbg x86_64-slc6-gcc62-dbg x86_64-slc6-gcc62-opt x86_64-slc6-gcc7-dbg
If you are on an SL6 UI and we want gcc62, we have to issue the following command :
$ source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_93c x86_64-slc6-gcc62-opt
Check the version of gcc :
$ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/cvmfs/sft.cern.ch/lcg/contrib/gcc/6.2.0native/x86_64-slc6/bin/../libexec/gcc/x86_64-pc-linux-gnu/6.2.0/lto-wrapper Target: x86_64-pc-linux-gnu Configured with: ../gcc-6.2.0/configure --prefix=/build/pmendez-sftnight/install-620local -with-system-zlib --disable-multilib --enable-languages=all Thread model: posix gcc version 6.2.0 (GCC)
Getting Python 3
Currently, Python 2.7 is available on our BEgrid UI, but let's say you absolutely need Python 3. Here are the steps to get it right :
$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh ... LCG_93python3 ...
The configuration LCG_93python3 seems to provide what we are looking for. To find the list of supported platforms for LCG_93python3 :
$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh LCG_93python3 Available <arch-os-complier> for LCG_93python3 : x86_64-centos7-gcc62-dbg x86_64-centos7-gcc62-opt x86_64-centos7-gcc7-dbg x86_64-centos7-gcc7-opt x86_64-slc6-gcc62-dbg x86_64-slc6-gcc62-opt x86_64-slc6-gcc7-dbg x86_64-slc6-gcc7-opt x86_64-ubuntu1604-gcc54-dbg x86_64-ubuntu1604-gcc54-opt
If you are working on an SL6 UI, you can choose any of the slc6 platforms. For example :
$ source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_93python3 x86_64-slc6-gcc62-opt
Check that Python3 is now available :
$ python3 Python 3.6.3 (default, Dec 19 2017, 22:45:30) [GCC 6.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>
Singularity images
In directory /swmgrs/nonvo/singularity, you will find some home-brew Singularity images that have been specifically designed and tested on our machines. Below, we describe how to use some of them.
tensorflow-gpu-centos7-iihe
Description
This image is based on the latest release of CentOS 7, and it mainly contains tensorflow-gpu 1.11.0 and keras, as well as the cuda libraries (version 9.0).
How to use it ?
First, you need to login to a GPU machine and prepare your environment like this :
$ qsub -I -q gpu2
or
$ qsub -I -q gpu6
where "gpu<X>" corresponds to the queue to reach machines with <X> GPU devices. After a few seconds, you'll get connected to a GPU machine. You need to prepare you environment to make GPU devices visible like this :
$ source /swmgrs/icecubes/set_gpus.sh
Note : you need to run these commands from a so-called "m-machines" (see this link for more details).
To launch a shell in the tensorflow container, use this command :
$ singularity shell --nv -B /swmgrs -B /cvmfs -B /scratch /swmgrs/nonvo/singularity/tensorflow-gpu-centos7-iihe.img
In this command, don't forget the option "--nv" to enable the NVIDIA support if you intend to use GPUs ! The role of "-B" options is to mount some directories into the container, so that you get a software environment very close to what you would have on the m-machines and/or the workernodes. By the way, you'll notice that your user directory in the container is the same as your user directory on the cluster.
