User Scientific Software

What is a software area ?

When computing on a cluster, you need a central place where to store scientific software required by compute jobs. This must be a directory shared between the workernodes of the cluster and the user interfaces, because users need to have the same software environment on the user interfaces, where they write and test their codes, as on the workernodes, where they will run their codes in production. Such a central shared software directory is what we call a software area.

BEgrid software area

BEgrid maintains a virtual organization called beapps that is open to all Belgian researchers. In the shell, the environment variable $VO_BEAPPS_SW_DIR gives you the actual path of this software area :

[vsc10003@ui01bgrid ~]$ echo $VO_BEAPPS_SW_DIR 
/sandbox/beappssgm

However, you don't really need to know this path, you can always use the environment variable instead in your job script or when working in the shell. For example, if you want to check the content of the beapps software area :

[vsc10003@ui01bgrid ~]$ ls -al $VO_BEAPPS_SW_DIR
total 5335586
drwxrwxr-x 20 22000 22000         33 Apr 30 17:22 .
drwxr-xr-x 22 root  root          24 Mar  3  2017 ..
drwxr-xr-x  3 22000 22050          3 Dec 17  2015 blast+
drwxr-xr-x 14 22000 22050         34 Feb 25  2013 gamess
...

This kind of environment variables is not specific to BEgrid, it is actually a convention adopted by all EGI sites for all VOs. For any VO and any site, you don't need to know what's the actual path of the software area, instead you can always use the environment variable VO_<VO_NAME>_SW_DIR where <VO_NAME> is the name of the VO.

Ordinary users don't have write access to the VO software area. If you need some software to be installed in it, please send us a mail and we will do it for you.

CVMFS software areas

CVMFS (or CernVM-File System) is a software deployment technology that has been created by CERN to distribute on sites the software needed by High-Energy Physics (HEP) researchers. Although it is mainly intended to HEP communities, it contains generic software (like Python, GCC, BLAS, LAPACK, FFT,...) that can be of interest for other disciplines than physics.

Instead of trying to explain in details how the /cvmfs sub-directories are organized, we will limit ourselves in this wiki to give some relevant links about the /cvmfs/sft.cern.ch (or SFT) software area :

• High-level description of the software packages and how they are grouped in LCG Configurations
• Website of the SFT group managing what's in /cvmfs/sft.cern.ch

The following sections will illustrate how to use the SFT software area.

Basic commands to use the software in SFT

The software packages are grouped in LCG Configurations. In each of these configurations, you will normally find set of software that are compatible with each others. To get the list of LCG Configurations :

$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh

Now, let's say you want to know more about the "LCG_93c" configuration :

$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh LCG_93c
Available <arch-os-complier> for LCG_93c : 
  x86_64-centos7-gcc62-opt
  x86_64-centos7-gcc7-dbg
  x86_64-slc6-gcc62-dbg
  x86_64-slc6-gcc62-opt
  x86_64-slc6-gcc7-dbg

If you are on an SL6 UI and we want gcc62, we have to issue the following command :

$ source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_93c x86_64-slc6-gcc62-opt

Check the version of gcc :

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/cvmfs/sft.cern.ch/lcg/contrib/gcc/6.2.0native/x86_64-slc6/bin/../libexec/gcc/x86_64-pc-linux-gnu/6.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ../gcc-6.2.0/configure --prefix=/build/pmendez-sftnight/install-620local -with-system-zlib --disable-multilib --enable-languages=all
Thread model: posix
gcc version 6.2.0 (GCC) 

Getting Python 3

Currently, Python 2.7 is available on our BEgrid UI, but let's say you absolutely need Python 3. Here are the steps to get it right :

$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh
...
LCG_93python3
...

The configuration LCG_93python3 seems to provide what we are looking for. To find the list of supported platforms for LCG_93python3 :

$ /cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh LCG_93python3
Available <arch-os-complier> for LCG_93python3 : 
  x86_64-centos7-gcc62-dbg
  x86_64-centos7-gcc62-opt
  x86_64-centos7-gcc7-dbg
  x86_64-centos7-gcc7-opt
  x86_64-slc6-gcc62-dbg
  x86_64-slc6-gcc62-opt
  x86_64-slc6-gcc7-dbg
  x86_64-slc6-gcc7-opt
  x86_64-ubuntu1604-gcc54-dbg
  x86_64-ubuntu1604-gcc54-opt

If you are working on an SL6 UI, you can choose any of the slc6 platforms. For example :

$ source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_93python3 x86_64-slc6-gcc62-opt

Check that Python3 is now available :

$ python3
Python 3.6.3 (default, Dec 19 2017, 22:45:30) 
[GCC 6.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Singularity images

In directory /swmgrs/nonvo/singularity, you will find some home-brew Singularity images that have been specifically designed and tested on our machines. Below, we describe how to use some of them.

tensorflow-gpu-centos7-iihe

Description

This image is based on the latest release of CentOS 7, and it mainly contains tensorflow-gpu 1.11.0 and keras, as well as the cuda libraries (version 9.0).

How to use it ?

First, you need to login to a GPU machine and prepare your environment like this :

$ qsub -I -q gpu2

or

$ qsub -I -q gpu6

where "gpu<X>" corresponds to the queue to reach machines with <X> GPU devices. After a few seconds, you'll get connected to a GPU machine. You need to prepare you environment to make GPU devices visible like this :

$ source /swmgrs/icecubes/set_gpus.sh

Note : you need to run these commands from a so-called "m-machines" (see this link for more details).

To launch a shell in the tensorflow container, use this command :

$ singularity shell --nv -B /swmgrs -B /cvmfs -B /scratch  /swmgrs/nonvo/singularity/tensorflow-gpu-centos7-iihe.img

In this command, don't forget the option "--nv" to enable the NVIDIA support if you intend to use GPUs ! The role of "-B" options is to mount some directories into the container, so that you get a software environment very close to what you would have on the m-machines and/or the workernodes. By the way, you'll notice that your user directory in the container is the same as your user directory on the cluster.