Data management

gLite Data Management

This section introduces file management in gLite.

Data management consists in:

• the mapping of logical filenames to "site URLs", done with the LFC using lfc-commands
• copying files to and from a SE, using lcg-commands
• to replicate files, using lcg-commands

Environment variables

Environment variables need to be set properly to ensure that the correct catalog service is used. Check the default environment variables :

 echo $LCG_GFAL_INFOSYS
 echo $LCG_CATALOG_TYPE
 echo $LFC_HOST

This should be:

[abulanza@gridy11 ~]$ echo $LCG_GFAL_INFOSYS
voms.begrid.be:2170
[abulanza@gridy11 ~]$ echo $LCG_CATALOG_TYPE
lfc
[abulanza@gridy11 ~]$ echo $LFC_HOST
gridy8.begrid.be

To set the empty value, please use these commands:

export LCG_GFAL_INFOSYS=voms.begrid.be:2170
export LCG_CATALOG_TYPE=lfc
export LFC_HOST=gridy8.begrid.be

LFC and LCG commands

• Listing file and directory

For each of the supported VO a separate "top level" directory exists under the "/grid" directory.

$ lfc-ls -l /grid/betest

This lists the contents of the /grid/betest directory.

• Define a HOME directory

$ export LFC_HOME=/grid/betest/tutorial

Avoid to type an absolute path for every file and directory.

• Create directory

Create a personal directory for storage by typing:

$ lfc-mkdir /grid/betest/yourusername

To check that you have created your directory, type:

$ lfc-ls -l 

The directory created is just a virtual directory and only exists within the catalog of lfn's.

• Upload file into SE

Upload a file into the directory you just created. First create locally a simple text file:

$ echo "datamanagement tutorial" > text_file.txt

The command used for this is lcg-cr (LCG copy and register). Find a suitable storage.

$ lcg-infosites --vo betest se
Avail Space(Kb) Used Space(Kb)  Type    SEs
----------------------------------------------------------
1401150000      13768           n.a     kg-se01.cc.kuleuven.be
1               1               n.a     kg-se01.cc.kuleuven.be
303610000       396459          n.a     gridse.atlantis.ugent.be
2231527613      3707338415      n.a     maite.iihe.ac.be
2231527613      3707338415      n.a     maite.iihe.ac.be
2231527613      3707338415      n.a     maite.iihe.ac.be
n.a             n.a             n.a     se01.cmi.ua.ac.be
1               1               n.a     se01.cmi.ua.ac.be

Store the file on the se01.cmi.ua.ac.be Storage Element :

[abulanza@gridy11 ~]$ lcg-cr --vo betest -l lfn:/grid/betest/abulanza/text_file.txt -d se01.cmi.ua.ac.be file://$HOME/text_file.txt

The guid is a unique identifier for each file (except for replicas).

The file physically exists on a SE and has an additional "virtual" filename in the catalog.

• Get the file SURL

lcg-lr (list-replicas) [lfn | guid]

$ lcg-lr --vo betest lfn:/grid/betest/username/text_file.txt
srm://se01.cmi.ua.ac.be/dpm/...

• Replicate file between SE

A file can be stored on multiple SE's. Running job can access the file on the closest SE.

$ lcg-rep --vo gilda -d maite.iihe.ac.be lfn:/grid/betest/username/text_file.txt

Check that the replica was created by using the LCG list replicas command:

lcg-lr --vo betest lfn:/grid/betest/username/text_file.txt

list all the replicas of the file

• Download a file from SE to UI

$ lcg-cp --vo betest lfn:/grid/betest/username/text_file.txt file://$HOME/text_file_copy.txt

• delete a file from SE with lcg-del

$ lcg-del -a  --vo betest lfn:/grid/betest/username/text_file.txt

This also removes the File Catalog entries.

• Remove the working directory on file catalog

$ lfc-rm -r /grid/betest/username