User General user info job glite
Log in on a BEgrid user interface
There are several user interfaces named the following way :
m<x>.iihe.ac.be with x=[0-9]
If you have carefully followed this procedure, you should be able to log in on one of them. As an example, let's try to log in on m6 :
ssh <your_login>@m6.iihe.ac.be
Creation of a VOMS proxy
If you are member of beapps
Prior to using the resources of the grid, you need a valid personal grid proxy. This proxy is kind of access token that will be used by the grid software to check your identity and your authorizations. It not only contains the DN of your personal certificate, but also the name of the virtual organization your are registered to.
To create such a proxy for the belgian VO "beapps", issue the following command :
voms-proxy-init --voms beapps
Grid proxies are limited in time. By default, it is valid for 12 hours. For longer duration, use the "--valid" option. Below, we request a proxy valid for 24 hours :
voms-proxy-init --voms beapps --valid 24:00
If you want to check a proxy you've just created, type the following command :
voms-proxy-info --all
If you can't register to beapps
Issue the following command :
/bin/createproxy.pl
And now, the magic of PUSP will show up with the command :
voms-proxy-info --all
that should result in something like this :
subject : /DC=org/DC=terena/DC=tcs/C=BE/O=Vrije Universiteit Brussel/CN=Robot - STEPHANE GERARD stgerard@vub.ac.be/CN=user:sgerard/CN=1598008192 issuer : /DC=org/DC=terena/DC=tcs/C=BE/O=Vrije Universiteit Brussel/CN=Robot - STEPHANE GERARD stgerard@vub.ac.be/CN=user:sgerard identity : /DC=org/DC=terena/DC=tcs/C=BE/O=Vrije Universiteit Brussel/CN=Robot - STEPHANE GERARD stgerard@vub.ac.be type : RFC3820 compliant impersonation proxy strength : 1024 path : /tmp/x509up_u20533 timeleft : 11:59:49 key usage : Digital Signature, Key Encipherment ==== VO beapps extension information ==== VO : beapps subject : /DC=org/DC=terena/DC=tcs/C=BE/O=Vrije Universiteit Brussel/CN=Robot - STEPHANE GERARD stgerard@vub.ac.be issuer : /DC=org/DC=terena/DC=tcs/C=BE/ST=Brussels/L=Brussels/O=Le reseau telematique belge de la recherche/CN=voms01.begrid.be attribute : /beapps/Role=NULL/Capability=NULL timeleft : 11:59:49 uri : voms01.begrid.be:18004
You've got a valid beapps proxy, you are ready for the next step...
Job submission
This section describes how to manage your jobs the glite-ce-* commands. With these commands, you must know beforehand on which CE you will send your jobs.
Choosing a Computing Elemement
First, you need to get the list of the CEs that are supporting the VO you belong to. You can do it like that :
[sgerard@m6 ~]$ lcg-infosites --vo beapps CE # CPU Free Total Jobs Running Waiting ComputingElement ---------------------------------------------------------------- 5947 0 0 0 0 cream02.iihe.ac.be:8443/cream-pbs-beapps 5832 226 2231 1986 245 creamce.gina.sara.nl:8443/cream-pbs-medium 5832 226 0 0 0 creamce.gina.sara.nl:8443/cream-pbs-short 5832 237 2239 1982 257 creamce2.gina.sara.nl:8443/cream-pbs-medium 5832 237 0 0 0 creamce2.gina.sara.nl:8443/cream-pbs-short 5832 237 2239 1982 257 creamce3.gina.sara.nl:8443/cream-pbs-medium 5832 237 0 0 0 creamce3.gina.sara.nl:8443/cream-pbs-short
Compose/edit a JDL file to describe your job
Jobs submitted to the grid are described using a specific language, the Job Description Language (JDL). Here we provide a very basic sample :
[
VirtualOrganisation = "beapps";
# the VO you belong to
JobType = "normal";
# in general this is normal or mpich
Type = "Job";
# this is a job :-)
Executable = "test_job.sh";
# name of the command without arguments
Arguments = "/proc/cpuinfo";
# arguments if necessary
StdOutput = "output";
# to put the standard output (display) in the file we call output
StdError = "error";
# to put the standard error in the file we call error
InputSandbox = "test_job.sh";
# to send the binary (or the script) and all necessary input files
OutputSandbox = {"output","error"};
# to receive the output files
OutputSandboxBaseDestURI = "gsiftp://localhost";
# to specify how the outputsandbox files will be transfered back to the user
]
More options for the JDL file can be found on this site: https://wiki.italiangrid.it/twiki/bin/view/CREAM/JdlGuide.
Remark: Don't send/receive input/output sandboxes greater than 20 MB. For greater files usage, look at section Advanced data access with gridftp.
Submit your job
To submit a job, just use this:
glite-ce-job-submit -a -r <target_resource> <your_jdl_file>
The <target_resource> is specified like this :
<full_hostname>:<port_number>/cream-<service>-<queue_name>
where <service> identifies the underlying resource management system (e.g. lsf, pbs, etc.). For example :
$ glite-ce-job-submit -a -r cream02.iihe.ac.be:8443/cream-pbs-beapps job.jdl 2016-02-09 14:44:44,459 WARN - VirtualOrganisation specified in the JDL but overriden with [beapps] https://cream02.iihe.ac.be:8443/CREAM979231935
The output gives you the jobID:
https://cream02.iihe.ac.be:8443/CREAM979231935
You will need this jobID in the following steps to monitor your job and get its ouput when it's finished.
Monitor your job
You can follow the different steps of your job execution with:
glite-ce-job-status <jobID>
Example of output:
$ glite-ce-job-status https://cream02.iihe.ac.be:8443/CREAM979231935 ****** JobID=https://cream02.iihe.ac.be:8443/CREAM979231935 Status = [IDLE]
Retrieve the results
When your job has reached the "DONE-OK" status, you are able to retrieve the output files (the ones you specified in the ouput sandbox of the jdl file). Use this command:
glite-ce-job-output [--dir <path to where you want your outputs>] <jobID>
If you don't specify the path for your outputs with the --dir option, then a directory containing your output sandbox files will be automatically created in the current directory.
Example :
$ glite-ce-job-output https://cream02.iihe.ac.be:8443/CREAM979231935 2018-03-08 15:26:41,256 INFO - For JobID https://cream02.iihe.ac.be:8443/CREAM979231935 output will be stored in the dir ./cream02.iihe.ac.be_8443_CREAM979231935
Data Management
- How to use storage element.
Other information
- Advanced data access with gridftp (for processing files > 40MB)
