User General user info WS-PGRADE

Introduction

WS-PGRADE is a Liferay-based web portal that allows you to submit jobs using a graphical interface, instead of command-lines.

Documentation

For a general introduction :

http://guse.hu/about/home

The documentation for users (the manual and the cookbook) can be found here :

http://guse.hu/documentation

Although the manual and the cookbook contain many explanations and examples, you may need some training material that you'll find here :

http://www.lpds.sztaki.hu/summerschool2013/?m=4

(hands-on sessions on WS-PGRADE given during the Summer School on Grid and Cloud Workflows and Gateways organized by LPDS-STAZKI)

Getting access to our WS-PGRADE portal

Go to this URL :

http://guse.iihe.ac.be/

To request an account, click on "Sign In", and then on "Create Account".

Fill in the form and click on "Save" button. You will have to wait a few hours before your request get approved...

Your personal computer must be equipped with Java.

Java settings on your personal computer

Some tools, like the graph editor, are in fact Java portlets, and so, you need a Java JVM on your personal computer.

It should work with Oracle Java 1.7 :

Java 7u51 introduced new security policies that deny the execution of the Graph Editor Web Start application in the default configuration.
In order to make the Graph Editor usable again on your machine, there are two possibilities. In either case, please start the Java Control Panel
on your machine, and select the "Security" tab:

    lower the security level: move the "Security Level" slider to "Medium", or
    add the gateway's base URL (e.g. https://foo.bar) to the "Exception Site List". For this, click on the "Edit Site List..." button, then click "Add", and enter the gateway's base URL to the "Location" field.

Once set, click "OK". You can get the used gateway's base URL from the gateway's administrator.

With Oracle Java 1.8, the security level "Medium" has been removed from ControlPanel. However, you can still create an exception for the portal.

To start the Java Control Panel under Linux, try to locate the binary with :

locate ControlPanel
--> /usr/java/jdk1.7.0_60/jre/bin/ControlPanel

and then simply the full path of the binary in the shell :

/usr/java/jdk1.7.0_60/jre/bin/ControlPanel

Here is what you should see in the ControlPanel after having configured an exception for the WS-PGRADE portal :

Image(java_configure_exception.png)

The Java from OpenJDK seems to be less problematic (for example, it works fine with java-1.8.0-openjdk).

Getting access to the resources

You got an access to the portal, but it doesn't mean that you got an access to the computing and/or storage resources behind that portal.

Access to local pbs batch system (cream02)

If you've got an account on our UIs (the so-called m-machines) :

• Go to the "Security" menu in the main bar and click on "Public key".
• An ssh keypair was generated automatically for you by the portal. Copy/paste the public key and send it by email to the BEgrid Team.
• You can also copy/paste the public key in your ~/.ssh/authorized_keys file on one of the UIs, but if you do so, keep in mind that it is temporary (for a permanent change, you need to go via the grid-admins as explained in previous bullet).
• Your public key must be linked to a resource and a valid account on that resource : choose the resource (cream02) in the list "URL of resource", and type your login in the field "User account" next to it, and click on the "Add" button.

Access to the grid

If you've already used the grid, you know that you need a valid proxy in order to access computing and/or storage resources from a VO. You will create this proxy on one of our UIs, and then you will upload this proxy to a myproxy server. After that, you will be able to download a proxy in the portal and to associate it to a VO.

Here is an example the beapps VO :

• Log in to a UI and type the following :

voms-proxy-init --voms beapps

• Upload the proxy to a myproxy server :

myproxy-init -l sgerard -t 150 -s px.grid.sara.nl

• In the portal, go to the menu "Security" -> "Certificate". Click on the "Download" button, and specify the hostname (px.grid.sara.nl), the login (sgerard), the lifetime in hours, and the password. Click on "Download" button and wait a few seconds... Click on "Associate to VO" and choose the "beapps" VO in the list and click on "OK".

Example of a parameter study workflow

If you are really new to WS-PGRADE, we strongly advise you to make your first steps using the manual and/or the cookbook and/or the training material.

In this section, we present a detailed example of a typical generic use case for researchers doing distributed computing : apply the same command on many different data files.

Step 0 : before starting

So, you need a list of datafiles (data_files_list), under the form a simple text file. You need a script (gen_output.pl) that will generate the inputs from the list. You also need a script (ls_file.pl) that contains the code you want to execute on each datafile.

You can download these examples in the attachment section of this page.

Step 1 : create a graph

The graph described the inputs and outputs of your jobs and how they will be linked.

In the present case, we need a first generator job whose role will be to generate the input for the second job which will be the real workhorse.

Job0 has one out port named "OUTPUT", Job1 has one in port named "INPUT" and one out port named "OUTPUT". The output of Job0 is linked to the input of Job1.

Here is a screenshot that shows you how to connect (with drag-and-drop) the inputs and the outputs of your jobs :

Image(create_graph.png)

Step 2 : create a concrete workflow

A concrete workflow is based on a graph : for each element of the graph, you will specify what is the concrete resource : - for each job (represented by an orange box) in the graph, you need to specify what is the binary or script, as well as the eventual parameters or arguments, and on which resource (or DCI for Distributed Computing Infrastructure) it will be run; - for each input or outut, you need to specify the files or parameters.

Go to the "Create concrete" option in the "Workflow" menu, specify the graph on which your concrete workflow is based and give it a name :

Image(create_concrete.png)

After clicking on "Ok" button, you'll be warned that the concrete has been created. You can go back to the "Concrete" window ("Concrete" option in the "Workflow" menu). Click on the "Configure" button for the newly created workflow. You get this view :

Image(configure_concrete.png)

Click on the Job0 box in the graph. This window appears :

Image(configure_Job0.png)

Specify the type of resource (pbs), the resource (our batch system : cream02.iihe.ac.be), and the JobManager, i.e. the queue name (localgrid). In this case, the executable code will be contained in a tarball named gen_output.pl.app.tgz that contains the script gen_output.pl and the list of datafiles (date_files_list). Note that the name of the tarball cannot be arbitrary, it must be the name of the script or binary file followed by ".app.tgz". Create this tarball and upload it with the "Browse" button. The value of parameter is 19 because there are 19 data files in the list. This parameter will be passed as an argument to the script gen_output.pl, but if you look at the script, it is not used in fact :

#!/usr/bin/perl

my $list = './data_files_list';

open(FILE, "<", "$list") or die "cannot open < list : $!";

my $cpt = 0;
while(<FILE>) {
    chomp;
    my $res = system("echo $_ > OUTPUT_$cpt");
    $cpt += 1;
}

close(FILE);

From this code, you also see more clearly that the role of Job0 is to generate a set of output files which name must follow the rule "OUTPUT_<index>" where <index> goes from 0 to 18. The content of each file will be the path of a data file (being understood that the file will be reachable from the nodes where Job1 will be run).

Don't forget to click on green round button otherwise your changes won't be kept !!!

Now click on the "Job I/O" tab, and set the "Generator" field to "Yes", and save this change by clicking on the round green button. This will inform the WS-PGRADE engine that this job is in fact a generator.

Going back to the main window, let's specify thee properties of Job1 :

Image(configure_Job1.png)

For the executable code, use the browse button to upload the script ls_file.pl, and save with the green button. (Nothing is to be done with the Job I/O.)

Just passing by, let's have a look at the code of this script :

#!/usr/bin/perl

my $file = <tt>cat INPUT</tt>;

my $res = system("ls -al $file > OUTPUT");

You can guess from that code that for each OUTPUT_<index> generated by Job0, an instance of Job1 will be created and submitted, and the content of OUPUT_<index> will be sent to INPUT of the instance of Job1. The command executed on each file is just "ls -al".

Close the window to configure your job, you are now back to the main screen of your concrete, you should see a red box with an icon representing a floppy :

Image(save_concrete.png)

If you have already submitted the concrete in the past, you might want to remove all the reliefs of these previous runs. Then choose the first option ("Delete old instances"). Otherwise, keep the second option ("Do not delete old instances"). Click on the floppy disk icon, otherwise all the changes you've just made will be lost.

Step 3 : submit and monitor the workflow

Let's go back to the concrete screen, and click on the "Submit" button in front of your concrete. You'll be asked to confirm.

Step 4 : get the results

By clicking on the "Details" button in front of your workflow, you can see the actual status of each job. The button "View finished" allows you to check the std.err and std.out of a job, and to download the outputs under the form of a tarball.