Documentation

Manuals

Here you'll find the user's manual on how to use the cluster.

1. System access

System Access

To ensure a secure login session, users must connect to the login node using the secure shell SSH protocol. Data movement must be done using the secure protocols sftp and/or scp.

Before any login sessions can be initiated using ssh, a working SSH client needs to be present in the local machine. Wikipedia is a good source of information on SSH in general and provides information on the various clients available for your particular operating system, but here we'll cover common examples for Windows 7, Ubuntu Linux and Mac OS Mavericks.

While you can send batch jobs to the whole SNOW cluster, user interactive sessions are restricted to the login node, called hpc.dtic.upf.edu 

Connecting to SNOW from a Windows client:

You need to install on your workstation a set of free tools to interact with the cluster:

  • Putty
  • XMing
  • FileZilla / WinSCP

Alternatively, you can choose the XManager Enterpsuite, that provides all tools on a single package. This tool is not free, but it will run without limitations for a 30-day period. 

Program URL Purpose
Putty www.putty.org Free SSH client
X-Ming sourceforge.net/projects/xming      X client, to be used with Putty
FileZilla www.filezilla.org FTP / FTPs / SFTP client
X-Manager enterprise   www.netsarang.com Enterprise class SSH suite. Includes all tools on a single package 

In this tutorial, we'll work with Putty and Xming. 

  • Download and install the Putty client. It's a single .exe file, place it on some folder on your hard disk
  • Execute putty, and configure X forwarding options

  • Download and install XMing. The following options are recommended:

  • Depending on your Windows firewall configuration, Windows will  need to be configured to have the ports open. If you use a firewall software, please refer to software manufacturer's support to allow a program through the firewall.

  • Open a ssh connection to the server hpc.dtic.upf.edu. The first time, you'll geta warning for the SSL signature key coming from hpc.dtic. Accept it permanently to avoid this window coming out every time:

  • Use your DTIC username and password to get access to the server. You'll get additional information about the DTIC username here.

Connecting to SNOW from a Linux client:

Connecting to SNOW from a linux machine is pretty easy, as everything is built in by default in every distribution. Simply open a terminal and type 'ssh -X username@hpc.dtic.upf.edu', where:

  • username is your DTIC user, in the usual form nsurname
  • -X tells your Linux client to forward and manage the graphical X sessions you eventually need to open over hpc.dtic.upf.edu
ivan@ubuntuvm1$ ssh -X ijimenez@hpc.dtic.upf.edu

Connecting to SNOW from a Mac OS client:

You'll need to install a couple of packages to be able to open SSH connections and make SFTP / FTPs connections:

Program URL Purpose
XQuartz xquartz.macosforge.org/landing/                   Free SSH client with X embedded support                                            
Cyberduck                         cyberduck.io Free FTPs / SFTP client
  • Download XQuartz from the project webpage
  • Double click to install the package with the default options
  • Once installed, the application will dock on the application menu. Click to launch it
  • Once opened, the procedure is similar to Linux. Type the following command (type your username in place of 'uhpc')
bash-3.2$ ssh -X uhpc@hpc.dtic.upf.edu

 

2. Computing environment

Unix Shell

By default user shell is dticsh, it is a symbolic link to /bin/bash. In other servers of the DTIC , 'dticsh' shell is a restricted shell which encapsulates user connections in sftp, rsh or even rsync. All users are defined in DTIC LDAP directory. The environment variable $SHELL shows your shell interpreter:

$ echo $SHELL
/bin/dticsh

Environment variables

By default user common variables printed out by the env command:

ijimenez@login:~$ env
MANPATH=/sge/man:/usr/man
CLUES_PATH=/usr/local/clues/0.89
TERM=xterm
SHELL=/bin/bash
SGE_JSV_TIMEOUT=600
XDG_SESSION_COOKIE=6e0d0fdd5d4d80844f93739852541bee-1387193389.659370-725688603
MODULEPATH_ROOT=/soft/modulefiles
SSH_CLIENT=10.80.1.106 50897 22
LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
SGE_CELL=default
SSH_TTY=/dev/pts/4
http_proxy=http://proxy:8080
USER=ijimenez
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
LMOD_sys=Linux
LD_LIBRARY_PATH=/sge/lib/linux-x64
ftp_proxy=http://proxy:8080
LMOD_PREPEND_BLOCK=normal
MAIL=/var/mail/ijimenez
PATH=/sge/bin/linux-x64:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/clues/0.89:/soft/MATLAB/R2013b/bin:/sge/matlab
LMOD_SETTARG_CMD=:
PDSH_RCMD_TYPE=ssh
PWD=/homedtic/ijimenez
LANG=en_US.UTF-8
MODULEPATH=/soft/modulefiles/Linux:/soft/modulefiles/Core:/soft/lmod/lmod/modulefiles/Core
LMOD_SETTARG_SUPPORT=no
SGE_ROOT=/sge
LMOD_CMD=/soft/lmod/lmod/libexec/lmod
SHLVL=1
HOME=/homedtic/ijimenez
LANGUAGE=en_US:en
BASH_ENV=/soft/lmod/lmod/init/bash
LMOD_arch=x86_64
LOGNAME=ijimenez
SSH_CONNECTION=10.80.1.106 50897 193.145.51.37 22
SGE_CLUSTER_NAME=p6444
DISPLAY=localhost:12.0
LMOD_DIR=/soft/lmod/lmod/libexec/
LMOD_COLORIZE=yes
_=/usr/bin/env
ijimenez@login:~$ 

From that extensive list, you'll basically work with the following ones:

HOME=/homedt<CONTAINER>/<USERNAME>
PATH=/homedtXX/username/bin:/sge/bin/linux-x64:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/clues/0.89
LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
LD_LIBRARY_PATH=/sge/lib/linux-x64

Additionally, the example above shows some variables that are not included by default: 

MATLAB_JAVA=/soft/jdk1.7.0_40/jre/
PATH=$PATH:/soft/MATLAB/R2013b/bin
OLDPWD=/soft/MATLAB/R2013b/sys/jxbrowser/glnxa64/xulrunner/xulrunner-linux-64

The environment variables are managed by the system and inherited from several places at login time. To add or modify the environment variables, we suggest this way:

Edit the file ~/.bashrc. It is placed on your home directory:

ijimenez@login:~$ vi $HOME/.bashrc

and go to the last lines of the file. In the following example, we'll add the environment variables for the Matlab Java JVM and we'll add Matlab to our application path:

# Create a new environment variable to override Matlab's own Java and use our own:
# -------------------------------------------------------------------------------
export MATLAB_JAVA=/soft/jdk1.7.0_40/jre/
#
# Add the path of matlab's binary to the system PATH
# -------------------------------------------------- 
export PATH=$PATH:/soft/MATLAB/R2013b/bin:/sge/matlab

To process the new version of our .bashrc, we have to logout and login again, as .bashrc is processed on login time.

Modules

The Modules package provides for the dynamic modification of the user's environment via modulefiles.

$ module avail
------------------------- /soft/lmod/lmod/modulefiles/Core ---------------------
 lmod/lmod    python/2.7.5    python/3.3.2   (D)    settarg/settarg    stress/1.0.4    x86_open64/4.5.2.1
The next command show wich modules are available:
$ module spider
module command with parameter show wich packages are available:
$ module spider <modulename>

to load and list modules execute the following command:

$  module load<modulename>
$ module list

The modules tool is useful to load and unload environment variables without worrying about .bashrc, but shows its best behavior if you want to dinamically change from two different installed versions of a package in the system. If you look carefully, you'll see that we have two differetn versions of python, and default is 3.3.2:

ijimenez@login:~$ module avail ------------------------------------------------------ /soft/lmod/lmod/modulefiles/Core ----------------------------------------------------------------------
   lmod/lmod    python/2.7.5    python/3.3.2   (D)    settarg/settarg    stress/1.0.4    x86_open64/4.5.2.1   Where:
   (D):  Default Module ijimenez@login:~$ module load python/3.3.2
ijimenez@login:~$ python --version
Python 3.3.2

Let's change the python version:

ijimenez@login:~$ module load python/2.7.5 ijimenez@login:~$ python --version
Python 2.7.5

3. Transferring files

Transferring files

Data movement has to be done through SFTP because SSH is the only allowed method to connect to the cluster. 

To upload / download files to the cluster using a Mac / Windows client, follow these steps:

Mac / Windows:

Download Filezilla Client from https://filezilla-project.org/

Install Filezilla client and configure new site. File > Site Manager

Click connect and accept RSA key:

Accept, your remote files are listed on the right column. Drag and drop or copy and paste from left column (local directory) to the right:

 

Linux has native SFTP support, you can choose to use it from a terminal or with graphical support:

sftp is an interactive file transfer program, use sftp client from your Linux terminal.

$ sftp uhpc@hpc.dtic.upf.edu
uhpc@hpc.dtic.upf.edu's password: 
Connected to hpc.dtic.upf.edu.
sftp> pwd
Remote working directory: /homedtic/uhpc

Alternatively you can use the graphical file browser of gnome:

Login to hpc.dtic.upf.edu with your username writing the next command on your file manager sftp://<username>@hpc.dtic.upf.edu

Type your password and click to log in.

Browse your files when you are connected:

4. Using the Grid Engine

4.1. Cluster queues, resources and limits

Cluster queues

A cluster queue is a resource that can handle and execute user jobs. Depending on the job's demands, the job will be executed on a given queue or another. Every queue has its own limits, behavior and default values. Currently, Snow cluster has six different queues shown on the following table:

Queue name Allowed use Comment
short.q Batch processing Intended for short time, low-cpu jobs, that must be processed and dispatched fast. 
default.q Batch processing Intended for long time, high cpu, high memory demanding jobs in amd processors.
intel.q Batch processing Intended for long time, high cpu, high memory demanding jobs in intel processors.
cuda.q Batch processing Intended for solve computationally and data-intensive problems using multicore processors GPUs.
inter.q Interactive sessions Intended to manage interactive sessions on cluster nodes. Limited resources.
all.q Batch and interactive For testing and administration purposes only.

All queues are defined with some common parameters. Unless specified otherwise, these parameters are inherited by all the jobs that run on these queues. This imposes limits, for example, on time or consumed resources for the jobs that run inside a given queue. Let's see, for example, the configuration of the queue short.q:

ijimenez@login:~$ qconf -sq short.q
qname                 short.q
hostlist              @allhosts
seq_no                1
load_thresholds       np_load_avg=1.75
suspend_thresholds    NONE
nsuspend              1
suspend_interval      00:05:00
priority              0
min_cpu_interval      00:05:00
processors            UNDEFINED
qtype                 BATCH
ckpt_list             NONE
pe_list               make smp ompi matlab
rerun                 FALSE
slots                 1,[@abudhabi=64]
tmpdir                /scratch
shell                 /bin/bash
prolog                NONE
epilog                NONE
shell_start_mode      posix_compliant
[ ... ]
terminate_method      NONE
notify                00:00:60
[ ... ]
initial_state         default
s_rt                  01:55:00
h_rt                  02:00:00
s_cpu                 127:50:00
h_cpu                 128:00:00
s_fsize               INFINITY
h_fsize               INFINITY
s_data                INFINITY
h_data                INFINITY
s_stack               INFINITY
h_stack               INFINITY
s_core                INFINITY
h_core                INFINITY
s_rss                 INFINITY
h_rss                 INFINITY
s_vmem                INFINITY
h_vmem                INFINITY
ijimenez@login:~$ 

The parameters s_rt, h_rt, s_cpu and h_cpu force all the jobs submitted to this queue to have the corresponding limits.

Cluster limits

When a given user registers a job on the scheduler, limits are applied. If the job's requeriments are higher than the available resources, the job will wait on the queue until the resources get free. But if the job's requeriments are higher than the limits, the job cannot be registered. The limits are setup at three different levels: user, research group and queue.

Cluster limits are defined as resource quotas, and are explained in the next tables:

Table 1. short.q limits

Item     Limit  Comment
Queue type Batch processing No user interactive usage allowed
Wall time 2 hours Every job can run for two hours in the cluster, no matter how many CPUs it’s going to use
CPU time 128 hours Every job can use a total time of 128 hours. That is, we allow a job to be on the system for 2 hours using all queue resources: 2 hours * 64 cores = 128 hours of calculation
Maximum user alllocatable slots   32 cores A single user can allocate up to 64 slots per job, so maximum parallelism allowed in this queue is 32
Maximum research group allocatable slots 256 Researchers on the same research group can allocate up to 256 processors on this queue

Table 2. default.q limits

Item Limit  Comment
Queue type             Batch processing     No user interactive usage allowed
Wall time Not set No time limit is set
CPU time Not set No time limit is set
Maximum user alllocatable slots   320 cores A single user can allocate up to 320 slots per job, so maximum parallelism allowed in this queue is 320
Maximum research group allocatable slots 576 cores  Researchers on the same research group can allocate up to 576 processors on this queue

Table 3. interactive.q limits

Item Limit Comment
Queue type             Interactive    User interaction managed by the scheduler
Wall time 4 hours Every user can have up to 4 hours of interactive session in the node
CPU time  64 hours Every user can have up to 64 CPU hours of interactive session in the node
Minimum slots available 16 cores In case of heavy cluster usage, al least 16 processors are always reserved for this queue.
Maximum user allocatable slots 4 cores A single user can allocate up to 4 processors on a single interactive session
Maximum research group allocatable slots  8 Researchers on the same research group can allocate up to 8 processors on this queue   


This behavior is modeled as a resource as shown below:

{
   name         maxslots
   description  "Max slots per user"
   enabled      TRUE
   limit        users {*} queues short.q to slots=32
   limit        projects {*} queues short.q to slots=256
   limit        users {*} queues default.q to slots=256
   limit        projects {*} queues default.q to slots=576
   limit        queues !inter.q to slots=688
}

4.2. Submitting basic jobs

Submitting basic jobs

The basic command to send a job is qsub. If don't specified otherwise, the scheduler is going to guess as best as it can in which queue the job is gonna be registered. 

ijimenez@login:~$ qsub sleeper.sh 
Your job 153624 ("Sleeper") has been submitted
ijimenez@login:~$ 

However, there is an exception when submitting jobs to the Intel processor nodes, because they only support the intel.q queue. Thus, it is composory to specify this argument in the qsub command format.

ijimenez@login:~$ qsub -q intel.q Tasa.sh 
Your job 673725 ("Tasa") has been submitted
ijimenez@login:~$

Once registered, the scheduler tells the job ID. Keep it handy, in case of problems you'll need this value to debug what happens.

Although you can specify the options when calling qsub, when submitting a job execution request it is strongly advised that all options be provided within a job definition file (in these examples the file will be called "job.sge"). This file will contain the command you wish to execute and any Grid Engine resource request options that you need. 

ijimenez@login:~$ vi job.sge

All Grid Engine options are preceded by the string "#$ ", and all other lines in the file are executed by the shell (the default shell is /bin/bash):

#!/bin/bash
#$ -N Test
#$ -q short.q
#$ -cwd
uname -a

The "-N" option sets the name of the job. This name is used to create the output log files for the job. We recommend using a capital letter for the job name is order to distinguish these log files from the other files in your working directory. This makes it easier to delete the log files later.

The "-q" option requests the queue in which the job should run.  The "-cwd" option instructs the Grid Engine to execute your jobs from the directory in which you submit the job (using qsub). This is recommended because it eliminates the need to "cd" to the correct sub-directory for job execution, and it ensures that you log files will be located in the same directory as your job-definition file.

Remember to set the "-q" parameter pointing to intel.q to use the Intel proceesor nodes.

#!/bin/bash
#$ -N Test
#$ -q intel.q
#$ -cwd
uname -a

We can monitor how our job is doing with the qstat command:

ijimenez@login:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
153646 0.00000 Test       ijimenez     qw    12/18/2013 17:12:00                                    1       
ijimenez@login:~$ 

ijimenez@login:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
 673734 0.62465 Tasa       ijimenez         r     04/15/2015 17:45:36 intel.q@node14                     1    

The qstat output shows the task ID of our job (job-IF), priority (prior) who has launched it (user), which is the state of the job (state) , when and where it has been registered (submit and queueu), and how many slots it's been using (slots). The most informatinal column is state, as it shows what is actually the job doing.

Most common states are described in the following table:

Fig 1. Description of job states

Abbreviate State Comment
qw Queue waiting The job is waiting to be assigned to a given queue
t Transferring Job is assigned to a queue and it's being transferred to one or more execution hosts.
r Running The job is running on the execution host
E Error The job has failed for some reason and it's still running. Output is being sent to the file specified with -e flag, to the default error file otherwise.
h Hold The job is being hold for some reason. Most common is to nbe waiting for another job to finish
R Restarted The job has been restarted for some reason. Most common reason are errors on the execution host, and the job is sent to another execution host to be processed again.

Job run time

Each queue has specific policies that enforce how long jobs are allowed to execute: short* queues allow up to 2 hours, medium* queues allow up to 24 hours, and long* queues have no runtime limit. When you submit a job, you are either implicitly or explicitly indicating how long the job is expected to run. The first way to to indicate the maximum runtime using the format

#!/bin/bash
#$ -N Test
#$ -cwd
#$ -l s_rt=04:30:00
#$ -l h_rt=05:00:00
uname -a

This way, the scheduler will check which one of our queues is matching the job requeriments, and will send the job to the most appropiate queue. In this case, the job is going to be processed by the default.q, as long as we're requesting a soft limit of 4:30 hours and a hard limit of 05:00 hours and it exceeds the short.q threshold:

ijimenez@login:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                slots ja-task-ID
--------------------------------------------------------------------------------------------------------
153662 0.50000 Test       ijimenez     r     12/18/2013 17:51:08 default.q@node10        1  

The main resources to control job time  executions are the flags shown below:

Fig 2. Parameters to control execution time

Flag  Request Comment
-l s_rt=hh:mm:ss Sets a soft-limit walltime of hh:mm:ss  The system will send a soft SIGN if execution time limit, set to hh:mm:ss is exceeded
-l h_rt=hh:mm:ss Sets a hard-limit walltime of hh:mm:ss The execution time limit for the job will be set to hh:mm:ss. If exceeded, the job will error and die.
-l s_cpu=hh:mm:ss Sets a soft-limit CPU of hh:mm:ss The system will send a soft SIGN if CPU time limit, set to hh:mm:ss is exceeded
-l h_cpu=hh:mm:ss Sets a hard-limit CPU of hh:mm:ss The CPU time limit for the job will be set to hh:mm:ss. If exceeded, the job will error and die.
-q <queue_name> Requests an specific queue for the job Forces the scheduler to register the job on a given queue. If job requeriments does not fit the queue, the job stands on qw state forever unless deleted
  • Walltime: is the 'real time' a job is running.
  • CPU time: is the CPU time a job is running
  • If a job is not parallelized, the walltime and the CPU time are the same; but ifa given program lauches N threads, the CPU time is the real time spent in execution * N threads launched.

Redirection output and error files

By default, if not specified otherwise, the scheduler will redirect the output of anuy job you launch to a couple of files, placed on yout $HOME, called <job_name>.e<job_id> and <job_name>.o<job_id>. After a few executions and test, probably your $HOME will look like this:

ijimenez@login:~$ ls
examples             hostname.sh.o153601  Program settings  Sleeper.e153624  Sleeper.o153624  Test.e153646  Test.e153662  Test.o153648
__hostname.err       job.sge              scripts           Sleeper.e153625  Sleeper.o153625  Test.e153647  Test.o153627  Test.o153649
__hostname.out       Matlab               Sleeper.e153622   Sleeper.o153622  sleeper.sh       Test.e153648  Test.o153646  Test.o153662
hostname.sh.e153601  programari           Sleeper.e153623   Sleeper.o153623  Test.e153627     Test.e153649  Test.o153647  user-scripts
ijimenez@login:~$ 

As a general rule, you are advised to use the following flags to redirect the input and error files:

Fig 3. Redirecting output and error files

Flag  Request Comment
-e <path>/<filename> Redirect error file The system will create the given file on the path specified and will redirect the job's error file here. If name is not specified, default name will apply.
-o <path>/<filename> Redirect output file The system will create the given file on the path specified and will redirect the job's output file here. If name is not specified, default name will apply.
-cwd Change error and output to the working directory The output file and the error file will be placed in the directory from which 'qsub' is called.

Of course, we can place these option on our job definition file:

ijimenez@login:~/examples$ vi job.sge 
#!/bin/bash
#$ -N Test
#$ -cwd
#$ -l s_rt=04:30:00
#$ -l h_rt=05:00:00
#$ -e $HOME/examples/uname/error
#$ -o $HOME/examples/uname/output
uname -a

And when launching the job, we'll see the output files created at $HOME/example/uname/output. If no error is reported, an empty file will be created.

ijimenez@login:~/examples/uname$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
153670 0.50000 Test       ijimenez     r     12/18/2013 18:19:08 default.q@node10                   1 
ijimenez@login:~/examples/uname$ cd output/
ijimenez@login:~/examples/uname/output$ cat Test.o153670
Linux node10 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
ijimenez@login:~/examples/uname/output$ 

Sending notifications

These examples run in seconds, buy mayhaps you're running jobs and simulations for hours or days, even weeks. The cluster Snow is geared with an exim4 email system, and you can configure your jobs to send you notifications once they're finished or if something goes wrong:

Fig 4. Sending notifications

Flag  Request Comment
-M <valid_email> Activate mailing feature and emal to address <valid_email> The system will send an email to the provided email when any of the the switches provided happen
-m b Begin Mail is sent at the beginning of the job, once the job enters the state 'r'
-m e End Mail is sent at the end of the job, when the job unregisters from the scheduler and no error is reported (no 'E' state)
-m a Aborted or rescheduled Mail is sent if job is aborted or rescheduled ('E' state appears)
-m s Suspended Mail is sent if job is suspended (usually by a user with higher privileges, 's' state appears)

Of course, we'll add these options to our job defintiion and we'll run:

#!/bin/bash
#$ -N Test
#$ -cwd
#$ -l s_rt=04:30:00
#$ -l h_rt=05:00:00
#$ -e $HOME/examples/uname/error
#$ -o $HOME/examples/uname/output
#$ -M ivan.jimenez@upf.edu
#$ -m bea
uname -a
ijimenez@login:~/examples/uname$ qsub job.sge
Your job 153676 ("Test") has been submitted

Deleting and modifying jobs

We can modify the requeriments of a job while it's waiting to be processed. Once it's

Fig 4. Deleting a job

Flag  Request Comment
qdel <job_id> Delete the job The system will remove the job and all its dependencies from the queues and the execution hosts

 

4.3. Submitting advanced jobs

1. Submitting an array of jobs

2. Submitting Parallel jobs

- Open MPI


1. Submitting an array of jobs

Sometimes you'll need to submit a lot of jobs in a single script. In this case, perhaps you'll need to handle thousands of independent jobs and its own input, output and error files. As a general rule, it's not a good idea generate thousands of separate job scripts which are submitted to the cluster.

Sun Grid Engine allows users to submit a special kind of job which executes a single scripts with N different input files. This is called 'array job' and it's submitted to the cluster only once and can be managed by as a single job. 

To create an array job is slightly different than create a single job, as long as we have to manage the N different input jobs. In this example, we'll simply copying the contents of the N different input files to the output files:
 
First, we create the generic files to place the input data. As a general rule, the easiest way to name the input files is to append an integer at the end of the input file, and use the environment variable SGE_TASK_ID to handle them. Whit the following 'for' command we create ten input, empty files:
for i in {1..10}; do echo "File $i" > data-$i.log; done

Let's verify the input files:

uhpc@login:~/Source/array-example$ ls -l
total 0
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-10.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-1.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-2.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-3.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-4.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-5.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-6.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-7.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-8.log
-rw-r--r-- 1 uhpc info_users 0 Dec 18 17:23 data-9.log
When creating an array job, -t flag will handle the way the scheduler will take the input files:

Fig 1. Basic parameters for array job

Flag  Request Comment
-N <job_name> Job Name Definition
The name of the job. The name should follow the "name" definition in sge_types(1).  Invalid job names will  be  denied  at submit time.
-t  n[-m[:s]] Array index limits
The option arguments n, m and s will be  available  through  the  environment  variables              SGE_TASK_FIRST, SGE_TASK_LAST and  SGE_TASK_STEPSIZE.

In the example, we want to pick ten files, starting on 1, ending at 10, and pick them one after another. So, out -t option has to be -t 1-10:1 (start on one, end on 10, increment 1 each time):  

#!/bin/bash
#$ -N FileArray
#$ -t 1-10:1
FILESPATH=/homedtic/uhpc/Source/array-example
cat $FILESPATH/data-${SGE_TASK_ID}.log > $FILESPATH/output-${SGE_TASK_ID}.log
exit 0;

Submit array job using qsub command:

uhpc@login:~/Source/array-example$ qsub submit-array.sh
Your job-array 153660.1-10:1 ("FileArray") has been submitted
Monitor job status using qstat command, 10 jobs running on short.q queue. You can see duplicated 10 times the job-ID 153672 on different nodes in the cluster. This time, the qstat commands shows additional output at the task-ID column:
uhpc@login:~/Source/array-example$ qstat
job-ID  prior   name       user         state submit/start at     queue          slots ja-task-ID
-------------------------------------------------------------------
153672 0.50000 FileArray  uhpc         r     12/18/2013 18:30:38 short.q@node10                 1 1
153672 0.50000 FileArray  uhpc         r     12/18/2013 18:30:38 short.q@node05                 1 2
153672 0.50000 FileArray  uhpc         r     12/18/2013 18:30:38 short.q@node09                 1 3
153672 0.50000 FileArray  uhpc         r     12/18/2013 18:30:38 short.q@node03                 1 4
153672 0.50000 FileArray  uhpc         t     12/18/2013 18:30:38 short.q@node10                 1 5
153672 0.50000 FileArray  uhpc         t     12/18/2013 18:30:38 short.q@node05                 1 6
153672 0.50000 FileArray  uhpc         t     12/18/2013 18:30:38 short.q@node09                 1 7
153672 0.50000 FileArray  uhpc         t     12/18/2013 18:30:38 short.q@node03                 1 8
153672 0.50000 FileArray  uhpc         r     12/18/2013 18:30:38 short.q@node04                 1 9
153672 0.50000 FileArray  uhpc         t     12/18/2013 18:30:38 short.q@node10                1 10
Output files are stored where script file have output-* defined.
uhpc@login:~/Source/array-example$ ls -l output-*
-rw-r--r-- 1 uhpc info_users 8 Dec 18 18:30 output-10.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-1.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-2.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-3.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-4.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-5.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-6.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-7.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-8.log
-rw-r--r-- 1 uhpc info_users 7 Dec 18 18:30 output-9.log
The content of every file is the same than his parent input file, as we have specified in our script:
uhpc@login:~/Source/array-example$ cat output-1.log
File 1
uhpc@login:~/Source/array-example$ cat data-1.log
File 1
uhpc@login:~/Source/array-example$ cat output-1.log
File 1
uhpc@login:~/Source/array-example$ cat data-10.log
File 10
uhpc@login:~/Source/array-example$ cat output-10.log
File 10
To delete all jobs of an array, execute command qdel specifying the job-ID of the whole array job. It will mark all ten independent jobs for deletion:
uhpc@login:~/Source/array-example$ qdel 153677
uhpc has registered the job-array task 153677.1 for deletion
uhpc has registered the job-array task 153677.2 for deletion
uhpc has registered the job-array task 153677.3 for deletion
uhpc has registered the job-array task 153677.4 for deletion
uhpc has registered the job-array task 153677.5 for deletion
uhpc has registered the job-array task 153677.6 for deletion
uhpc has registered the job-array task 153677.7 for deletion
uhpc has registered the job-array task 153677.8 for deletion
uhpc has registered the job-array task 153677.9 for deletion
uhpc has registered the job-array task 153677.10 for deletion
To delete a single job of an array, specify the TASK_ID at the ending of the job-ID. The syntax should follow this way: qdel array-job-id <dot> element_job_id: 
uhpc@login:~/Source/array-example$ qdel 153679.1
uhpc has registered the job-array task 153679.1 for deletion

 

2. Submitting Parallel jobs
 
 
Open MPI divides resources in cluster slots, the next example show as you should execute an OMP binary in a parallel environment with Open MPI (mpiexec). The next example  show how you can submit a job with NPB performance tools.
 
Write an script for qsub command (see basic example to build it):
#!/bin/bash
#$ -pe ompi* 256
#$ -N bt.B
export OMP_NUM_THREADS=256
/soft/openmpi/openmpi-1.6.5/bin/mpiexec  /PATH/TO/BIN/NPB3.2/NPB3.2-OMP/bin/bt.B

Fig 2. Parameters Open MPI Script

Flag  Request Comment
-N <job_name> Job Name Definition The name of the job. The name should follow the "name" definition in sge_types(1).  Invalid job names will  be  denied  at submit time.
-pe ompi* <slots> Parallel enviroment 

slots = set integer number if you submit an MPI job that requests a lot of slots (ex. more than 16).

You can ask to SGE how many slots are available by using a range descriptor

#$ -pe ompi* 8,16 (schedule when the system can provide either 16 or 8 slots)
#$ -pe ompi* 4-16 (requests anywhere from 4 to 16 slots on a single host)

 
In this example bt.B is running with 256 slots on node02:
$ qsub pe-qsub.sh
Your job 153804 ("bt.B") has been submitted

You can see the number of slots using qstat command:

qstat
job-ID  prior   name       user         state submit/start at     queue          slots ja-task-ID
--------------------------------------------------------------------
153804 0.52500 bt.B       uhpc      r     12/18/2013 20:16:40 default.q@node02              256       

Node02 has 256 processes running  for bt.B program.

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+ COMMAND
41020 uhpc   20   0  321m 175m  944 R   168  0.1   0:33.14 bt.B
41664 uhpc   20   0  321m 175m  944 S   162  0.1   0:31.71 bt.B
41369 uhpc   20   0  321m 175m  944 S   159  0.1   0:32.39 bt.B
41215 uhpc   20   0  321m 175m  944 S   145  0.1   0:32.66 bt.B
41942 uhpc   20   0  321m 175m  944 R   144  0.1   0:29.93 bt.B
40828 uhpc   20   0  321m 175m  944 S   142  0.1   0:32.28 bt.B
42872 uhpc   20   0  321m 175m  944 R   139  0.1   0:28.68 bt.B
41552 uhpc   20   0  321m 175m  944 R   138  0.1   0:30.52 bt.B
42569 uhpc   20   0  321m 175m  944 S   137  0.1   0:33.24 bt.B
42817 uhpc   20   0  321m 175m  944 R   136  0.1   0:31.24 bt.B
42239 uhpc   20   0  321m 175m  944 R   134  0.1   0:30.50 bt.B
40535 uhpc   20   0  321m 175m  944 S   133  0.1   0:31.00 bt.B            

Many ouput files are written when the job finishes:

bt.B.e153808  bt.B.o153808  bt.B.pe153806  bt.B.pe153808  bt.B.po153806  bt.B.po153808

 

 

4.4. Submitting interactive jobs

Submitting interactive jobs

Sometimes users need more computational power to run commands from the command line. As a general rule, it is NOT a good idea to launch these commands from the login node, because the login node is a server designed to manage user sessions, it has limited resources and it does not behave well with CPU extensive sessions. Additionally, intensive CPU usage on the login node could reduce session speed and performance for the rest of the users. 

To solve this need, users can request an interactive job on a computing node with the qrsh command. Interactive jobs are only allowed on a single queue. qrsh behaves the same way as the rest of the scheduler commands: it inherits values from the queue where the job runs, but also user options can be specified.

Fig 1. inter.q values

Item Limit  Comment
Queue type             Interactive    User interaction managed by the scheduler.
Wall time 4 hours Every user can user can have up to 4 hours of interactive session in the node.
CPU time  64 hours Every user can user can have up to 64 CPU hours of interactive session in the node.
Architectures AMD/INTEL Node 11 for AMD architecture, Node 12 for INTEL architecture. 6 slots for each host.
Minimum slots available   12 cores

In case of heavy cluster usage, 16 processors are always reserved for this queue.

Maximum user allocatable slots 4 cores A single user cal allocate up to 4 processors on a single interactive session.
Maximum research group allocatable slots  8 Researchers on the same research group can allocate up to 576 processors on this queue. 
 
ijimenez@login:~$ qrsh< Linux node09 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64   The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.   Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Fri Dec 13 13:31:31 2013 from 192.168.7.6 ijimenez@node09:~$ 

The following command allows us to use an specific node: qrsh -l h='node11' -now no

This way, I request an interactive job inheriting the inter.q parameters and limits. As long as the session is, internally, a job registered on a queue, the qstat command will show a running job:

ijimenez@node10:~$ qstat 
job-ID prior name user state submit/start at queue slots ja-task-ID 
----------------------------------------------------------------------------------------------------------------- 
153890 0.47500 QRLOGIN ijimenez r 12/27/2013 09:31:40 inter.q@node10 1 
ijimenez@node10:~

The common qsub modifiers can be used when calling qrsh. Additionally, if the connection to the login node was configured to handle X graphics, the qrsh command on the nodes will also forward the graphical commands. Most common usage is to call Matlab within a node to take advantage of the higher amount of memory available:

 

4.5. Monitoring

Monitoring jobs

We can monitor our jobs with the qstat command. If we call qstat without arguments, it will show the state of the jobs for the current user:

ijimenez@login:~/Matlab/Mandelbrot$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 154084 0.47598 Mandelbrot ijimenez     r     01/07/2014 15:37:59 short.q@node11                     6        
ijimenez@login:~/Matlab/Mandelbrot$ 
 
Next table shows the output for qstat command:
 

Fig 1. qstat information

Field Description Comment
job-ID Numerical ID of the job Numerical identifier of the job 
prior Priority Priority of the job, from lowest (-20) to highest (20). Similar to Linux 'nice' command, is used for the cluster administrators to priorize jobs on a given queue
name Name of job owner The name of the user who owns the job
state Job status Job status. Available states are shown Table 3.
submit/start at Start time Date and time when the scheduler has registered the job. To register the job does not imply to execute it. Once the job is registered, it must wait for the selected resources to be free.
queue Queue Queue where the job runs. 
slots Slots Number of processors used by the job
ja-task-ID Parallel job Task ID Task-ID. Only shown in the case of array jobs,
 
 

Fig 2. Description of job states

Abbreviate State Comment
qw Queue waiting The job is waiting to be assigned to a given queue
t Transferring Job is assigned to a queue and it's being transferred to one or more execution hosts.
r Running The job is running on the execution host
E Error The job has failed for some reason and it's still running. Output is being sent to the file specified with -e flag, to the default error file otherwise.
h Hold The job is being hold for some reason. Most common is to nbe waiting for another job to finish
R Restarted The job has been restarted for some reason. Most common reason are errors on the execution host, and the job is sent to another
 
The qstat command accepts several arguments:
 
Fig 3 qstat command options
 
Command Option Comment
-f Show full output Show full information of all used slots
-F  Show full output Show complete information of all used slots
-u <user> Show jobs for a given user Show the jobs for a given users
-j <job_id> Show schedule information Shows scheduling options for a given job_id

 

 

4.6. Submitting cuda jobs

In this section, we have submitted a basic job through the cuda queue using the qsub command.

Cuda is a parallel computing language and programming model invented by NVIDIA. It enables dramatic improvements in computing performance by harnessing the power of the Graphics Processing Unit (GPU). 

First of all, we have created the file as we have explained before, all Grid Engine options are preceded by the string #$, and all other lines in the file are executed by the shell (the default shell is /bin/bash):

#!/bin/bash

#$ -N CudaSample
#$ -l gpu=1
#$ -q default.q
#$ -e $HOME/logs/$JOB_NAME-$JOB_ID.err
#$ -o $HOME/logs/$JOB_NAME-$JOB_ID.out
module load cuda/7.5
/soft/cuda/NVIDIA_CUDA-7.5_Samples/0_Simple/vectorAdd/vectorAdd

IMPORTANT: This qsub sample script request a cuda resource, this means that your job will be executed in a single cuda card(all cuda cores of this card will be available for your job). Please do not request more than one gpu, cuda hosts only own one physical cuda card; if you change it, your job will be waiting forever.

Script options:

-N This script is called MyFirstJob2Cuda:

-l Resource type.

-q Parameter is set to cuda.q to use the multicore processors GPUs.

-e error files are created in the cuda/error

-o output files are created in the cuda/output and cuda/error directories.

 

 

 

 

 

 

 

To send the job through the cuda queue, it is necessary to load the module cuda/7.5.

module load cuda/7.5

This script executes a sample file.

/soft/cuda/NVIDIA_CUDA-7.5_Samples/0_Simple/vectorAdd/vectorAdd.

If you want to do some tests, in the following path you can find other samples to use:

/soft/cuda/NVIDIA_CUDA-7.5_Samples/

When the file is saved, we create the folders:

Finally, we launch the job and we monitor it with ‘qstat’ command:

In this case, we have obtained a success execution, you can check the output directory to see the results:

 

4.7. Select Debian 8 as resource

Compute nodes are currently under operating system upgrade process from Debian 7.9 to Debian 8.6; all nodes are in the same cluster and compute queues of the Open Grid Scheduler. 
 
Main differences and news:
  • - Last stable release of Debian 8 Release 6 (September 17th, 2016).
  • - Open Grid Scheduler integration.
  • - Same user homes for all nodes. 
  • - New software repository compiled on demand for HPC cluster.
    • - Python 3.5.3 (SimpleITK, tensorflow, theano, scipy, lmdb, pandas, ...)
    • - GCC compiler 6.2
    • - CUDA toolkit 8.0.4
    • - OpenMPI 2.0.1
    • - lmod 5.1.5
    • - 3D-Caffe
    • [If you need more software, just request it by CAU]
 
By default this systems cannot be selected, unless otherwise is stated; this guideline provide you the instructions to connect and compute in Debian 8.
 
Debian 8 is a resource of the cluster named as 'debian8' or shortcut 'd8', to use that it must be specified ondemand. 
 
Some common resources of the cluster:
- cpu
​- slots
- gpu
- mem_used and mem_free
- h_cpu and h_vmem
- debian8

Recomendation: First of all you should keep in mind if you have compiled software in your home, compiled software in Debian 8 might not work in older releases. Our recomendation is execute in the system that you compiled your software, if not, you must recompile.


* Interactive session:

To request an interactive session, first of all you must be logged in login node. Then execute next command with 'd8' resource:
<your_user>@login:~$ qrsh -l d8

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Jan 18 09:08:25 2017 from 192.168.7.6
<your_user>@node11:~$
Grid Scheduler avaluate your needs and select a node with this resource (interactive tasks can be checked as usual):
<your_user>@node11:~$ qstat

job-ID  prior   name       user         state submit/start at     queue             slots ja-task-ID 
----------------------------------------------------------------------------------------------
3832099 0.44872 QRLOGIN    <your_user>      r     01/18/2017 09:28:10 inter.q@node11               1        

* Batch session:

An example of file script to request (for example) gpu, debian8 and default.q queue:
 
#!/bin/bash
#$ -N CudaSample
#$ -l gpu=1
#$ -l debian8
#$ -q default.q
#$ -e $HOME/logs/$JOB_NAME-$JOB_ID.err
#$ -o $HOME/logs/$JOB_NAME-$JOB_ID.out

module load cuda/8.0
/soft/cuda/NVIDIA_CUDA-8.0_Samples/0_Simple/vectorAdd/vectorAdd
hostname
 
Grid Scheduler evaluate your needs and select a node to compute your batch job:
<your_user>@login:~/Scripts$ qstat

job-ID  prior   name       user         state submit/start at     queue            slots ja-task-ID 
---------------------------------------------------------------------------------------------------
3832177 0.30833 CudaSample <your_user>      r     01/18/2017 09:35:25 default.q@node11            1       
Result of the job executed:
<your_user>@login:~/Scripts$ cat ../logs/CudaSample-3832177.*

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
node11

5. Working with Matlab and SGE

5.1.Running basic Matlab jobs over SGE

MATLAB is a high-level language and interactive environment for numerical computation, visualization, and programming. Using MATLAB, you can analyze data, develop algorithms, and create models and applications. The language, tools, and built-in math functions enable you to explore multiple approaches and reach a solution faster than with spreadsheets or traditional programming languages, such as C/C++ or Java.

Before using Matlab, you should check if required environment variables are properly set. If they are not, set them following these steps:

Export Matlab variables on your user profile (you can define it on your .basrc profile):

export MATLAB_JAVA=/soft/jdk1.7.0_40/jre/
export PATH=$PATH:/soft/MATLAB/R2013b/bin
export OLDPWD=/soft/MATLAB/R2013b/sys/jxbrowser/glnxa64/xulrunner/xulrunner-linux-64

When setting environment vatiables in .bashrc, a new login event is required to have them processed. Logout, login again and check if envirionment variables are correclty set. The output shoud be as follows:

ijimenez@login:~$ which matlab
/soft/MATLAB/R2013b/bin/matlab
ijimenez@login:~$ echo $MATLAB_JAVA
ijimenez@login:~$ echo $OLDPWD
/soft/MATLAB/R2013b/sys/jxbrowser/glnxa64/xulrunner/xulrunner-linux-64<
ijimenez@login:~$ 

Once done, connect to a node with 'qrsh' command. You'll be redirected to a free computing node where you will run Matlab:

uhpc@login:~$ qrsh
Warning: Permanently added '[node08]:55494,[192.168.7.108]:55494' (ECDSA) to the list of known hosts.
Linux node08 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
uhpc@node08:~$ 

Running interactive Matlab sessions

Following is the easiest way of running Matlab in the cluster. This method should be only used to testing previously, outside-developed code purposes. Developing code directly on the cluster is not recommended and shoul be avoided as a general rule.

From the node, you can choose to run Matlab with or without graphical support (default is to have GUI). Execute Matlab GUI console using 'matlab &' command on your shell:

Your graphical pannel looks like this:

On the left panel you'll see your $HOME files. You can run an script from matlab gui or drag and drop example.m script to Command Window.

Run command is executed automatically:

If oyu don't need graphical Matlab support you can call Matlab with the -nosplash -nodesktop modifiers:

ijimenez@node06:~$ matlab -nosplash -nodesktop
                             < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                   R2013b (8.2.0.701) 64-bit (glnxa64)
                               August 13, 2013
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.

>> 1+1
ans =      2 >> 

Fig 1. Matlab runtime modifiers

Flag  Request Comment
-help Help Shows Matlab help
-e Display environment variables Display ALL environment variables. If the status return is not 0, some corrective actions may be needed. Matlab is not run.
-n Display diagnostics Display environment variables, libraries, arguments and other diagnostic information to the standard output. Matlab is not run
-arch Architecture request Start Matlab assuming processor architecture arch.
-nodisplay Disable X Do not display any X command. Matlab's JVM is started
-nojvm Disable Matlab JVM Do not start Matlab's JAva virtual machine. Any Matlab extension depending on JVM will not run
-r <file.m> Run file inmediately Start Matlab and run file.m inmediately after
-logfile file.log Send output to file Make a copy of all output to file.log. This include crash reports.

To send Matlab jobs in the background, a combination of Matlab modifiers and SGE options will be used. In the next steps we'll cover how to send a Matlab job to a SGE queue:

1. Write your matlab source in a file with .m extension, the next is named as 'example.m'

x = [1 2 3 4];
fprintf('Example number = %i\n', x)

2. Write a submit job script named as 'matlab-job.sh'. Place it wherever on your home directory, and specify here the desired options.  

#!/bin/sh
#$ -N MatlabJob
#$ -cwd
#$ -o matlab.$JOB_ID.out
#$ -e matlab.$JOB_ID.err
/soft/MATLAB/R2013b/bin/matlab -nojvm -nodisplay -r "example;quit;"

Fig 2. Parameters used in matlab-job.sh

Flag  Request Comment
-N <job_name> Job Name Definition The name of the job. The name should follow the "name" definition in sge_types(1).  Invalid job names will  be  denied  at submit time.
-cwd Current working directory Use current working directory.
-o <file_name> Output file path Standard output to file path identified by JOB_ID.
-e <file_name> Error file path Error output to file path identified by JOB_ID.
-r <comand1;commandN> Command run Execute comands separated by ';'. Note that file extension is not defined.
-nojvm JVM disabled Does not start the JVM software and uses current window. Graphics will not work without the JVM.
-nodisplay Display disabled Also starts the JVM and does not start the desktop in addition it also ignores Commands and the DISPLAY environment variable in LINUX.
-r  Run inmediately Start Matlab and run file.m inmediately after
 

3. Submit your matlab job:

uhpc@login:~$ qsub matlab-job.sh
Your job 153873 ("MatlabJob") has been submitted

4. Monitor the job status:

uhpc@login:~$ qstat
job-ID  prior   name       user         state submit/start at     queue           slots ja-task-ID
----------------------------------------------------------------------------------------
153873 0.00000 MatlabJob  uhpc         qw    12/20/2013 16:41:34                             1       

5. When the job is finished, the output files are created in your Current Working Directory:

-rw-r--r-- 1 uhpc info_users    0 Dec 20 16:16 matlab.153873.err
-rw-r--r-- 1 uhpc info_users  403 Dec 20 16:16 matlab.153873.out

6. The standard output results of Matlab, file with stderr is empty(job without errors):

uhpc@login:~$ cat matlab.153867.out
                            < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                    R2013b (8.2.0.701) 64-bit (glnxa64)
                              August 13, 2013
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
Example number = 1
Example number = 2
Example number = 3
Example number = 4

5.2. Running Matlab advanced jobs over SGE

To achieve better results while running simulations on the cluster, some best practices should be followed. We'll show in this section some tips and techniques to speed up simulations and efficiently use the cluster.

Keep your data organized

As a general rule, plain directories (single folders containing hundreds or thousands of files) should be avoided, as it has performance impact on the NFS filer. An strategy is to create a folder for each experiment you do and, inside this folder, place the directories you need. 

Use the SSD drives when possible

The nodes are geared with SSD drives. The performance of your job will improve if you copy the data to the SSD of the node, process there, and send the results back rather to using always the NFS.

Paralellize when possible

Whenever possible, use SMP multi-processing, parallel loops or other techniques to achieve results faster

Clean the scratch

If you use the SSD drives as scratch, remember to clean the data you don't use after the job is done

In the next section, we'll show two different simulations and how to reach better results: 

Running in serial:

Monte Carlo methods (or Monte Carlo experiments) are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results, and fit perfectly to illustrate the serial vs. parallel method:

1. Create the folder where the experiment data will be placed:

ijimenez@login:~/Matlab$ mkdir MonteCarlo/
ijimenez@login:~/Matlab$ cd MonteCarlo/
ijimenez@login:~/Matlab$ mkdir data
ijimenez@login:~/Matlab$ mkdir out
ijimenez@login:~/Matlab$ mkdir script
ijimenez@login:~/Matlab$ mkdir job-out

2. Place the Matlab jobs in ./script:

ijimenez@login:~/Matlab$ cd script/
ijimenez@login:~/script$ vi montecarlo.m/
%% SERIAL DEMO

%% Init problem
iter = 100000;
sz = 55;
a = zeros(1, iter);

% MonteCarlo simulations
disp('Starting ...');
tic;
for simNum = 1:iter
a(simNum)= myFunction(sz);
end
toc;
 

We show also the code for myFunction:

ijimenez@login:~/script$ vi myFunction.m/

function out = myFunction(in)
        out=max(svd(rand(in)));

3. We'll write down a more complex submission script. Submission code is documented:

#!/bin/bash # Load modules directive
. /etc/profile.d/modules.s

# Copy sources to the SSD:

# First, make sure to delete previous versions of the sources:
# ------------------------------------------------------------
if [ -d /scratch/MonteCarlo ]; then
        rm -Rf /scratch/MonteCarlo
fi

# Second, replicate the structure of the experiment's folder:
# -----------------------------------------------------------
mkdir /scratch/MonteCarlo
mkdir /scratch/MonteCarlo/data
mkdir /scratch/MonteCarlo/error
mkdir /scratch/MonteCarlo/script
mkdir /scratch/MonteCarlo/out

# Third, copy the experiment's data:
# ----------------------------------
cp -rp /homedtic/ijimenez/Matlab/MonteCarlo/data/* /scratch/MonteCarlo/data
cp -rp /homedtic/ijimenez/Matlab/MonteCarlo/script/* /scratch/MonteCarlo/script

# Fourth, prepare the submission parameters:
# Remember SGE options are marked up with '#$':
# ---------------------------------------------
# Requested resources:
#
# Simulation name
# ----------------
#$ -N "MonteCarlo-serial"
#
# Expected walltime: five minutes maximum
# ---------------------------------------
#$ -l h_rt=00:05:00
#
# Shell
# -----
#$ -S /bin/bash
#
# Output and error files go on the user's home:
# -------------------------------------------------
#$ -o /homedtic/ijimenez/Matlab/MonteCarlo/job-out/montecarlo-serial.out
#$ -e /homedtic/ijimenez/Matlab/MonteCarlo/job-out/montecarlo-serial.err
#
# Send me a mail when processed and when finished:
# ------------------------------------------------
#$ -m bea
#$ -M  my.email@upf.edu
#

# Start script
# --------------------------------
#
printf "Starting execution of job $JOB_ID from user $SGE_O_LOGNAME\n"
printf "Starting at `date`\n"
printf "Calling Matlab now\n"
printf "---------------------------\n"
# Execute the script
/soft/MATLAB/R2013b/bin/matlab -nosplash -nojvm -nodesktop -r "run /scratch/MonteCarlo/script/montecarlo.m"
# Copy data back, if any
printf "---------------------------\n"
printf "Matlab processing done. Moving data back\n"
# cp -rf /scratch/MonteCarlo/out/montecarlo.out /homedtic/ijimenez/Matlab/MonteCarlo/out
printf "Job done. Ending at `date`\n"

4. We have the elapsed time in the contents of the .out file in the job-out folder. Running this simulation in serial has taken nearly a minute:

Starting execution of job 153972 from user ijimenez
Starting at Tue Dec 31 10:06:53 CET 2013
Calling Matlab now
Warning: No display specified.  You will not be able to display graphics on the screen.
                            < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                    R2013b (8.2.0.701) 64-bit (glnxa64)
                              August 13, 2013

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.

Starting ...
Elapsed time is 63.783560 seconds.
>> Matlab processing done. Moving data back
Job done. Ending at Tue Dec 31 10:07:59 CET 2013

Running in parallel:

Matlab Parallel Computing Toolbox product offers several features that simplify the development of parallel applications in MATLAB. It offers programming constructs such as parallel loops and distributed arrays that let you extend your serial programming into a parallel domain. You can use these constructs without the requirement of learning a complex parallel language or making significant changes to your existing code. The toolbox supports interactive development, which lets you connect to your cluster from a MATLAB session to interactively perform parallel computations or use them in batch. Currently, the Paralell Computing Toolbox license is limited to 12 local cores per worker. 

Let's run the same MonteCarlo simulation taking advantage of the paralellization. To do this, we must do two things:

a. Reserve a pool of N workers to register the job on SGE (where N is 12 at the maximum)

b. Tell Matlab to use the local pool of workers

1. Let's modify our Matlab script. We'll use the 'parfor' instead of 'for'. We can do it as long as the inner operations on the loop does not depend on the loop iterator, and we'll also use the directive parpool to create the pool using the profile 'local' with 6 slots. More information on Matlab cluster profiles here:

 
%% PARFOR DEMO
 
%% Init problem
% Init pe
parpool('local',6)
 
% MonteCarlo simulations
disp('Starting ...')
tic;
iter    = 100000;
sz      = 55;
a       = zeros(1, iter);
parfor (simNum = 1:iter, 6)
        a(simNum)= myFunction(sz);
end
toc;
delete(gcp)

2. Let's modify also our submission script. We'll include the -pe option to reserve as many slots as we have specified in the previous parfor command. We'll also modify the lines to call the job and the output and error files:

[...]
# Requested resources:
#
# Simulation name
# ----------------
#$ -N "MonteCarlo-paralell"
#
# Paralell environment: we'll use six cores
# on the paralell environment called 'smp'
# -----------------------------------------
#$ -pe smp 6
[...]
# Output and error files go on the user's home:
# -------------------------------------------------
#$ -o /homedtic/ijimenez/Matlab/MonteCarlo/job-out/montecarlo-paralell.out
#$ -e /homedtic/ijimenez/Matlab/MonteCarlo/job-out/montecarlo-paralell.err
[...]
This time, if we check the speedup achieved shown by the output files, we'll see that serial took nearly a minute, and paralell took 13 seconds. This shows a speedup of 1:6:
 
Serial: Elapsed time is 63.909396 seconds.

Paralell:Elapsed time is 13.476630 seconds.

6. External References

Here we sort some external references. Some of them could be useful:

Compiler Options for AMD processors

A quick list of options that can be used on several compilers to speed up and optimize the code on AMD processors:

http://developer.amd.com/wordpress/media/2012/10/CompilerOptQuickRef-63004300.pdf

 

AMD Linux Tuning guide

A complete documentation about the AMD processors. Altough it speaks of previous processor series (62xx, aka Interlagos) most of the options, ideas and suggestions can be applied to our  Abu Dhabi 63xx series.

http://developer.amd.com/wordpress/media/2012/10/51803A_OpteronLinuxTuningGuide_SCREEN.pdf

 

ACML: AMD Core Math Library

Full guide to ACML tool. It can be used as a replacement for generic Linux math libraries to take the most advantage of the AMD architecture

http://developer.amd.com/wordpress/media/2013/05/acml.pdf

 

AMD software optimization guide

This guide provides optimization information and recommendations for AMD Family 15h processors. 

http://developer.amd.com/wordpress/media/2012/10/47414_15h_sw_opt_guide.pdf

7. Asynchronous Job Operator (ajo)

Introduction

Asynchronous Job Operator (AJO) is a software tool designed to provide an easier and transparent method of operating with jobs in a HPC system.

It acts as a gateway between the user interface and the HPC system, and permits users to Submit, Retrieve, Query, Cancel and Erase jobs in a faster and easier way.

                                                                                 

 

Requirements

  • AJO works on GNU/Linux system.
  • Needs to install Ruby 1.9.1 or newer version.
  • Have an SSH client.

 

Installation

  • Download AJO.

            From the home page:  http://rdlab.cs.upc.edu/releases/ajo/

            From the svn repository using public_ajo as key:  

                   svn co http://svn-rdlab.lsi.upc.edu/subversion/ajo/public --username public_ajo

            From hpc.dtic cluster at /soft/public/  directory.

  • The downloaded folder will contain six files, of which the highest configuration is done in the config.rb file.

 

Configuration

Before configuring AJO it is recommended to establish a password-less SSH login connection to the cluster. This is possible through a pair of private/public keys.

  • Generates a pair of private and public key                     

ssh-keygen -t rsa

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /homedtic/noko/.ssh/id_rsa.

Your public key has been saved in /homedtic/noko/.ssh/id_rsa.pub.

 

  • Copy the public key in the cluster login server, by running the next command from the directory where the keys have been generated, 

ssh-copy-id  -i  id_rsa.pub user@hpc.dtic

 

  • Establish connection to the cluster.

ssh hpc.dtic

You should now be able to establish a password-less connection.

 

The main configuration is now done in six options of the config.rb file.

·         SSH options.

Here needs to configure the server to which AJO is going to establish  an ssh connection in the HPC system. In the hpc.dtic.upf.edu cluster is the login server.

Also needs to configure the username, by introducing the personal user name used to access the cluster.

The SSH_CMD line will remain as default, unless there is a problem of initializing the environment variables. In this case you may add the profile directory.

 

An example of the final configuration is found below.

 SERVER = "hpc.dtic.upf.edu"

  USER = <DTIC_USERNAME> # insert your Linux username between quotes

  SSH_CMD = "/usr/bin/ssh #{USER}@#{SERVER} '. /etc/profile;'"

  AJO_DIR = `#{SSH_CMD} 'echo $HOME'`.chomp("\n") + "/.executions"

 

·         SGE options

There is no need to do any modification here. In case there is a problem in finding the SGE_ROOT, the simple quote containing the echo in the first line may be removed.

SGE_ROOT = `#{SSH_CMD} echo $SGE_ROOT`.chomp "\n"

  SGE_ARCH = `#{SSH_CMD} '#{SGE_ROOT}/util/arch'`.chomp "\n"

  SGE_UTIL_PATH = SGE_ROOT + "/bin/" + SGE_ARCH

  QSUB_CMD = SGE_UTIL_PATH + "/qsub"

  QSTAT_CMD = SGE_UTIL_PATH + "/qstat"

  QACCT_CMD = SGE_UTIL_PATH + "/qacct"

  QDEL_CMD = SGE_UTIL_PATH + "/qdel"

 

·         Encryption options

It is recommended to change the User and Server values in the second and third line by something more complex to asure a secure communication between AJO and the HPC system.

The new values may be something random and must be in quotes, as shown next.

  CIPHER_SALT = "ajo"

  CIPHER_KEY = OpenSSL::PKCS5.pbkdf2_hmac_sha1("oy1382-2487*", CIPHER_SALT, 2000, 16)

  CIPHER_IV = OpenSSL::PKCS5.pbkdf2_hmac_sha1("oyserver2487*", CIPHER_SALT, 2000, 16)

 

·         Folders and files options.

This part enables to configure the folders and files that will be handle to and downloaded from the HPC system.

-   FOLDER_ARGS is the folder or folders used by AJO to collect the jobs and scripts to run in the HPC system. The example below shows that AJO collects the jobs and scripts in the folder "ajo3" and runs them in the Cluster.

FOLDER_ARGS = {

    :ajo2 => "/homedtic/#{USER}/ajo3/",

 

-  FILE_ARGS is also used to show AJO what to handle to the cluster. This allows to select a given file o files instead of a whole folder as the previous case. 

FILE_ARGS = {

    :ajofi2 => "/homedtic/#{USER}/ajo2/Proyecto.sge",

    :ajofi3 => "/homedtic/#{USER}/ajo2/Proyecto.m",

 

- FOLDER_OUTPUT: same like FOLDER_ARG but used to recover or retrieve results instead of handling.

FOLDER_OUTPUT = {

    :fajo2 => "/job/",

 

- FILE_OUTPUT: same like FILE_ARG but used to recover or retrieve result instead of handling.

FILE_OUTPUT = {

     :file4 => "job.out",

 

-  RETRIEVE enables users to define the file to be retrieve from the remote result folder.  AJO keeps the result of any execution in std.out and std.err by default.

RETRIEVE = {

   :resultado => "std.out",

   :error => "std.err"

 

·         Commands.

Here one can define the commands that will be executed in the cluster. The format of the command is:     “command #{argument[argument_name]}”

    [

      "sh #{folder_output[:fajo2]}",

      "sh #{folder_args[:ajo2]}/Proyecto.sge",

      "sh #{file_output[:file4]}",

    ]

  end

 

·         Running AJO.

- Summit jobs:

               ajo -c config.rb –s  -library libhpc.rb 

The installation directory has to be specified, making the previous command to become 

/soft/public/ajo_v1.x/ajo -c /soft/public/ajo_v1.x/config.rb -s --library /soft/public/ajo_v1.x/libhpc.rb 

- Retrieve:

When a job is summated correctly the job ID is offered. This will be use to recover the result. 

/soft/public/ajo_v1.x/ajo -c /soft/public/ajo_v1.x/config.rb -r  ID --library /soft/public/ajo_v1.x/libhpc.rb --log-all

An example of ID is d427e4b981f87875ba9389ff261ae272a2551151f5775861064be484480e38b8eb586d026634ff1987eeada7aff6f83a

 

Here is a summary of some important options available.

--api

    Ask AJO to print only concise output.

-c FILE or --config FILE

Tells AJO which is the configuration file.

-d DIR or --retrieve-directory DIR

    Download the results of the execution to a given directory (DIR).

--download-exec-folder ID

    Downloads the folder where the job is executed at the HPC system.

-e ID or --erase

    Erase the directory associated with the job at the server.

-l or –list

    List all the identifiers available for query and retrieval.

--library FILE

    Specify libhpc location.

--log-all
    Make AJO log every step it takes while executing tasks in the ajo.log file created by default in current directory.

-q ID or --query ID
    Get the information about the status of the job with id=ID.

-s or --submit
    Submit the job you have specified in the configuration file to the cluster

-x ID or --cancel ID
    Cancel a running job.

 

8. EULA

End User Agreement

To be added soon

Examples

1. Simple 'hostname' job

Setting up some folders to perform experiments

We'll setup a root directory, called 'examples', where we'll store all the input files, scripts and results we're going to test.

ijimenez@login:~/examples$ mkdir hostname
ijimenez@login:~/examples$ cd hostname
ijimenez@login:~/examples/hostname$

The easiest example: where do you run?

To illustrate that we have no prior knowledge of which node is goin to execute our tasks, we'll program a little script to send us the hostname who will be running our scripts. And within the file we do a simple echo of the environment variable $HOSTNAME. The hostname.sh file comes with only two lines:

ijimenez@login:~/examples/hostname$ vi hostname.sh
#!/bin/bash
echo "Hello, this is your script, running on $HOSTNAME"

And now we'll setup the job file for hostname.sh. Comments are explained on the file:

ijimenez@login:~/examples/hostname$ vi hostname.sub
#!/bin/bash ##########################################
# Options and parameters for SGE:
##########################################
# (1) Name of the job to identify (flag -N)
# The parameter passed to -N is an alias to the script
# -----------------------------------------
#$ -N The-easiest-job
#
# (2) We'll redirect the output files to
# our working directory (flags -cwd, -e, -o)
# ---------------------------------------
#$ -cwd
#$ -o __hostname.out
#$ -e __hostname.err
#
# (3)Finally, we call the script
# ------------------------------
sh hostname.sh

And now we simply run the job, telling the scheduler to register the .sub file:

ijimenez@login:~/examples/hostname$ qsub hostname.sub
Your job 153605 ("The-easiest-job") has been submitted
ijimenez@login:~/examples/hostname$ 

Finally, we check the contents of the output file:

    ijimenez@login:~/examples/hostname$ more __hostname.out 
    Hello, this is your script, running on node09

2. Mandelbrot with Matlab

In this example, we'll create a Mandelbrot fractal with Matlab. First, we'll test our code interactively on a qrsh session and then we'll create some Matlab and SGE scripts to run the process in batch.

1. Interactive session:

Xshell:\> ssh -X ijimenez@hpc.dtic.upf.edu
Looking up host 'hpc.dtic.upf.edu'...
Host 'hpc.dtic.upf.edu' resolved to 193.145.51.37.
Connecting to 193.145.51.37:22...
Connection established.
To escape to local shell, press 'Ctrl+Alt+]'.
Linux login 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64
[ ... ]
ijimenez@login:~$ qrsh
Linux node02 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
[ ... ]
Last login: Thu Dec 12 13:26:11 2013 from 192.168.7.6
ijimenez@node02:~$ matlab & 

This is the source for the  Mandelbrot set we're going to create: http://www.mathworks.com/help/distcomp/examples/illustrating-three-approaches-to-gpu-computing-the-mandelbrot-set.html

The Mandelbrot code depends on four variables we set at the beginning, and will model the behavior of the Mandelbrot set:

>> maxIterations = 500;
gridSize = 1000;
xlim = [-0.748766713922161, -0.748766707771757];
ylim = [ 0.123640844894862,  0.123640851045266];

The full Matlab code is shown on the next figure:

If we execute the code, we'll obtain this figure:

 

2. Mandelbrot set in serial batch: 

Let's create the usual directory struct:

ijimenez@node04:~/Matlab$ mkdir Mandelbrot
ijimenez@node04:~/Matlab$ cd Mandelbrot
ijimenez@node04:~/Mandelbrot$ mkdir data
ijimenez@node04:~/Mandelbrot$ mkdir script
ijimenez@node04:~/Mandelbrot$ mkdir job-out
ijimenez@node04:~/Mandelbrot$ mkdir out

Let's create now the Matlab script:

ijimenez@node04:~/Mandelbrot$ cd script 
ijimenez@node04:~/Mandelbrot$ matlab

We can write it using Matlab's own editor. We'll introduce some modifications, as long as we are not going to handle a drawing but an image object:

% Show
cpuTime = toc( t );
%set( gcf, 'Position', [200 200 600 600] );
% imagesc( x, y, count );
% axis image
% colormap( [jet();flipud( jet() );0 0 0] )
% title( sprintf( '%1.2fsecs (without GPU)', cpuTime ) );
imwrite((count/max(count(:)))*255,jet,'/homedtic/ijimenez/Matlab/Mandelbrot/out/mandelbrot.png');

The full code of our simulation is shown below:

Some notes on the script:

a) we'll control the time trhourh the tic() - toc() commands.

b) The 'imwrite' command gets three values: the image matrix, the colormap and the file where we want to write the data. We apply a transformation to every element of the matrix to 'colorize' the value. Also, we'll use Matlab's builtin colormap 'jet'. 

c) We're going to write the output file in the /scratch/Mandelbrot/out rather than on $HOME, and we'll move the data back with the submission script:

#!/bin/bash
 
# Load modules directive
. /etc/profile.d/modules.sh
 
# Copy sources to the SSD:
#
 
# First, make sure to delete previous versions of the sources:
# -----------------------------------------------------------
if [ -d /scratch/Mandelbrot ]; then
        rm -Rf /scratch/Mandelbrot
fi
 
# Second, replicate the structure of the experiment's folder:
# -----------------------------------------------------------
mkdir /scratch/Mandelbrot
mkdir /scratch/Mandelbrot/data
mkdir /scratch/Mandelbrot/error
mkdir /scratch/Mandelbrot/script
mkdir /scratch/Mandelbrot/out
 
# Thir, copy the experiment's data:
# ----------------------------------
cp -rp /homedtic/ijimenez/Matlab/Mandelbrot/data/* /scratch/Mandelbrot/data
cp -rp /homedtic/ijimenez/Matlab/Mandelbrot/script/* /scratch/Mandelbrot/script
 
# Fourth, prepare the submission parameters:
# Remember SGE options are marked up with '#$':
# ---------------------------------------------
# Requested resources:
#
# Simulation name
# ----------------
#$ -N "Mandelbrot"
#
# Expected walltime: ten minutes maximum
# ---------------------------------------
#$ -l h_rt=00:10:00
#
# Shell
# -----
#$ -S /bin/bash
#
# Output and error files go on the user's home:
# -------------------------------------------------
#$ -o /homedtic/ijimenez/Matlab/Mandelbrot/job-out/Mandelbrot.out
#$ -e /homedtic/ijimenez/Matlab/Mandelbrot/job-out/Mandelbrot.err
#
# Send me a mail when processed and when finished:
# ------------------------------------------------
#$ -m bea
#$ -M  my.email@upf.edu
# Start script
# --------------------------------
#
# Print some informational data:
# ------------------------------
printf "Starting execution of job Mandelbrot with ID: $JOB_ID from user: $SGE_O_LOGNAME\n"
printf "Starting at `date`\n"
printf "Calling Matlab now\n"
printf "_____________________\n"
#
# Execute the Matlab script
# -------------------------
/soft/MATLAB/R2013b/bin/matlab -nosplash -nojvm -nodesktop -r "run /scratch/Mandelbrot/script/Mandelbrot.m"
#
# Copy data back, if any
# ----------------------
printf "Matlab processing done. Moving data back\n"
cp -rf /scratch/Mandelbrot/out/* /homedtic/ijimenez/Matlab/Mandelbrot/out
printf "_________________\n"
#
# Clean the crap:
# ---------------
printf "Removing local scratch directories...\n"
if [ -d /scratch/Mandelbrot ]; then
        rm -Rf /scratch/Mandelbrot
fi
printf "Job done. Ending at `date`\n"

That's it!. Submit the job:

ijimenez@login:~/Matlab/Mandelbrot$ qsub mandelbrot.sub 

and wait for it to end:

ijimenez@login:~/Matlab/Mandelbrot$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 154071 0.47500 Mandelbrot ijimenez     r     01/07/2014 11:47:59 short.q@node06                     1  

Once it finishes, we'll get the output time in ./job-out/Mandelbrot.out:

Starting execution of job Mandelbrot with ID: 154071 from user: ijimenez
Starting at Tue Jan  7 11:47:59 CET 2014
Calling Matlab now
_____________________
Warning: No display specified.  You will not be able to display graphics on the screen.
Warning: No window system found.  Java option 'MWT' ignored.
 
                            < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                    R2013b (8.2.0.701) 64-bit (glnxa64)
                              August 13, 2013
 
 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
Elapsed time is 109.338394 seconds.
Matlab processing done. Moving data back.
_________________
Removing local scratch directories...
Job done. Ending at Tue Jan  7 11:49:53 CET 2014

and the output image file will be in ./out/mandelbrot.png. We can use the gpicview utility to view the file:

ijimenez@login:~/Matlab/Mandelbrot/out$ gpicview mandelbrot.png 

 

 

3. Montecarlo with Matlab and MDCS

An illustrating example for configuring and running jobs on Matlab Distributed Computing Server (MDCS).

 

MDCS is already installed in the cluster. You only need to configure Matlab on your workstation inorder to run Matlab codes on the cluster.

The example runs a Montecarlo simulation on the workstation before, then on the cluster.

 
 
 
 
Running Montecarlo on the local Matlab (workstation)
 
 
Configuring and Running Montecarlo using MDCS.
 
 
1- Download snow.remote.r2013b.zip or snow.remote.r2014b.zip (depending on your Malab version) for windows systems. And snow.remote.r2013b.tar or snow.remote.r2014b.tar for Linux/Mac systems from the cluster /SOFT/Matlab/MDCS/ directory. You can download it to your worksatation using any SFTP client.
 
 
 
 
 

2- Uncompress the download and place contain into "MATLABROOT/toolbox/local", where MATLABROOT refers to the local Matlab installation directory.

Refer to the following table to find Matlab default directory.

Operating system Matlab directory
Windows C:\Program Files\MATLAB\R20xx\
Linux /usr/local/MATLAB/R20xx/
Mac /Applications/Matlab/R20xx/

xx -> matlab version

The download will contain four files and one folder as shown in the image.

 

3- Start Matlab on the local machine and configure it to submit jobs directly to the cluster, by calling configCluster.  And set your cluster user name.

You only have to do this once per Matlab version.

 

Depending on the job type, the sending method differs slightly. Basically, there are two types of jobs:

  • Serial: The job is atomic (uses only one core) and does not parallelize.

  • Communicating: The job is internally divided on threads and communicate themselves.

Additionally, you have to setup a secure communication channel between your workstation and the cluster, to be able to send your scripts and source files over the network and to retrieve the results.

4 – Submit jobs to the cluster (batch command).

Serial Job handling   

 

c = parcluster;

j = c.batch(@filename, 1, {});
 

 

The first time you submit a job, you will be prompt to select the cluster authentication method. You can either choose to use the cluster user password or to establish a password-less authentication method.

 

If you decide to use your cluster password press “No” in the User Credentials box, then introduce your password on the appearing dialog box.

 
 

Otherwise, if you decide to use a password-less authentication method select "yes" in the previous User Credential box, then indicate the directory to find the private key.

 

 

You are also asked to supply the passphrase. Select “yes” and introduce the phrase in the case anyone was provided when generating the key pair, otherwise select “no”.

 
 

Parallel job handling  

           c = parcluster;

           nan = c.batch(@filename, 1, {}, ‘matlabpool’, number_workers);

The procedure is the same as serial jobs, with two main differences:

  • The for loop in the code has to be replaced with the parfor loop.

  • Ant the Matlabpool function has to be addede in the exexution line, with the number of wokers needed.

matlabpool shows that is a parallelized job. number_workers cannot be higher than 31, because the number of workers is limited to 32.

 
 
 

Recover results

 

fetchOutputs is the function used to recover the result of a submitted job.

 

j.fetchOutputs{:}


 

Other important functions are submit, wait, diary and load.

 

Configure jobs

 

Additional information can be specified before submitting a job.


- Email Notification (when the job is running, exiting, or aborting).

     ClusterInfo.setEmailAddress('noko@upf.edu')


- Memory Usage.

     ClusterInfo.setMemUsage('2G')


- Queue name.

     ClusterInfo.setQueueName('default.q')


- Walltime.

     ClusterInfo.setWallTime('00:30:00')

 
 

4. Password-less connection

UNIX/Linux.

  • Generate a pair of private and public key in your local machine with the keygen command. Do not enter any passphrase, just press Enter.

         

  nicul@pc1nico:~$ ssh-keygen -t rsa

  Enter passphrase (empty for no passphrase):

  Enter same passphrase again:

  Your identification has been saved in /homedtic/noko/.ssh/id_rsa.

  Your public key has been saved in /homedtic/noko/.ssh/id_rsa.pub.


 

  • Copy the public key in the authentication system of the cluster.

 nicul@pc1nico:~/.ssh$ ssh-copy-id  -i  id_rsa.pub user@hpc.dtic.upf.edu

 

  • Establish connection to the cluster.

 

  ssh hpc.dtic.upf.edu

 

Windows

  • Generate a pair of private and public key in the cluster with the keygen command, just like you did for Linux but in the cluster and not on the user local machine.

 

 noko@login:~$ ssh-keygen -t rsa

 

  • Copy contain of the public key in the authorized_keys file found at homedtic/user/.ssh/ directory of the cluster.

 

 noko@login:~/.ssh$ cat id_rsa.pub /homedtic/noko/.ssh/authorized_keys

 

  • If the authorized_keys file does not exist, create it.

 

 noko@login:~/.ssh$ vi authorized_keys

 

  • Copy the private key to any directory of your choice on the local machine. This is the directory where you will redirect the Matlab configuration in case you select this password-less authentication method.

5. Performance AMD and Intel

We use a simple matlab function to compare the performance of the two processor models available in the cluster.

Pocessor Sockets Cores Speed Year Technology
Intel 2 4 2.0 2009 Dedicated
AMD 4 16 2.4 2012 Shared

 

The simulation code simply calculate the signal power received by a user at some distance away from the antenna. User and antenna positions are randomlly choosen from two sets of random generated values. 

 

We run the simulation in both processors queue types, and find out that Intel processor are a little bit fatser than the AMD processors, as they can run the simulation with less time.

 

Conclusion

Despite the difference manufacturing years, both processors perform well. Dealing with floating-point operations that require a single core, Intel processors nodes are advisable. However, when performing tasks that require various cores, the AMD processors nodes are preferable, because Intel cores are few and will soon be exhausted.

  

Changelog

Here we will keep you updated with the future commits and software installs.

17/1/2014: New queue available

On 17/1/2014, a new queue has been setup on the cluster, called skip_first.q

Each node of the cluster is geared with four AMD Abu-Dhabi 16-core processors. When a given job is processed on an empty processoir (no process running anywhere on any core) it is processed by the core 0 and speeds up by gaining high, faster accesses to the shared L2 and L3 caches. This behavior can introduce noise on the jobs who measure time, as long as there is no prior knowledge of the state of the processor: it can be empty or not, thus the same algorith can perform differently due to external reasons.

We are still testing performance, limits and correct behavior of this queue, but limit are:

- for users: 64 processors per job
- for research groups: 128 processors per all the group's jobs
- maximum cores available to the queue: 688 slots
- no limits for RAM or time