Artemis User Guide Information and Communications Technology Sydney Informatics Hub 16th March, 2017 © The University of Sydney Table of Contents Getting Started ............................................................................................................. 2 Access to Artemis ....................................................................................................... 2 Linux Command Line ................................................................................................. 2 Linux Text Editors ....................................................................................................... 2 Projects and UniKeys ................................................................................................. 2 Opening GUI windows on Artemis ............................................................................. 3 Software ..................................................................................................................... 3 Artemis compute nodes .............................................................................................. 4 Core allocations per Artemis queue ........................................................................... 4 Data Storage on Artemis ............................................................................................. 4 /home .......................................................................................................................... 4 /project ........................................................................................................................ 5 /scratch ....................................................................................................................... 5 Software/Programs/Applications ............................................................................... 6 Modules (or how to load installed software) .............................................................. 6 Installing software ....................................................................................................... 6 Submitting Jobs to Artemis ........................................................................................ 7 Serial jobs (1 core) ..................................................................................................... 7 Parallel jobs using OpenMP ....................................................................................... 7 Parallel jobs using MPI ............................................................................................... 8 Interactive jobs ........................................................................................................... 8 Array jobs ................................................................................................................... 9 Job monitoring/management ................................................................................... 12 Job Scheduling Priority: “Fair Share” ..................................................................... 13 Job Queues ................................................................................................................. 13 Queues available to all users ................................................................................... 13 Queue resource limits .............................................................................................. 15 Strategic Allocations ................................................................................................. 15 Data Transfer Queue (dtq) ........................................................................................ 16 Automating data transfers ........................................................................................ 16 Data transfer script: dt-script .................................................................................... 18 Example dt-script workflow ...................................................................................... 19 Compiling on Artemis................................................................................................ 20 GNU .......................................................................................................................... 20 PGI ............................................................................................................................ 21 Intel ........................................................................................................................... 21 Java .......................................................................................................................... 21 GPU .......................................................................................................................... 22 Research data store (RDS) ....................................................................................... 23 RCOS ....................................................................................................................... 23 Classic RDS ............................................................................................................. 24 Glossary ...................................................................................................................... 26 Date published: 16th March, 2017. © The University of Sydney 1 Getting Started Access to Artemis For instructions to obtain an Artemis account, see the following page: http://sydney.edu.au/research_support/hpc/access/index.shtml Linux Command Line Artemis is a Linux-based HPC with a command-line interface. If you have no background in Linux or the Linux command line, there are a number of excellent tutorials online that will help you get started. A recommended starting tutorial can be found here: http://www.ee.surrey.ac.uk/Teaching/Unix/ Once you finish tutorial 4, you will have enough command line knowledge to start use Artemis. Linux Text Editors It is possible to edit text files on Artemis using text editors such as Nano, GEdit, Vim or Emacs. If you have logged into Artemis with an X server connection (that is, with the -X or -Y options to the ssh login command and a connection to an X server running on your computer), then you can use gedit to edit text files. If you don’t have an X server running, then the next simplest text editor is nano. If you find yourself editing text files frequently on Artemis and want to invest time to learn a more powerful text editor, then it is worth using either emacs or vim. Projects and UniKeys UniKey Once you have been granted access to Artemis, you log in using your UniKey credentials. Throughout the guide, we use the example UniKey abcd1234 whenever we need to specify a UniKey. Remember to replace abcd1234 with your UniKey wherever needed. Project You need to know what your project is called. You need to specify a project for every job submitted to Artemis. There are two styles of project name you can use: Your full project name has the form RDS-FAC-Project-RW, where FAC is your faculty code and Project is your abbreviated project name as specified in your RDMP. If you have access to Artemis, but don’t know your project name, you could ask your Lead CI or type groups into Artemis after logging in: [abcd1234@login4]$ groups linuxusers HPC-ACL RDS-ICT-PANDORA-RW RDN-USERS RDN-ICT-TEST RDS- ICT-TEST-RW The projects you are a member of are any entries that start with RDS-. In the above case, abcd1234 is a member of the projects PANDORA and TEST. For most applications on Artemis, you can use your project name without the RDS-FAC-…-RW parts. However, some applications require the full project name. If you have difficulties using your abbreviated project name, try using your full project name instead. Date published: 16th March, 2017. © The University of Sydney 2 Throughout this guide, we will use the project PANDORA in the examples provided. Remember to replace PANDORA with your project name when you use Artemis. Opening GUI windows on Artemis It is possible to open programs with a graphical user interface (GUI) on Artemis if you have an X server running on your computer and ssh into Artemis using either the -X or -Y options. Programs on Artemis with GUIs include gedit (a text editor), Matlab and Gnuplot. Linux If you are on a Linux computer, such as Ubuntu or Linux Mint, then you are already running an X server. Therefore, you can open programs with GUIs if you log into Artemis using the -X or - Y options. ssh -X [email protected] MacOS On MacOS, you need to install XQuartz. After installing XQuartz, you can then open a terminal window using MacOS’s built-in terminal program, then SSH to Artemis using the -X or - Y options, as was the case for Linux. Windows On Windows, there are a number of X servers to choose from. The University of Sydney has a site license for Starnet X-Win32. Follow the instructions to download and install X-Win32 here: http://staff.ask.sydney.edu.au/app/answers/detail/a_id/316/session/L2F2LzEvdGltZS8xN Dg2MzM0MTM5L3NpZC9vcE5MRnZhbg%3D%3D. Then use the new connection wizard to set up an Artemis connection: • Select SSH and give your connection a name. For example, “John’s Artemis connection” then click next. • The host to connect to is hpc.sydney.edu.au • The login name and password are your UniKey credentials • The command to run on host will be filled in correctly if you click the Linux option. • Click Finish Your connection will appear as one of the options in the list. To start a command line session on Artemis, click the connection you just made, then click Launch. A command line terminal connection to an Artemis login node will open. If everything was done correctly, you will be able to launch programs such as gedit from the command line. Software There are many pre-installed programs on Artemis, including various compilers, scripting languages (for example, Python, Perl and R), libraries and other specialised software. For full details, see the Software section. Date published: 16th March, 2017. © The University of Sydney 3 Artemis compute nodes Artemis is a 4264-core HPC system comprised of the following nodes: Artemis Phase 1 Nodes Artemis Phase 2 Nodes Standard High GPU Standard High Memory Memory Memory Memory Number of Nodes 56 2 5 80 3 Cores per node 2 x 12 2 x 12 2 x 12 CPUs 2 x 16 4 x 16 2 GPUs RAM per node 128 GB 512 GB 128 GB 128 GB 6 TB Core allocations per Artemis queue Some nodes of Artemis are restricted to certain groups:  Civil Engineering have ownership of 13 nodes (416 cores); they have exclusive access to these nodes.  Some researchers were granted strategic allocations on Artemis. These nodes are reserved for these projects. As of February 2017, 936 cores are reserved for strategic allocations. The remaining nodes (2912 cores in total) are available to all users of the system. Data Storage on Artemis Artemis uses a Lustre file system that is accessible from any compute node and provides excellent I/O performance (especially for large files). The Lustre file system contains the following storage spaces:  Home (/home/abcd1234)  Project (/project/PANDORA)  Scratch (/scratch/PANDORA) Remember to replace abcd1234 with your UniKey and PANDORA with your abbreviated project name. Data will not be backed up so we cannot offer any ability to retrieve lost data. For long term storage please store all data in the Research Data Store (RDS). /home Each researcher will be allocated their own home directory within Artemis. This can be used to store program code, batch scripts and other files. Please note that the home directory has limited space and is intended for storing code and configuration information. Data on /home is backed up in case of system failure (that is, if the Lustre filesystem fails), but files cannot be recovered if they are accidentally deleted. Important data should be stored in the research data store. Date published: 16th March, 2017. © The University of Sydney 4 To check your /home quota usage, run the following command: pquota /project The default allocation is 1 TB of project storage, shared between all members of the project. Please note that no additional space will be provided. If you require more disk space for job output, use /scratch instead. Since /project (or /scratch or /home) is not backed up, please back up all important data to the Research Data Store regularly. To check your quota and usage of your allocated storage on /project, use the pquota command. /scratch /scratch is a 204 TB pool of shared storage that is intended for data that needs to be saved during a job, but can be deleted once the job completes. As this directory is shared by all users, please do not leave your data in this area for longer than absolutely necessary. Transfer important files to /project if you need them after your job finishes. /scratch is not backed up, so please back up important data to the Research Data Store. To see how much /scratch space is available, use the following command: df -h /scratch To see how much space your project is using: lfs quota -hg RDS-ICT-PANDORA-RW /scratch Remember to replace RDS-ICT-PANDORA-RW with your project name, as described in the Projects and UniKeys section. For details about disk usage by specific subdirectories, use this command: du -sh /scratch/PANDORA/directory where the last argument is the directory you want to know the size of. Date published: 16th March, 2017. © The University of Sydney 5 Software/Programs/Applications Artemis has many pre-installed programs available. Your batch jobs can be set up to use these programs by loading "modules". When loaded, these modules quickly set up your batch job environment with paths to executables and other important settings so you don't have to manually configure your batch job environment or remember where programs are installed. A full list of all software on Artemis can be found on the software page. Modules (or how to load installed software) To see all available modules on Artemis, run the following command: module avail this will return a (long) list of modules. To narrow down the list to programs you’re interested in using, try using grep. The following example returns any modules with the word matlab in it: module avail |& grep -i matlab To load a module, for example, Matlab, use the following command: module load matlab this will add Matlab to your PATH and load any required dependencies. It will, by default, load the most commonly used version of Matlab. If you require a different version, specify the version as follows: module load matlab/R2014b Having multiple versions of the same program open can cause conflicts. For example, if Matlab 2014 and Matlab 2015 were simultaneously loaded, then the last version loaded will be used. For this reason, it is good practice to unload unnecessary modules: module unload matlab/R2014b Occasionally, some modules confict with others and create strange and difficult to diagnose errors. To unload all modules with one command, use module purge: module purge Installing software If the software you require isn’t installed on Artemis, you can submit a High Performance Computing request to install software via the ICT Self-Service portal. To access the form using the links provided here, click the ICT Self-Service portal link, log in with your unikey and password, then click the High Performance Computing request link. Alternatively, yo u can compile your own programs on Artemis. For information, see the Compiling section. Date published: 16th March, 2017. © The University of Sydney 6 Submitting Jobs to Artemis Artemis uses a modified version of PBS Pro to manage and queue jobs. Some minimal example scripts are provided below. Serial jobs (1 core) #!/bin/bash #PBS -P PANDORA # your project name #PBS -l select=1:ncpus=1:mem=4GB # select one chunk with 1 CPU and 4 GB memory #PBS -l walltime=10:00:00 # actual time your job will run for cd $PBS_O_WORKDIR my_program > results.out Save this script to a file (for example MyScript.pbs), then then submit it to the PBS scheduler using the following command on the command line: qsub MyScript.pbs there are other #PBS directives that you may optionally use. For a full list, see the PBS Professional user manual. Parallel jobs using OpenMP OpenMP is a shared memory parallelism model, so OpenMP parallelised programs can only be run on one chunk (a “chunk” is a virtual node) and either 24, 32 or 64 cores, depending on the node. An example script to run an OpenMP parallelised program is: #!/bin/bash #PBS -P PANDORA # your project name #PBS -l select=1:ncpus=4:mem=4GB # select one chunk with 4 CPUs and 4 GB memory #PBS -l walltime=10:00:00 # actual time your job will run for cd $PBS_O_WORKDIR my_program > results.out Note: OMP_NUM_THREADS is automatically set to the number of cores per chunk on Artemis, so you don’t have to manually set the value of OMP_NUM_THREADS in your PBS script. Date published: 16th March, 2017. © The University of Sydney 7 Parallel jobs using MPI MPI (Message Passing Interface) allows jobs to communicate across several nodes. A basic MPI job script is shown below. #!/bin/bash #PBS -P PANDORA # your project name #PBS -l select=1:ncpus=4:mpiprocs=4:mem=4GB # select 1 chunk, 4 MPI cores, 4 GB memory #PBS -l walltime=10:00:00 # actual time your job will run for cd $PBS_O_WORKDIR # change to current working directory module load siesta # Also automatically loads the Intel MPI library mpirun -np 4 siesta < h2o.fdf > h2o.out The above file will choose one chunk with 4 cores and 4 GB of memory. All 4 cores have been made available to MPI as specified by the mpiprocs=4 option. It is recommended to set ncpus and mpiprocs to the same value, unless you know you can set them differently. Note: load the correct MPI implementation for your program. Most modules will automatically load the appropriate MPI implementation. However, you can load your own MPI implementation if you are compiling your own MPI programs or if you know you can use an alternative MPI implementation. With MPI, you can select more than one chunk: #!/bin/bash #PBS -P PANDORA # your project name # select 5 chunks, 8 MPI cores per #PBS -l select=5:ncpus=8:mpiprocs=8:mem=4GB chunk, 4 GB memory per chunk #PBS -l walltime=10:00:00 # actual time your job will run for cd $PBS_O_WORKDIR # change to current working directory # Also automatically loads the Intel module load siesta MPI library mpirun -np 40 siesta < h2o.fdf > h2o.out This script requests 5 chunks with 8 cores each (all cores are available to MPI) and 4 GB of memory per chunk, meaning your job will be allocated 40 cores and 20 GB of memory in total. Interactive jobs You can request an interactive job using the qsub -I directive from the command line. For example: qsub -I –P PANDORA -l select=1:ncpus=1:mem=4GB,walltime=1:00:00 Remember to replace PANDORA with your project name. Note: You cannot use a PBS script to start an interactive job. Only qsub -I works. Date published: 16th March, 2017. © The University of Sydney 8 Running programs with a GUI in an interactive job You can open a GUI on Artemis in an interactive session by firstly logging in as described in getting started, then requesting an interactive job using the qsub -I -X command. An example qsub command is (all on one line): qsub –I -X –P PANDORA -l select=1:ncpus=4:mem=10GB,walltime=1:00:00 Users with strategic allocations can run interactive jobs in their allocated area by specifying a queue in the interactive qsub command. For example, members of the project porous can request an interactive session in their allocation with the following command (all on one line): qsub –I -X –P porous -q condo-civil –l select=1:ncpus=4:mem=10GB,walltime=1:00:00 Array jobs PBS array jobs are a convenient way of running many computations with a single script. They are particularly suitable for jobs which need to be run many times, but with different parameters or on different input files. To submit an array job, you need to include this PBS directive in your PBS script: #PBS -J start_index-end_index:increment For example: #PBS -J 1-10 will start an array job with 10 elements. Each job will be assigned a single value of $PBS_ARRAY_INDEX from 1 to 10. The array indices must be integers. Another example is: #PBS -J 1-10:2 This directive will start an array job containing 5 elements, with values of $PBS_ARRAY_INDEX of 1, 3, 5, 7, and 9. Two common uses of array jobs are to run a program multiple times with different input parameters or different input files. Some example array job PBS scripts are shown below. Single Parameter Change This script will start an array job with 10 elements, each element will be assigned a different value of $PBS_ARRAY_INDEX from 1 to 10. In this case, $PBS_ARRAY_INDEX is used directly as a command line argument to the program called my_program. Date published: 16th March, 2017. © The University of Sydney 9