flight-appliance-docs Documentation Release 1.0 Alces Flight Ltd Jul 25, 2019 Using your Flight Compute cluster 1 License 3 2 Prerequisites 5 i ii flight-appliance-docsDocumentation,Release1.0 This site holds documentation designed to help users create a simple research compute environment using popular public and private cloud platforms. As well as launching and accessing the environment, guides and tutorials are includedtohelpend-usersinstallsoftwareapplications,managetheirdataandrundifferentworkloads. UsingyourFlightComputecluster 1 flight-appliance-docsDocumentation,Release1.0 2 UsingyourFlightComputecluster 1 CHAPTER License ThisdocumentationisreleasedundertheCreative-Commons: Attribution-ShareAlike4.0Internationallicense. The AlcesFlightComputesoftwareisreleasedunderthetermsoftheAlcesFlightEULA.Pleasereadtherelevantlicense textbeforeproceedingtouse. 3 flight-appliance-docsDocumentation,Release1.0 4 Chapter1. License 2 CHAPTER Prerequisites We recommend that users wishing to use Flight Appliances have basic Linux skills. The ability to move about in a filesystem,copyanddeletefiles,readandeditfilesonthecommand-linewillbeneededinordertogetthebestoutof theFlightsoftware. 2.1 Getting started as quickly as possible Thisdocumentationisdesignedtoprovideaquick-startguidetoyourpersonalFlightComputecluster,butifyou’ve used clusters before then you might just need a super-quick guide to help you get started. Here are just the basics, alongwiththingsyoushouldprobablylookatinmoredetailwhenyouhavetime.Usefulcommandsareincludedwith eachstep-pleaserefertothefulldocumentationforfurtherdetails. 1. LaunchyourpersonalAlcesFlightComputeclusteronAWSoracompatibleOpenStackprivatecloud. Touse theAWSplatform,usetheprovidedCloudFormation(CFN)templatetostartapersonalclusterinanewVPC. Youwillbeaskedvariousquestionsabouthowyouwantyourclustertolook-mostoptionsareexplained,with sensibledefaultsshown. 2. LogintoyourclusterviaSSHusingyourchosenusername,andtheSSHkeyregisteredwithyourcloudprovider. YourcloudserviceshouldreporttheaccessIPaddresstologintoonceyourclusterislaunched. Ifyouenabled clusterauto-scaling,computenodesareautomaticallyadded/removedbasedonthestatusofyourjob-scheduler queue. Use alces configure autoscaling disable to turn off scaling if you’ll be running jobs manually. 3. Start a graphical desktop session with the command alces session start gnome. Connect using a VNCclient,withtheone-timepasswordshownwhenyoustartedthesession. Multipleuserscanconnecttothe samesessionforcollaborativeprojects. Usethexrandrcommandtochangethescreenresolutioninarunning desktopsession. 4. Your user has full sudo access, and yum is configured to install Linux packages. Load the PDSH module with the command module load services/pdsh, then use the nodeattr -n nodes command to see your list of compute node hostnames, and pdsh -g nodes <command> to run commands across all nodesinyourcluster. TheFlightComputeSoloproductisdesignedtosupportasingleuser,althoughmultiple userscanloginusingthesameaccount. 5 flight-appliance-docsDocumentation,Release1.0 5. Your cluster has an NFS shared filesystem mounted across all nodes, which your home-directory is part of. CopydatatoandfromtheclusterusingSCP/SFTP.Thealces storagecommandsprovideaccesstoobject storage services including S3, Swift and Dropbox. Clusters can also be configured to use other storage types includingparallelfilesystemsandsharedfilers,dependingonyourcloudplatform. 6. Usethealces gridwarecommandtoviewandinstallsoftware.Themainrepositorylistsstablepackages, withalargerlistofapplicationsavailableaspartofthevolatilerepository. 7. Your cluster comes with an installation of a batch job scheduler; the default is SLURM, and a range of other schedulersarealsoavailable.Usethealces templatecommandforalistoftemplatejob-scriptstogetyou started. 8. Terminatetheclusterstackviayourcloudproviderwhenyou’vefinishedworking. Remembertocopyanydata youwanttokeepofftheclustertosecurestoragebeforeterminatingit. 2.2 What’s New 2.2.1 2017.2 • Choosethejobschedulertobeconfiguredonyourcluster(SLURM,GridEngine,OpenLava,TORQUEorPBS) • PreselectGridwaredepotstoinstallonthecluster-yoursoftwareisreadytousefromthewordgo • Additionaloptionsforformattingandstructuringthestoragelayoutofthecluster,including: – EBSstoragevolumetype – Separatingpartitionsforhomeandapplicationsdirectories – Setingencryptionpoliciesonvolumes – Configuringlocalscratchspace • ExtendedAlcesGridwareDockerfeatures-including – Imagesharing – Localregistries – MPIcontainerexecution – Customvolumemounts • Thenewc5,h1,p3,m5&x1eAWSinstancesarenowavailableforselection • FlightcanbelaunchedinthenewAWSregioneu-west-3(Paris) • PersonalGridware-installanyoftheGridwareappstoyouruserdirectory • Newclusterdashboard-FlightLaunch-replacesthepreviousdashboard 2.2.2 2017.1 • Newdefaultjobscheduler-SLURM • Theloginnodeisnowsettoamorepowerfulnodetype • Evenmorenodetypestochoosefrom • TheAlcesGridwarecommandnowsupportsbuildingofDockercontainers • Extendedcustomizationprofilefunctionality-setupyourenvironmenthowyouwant 6 Chapter2. Prerequisites