IBM Plat LSF家族安裝和配置簡介.V1.0.doc

_IBM Platform LSF家族安裝和配置簡介V1.0版 馬雪潔2013.5.7目錄1集群結構11.1單純LSF環(huán)境（命令行提交）11.2LSF+PAC環(huán)境（WEB提交）11.3LSF+PM環(huán)境（PM提交）32LSF安裝和基本配置舉例32.1LSF安裝步驟32.1.1獲得LSF和PAC安裝包32.1.2解壓縮lsfinstall安裝腳本文件42.1.3修改install.config配置文件42.1.4執(zhí)行安裝42.1.5配置開機自啟動42.1.6測試安裝42.1.7啟動/停止LSF進程（三種方式）52.1.8測試提交作業(yè)52.1.9使能root提交作業(yè)52.1.10修改配置文件后reconfig62.1.11日志和debug62.2配置文件說明62.3常用命令72.4配置公平競爭調(diào)度策略72.4.1添加輪循調(diào)度隊列72.4.2添加層次公平競爭策略82.4.3多隊列公平競爭策略82.4.4使能配置92.5配置搶占調(diào)度策略92.6配置全局限制策略102.6.1限制用戶運行的作業(yè)數(shù)目102.6.2限制節(jié)點運行作業(yè)數(shù)目102.6.3限制隊列作業(yè)的運行限制102.6.4設定General limits112.6.5使能配置122.7配置提交控制腳本esub122.8配置資源管理elim示例122.8.1匯報home目錄空閑大小122.8.2匯報root進程數(shù)目122.8.3匯報應用程序許可證數(shù)目132.8.4測試elim腳本142.8.5添加資源定義和資源地圖142.8.6查看資源數(shù)目143LSF命令行集成應用示例143.1CFD+集成（spooling file）143.1.1CFD+安裝和許可證143.1.2集成許可證管理elim143.1.3添加CFD+ job starter153.1.4添加CFD APP profile173.1.5CFD+命令行提交腳本實例173.2GAUSSIAN集成方式（spooling file）183.2.1Gaussian安裝和許可證183.2.2Gaussian命令行提交腳本實例183.3Abaqus的腳本集成（bsub命令）183.4Platform MPI作業(yè)193.5Openmpi作業(yè)203.6Intel MPI作業(yè)203.6.1Express版本不記賬方式203.6.2Express版本blaunch記賬方式223.6.3Standard版本PAM集成方式274安裝PAC295使用PAC進行應用程序集成305.1Gaussian界面集成過程315.2CFD+集成后界面和后臺腳本365.3在PAC中監(jiān)控許可證386安裝License Scheduler406.1基本安裝測試406.2基本配置舉例406.2.1添加許可證服務器地址406.2.2映射許可證feature：406.2.3使用許可證資源416.2.4配置許可證調(diào)度策略417常見問題418使用man page419售后技術支持411 集群結構較大的集群都會設計 單獨的登錄節(jié)點，用戶只能ssh到登錄節(jié)點，不能直接ssh到集群的任何主節(jié)點和計算節(jié)點。同時配置用戶在計算節(jié)點之間的ssh互信，為了并行作業(yè)的運行。登錄節(jié)點也安裝LSF，配置為LSF 靜態(tài)Client或者MXJ值為0，也即不運行作業(yè)的客戶端。集群的WEB節(jié)點與辦公訪問局域網(wǎng)一個網(wǎng)段。如需使用浮動client，主節(jié)點網(wǎng)卡需要1.1 單純LSF環(huán)境（命令行提交）1.2 LSF+PAC環(huán)境（WEB提交）用戶通過portal提交作業(yè)：1.3 LSF+PM環(huán)境（PM提交）2 LSF安裝和基本配置舉例2.1 安裝前的準備工作NIS ready；NFS/GPFS ready；2.2 LSF安裝步驟Use root to install.Get NIS and NFS/GPFS ready.2.2.1 獲得LSF和PAC安裝包lsf8.3_linux2.6-glibc2.3-x86_64.tar.Z lsf8.3_lsfinstall_linux_x86_64.tar.Z pac8.3_standard_linux-x64.tar.Z 許可證文件platform_hpc_std_entitlement.dat2.2.2 解壓縮lsfinstall安裝腳本文件 Put the package under /root/lsfrootS2 lsf# gunzip lsf8.3_lsfinstall_linux_x86_64.tar.Z tar -xvf lsf8.3_lsfinstall_linux_x86_64.tar2.2.3 修改install.config配置文件首先添加集群管理員lsfadmin。cd lsf8.3_lsfinstallvi install.configrootS2 lsf8.3_lsfinstall# cat install.config LSF_TOP=/opt/lsf (安裝目錄) LSF_ADMINS=lsfadmin (先創(chuàng)建lsfadmin的用戶名) LSF_CLUSTER_NAME=platform (集群名稱，任意指定) LSF_MASTER_LIST=s2 s3 (LSF管理節(jié)點) LSF_ENTITLEMENT_FILE=/root/lsf/platform_hpc_std_entitlement.dat (安裝源許可證的地址) LSF_TARDIR=/root/lsf/ (安裝源文件包的地址)2.2.4 執(zhí)行安裝./lsfinstall -f install.config2.2.5 配置開機自啟動/opt/lsf/9.1/installhostsetuprhostsetup2.2.6 測試安裝安裝目錄下的/conf目錄rootS2 conf# source profile.lsfAdd source profile.lsf to /etc/profileif no rsh, set ssh in lsf.confrootS2 conf# tail lsf.confLSF_RSH=ssh2.2.7 啟動/停止LSF進程（三種方式）rootS2 conf# lsfstartup/lsfstop或者lsadmin limstatup/limshutdownlsadmin resstartup/resshutdownbadmin hstartup/hshutdown或者lsf_daemons start/stoprootS2 conf# lsidIBM Platform LSF Express 8.3 for IBM Platform HPC, May 10 2012Copyright Platform Computing Inc., an IBM Company, 1992-2012.US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.My cluster name is platformMy master name is s2You have new mail in /var/spool/mail/rootrootS2 conf# lsloadHOST_NAME status r15s r1m r15m ut pg ls it tmp swp mems2 ok 0.0 0.0 0.0 1% 0.0 1 0 151G 20G 61Gs4 ok 0.0 0.0 0.0 2% 0.0 1 2 183G 20G 62Gs6 ok 0.0 0.0 0.0 3% 0.0 1 2 3734M 2G 30Gs5 ok 0.0 0.0 0.0 5% 0.0 1 2 3468M 2G 30G2.2.8 測試提交作業(yè)bsub sleep 1000002.2.9 使能root提交作業(yè)enable root to submit job: LSF_ROOT_REX=local重啟LSF進程。2.2.10 修改配置文件后reconfig修改lsf.*配置文件后lsadmin reconfig修改lsb.*配置文件后badmin reconfig部分參數(shù)需要重啟LSF主調(diào)度或者其他進程：badmin mbdrestart; lsadmin limrestart; lsadmin resrestart; badmin hrestart2.2.11 日志和debugFind the logs under log directory. LSF will run mainly 3 processes on each node, on master node will have 2 more.Master: lim,res,sbatchd,mbatchd,mbschedCompute:lim,res,sbatchdTurn on debug in command line:Run lim -2 directly on node to check why lim not startup.2.3 配置文件說明目錄/etc/init.d:/etc/init.s/lsf lsf服務自啟動腳本目錄/apps/platform/8.3/lsf/conf:lsf.conf lsf配置文件lsf.cluster.cluster83 集群配置文件lsf.shared 共享資源定義文件./lsbatch/cluster83/configdir/lsb.* 調(diào)度系統(tǒng)配置文件lsb.users lsf用戶與用戶組配置文件lsb.queues lsf隊列配置文件lsb.params lsf調(diào)度參數(shù)配置文件lsb.applications lsf應用配置文件lsb.hosts lsf機器與機器組配置文件lsb.resources lsf資源配置文件lsb.modules lsf模塊配置文件2.4 常用命令bsub:提交作業(yè)；bjobs:查看作業(yè)信息；bhist:查看作業(yè)歷史；lshosts:查看節(jié)點靜態(tài)資源；bhosts，lsload:查看節(jié)點狀態(tài)和資源信息；bqueues ：查看隊列配置；blimits：查看限制limit信息；lsid:集群版本和主節(jié)點；bmod:修改bsub option；等等。2.5 基于資源的調(diào)度策略bsub R “(type=LINUX2.4 & r1m 2.0)|(type=AIX & r1m 1.0)”或者在隊列l(wèi)sb.queues或者lsb.application文件定義：RES_REQ=select(type=LINUX2.4 & r1m 2.0)|(type=AIX & r1m =300 & mem500 orderswap:mem rusageswap=300,mem=500 job1bsub R rusagemem=500:app_lic_v2=1 | mem=400:app_lic_v1.5=1 job1bsub R selecttype=any & swp=300 & mem500 ordermem job12.6 配置公平競爭調(diào)度策略2.6.1 添加輪循調(diào)度隊列Modify lsb.queues, add followingBegin QueueQUEUE_NAME = roundRobin PRIORITY = 40FAIRSHARE = USER_SHARESdefault,1#USERS = userGroupA Define your own usergroupEnd QueueRun badmin reconfig to enable the change.Run bqueues l to check the queues configure2.6.2 添加層次公平競爭策略Add following queue to add hierarchicalshare policy:Begin QueueQUEUE_NAME = hierarchicalShare PRIORITY = 40USERS = userGroupB userGroupC FAIRSHARE = USER_SHARESuserGroupB,7 userGroupC,3End Queue2.6.3 多隊列公平競爭策略在lsb.queues中添加下列隊列，注意節(jié)點組和用戶組定義。Begin QueueQUEUE_NAME = verilog DESCRIPTION = master queue definition cross-queuePRIORITY = 50FAIRSHARE = USER_SHARESuser1,100 default,1FAIRSHARE_QUEUES = normal shortHOSTS = hostGroupC# resource contention #RES_REQ = rusageverilog = 1End QueueBegin QueueQUEUE_NAME = shortDESCRIPTION = short jobs PRIORITY = 70 # highestHOSTS = hostGroupC RUNLIMIT = 5 10End QueueBegin QueueQUEUE_NAME = normalDESCRIPTION = default queuePRIORITY = 40 # lowestHOSTS = hostGroupC End Queue2.6.4 使能配置badmin reconfig提交作業(yè)，并查看隊列的用戶動態(tài)優(yōu)先級變化：bqueues rl normal2.7 配置搶占調(diào)度策略配置最基本的slots搶占：Begin QueueQUEUE_NAME = shortPRIORITY = 70HOSTS = hostGroupC # potential conflictPREEMPTION = PREEMPTIVEnormalEnd QueueBegin QueueQUEUE_NAME = normalPRIORITY = 40HOSTS = hostGroupC # potential conflict PREEMPTION = PREEMPTABLEshortEnd Queue向兩個隊列提交作業(yè)，查看被preempt的作業(yè)的pending原因。2.8 配置全局限制策略2.8.1 限制用戶運行的作業(yè)數(shù)目在lsb.users文件中添加：Begin User USER_NAME MAX_JOBS JL/P user1 4 - user2 2 1 user3 - 2 groupA 8 - groupB 1 1 Default 2 - End User2.8.2 限制節(jié)點運行作業(yè)數(shù)目在lsb.hosts文件中：Begin HostHOST_NAME MXJ JL/Uhost1 4 2host2 2 1host3 ! -End Host2.8.3 限制隊列作業(yè)的運行限制在lsb.queues中添加：Begin QueueQUEUE_NAME = myQueue HJOB_LIMIT = 2PJOB_LIMIT = 1UJOB_LIMIT = 4HOSTS = hostGroupA USERS = userGroupA End Queue2.8.4 設定General limits在lsb.resources文件定義全局general limits示例：Begin LimitUSERSQUEUESHOSTS SLOTS MEM SWP user1 -hostB - - 20% user2normalhostA - 20 - End LimitBegin Limit NAME = limit1 USERS = user1 PER_HOST = hostA hostC TMP = 30% SWP = 50% MEM = 10%End LimitBegin Limit PER_USER QUEUES HOSTS SLOTS MEM SWP TMP JOBS groupA - hgroup1 - - - - 2 user2 normal - - 200 - - - - short - - - - - 200 End Limit 2.8.5 使能配置badmin reconfig2.9 配置提交控制腳本esub全局esub腳本在作業(yè)被提交是調(diào)用，可以被自動的或者顯式的調(diào)用從而控制用戶作業(yè)提交的行為。編輯ject文件在$LSF_SERVERDIR下面（chmod為可執(zhí)行）：#!/bin/shif _$LSB_SUB_PARM_FILE != _ ; then . $LSB_SUB_PARM_FILE if _$LSB_SUB_PROJECT_NAME = _ ; then echo You must specify a project! &2 exit $LSB_SUB_ABORT_VALUE fifiexit 0在lsf.conf中定義LSB_ESUB_METHOD=”project”2.10 配置資源管理elim示例2.10.1 匯報home目錄空閑大小編輯elim文件elim.home，放置在$LSF_SERVERDIR下面。chmod為可執(zhí)行。#!/bin/shwhile true ; do home=df -k /home | tail -1 | awk printf %4.1f, $4/(1024*1024) echo 1 home $home sleep 30done2.10.2 匯報root進程數(shù)目編輯elim.root，放置在$LSF_SERVERDIR下面。chmod為可執(zhí)行。#!/bin/shwhile true ; do root=ps -ef | grep -v grep | grep -c root echo 1 rootprocs $root sleep 30done2.10.3 匯報應用程序許可證數(shù)目#!/bin/shlic_X=0 ; num=0while true ; do # only want the master to gather lic_X if $LSF_MASTER = Y ; then lic_X=lmstat a c lic_X.dat | grep .&2 fi # only want training8, training1 to gather simpton licenses if hostname = training8 o hostname = training1 ; then num=lmstat a c simpton_lic.dat | grep .&2 fi # all hosts including master, will gather the following root=ps efw | grep v grep | grep c root1&2 tmp=df k /var/tmp | grep var | awk print $4 /1024&2if $LSF_MASTER = Y ; then echo 4 lic_X $lic_X simpton $num rtprc $root tmp $tmp else echo 3 simpton $num rtprc $root tmp $tmp fi # the same INTERVAL values defined in lsf.shared sleep 60done2.10.4 測試elim腳本直接運行./elim.root查看elim輸出是否正確。2.10.5 添加資源定義和資源地圖在lsf.shared文件中添加rootprocs 定義，并在lsf.cluster resources Map中添加資源和節(jié)點的映射關系。使能配置：lsadmin reconfig; badmin reconfig2.10.6 查看資源數(shù)目lsload l3 LSF命令行集成應用示例本節(jié)例舉幾個應用的不同集成方式。使用spooling文件或者bsub命令行都可以自由轉換。3.1 CFD+集成（spooling file）3.1.1 CFD+安裝和許可證安裝路徑：ln-3620-4許可證：/gpfs/software/cfdpp/mbin/Metacomp.lic 許可證服務器：ln-3620-4啟動許可證服務器：hpcadminmn-3650 jessi$ ssh ln-3620-4Last login: Tue Mar 26 19:19:24 2013 from mn-3650.private.dns.zone hpcadminln-3620-4 $ /gpfs/software/cfdpp/mbin/lmgrd -c /gpfs/software/cfdpp/mbin/Metacomp.lic確認許可證服務器是否正常運行：/gpfs/software/cfdpp/mbin/lmutil lmstat -a -c /gpfs/software/cfdpp/mbin/Metacomp.lic3.1.2 集成許可證管理elim添加elim方法：（elim全集群只需運行一個，因此只在頭節(jié)點放置elim腳本即可）在頭節(jié)點：cd $LSF_SERVERDIR添加如下文件：elim.lic:rootmn-3650 jessi# cd $LSF_SERVERDIRrootmn-3650 etc# pwd/opt/lsf/8.3/linux2.6-glibc2.3-x86_64/etcrootmn-3650 etc# cat elim.lic#!/bin/shtotallicences=/gpfs/software/cfdpp/mbin/lmutil lmstat -a -c /gpfs/software/cfdpp/mbin/Metacomp.lic |grep Users of CFD+_SOLV_Ser | /bin/cut -d -f7while truedousedlicences=/gpfs/software/cfdpp/mbin/lmutil lmstat -a -c /gpfs/software/cfdpp/mbin/Metacomp.lic | /bin/grep Users of CFD+_SOLV_Ser | /bin/cut -d -f13 cfd_lic=$( $totallicences-$usedlicences )echo 1 cfd_lic $cfd_lic/bin/sleep 30done rootmn-3650 etc# chmod a+x elim.lic修改如下的配置文件：rootmn-3650 etc# vi $LSF_ENVDIR/lsf.shared添加如下一行：cfd_lic Numeric 30 Y (CFD+ License)rootmn-3650 etc# vi $LSF_ENVDIR/lsf.cluster在resourcemap一段添加如下一行：Begin ResourceMap RESOURCENAME LOCATION cfd_lic allhostid default rootmn-3650 etc# lsadmin reconfig; badmin reconfig3.1.3 添加CFD+ job starter如果使用spooling file可不用添加。（Portal集成方式使用）添加jobstarter可執(zhí)行文件：hpcadminmn-3650 jessi$ cat /opt/lsf/jobstarter/cfd_starter #!/bin/shMPI_RUN=/gpfs/software/cfdpp/hpmpi/bin/mpiruncase $PRESSION in SINGLE_PRESSION) CFD_CMD=/gpfs/software/cfdpp/mbin/mcfd.11.1/r4_hpmpimcfd ; DOUBLE_PRESSION) CFD_CMD=/gpfs/software/cfdpp/mbin/mcfd.11.1/hpmpimcfd ; esacCMD=$* -hostfile $LSB_DJOB_HOSTFILE $CFD_CMDeval $CMD45 | Page3.1.4 添加CFD APP profilerootmn-3650 etc# vi $LSF_ENVDIR/lsf.shared添加如下配置：Begin ApplicationNAME = cfdJOB_STARTER=/opt/lsf/jobstarter/cfd_starterRES_REQ=rusagecfd_lic=1End Applicationbadmnin reconfig使得此文件生效，使用bapp l cfd查看是否成功：rootmn-3650 bin# bapp -l cfdAPPLICATION NAME: cfd - No description provided.STATISTICS: NJOBS PEND RUN SSUSP USUSP RSV 12 12 0 0 0 0PARAMETERS:JOB_STARTER: /opt/lsf/jobstarter/cfd_starterRES_REQ: rusagecfd_lic=13.1.5 CFD+命令行提交腳本實例hpcadminmn-3650 jessi$ cat cfd.sh#!/bin/sh#BSUB -n 12#BSUB -o %J.out#BSUB -e %J.err#BSUB -app cfd#BSUB -R rusagecfd_lic=1cd /gpfs/software/cfd+/test/ogive/gpfs/software/cfdpp/hpmpi/bin/mpirun -hostfile $LSB_DJOB_HOSTFILE /gpfs/software/cfdpp/mbin/mcfd.11.1/hpmpimcfd然后bsub cfd.sh 提交作業(yè)。3.2 GAUSSIAN集成方式（spooling file）3.2.1 Gaussian安裝和許可證路徑：/gpfs/software/Gaussian/許可證：無許可證版本，單個作業(yè)只能單機運行。3.2.2 Gaussian命令行提交腳本實例下面腳本：g03.sh#!/bin/sh#BSUB -q qchem#BSUB -n 4#BSUB -R spanhosts=1#BSUB -cwd .#BSUB -e %J.err#BSUB -o %J.outJOB=Full_codes_112_ipr_C1_0588.comJOBNAME=basename $JOB .comexport g03root=/gpfs/software/Gaussianexport GAUSS_SCRDIR=/tmpsource $g03root/g03/bsd/file/gpfs/software/Gaussian/g03/g03 $JOBNAME.log提交作業(yè)：bsub g03.sh3.3 Abaqus的腳本集成（bsub命令）編輯腳本abaqus_run.sh腳本#!/bin/sh# version: 1.3.0export ABAQUS_CMD=/gpfs/software/Abaqus/Commands/abaqusexport LM_LICENSE_FILE=/gpfs/software/Abaqus/License/abq612.lic#指定cpu number,注意要與bsub命令行中-n指定的cpu個數(shù)一致export NCPU=16#指定輸入文件export INPUT_FILE=beam.inp#指定作業(yè)名export JOB_NAME=abaqus_job3$ABAQUS_CMD job=$JOB_NAME cpus=$NCPU input=$INP_INPUT_FILE2)通過LSF提交#進入輸入數(shù)據(jù)所在目錄，執(zhí)行bsub命令bsub -q qeng -n 16 ./abaqus_run.sh3.4 Amber作業(yè)（blaunch集成，可記賬）針對intelmpi，編寫mpdboot.lsf腳本。變?yōu)榭蓤?zhí)行，放置在$LSF_SERVERDIR下面編寫提交作業(yè)腳本：ymeimnis test$ cat new.sh #!/bin/sh#BSUB -q small#BSUB -n 128#BSUB -o %J.out#BSUB -e %J.err#BSUB -J IMPI#BSUB -x #export PATH=/gpfs01/software/intel/impi/24/intel64/bin:$PATH#/gpfs01/home/ymei/jessi/mpdboot.lsfmpdboot.lsfexport I_MPI_DEVICE=ssm#export I_MPI_FABRICS=shm:ofa#export I_MPI_FAST_STARTUP=1#export I_MPI_DEVICE=rdssm#mpiexec -np $LSB_DJOB_NUMPROC /gpfs01/software/intel/impi/24/test/helloword mpiexec -np $LSB_DJOB_NUMPROC $AMBERHOME/bin/sander.MPI -ng 32 -groupfile remd10.groupfilempdallexit提交作業(yè)：bsub new.sh3.5 Platform MPI作業(yè)3.5.1 安裝Platform MPI確認用戶無密碼訪問ssh OK。安裝Platform MPI到共享目錄下：sh platform_mpi-0-0320r.x64.sh -installdir=/opt/pmpi norpm如果缺失C Compiler，執(zhí)行：yum install gcc3.5.2 LSF外面驗證安裝OK設置環(huán)境變量：export MPI_REMSH=ssh -xexport MPI_ROOT=/opt/pmpi/opt/ibm/platform_mpi/編譯helloworld示例程序：/opt/pmpi/opt/ibm/platform_mpi/bin/mpicc -o helloworld /opt/pmpi/opt/ibm/platform_mpi/help/hello_world.crootserver3 help# /opt/pmpi/opt/ibm/platform_mpi/bin/mpirun -f ./help/hosts warning: MPI_ROOT /opt/pmpi/opt/ibm/platform_mpi/ != mpirun path /opt/pmpi/opt/ibm/platform_mpiHello world! Im 1 of 4 on server3Hello world! Im 0 of 4 on server3Hello world! Im 3 of 4 on computer007Hello world! Im 2 of 4 on computer007rootserver3 help# cat ./help/hosts -h server3 -np 2 /opt/pmpi/opt/ibm/platform_mpi/help/helloworld -h computer007 -np 2 /opt/pmpi/opt/ibm/platform_mpi/help/helloworld3.5.3 通過LSF提交export MPI_REMSH=blaunch$ mpirun -np 4 -IBV /helloworld$ mpirun -np 32 -IBV /helloworld$ mpirun -np 4 -TCP /helloworld或者 rootserver3 conf# bsub -o %J.out -e.%J.err -n 4 /opt/pmpi/opt/ibm/platform_mpi/bin/mpirun -lsb_mcpu_hosts /opt/pmpi/opt/ibm/platform_mpi/help/helloworldJob is submitted to default queue .rootserver3 conf# bjobsJOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME210 root PEND normal server3 *elloworld May 9 10:55 rootserver3 conf# cat 210.out Sender: LSF System Subject: Job 210: in cluster DoneJob was submitted from host by user in cluster .Job was executed on host(s) , in queue , as user in cluster . was used as the home directory. was used as the working directory.Started at Thu May 9 18:49:06 2013Results reported at Thu May 9 18:49:07 2013Your job looked like:-# LSBATCH: User input/opt/pmpi/opt/ibm/platform_mpi/bin/mpirun -lsb_mcpu_hosts /opt/pmpi/opt/ibm/platform_mpi/help/helloworld-Successfully completed.Resource usage summary: CPU time : 0.23 sec. Max Memory : 2 MB Average Memory : 2.00 MB Total Requested Memory : - Delta Memory : - (Delta: the difference between total requested memory and actual max usage.) Max Swap : 36 MB Max Processes : 1 Max Threads : 1The output (if any) follows:Hello world! Im 2 of 4 on computer007Hello world! Im 0 of 4 on co