JobServicesandDataServicesGiuseppeLaRoccaINFNTRGrid就業(yè)服務(wù)和數(shù)據(jù)服務(wù)的GiuseppeLa羅卡INFNTR網(wǎng)格課件

1、EGEE is a project funded by the European Union under contract IST-2003-508833EGEE TutorialTurin - ITALY18 19 January 2005Job Services and Data Services Giuseppe La RoccaINFN Catania第1頁，共46頁。Overview Files on the RLS - Name ConventionJob Description Language - OverviewRelevant attributes JDL Examples

2、Job without Data RequirementsJob with Output DataJob with Input and Output DataOther examples using the lcg-* commands BibliographySummary and Conclusions第2頁，共46頁。Files on the RLS - Name ConventionLogical File Name 1Logical File Name nGUIDPhysical File SURL nPhysical File SURL 1.Globally Unique Iden

3、tifier (GUID) A non-human-readable unique identifier for a file, e.g.“guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6”Site URL (SURL) (or Physical/Site File Name (PFN/SFN)The location of the actual file on a storage system, e.g. “sfn:/lxshare0209.cern.ch/data/alice/ntuples.dat” Logical File Name (LFN) An

4、alias created by a user to refer to some file, e.g. “l(fā)fn:cms/track1”Transport URL (TURL)A temporary URL which can be used to access a particular data file contained in a Storage Element e.g.“gsiftp:/lxshare0209.cern.ch/data/alice/ntuples.dat”第3頁，共46頁。Job Description Language (1) In LCG-2 Job Descrip

5、tion Language (JDL) are used to describe jobs for execution on Grid.The JDL adopted within the LCG-2 middleware is based upon Condors CLASSified Advertisement language (ClassAd).A ClassAd is a record-like structure composed of a finite number of attribute separated by semi-colon (;)A ClassAd is high

6、ly flexible and can be used to represent arbitrary servicesThe JDL is used in LCG-2 to specify the desired job characteristics and constrains, which are used in by match-making process to select the resources that the job will use.第4頁，共46頁。The JDL syntax consists on statements like:Attribute = value

7、;Comments must be preceded by a sharp character ( # ) or have to follow the C+ syntax WARING: The JDL is sensitive to blank characters and tabs. No blank charactersor tabs should follow the semicolon at the end of a line.Job Description Language (2) 第5頁，共46頁。In a JDL, some attributes are mandatory w

8、hile others are optional.A essential JDL is the following:Executable = “test.sh”; StdOutput = “std.out”;StdError = “std.err”; InputSandbox = “test.sh”;OutputSandbox = “std.out”,”std.err”;If needed, arguments to the executable can be passed:Arguments = “Hello World!”;Job Description Language (3) 第6頁，

9、共46頁。If the arguments contains quoted strings, the quotes must be escaped with a backslash e.g. Arguments = “”Hello World!“ 10”;Special characters such as &, |, , 4); # If more than one resource matches, the resource with the largest # number of CPU is chosen.Rank = (other.GlueCEStateFreeCPUs);Job W

10、ithout Data Requirements(2)第25頁，共46頁。script.sh#!/bin/sh/bin/echo Hello $1 and Welcome to the EGEE Tutorial!第26頁，共46頁。ExamplesJDL WithOutput Data第27頁，共46頁。Job with Output Data (1) Type = “job”;JobType = “Normal”;Executable = “scriptOutput.sh;Arguments = “Giuseppe”;VirtualOrganisation = “gilda”; StdOu

11、tput = “sim.out”;StdError = “sim.err”;InputSandbox = scriptOutput.sh;第28頁，共46頁。OutputSandbox = “sim.err”, “sim.out”;OutputData = Outputfile = “TorinoXX.out”;LogicalName = “l(fā)fn:TorinoXX.out”;StorageElement = “grid009.ct.infn.it”; Requirements=(other.GlueCEInfoTotalCPUs4); Rank=(other.GlueCEStateFreeC

12、PUs); RetryCount = 0;Job with Output Data (2)第29頁，共46頁。scriptOutput.sh#!/bin/sh/bin/echo Hello $1 and Welcome to the EGEE Tutorial! TorinoXX.out第30頁，共46頁。ExamplesJDL With Input Data第31頁，共46頁。Job with Input Data (1) Type = “job”;JobType = “Normal”;Executable = “scriptInput.sh;Arguments = “Francesco”;

13、VirtualOrganisation = “gilda”; StdOutput = “std.out”;StdError = “std.err”;第32頁，共46頁。InputSandbox = “scriptInput.sh;OutputSandbox = “std.err”, “std.out”;InputData = “l(fā)fn:myoutdata.1”;DataAccessProtocol = “gridftp”,”rfio”;Requirements=(other.GlueCEInfoTotalCPUs4);Rank=(other.GlueCEStateFreeCPUs); Retr

14、yCount = 0;Job with Input Data (2)第33頁，共46頁。scriptInput.sh#!/bin/shlcg-cp vo gilda lfn:myoutdata.1 file:pwd/dataset1.outecho “Before updating.”cat dataset1.out#Adding new entry on the dataset1.out file./bin/echo Hello $1 and Welcome to the EGEE Tutorial! dataset1.outecho “After updating.”cat dataset

15、1.out第34頁，共46頁。ExamplesInteract with theRLS using thelcg-* commands第35頁，共46頁。StoreData.jdlType=Job;JobType=Normal;Executable=StoreData.sh;Arguments=myfile myfile grid009.ct.infn.it;VirtualOrganisation=gilda;StdOutput=std.out;StdError=std.err;InputSandbox=StoreData.sh,myfile;OutputSandbox=std.out,std

16、.err;Requirements=(other.GlueCEInfoTotalCPUs 4);Rank=other.GlueCEStateFreeCPUs;第36頁，共46頁。#!/bin/shARGS=3ERROR_BADARGS=10# Bad arguments errorERROR_FILE=20# File to copy on RLS doesnt existERROR_STORAGE=30# Storage Element doesnt exist in this Catalog.# Check for proper no. of command line args.if $#

17、 -ne $ARGS thenecho Usage: basename $0 text-file2copy lfn-file2copy se-nameexit $ERROR_BADARGSStoreData.sh (1)第37頁，共46頁。else # Setting the variables for the script.FILE2COPY=$1LFN2COPY=$2SENAME=$3fi# Check if the $FILE2COPY exists on the path.if ! -e pwd/$FILE2COPY thenecho Error: basename $0. $FILE

18、2COPY does not exist on the path.exit $ERROR_FILEfiStoreData.sh (2)第38頁，共46頁。# Checking if the SENAME is a good SE where to store data.lcg-infosites -vo gilda closeSE | grep $SENAMEif $? -eq 0 then# Uploading the copy on the RLS.lcg-cr -vo gilda -d $SENAME -l $LFN2COPY file:pwd/$FILE2COPYelseecho So

19、rry, but you have to specific a good Storage Element.exit $ERROR_STORAGEfiStoreData.sh (3)第39頁，共46頁。Type=Job;JobType=Normal;Executable=“UpdateData.sh;Arguments=myfile Julia grid009.ct.infn.it;VirtualOrganisation=gilda;StdOutput=std.out;StdError=std.err;InputSandbox=“UpdateData.sh;OutputSandbox=std.o

20、ut,std.err;Requirements=(other.GlueCEInfoTotalCPUs 4);Rank=other.GlueCEStateFreeCPUs;UpdateData.jdl第40頁，共46頁。UpdateData.sh (1)#!/bin/shARGS=3ERROR_BADARGS=10# Bad arguments error.ERROR_BADLFN=20# LFN doesnt exist on the RLS.ERROR_BADSTORAGE=30# Storage Element doesnt exist in this Catalog.# Check fo

21、r proper no. of command line args.if $# -ne $ARGS thenecho Usage: basename $0 lfn-file2copy argument se-nameexit $ERROR_BADARGSelse # Setting the arguments for the script.LFN2RETRIEVE=$1ARGUMENT=$2SENAME=$3fi第41頁，共46頁。# Check if the $LFN2RETRIEVE exists on the RLS.lcg-lr -vo gilda lfn:$LFN2RETRIEVEi

22、f $? -eq 1 thenecho Sorry, but the LFN that you have specified does not exists on the RLS.exit $ERROR_BADLFNelse# Retrieve the file from the catalog.lcg-cp -vo gilda lfn:$LFN2RETRIEVE file:pwd/myfile# .adding new data to myfile.echo Hello $ARGUMENT and Welcome to the EGEE Tutorial! pwd/myfilefi# Che

23、cking if the SENAME is a good SE where to store data.lcg-infosites -vo gilda closeSE | grep $SENAMEUpdateData.sh (2)第42頁，共46頁。if $? -eq 1 thenecho Sorry, but you have to specific a good Storage Element.exit $ERROR_STORAGEelse# Setting the variables for the new copy.FILE2COPY=pwd/myfileLFN2COPY=$LFN2RETRIEVE# Removing the old copy form the RLS.lcg-del -vo gilda -a lfn:$LFN2COPY# . and stagi