PowerPath Best Practice for AIX v7.5

1、 PowerPath Best Practice for AIX2008 年年 8 月月EMC Global Service TIS版本記錄版本記錄版本號版本號版本日期版本日期修改者修改者說明說明0.12006 年 9 月 13 日王進創(chuàng)建 PowerPath Best Practice for AIX 模板0.12006 年 10 月 8 日夏衛(wèi)東添加內容2.02006 年 11 月 22 日夏衛(wèi)東添加項目 2.3, 3.3, 3.4, 3.5, 3.6, 3.7, 4.23.02007 年 1 月 2 日夏衛(wèi)東在版本記錄中標明添加項目；添加項目 3.8, 3.9, 4.34.02007 年

2、 1 月 27 日王進增加一個章節(jié) 4.4: emc1081275.02007 年 2 月 28 日夏衛(wèi)東增加 AIXV5 patch level 信息，4GB 光纖卡信息，增加一個章節(jié) 4.5：emc1116356.02007 年 4 月 2 日夏衛(wèi)東增加 1.2：PP5.0 AIX requirement；增加3.10：如何修改 VCMDB 使不同操作系統(tǒng)共享FA ports；更新 2.2: EMC69100 內容6.12007 年 4 月 16 日夏衛(wèi)東增加 3.12：關于 AIX4.3.3 的內容7.02007 年 5 月 28 日夏衛(wèi)東增加 3.11: TF/Clone experi

3、ence sharing4.6：use PowerPath instead of AIX MPIO4.7：emc116729 FSCSI_ERR104.8: emc155602 iostat incorrect7.12008 年 1 月 6 日李曉川陳銘更新 AIX 的版本增加 4.9：EMC163961: AIX Host Lost access to storage during powermt restore( same as powermt coinfig / syminq / symrdf / symsnap)7.22008 年 2 月 14 日陳銘更新 AIX 的版本增加 4 po

4、werpath 5.1 和 SP1 對相關問題的改進7.32008 年 4 月 26 日陳銘更新 AIX 的版本增加 3.10 關于 Symmetrix 鎖的相關案例7.42008 年 6 月 2 日陳銘針對 HACMP5.4 再次更新 2.2: EMC69100 內容刪除 2.3 Starting from HACMP5.4, Primus69100 “emcpowerreset” not needed添加 2.3 EMC156011 推薦在 HACMP5.4 繼續(xù)使用 emcpowerreset7.52008 年 8 月 12 日陳銘增加 3.11 emc81931 emc81930 通過

5、Solution Enabler 清除存儲的 SCSI Reservation Lock版權說明版權說明本文件中出現(xiàn)的任何文字敘述、文檔格式、插圖、照片、方法、過程等內容，除另有特別注明，版權均屬美國EMC 公司所有，受到有關產權及版權法保護。任何個人、機構未經美國 EMC 公司的書面授權許可，不得復制、引用或傳播本文件的任何片斷，無論通過電子形式或非電子形式。目目 錄錄1AIX HOST CONNECTIVITY.61.1DOWNLOADING LATEST AIX ODM FROM EMC WEBSITE:.61.2AIX FIX LEVEL INFORMATION:.61.3FC ADA

6、PTER USEFUL COMMANDS:.81.4USEFUL SETTING AND PARAMETERS:.121.4.1FSCSI device Fast_Fail & Dyntrk.121.4.2Set HACMP “break reserves in parallel =true” (Recommended).121.5USEFUL SMALL SCRIPTS: .121.5.1刪除所有 hdiskpower 設備：.121.5.2刪除從 hdiskpower0 到 hdiskpower100 的設備： .131.5.3刪除所有 hdisk 設備：.131.5.4刪除某塊光

7、纖卡的 hdisk 設備.132HACMP.142.1HACMP 切換非常慢，每個 EMC 設備的切換時間是 2-3 分鐘。 .142.2EMC 69100: HACMP 手工的TAKEOVER切換成功，但是當一臺主機宕機時，EMC 硬盤的切換失敗，HACMP 無法清除硬盤的 SCSI RESERVATION狀態(tài)。 .172.3EMC 建議在 HACMP5.4 中繼續(xù)使用EMCPOWERRESET.272.4TOPAS 或 IOSTAT 顯示 EMC 硬盤 100%忙，但是實際的 I/O 卻非常小或者沒有。.292.5即使重啟也無法刪除HDISKPOWER設備,系統(tǒng)始終顯示設備 BUSY.29

8、2.6EMC143075, “SET_SCSI_ID” HAS BEEN REPLACED BY “CFGSCSI_ID” WITH CLARIION LUNS IN HACMP/PP ENVIRONMENT.292.7CLARIION LUN 在 AIX 操作系統(tǒng)上的識別順序總結-AUTHOR： DANNY LI.332.8AIX CHDEV COMMAND CAUSE DISK OPERATION ERROR ON CLARIION PASSIVE(STANDBY) PATH - AUTHOR ：DANNY LI.342.9關于 QUEUE DEPTH 參數(shù)-AUTHOR： DANNY L

9、I.372.10AIX JFS2 MOUNT OPTION CIO CAUSE VERY LOW PERFORMANCE FOR CP AND DD COMMAND- AUTHOR： DANNY LI.382.11EMC PROCEDURES WHEN REPLACING HBA CARD ON AIX- BY DANNY LI.392.12STEPS FOR TF/SNAP IMPLEMENTATION - BY DANNY LI.392.13如何修改 VCMDB 使不同操作系統(tǒng)共享 FA PORTS:.402.14COOKBOOK FOR TIMEFINDER INCREMENTAL CL

10、ONE AND CLONE RESTORE.402.15AIX433 環(huán)境下的技術問題.403POWERPATH .403.1FAST I/O FAILURE AND DYNAMIC TRACKING.413.2EMC145919: POWERPATH 4.5.X: AIX LVM I/O ERRORS DUE TO MISSING DEVICE RESERVATIONS.453.3SUGGEST NOT TO CHANGE “FC_ERR_RECOV” DEFAULT VALUE.473.4EMC108127: METHOD ERROR /ETC/METHODS/CFGPOWER -L PO

11、WERPATH0 0514-040 ERROR INITIALIZING A DEVICE INTO THE KERNEL IN AIX WITH MINOR NUMBER MISMATCH.473.5EMC111635: HOW TO REMOVE LUNZ DEVICES ON AN AIX HOST.503.6USE POWERPATH INSTEAD OF AIX MPIO.523.7EMC116729: FSCSI_ERR10 CONFIGURATION MISMATCH ERRORS IN ERRPT AFTER TURNING ON DYNAMIC TRACKING.533.8E

12、MC155602: AIX HOST WITH POWERPATH DOESNT UPDATE DKSTAT CORRECTLY THUS IOSTAT SUMMARY APPEARS INCORRECT.553.9EMC163961: AIX HOST LOST ACCESS TO STORAGE DURING POWERMT RESTORE( SAME AS POWERMT COINFIG / SYMINQ / SYMRDF / SYMSNAP).553.10關于 SYMMETRIX鎖的相關案例.563.11通過 SOLUTION ENABLER 清除存儲的 SCSI RESERVATIO

13、N LOCK.581AIX Host Connectivity1.1Downloading latest AIX ODM from EMC website:ftp:/ FIX level information:PowerPath5.0 AIX Maintenance Level requiremnt：AIX 6.1 service pack:NameTypePrereqsDate6100-00-04-0815 Service Pack ? 2008 6100-00-03-0808 Service Pack ? 2008 6100-00-02-0750 Service Pack ? 2007

14、6100-00-01-0748 Service Pack ? 2007 AIX 5.3 service pack:NameTypePrereqsDate5300-07-03-0811 Service Pack 5300-07-00-0747 ? 2008 5300-07-02-0806 Service Pack 5300-07-00-0747 ? 2008 5300-07-01-0748 Service Pack 5300-07-00-0747 ? 2007 5300-07-00-0747 Technology Level ? 2007 5300-06-06-0811 Service Pack

15、 5300-06 ? 2008 5300-06-05-0806 Service Pack 5300-06 ? 2008 5300-06-04-0748 Service Pack 5300-06 ? 2007 5300-06-03-0732 Service Pack 5300-06 ? 2007 5300-06-02-0727 Service Pack 5300-06 ? 2007 5300-06-01-0722 Service Pack 5300-06 ? 2007 5300-06 Technology Level ? 2007 5300-05-CSP Concluding Service P

16、ack 5300-05 ? 2007 5300-05-06 Service Pack 5300-05 ? 2007 5300-05-05 Service Pack 5300-05 ? 2007 5300-05-04 Service Pack 5300-05 ? 2006 5300-05-03 Service Pack 5300-05 ? 2006 5300-05-02 Service Pack 5300-05 ? 2006 5300-05-01 Service Pack 5300-05 ? 2006 5300-05 Technology Level ? 2006 5300-04-CSP Con

17、cluding Service Pack 5300-04 ? 2006 5300-04-03 Service Pack 5300-04 ? 2006 5300-04-02 Service Pack 5300-04 ? 2006 5300-04-01 Service Pack 5300-04 ? 2006 5300-04 Technology Level ? 2006 5300-03 Maintenance Level ? 2005 5300-02 Maintenance Level ? 2005 5300-01 Maintenance Level ? 2005 AIX5.2 service p

18、ack:Technology Levels and Service PacksNameTypePrereqsDate5200-10-04-0750Service PackTL 5200-10December 20075200-10-03-0744Service PackTL 5200-10October 20075200-10-02-0730Service PackTL 5200-10July 20075200-10-01-0722Service PackTL 5200-10June 20075200-10Technology Level June 20075200-09-CSPConclud

19、ing Service PackTL 5200-09June 20075200-09-06Service PackTL 5200-09March 20075200-09-05Service PackTL 5200-09February 20075200-09-04Service PackTL 5200-09January 20075200-09-03Service PackTL 5200-09November 20065200-09-02Service PackTL 5200-09October 20065200-09-01Service PackTL 5200-09September 200

20、65200-09Technology Level August 20065200-08-CSPConcluding Service PackTL 5200-08August 20065200-08-02Service PackTL 5200-08April 20065200-08-01Service PackTL 5200-08February 20065200-08Technology Level February 20065200-07-CSPConcluding Service Pack5200-07March 2006Maintenance Level packages (legacy

21、)NameTypePrereqsDate5200-07Maintenance Level September 20055200-06Maintenance Level May 20055200-05Maintenance Level January 20055200-04Maintenance Level December 20045200-03Maintenance Level May 20045200-02Maintenance Level October 20035200-01Maintenance Level May 2003AIX 5.1 service pack:ML5100-09

22、September, 2005 SW fixesML5100-08May, 2005 SW fixesML5100-07October, 2004 SW fixes1.3FC adapter useful commands:#lsdev -Cc adapter|grep fcs 查找系統(tǒng)所安裝的光纖卡查找系統(tǒng)所安裝的光纖卡fcs0 Available 20-58 FC Adapterfcs1 Available 20-60 FC Adapter#lscfg -vl fcsX查看光纖卡硬件信息，例如查看光纖卡硬件信息，例如:卡的類型卡的類型, WWN, 微碼版本微碼版本Where X is th

23、e number of the adapter returned by the previous lsdev command.4GB single port Fiber channel card:FC5758 ( 卡上標簽為卡上標簽為 280D,1905 )fcs3 P1-I1/Q1 FC AdapterPart Number.03N5014EC Level.ASerial Number.1E313BB001Feature Code/Marketing ID.280DManufacturer.001EFRU Number. 03N5014Device Specific.(ZM).3Networ

24、k Address.10000000C9488F4C （FC card WWN）ROS Level and ID.02C82135Device Specific.(Z0).1036406DDevice Specific.(Z1).00000000Device Specific.(Z2).00000000Device Specific.(Z3).03000909Device Specific.(Z4).FFC01158Device Specific.(Z5).02C82135Device Specific.(Z6).06C32135Device Specific.(Z7).07C32135Dev

25、ice Specific.(Z8).20000000C9488F4CDevice Specific.(Z9).BS2.10X8 =微碼版本：微碼版本：2.10X8Device Specific.(ZA).B1D2.10X8 Device Specific.(ZB).B2D2.10X8 Device Specific.(YL).P1-I1/Q14GB dual port Fiber channel card:FC5759 ( 卡上標簽為卡上標簽為 5759,1910 )fcs7 P1-I2/Q1 FC Adapter Part Number.03N5029 EC Level.A Serial N

26、umber.1F5310C048 Manufacturer.001F Feature Code/Marketing ID.5759 FRU Number. 03N5029 Device Specific.(ZM).3 Network Address.10000000C94ABA08 (FC card WWN) ROS Level and ID.02C82132 Device Specific.(Z0).1036406D Device Specific.(Z1).00000000 Device Specific.(Z2).00000000 Device Specific.(Z3).0300090

27、9 Device Specific.(Z4).FFC01155 Device Specific.(Z5).02C82132 Device Specific.(Z6).06C12132 Device Specific.(Z7).07C12132 Device Specific.(Z8).20000000C94ABA08 Device Specific.(Z9).BS2.10X8 = firmware level Device Specific.(ZA).B1F2.10X2 Device Specific.(ZB).B2F2.10X2 Device Specific.(YL).P1-I2/Q12G

28、B single port Fiber card: FC5716 ( 卡上標簽為卡上標簽為 280B,197E,1957 ) 常用于常用于 P5P5 系列主機上：系列主機上：fcs1 P1-C1-T1 FC Adapter Part Number.80P4543 EC Level.A Serial Number.1A31600450 Manufacturer.001A Feature Code.280B 或 197E 或 1957 FRU Number. 80P4544 Device Specific.(ZM).3 Network Address.10000000C9332A79 (FC ca

29、rd WWN) ROS Level and ID.02881914 Device Specific.(Z0).1001206D Device Specific.(Z1).00000000 Device Specific.(Z2).00000000 Device Specific.(Z3).03000909 Device Specific.(Z4).FF801315 Device Specific.(Z5).02881914 Device Specific.(Z6).06831914 Device Specific.(Z7).07831914 Device Specific.(Z8).20000

30、000C940F419 Device Specific.(Z9).TS1.90A4 = firmware level Device Specific.(ZA).T1D1.90A4 Device Specific.(ZB).T2D1.90A4 Device Specific.(YL).U7879.001.DQD0JHD-P1-C1-T1 2GB single port Fiber card:FC6239（卡上標簽為（卡上標簽為 5704） 常用于常用于 P6XX 系列主機上：系列主機上：fvs0 3F-08 FC Adapter Part Number.80P6415 EC Level.A Se

31、rial Number.1D51406697 Manufacturer.001D Feature Code/Marketing ID.5704 FRU Number. 80P6416 Device Specific.(ZM).3 Network Address.10000000C9492EF6 ROS Level and ID.02E01971 Device Specific.(Z0).2003806D Device Specific.(Z1).00000000 Device Specific.(Z2).00000000 Device Specific.(Z3).03000909 Device

32、 Specific.(Z4).FF601410 Device Specific.(Z5).02E01971 Device Specific.(Z6).06631971 Device Specific.(Z7).07631971 Device Specific.(Z8).20000000C9492EF6 Device Specific.(Z9).HS1.91X1 = firmware level Device Specific.(ZA).H1D1.91X1 Device Specific.(ZB).H2D1.91X1 Device Specific.(YL).P1-I5/Q1FC6228 比較老

33、型號的光纖卡，比較老型號的光纖卡，2GB 速率速率：fcs0 00-04 IBM Gigabit Fiber Channel PCI Adapter for 64 bit busPart Number.00P4494EC Level.ASerial Number.1A25001206Manufacturer.001AFRU Number. 00P4495Network Address.10000000C930D02BROS Level and ID.02C03951Device Specific.(Z0).2002606DDevice Specific.(Z1).00000000Device

34、Specific.(Z2).00000000Device Specific.(Z3).03000909Device Specific.(Z4).FF401210Device Specific.(Z5).02C03951Device Specific.(Z6).06433951Device Specific.(Z7).07433951Device Specific.(Z8).20000000C930D02BDevice Specific.(Z9).CS3.91X4 Device Specific.(ZA).C1D3.91X4 Device Specific.(ZB).C2D3.91X4FC622

35、7老型號光纖卡，老型號光纖卡，1GB 速率：速率：fcs0 00-04 IBM Gigabit Fiber Channel PCI Adapter ROS Level and ID.02903291 Device Specific (Z9).SS3.22A1 Part Number.09P4038, 09P1162, 03N4167, 24L0023 Serial Number.00000000 Device Specific.(YL).P2-I5#lsattr -El fscsi0查看查看 fscsi 設備的連接狀態(tài)和設備的連接狀態(tài)和 fastfail, dyntrk 等參數(shù)設置等參數(shù)設置s

36、csi_id 0 x10300 Adapter SCSI ID Falseattach switch How this adapter is CONNECTED Falsesw_fc_class 3 FC Class for Fabric Truefc_err_recov delayed_fail FC Fabric Event Error RECOVERY Policy True#rmdev -dl fscsi0 刪除光纖協(xié)議設備，不建議刪除刪除光纖協(xié)議設備，不建議刪除 fcs0 設備。設備。#rmdev -dl fscsi0 -R 刪除光纖卡及其所連接設備，例如刪除光纖卡及其所連接設備，例

37、如 hdiskpower, hdisk, FC tape#cfgmgr -vl fcs0配置某光纖卡連接設備配置某光纖卡連接設備#cfgmgr -v配置所有設備配置所有設備#/usr/lpp/EMC/Symmetrix/bin/emc_cfgmgr 配置配置 EMC Hdisk 設備設備, 尤其是設備尤其是設備如果某臺主機只訪問如果某臺主機只訪問 BCV 設備，需要在設備，需要在/etc/inittab 文件結尾增加一行命令：文件結尾增加一行命令：$/mkbcv -a。這樣當該主機重啟時，。這樣當該主機重啟時，BCV 設備會自動變?yōu)樵O備會自動變?yōu)锳vailable 狀態(tài)。狀態(tài)。1.4Usefu

38、l Setting and Parameters:1.4.1FSCSI device Fast_Fail & Dyntrk 相關說明參見 2.1& 4.3：chdev -l fscsi0 -a fc_err_recov=fast_fail -Pchdev -l fscsi0 -a dyntrk=yes -PNote: Only enable it in “multipath software”(such as PowerPath) environment. Only enable it in Switch environment.1.4.2Set HACMP “break re

39、serves in parallel =true” (Recommended)相關說明參見 4.1 ：smitty hacmp - extended configuration - extended resource configuration - HACMP Extended resources configuration - configure custom disk methods - change/show Custom disk methods -select disk/pseudo/power then scroll down to break reserves in parall

40、el and change the value to true.1.5useful small scripts:1.5.1刪除所有刪除所有 hdiskpower 設備：設備： for i in lsdev -Cc disk|grep hdiskpower|cut -c 1-13dormdev -dl $idone1.5.2刪除從刪除從 hdiskpower0 到到 hdiskpower100 的設備：的設備：# i=0# while i -le 100 dormdev -dl hdiskpower$ilet i=i+1done1.5.3刪除所有刪除所有 hdisk 設備：設備：# for i

41、in lsdev -Cc disk|grep EMC|cut -c 1-8dormdev -dl $idone1.5.4刪除某塊光纖卡的刪除某塊光纖卡的 hdisk 設備設備# lsdev -Cc adapter|grep fcs0fcs0 Available 20-58 FC Adapter# for i in lsdev -Cc disk|grep 20-58|cut -c 1-8dormdev -dl $idone2HACMP2.1 HACMP 切換非常慢，每個切換非常慢，每個 EMC 設備的切換時間是設備的切換時間是 2-3 分鐘。分鐘。Answer 1: set break rese

42、rves in parallel = trueID: emc122557Domain: EMC1Solution Class: 3.X CompatibilityGoal How do I set up break reserves in parallel on AIX HACMP? Fact OS: IBM AIX 5.xFact Application SW: HACMP 5.XFact break reserves in parallel = falseSymptom HACMP Cluster failover is slowFix set break reserves in para

43、llel = trueNote To change this setting enter: smitty hacmp - extended configuration - extended resource configuration - HACMP Extended resources configuration - configure custom disk methods - change/show Custom disk methods select disk/pseudo/power then scroll down to break reserves in parallel and

44、 change the value to true. Repeat this on all nodes in the cluster, then resync the cluster.Note: This will break reserves for all disks in each volume group in parallel. Thus a cluster with a large number of disks in each volume group will see substantially better improvement than a cluster where e

45、ach volume group contains a small number of disks.Note If disk/pseudo/power is not defined as a custom disk method on your cluster and you have PowerPath installed, please see EMC69100Answer 2: “set_scsi_id” is not installed in CLARiiON environmentID: emc97234Domain: EMC1Solution Class: 3.X Compatib

46、ilityGoal New set_scsi_id script is now required with CLARiiON LUNs in an HACMP / PP environment.Goal How to set up HACMP on AIX with CLARiiON LUNs under PowerPath control.Goal Example of the script set_scsi_id.sh needed with the PowerPath emcpowerreset utility.Fact Powerpath implemented with HACMP

47、and emcpowerreset.Fact EMC SW: PowerPath 4.3 and aboveFact Application SW: HACMPFact emcpowerreset utility usedFact Product: CLARiiON Storage ArraySymptom HACMP Cluster failover is slowSymptom emcpowerreset gets called on cluster initial start with no devices reserved.Symptom emcpowerreset gets call

48、ed during node failure event for devices with no disk reservationsSymptom emcpowerreset takes a longer period of time to execute.Cause The scdiskutil binary uses the PowerPath pseudo devices scsi_id ODM attribute but sends ioctl(s) directly to the adapter/HBA interface instead of using the PowerPath

49、 interface. In some cases this scsi_id ODM attribute will correspond to the CLARiiON passive not ready?path rather then the Active?path to the device required. When HACMP tries to access a LUN down the passive not ready?path, the following problems can be encountered:1) In a Cascading HACMP env, the

50、 HACMP failover time will increase for each device accessed down the passive path.2) In a Concurrent Access env, the emcpowerreset?utility might get run on some devices down the passive path resulting in a Permanent SCSI Reservation on LUNs which should never have SCSI Reservation. See Primus case e

51、mc104555 for more details on this issue.Fix Fixed in PP 4.6 (ETA Q3/2006). In the mean time Customer can get the fix from EMCs PowerLink web site.All of the above problems have been fixed with a new version on the set_scsi_id?script. To address a number of backward and forward compatibility issues,

52、this new script now combines the original set_scsi_id.sh script with part of the existing rc.emcpower.Also note:1.This new set_scsi_id?script is now required for ALL CLARiiON attach devices to an AIX node with HACMP and PP. 2.This new set_scsi_id?script will replace the existing versions of the set_

53、scsi_id.sh?and rc.emcpower?scripts. 3.With this new set_scsi_id?script the old rc.emcpower utility is no longer required. 4.This new set_scsi_id?script is backward compatible with PP 4.3 and higher. 5.The name of the file has changed from set_scsi_id.sh?to set_scsi_id? 6.The default location that th

54、e file will be installed in when added via a PP install CD has changed. It was placed in ?usr/lpp/EMC/CLARiiON/bin? it will now be put in /etc.Customers adding this new set_scsi_id?script to an AIX node for the first time should follow the steps below to acquire the file from EMCs PowerLink website

55、and then use the claddcustom command, as shown in the procedure below, to add the set_scsi_id?script into their HACMP environment.Customers who already have the previous version of the set_scsi_id.sh?script configured in their HACMP env will need to delete the old set_scsi_id.sh?script and modify th

56、eir HACMP env to point to the new file in the new location. This can be done a number of ways. EMC recommends following the exact same steps in the procedure below except, in Step 3, replace claddcustom? with clchcustom ?to change the current HACMP configuration to point to the new file name in the

57、new location.1. Copy the set_scsi_id.zip?file from EMCs PowerLink web site to a windows based system with WinZip. set_scsi_id.zip or http:/ Select Support - Downloads and Patches - Downloads D-R - PowerPath for UNIX. Then select the file set_scsi_id.zip?and follow the pop up to save the file to disk

58、.2. Uncompress the file set_scsi_id.zip?using unzip.3. Copy the script set_scsi_id?to the /etc directory of all AIX node needing it and make it executable. chmod +x /etc/set_scsi_id4. Add the custom cluster event to your configuration. This event is the name given to the script that will be added la

59、ter to select pre-defined HACMP events. Note that is is one long command./usr/es/sbin/cluster/utilities/claddcustom -t event -nset_scsi_id -ISet correct scsi id on EMC CLARiiON pseudo devices.-v/etc/set_scsi_id5. Verify your custom cluster event was added.(0)odmget HACMPcustomHACMPcustom: name = set

60、_scsi_id type = event description = Set correct scsi id on EMC CLARiiON pseudo devices. value = /etc/set_scsi_id relation = status = 06. Modify the pre-defined HACMP event by giving the event command your custom cluster event as a pre-event command./usr/es/sbin/cluster/utilities/clchevent -Onode_up -s/usr/es/s