weblogic中間件通用應(yīng)急專項(xiàng)預(yù)案

Weblogic〔含業(yè)務(wù)應(yīng)用〕通用應(yīng)急預(yù)案一、應(yīng)急處理標(biāo)準(zhǔn)和工作步驟weblogic集群方法布署，經(jīng)過(guò)中間件把握臺(tái)對(duì)各應(yīng)用節(jié)點(diǎn)進(jìn)展統(tǒng)一治理和監(jiān)控，此次中間件應(yīng)急預(yù)案以“先搶通、后搶修”為處理標(biāo)準(zhǔn)，首先恢復(fù)業(yè)務(wù)應(yīng)用，保存相關(guān)日志，后期再依據(jù)日志進(jìn)展具體分析和故障定位。weblogic中間件是應(yīng)用程序公布平臺(tái)，相比照較穩(wěn)定。業(yè)weblogicweblogic中間件JDBC等配置，實(shí)現(xiàn)前端應(yīng)用和后臺(tái)數(shù)據(jù)庫(kù)之間交互。所以不管是程序代碼問(wèn)題、硬件問(wèn)題、數(shù)據(jù)庫(kù)問(wèn)題全部有可能表現(xiàn)在weblogic中間件節(jié)點(diǎn)特別；而且由于程序代碼效率低下、參數(shù)weblogic中間件節(jié)點(diǎn)特別。這種狀況下，weblogic具體緣由能夠經(jīng)過(guò)備份日志，事后進(jìn)展分析排查。本應(yīng)急預(yù)案中包含到程序包路徑，均是依據(jù)信通企業(yè)weblogic中間件安裝標(biāo)準(zhǔn)中指定路徑。假設(shè)未依據(jù)標(biāo)準(zhǔn)安裝中間件，請(qǐng)?jiān)儐?wèn)對(duì)應(yīng)系統(tǒng)專責(zé)幫助處理。二、weblogic6步：1IE掃瞄器訪問(wèn)節(jié)點(diǎn)應(yīng)用確定節(jié)點(diǎn)可用性。2weblogic把握臺(tái)確定節(jié)點(diǎn)狀態(tài)。3、搜集故障點(diǎn)日志，備份節(jié)點(diǎn)日志。4、查看節(jié)點(diǎn)日志，依據(jù)報(bào)錯(cuò)信息確定故障緣由。5、依據(jù)故障緣由，實(shí)施應(yīng)急操作。6、組織人員進(jìn)展具體緣由分析并整改。三、故障場(chǎng)景Weblogic中間件日志文件中常見(jiàn)錯(cuò)誤以下：序號(hào)錯(cuò)誤信息序號(hào)錯(cuò)誤信息緣由消滅時(shí)機(jī)實(shí)施步驟1java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError內(nèi)存溢出節(jié)點(diǎn)運(yùn)行過(guò)程中內(nèi)存溢出節(jié)點(diǎn)開(kāi)啟過(guò)程中2more than the configured time 線程堵塞節(jié)點(diǎn)運(yùn)行過(guò)程中1,2,3,4,59,121,2,3,4,53(StuckThreadMaxTime)of“600“secondserroroccured:weblogic.jdbc.extensions.PoolLimitSQLException:weblogicmon.resourcepool.ResourceLimitException:NoresourcescurrentlyavailableinpooldgNonPooltoallocatetoapplications,pleaseincreasethesizeofthepoolandretry..BEA-000627Reachedmaximumcapacitypool“JDB-HQGL”make0newresourceinstances<BEA-000628><Created“1“resourcesforpool“JDBCDataSource-0“,outofwhich“0“areavailableand“1“areunavailable.ToomanyopenfilesJDBC連接池滿節(jié)點(diǎn)開(kāi)啟過(guò)程中6,124JDBC連接池滿節(jié)點(diǎn)運(yùn)行過(guò)程中1,2,3,4,5,65翻開(kāi)文件超節(jié)點(diǎn)運(yùn)行過(guò)程中1,2,3,4,5,7,11,12出系統(tǒng)限制6<BEA-000386> <Server subsystemfailed. Reason:weblogic 密碼錯(cuò)誤節(jié)點(diǎn)開(kāi)啟過(guò)程中8,127weblogic.security.SecurityInitializationException:Authenticationforuserweblogicdeniedweblogic.security.SecurityInitializationException:Authenticationforuserweblogicdenied/home/weblogic/Oracle/Middleware/user 人為停頓節(jié) 節(jié)點(diǎn)運(yùn)行過(guò)程中1,2,3,4,12_projects/domains/test_domain/bin/start 點(diǎn)WebLogic.sh:line180:64034Killed${JAVA_HOME}/bin/java${JAVA_VM}${MEM_ARGS}-Dweblogic.Name=${SERVER_NAME}-Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy${JAVA_OPTIONS}${PROXY_SETTINGS}${SERVER_CLASS}8 <Aug10,1:41:37AMCST><Error><EmbeddedLDAP><BEA-000000><Error權(quán)限問(wèn)題節(jié)點(diǎn)開(kāi)啟過(guò)程中10,11,12openingtheTransactionLog:/home/weblogic/Oracle/Middleware/user_projects/domains/test_domain/servers/server1/data/ldap/ldapfiles/EmbeddedLDAP.tran(Permissiondenied)>9 <Aug10,3:15:07AMCST><Error><JDBC><BEA-001112><Test“SELECT1數(shù)據(jù)庫(kù)單節(jié)節(jié)點(diǎn)運(yùn)行過(guò)程中13,11,12FROMDUAL“setupforpool“JDBCData無(wú)法全部正Source-0“failedwithexception:“java.s常對(duì)外供給ql.SQLRecoverableException:Nomored效勞atatoreadfromsocket“.><Aug10,3:15:07AMCST><Warning><JDBC><BEA-001129><Receivedexceptionwhilecreatingconnectionforpool“JDBCDataSource-0“:IOError:TheNetworkAdaptercouldnotestablishtheconnection.>10 TheJREwasnotfoundindirectory/usr/java/k1.6.0_38.(JAVA_HOME)Java環(huán)境變量配置不正節(jié)點(diǎn)開(kāi)啟過(guò)程中14,12Pleaseedityourenvironmentandsetth確eJAVA_HOMEvariabletopointtotherootdirectoryofyourJavainstallation.11 <BEA-002621><Connection rejected,theserverlicenseallowsconnections版license未節(jié)點(diǎn)運(yùn)行過(guò)程中1,2,3,4,5,15,11,12fromonly5uniqueIPaddresses.>授權(quán)12 VM Version 1.5.0_12-b04 from MicrosystemsInc.><-8-11下午04時(shí)50分58秒CST><Critical> <WebLogicServer><BEA-000362><Serverfailed.Reason:Thereare1nestederrors:weblogic.management.ManagementExce

節(jié)點(diǎn)未正常停頓

節(jié)點(diǎn)開(kāi)啟過(guò)程中 5，11，12ption: Unable to obtain lock on13ption: Unable to obtain lock on13/home/weblogic/bea/user_projects/domains/bzcb_domain/servers/AdminServer/tmp/AdminServer.lok.Servermayalreadyberunning因硬件或系統(tǒng)故障，應(yīng)用效勞器全部無(wú)法16開(kāi)啟四、應(yīng)急處理步驟IE登陸訪問(wèn)對(duì)應(yīng)故障節(jié)點(diǎn)應(yīng)用地址，驗(yàn)證故障現(xiàn)象，預(yù)防誤告警消滅。weblogic把握臺(tái)，確定故障節(jié)點(diǎn)名稱。IEweblogic://IP:端口，依據(jù)特別端口號(hào)確定特別節(jié)點(diǎn)名稱，以以以下圖。weblogic用戶登陸操作系統(tǒng)。備份對(duì)應(yīng)節(jié)點(diǎn)日志。$$cd/app/logs/hqgl/server1.log ---進(jìn)入對(duì)應(yīng)日志名目$cprserver1.log0808server1.log 依據(jù)日期備份節(jié)點(diǎn)日志查看節(jié)點(diǎn)進(jìn)程狀態(tài)。$$ps–ef|grepserver1(server1為節(jié)點(diǎn)名稱)12：weblogic590862803022:12pts/100:00:00grepserver111：JDBC連接池上限JDBCJDBC整initialcapacitymaximumcapacityinitialcapacitymaximumcapacity表示最大連接數(shù)，關(guān)鍵調(diào)整增大MaximumCapacity參數(shù)，再原有根底上增加20〔具體要依據(jù)對(duì)應(yīng)數(shù)據(jù)庫(kù)processweblogic*MaximumCapacityprocess80%〕以以以下圖所表示：JDBC放數(shù)據(jù)庫(kù)連接資源，達(dá)成搶通業(yè)務(wù)目標(biāo)。調(diào)整操作系統(tǒng)用戶翻開(kāi)文件數(shù)量上限使用使用root#vi/etc/security/limits.Confweblogic- nofile10240 增該內(nèi)容，或調(diào)整該值上限。#suweblogic weblogic$ulimit–Hn 10240修改weblogicweblogic文件。$cd/app/weblogic/Oracle/Middleware/user_projects/domain名稱/servers/節(jié)點(diǎn)名稱/security/$vipertiesUsername=weblogicUsername=weblogic 登錄把握臺(tái)使用用戶名Password=12345qwert 登錄把握臺(tái)使用密碼，在開(kāi)啟節(jié)點(diǎn)后會(huì)自動(dòng)進(jìn)展加密。調(diào)整JVM使用使用weblogic$cd/home/weblogic/startsh/ 進(jìn)入開(kāi)啟腳本名目$viserver1.sh JVM物理內(nèi)存確定，標(biāo)準(zhǔn)上該效勞器上運(yùn)行java80%。exportUSER_MEM_ARGS=“-Xms2048m-Xmx2048m-XX:PermSize=512m-XX:MaxPermSize=512m“WL_LOG=/app/logs/hqgl/server1.lognohup/home/weblogic/Oracle/Middleware/user_projects/domains/test_domain/bin/startManagedWebLogic.shserver1 ://:7001>$WL_LOG&tailf$WL_LOG修改文件權(quán)限使用使用root#chown–Rweblogic:bea/app/source_program/hqgl/----修改程序包權(quán)限#chown–R/app/weblogic/Oracle/Middleware/user_projects/domains/domain稱 修改weblogic相關(guān)文件權(quán)限#chown–R/app/logs/hqgl/ 修改開(kāi)啟日志文件權(quán)限停頓特別節(jié)點(diǎn)進(jìn)程weblogic@testweblogic@teststartsh]$kill-961658----停頓特別節(jié)點(diǎn)進(jìn)程開(kāi)啟特別節(jié)點(diǎn)weblogic@testweblogic@teststartsh]$cd/home/weblogic/startsh/[weblogic@teststartsh]$ls--進(jìn)入開(kāi)啟腳本路徑admin.shserver1.sh[weblogic@teststartsh]$./server1.sh ---開(kāi)啟節(jié)點(diǎn)修改JDBC假設(shè)在數(shù)據(jù)庫(kù)單節(jié)點(diǎn)或假設(shè)在數(shù)據(jù)庫(kù)單節(jié)點(diǎn)或VIPRAC節(jié)點(diǎn)無(wú)法全部正常對(duì)外供給效勞時(shí)，修改JDBC登陸weilogicJDBC配置頁(yè)面，依據(jù)數(shù)據(jù)庫(kù)專責(zé)供給可用連接串，修改JDBC連接串地址為jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.1.80.141)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=test)(INSTANCE_NAME=test)))Weblogic$java –version --------查詢 weblogic 用戶下 jdk 信息javaversion“1.6.0_37“Java(TM)SERuntimeEnvironment(build1.6.0_37-b06)JavaHotSpot(TM)64-BitServerVM(build20.12-b01,mixedmode)$whichjava --- 查 詢 jdk 安 裝 路 徑/usr/j