CUDB和PG閏秒調(diào)整方案(2015.6.29)

1、 中國移動通信集團(tuán)廣東有限公司網(wǎng)絡(luò)管理中心核網(wǎng)絡(luò)維護(hù)室CUDB和PG閏秒調(diào)整方案中國移動廣東公司網(wǎng)管維護(hù)中心核心網(wǎng)絡(luò)維護(hù)室2015年6月 目錄一、網(wǎng)絡(luò)變更簡報41.1網(wǎng)絡(luò)變更背景41.2網(wǎng)絡(luò)變更內(nèi)容4二、變更風(fēng)險點與業(yè)務(wù)影響評估52.1本次網(wǎng)絡(luò)變更的主要風(fēng)險點52.2本次變更實施人員52.3是否為首次變更52.4前期測試結(jié)果52.5業(yè)務(wù)影響評估52.5.1廠家意見52.5.2變更負(fù)責(zé)人意見52.6 綜合影響評估62.6.1 計費影響62.6.2 綜資及話務(wù)網(wǎng)管影響62.6.3 支撐系統(tǒng)調(diào)整62.6.4 性能統(tǒng)計影響62.6.5 業(yè)務(wù)流程影響62.7 領(lǐng)導(dǎo)決策點6三、操作內(nèi)容與流程63.1 相關(guān)

2、拓?fù)鋱D63.1 操作內(nèi)容與流程7四、詳細(xì)實施步驟84.1 變更前準(zhǔn)備84.2 詳細(xì)實施步驟9五、測試與驗證方案105.1 驗證方案（定期撥測）105.1 驗證方案（定期指標(biāo)）10六、應(yīng)急方案116.1 本次變更可能出現(xiàn)的問題116.2 應(yīng)急與倒回方案11七、BIBYSITTING監(jiān)測要求13八、聯(lián)系人表13九、備注14一、網(wǎng)絡(luò)變更簡報1.1網(wǎng)絡(luò)變更背景本次操作是針對新增NGBOSS服務(wù)開通主機(jī)的靜態(tài)路由配置的方案1.2網(wǎng)絡(luò)變更內(nèi)容修改網(wǎng)絡(luò)配置 添加和NGBOSS相關(guān)的兩條靜態(tài)路由變更涉及網(wǎng)元網(wǎng)絡(luò)變更內(nèi)容網(wǎng)絡(luò)影響情況變更操作時間業(yè)務(wù)影響時間業(yè)務(wù)受影響網(wǎng)元修改PG和CUDB的NTP配置無2015/

3、06/30 00:00 05:00無無二、變更風(fēng)險點與業(yè)務(wù)影響評估2.1本次網(wǎng)絡(luò)變更的主要風(fēng)險點序號風(fēng)險點具體描述變更負(fù)責(zé)人意見1局?jǐn)?shù)據(jù)變更修改NTP服務(wù)器啟動配置22.2本次變更實施人員2.3是否為首次變更是2.4前期測試結(jié)果修改配置后，觀察設(shè)備的性能統(tǒng)計和告警是否正常2.5業(yè)務(wù)影響評估2.5.1廠家意見預(yù)計不會對網(wǎng)絡(luò)業(yè)務(wù)產(chǎn)生不良影響。2.5.2變更負(fù)責(zé)人意見預(yù)計不會對網(wǎng)絡(luò)業(yè)務(wù)產(chǎn)生不良影響。2.6 綜合影響評估2.6.1 計費影響不會對計費產(chǎn)生影響變化。2.6.2 綜資及話務(wù)網(wǎng)管影響相關(guān)網(wǎng)元信息在綜資和話務(wù)網(wǎng)管中不需要更新。2.6.3 支撐系統(tǒng)調(diào)整不需要在支撐系統(tǒng)上補充配置，例如億陽主動性能

4、監(jiān)控、IPNET網(wǎng)管等。2.6.4 性能統(tǒng)計影響不會對網(wǎng)元或網(wǎng)管上的性能統(tǒng)計報表產(chǎn)生影響。2.6.5 業(yè)務(wù)流程影響不會對業(yè)務(wù)流程產(chǎn)生變化。2.7 領(lǐng)導(dǎo)決策點序號需要領(lǐng)導(dǎo)決策點具體描述變更負(fù)責(zé)人意見12三、操作內(nèi)容與流程3.1 相關(guān)拓?fù)鋱D3.1 操作內(nèi)容與流程序號操作內(nèi)容實施人實施時間結(jié)果確認(rèn)1修改NTP服務(wù)器啟動配置2015/06/30 00:00 2四、詳細(xì)實施步驟4.1 變更前準(zhǔn)備PG:序號變更準(zhǔn)備項實施人配合人實施時間結(jié)果確認(rèn)1設(shè)備廠家網(wǎng)元類型序號核查項核查及判定標(biāo)準(zhǔn)備注說明愛立信PG1dvemon status n allCAI3G ports should be E/E on all

5、 PL.MML ports should be E/E on all PL.CLI ports should be D/D on all PL.OAM ports should be E/E on all PL.Application server should be E/E on all PL.Deployment error number should be 0 on all PL.ESA processes should be UP on all the nodes.License server should be UP on both SC.Block proxy should be

6、UP on PL3.Reporter should be UP on all PL.Zookeeper should be UP on both SC and PL-3.EMA愛立信PG2fmacivealarmsThere should be no critical alarms.EMA2檢查PG的proclog，確認(rèn)各類指令的發(fā)放成功率指標(biāo)檢查，精確到指令級別（ADD_SUB、LST_SUB等）3備份所有GEP3板卡的NTP配置cp /etc/init.d/ntp /etc/init.d/ntp.bak4CUDB:序號變更準(zhǔn)備項實施人配合人實施時間結(jié)果確認(rèn)1設(shè)備廠家網(wǎng)元類型序號核查項核查及

7、判定標(biāo)準(zhǔn)備注說明愛立信CUDB1cudbSystemStatus(1) The time and date is right,(2)CUDB software version is right, (3)11B:System Monitor threads status on each system controller board are OK; 13B:System Monitor leader and BC Server status are OK.(4)Clusters status on each PL and DS is OK, the memory usage lower than

8、 80% (5) There are 4 NDBs process are running on PL Unit and 2 NDBs process are running on each DS Unit, (6) Replication Channels are OK, 'M': Master, 'Sn': Slave replication channel #n is active. (7)There is no new alarm is printed, more information refer to chapter 2.2.16 (5) MySQL

9、 servers connection are listed as “OK”; (6) CUDB Processes in OAMs, PLs and DSs are listed as “running”;愛立信CUDB2fmacivealarmsThere should be no critical alarms.23備份所有GEP3板卡的NTP配置cp /etc/sysconfig/ntp /etc/sysconfig/ntp.bak44.2 詳細(xì)實施步驟步驟要求細(xì)化到指令級。PG:序號詳細(xì)實施步驟（細(xì)化到指令級）實施人配合人實施時間結(jié)果確認(rèn)1登錄PG的第一塊GEP32更改NTP工作模式

10、，修改ntp配置文件sed -i 's/-ng/-xng/g' /etc/init.d/ntp3然后重啟板卡上的ntp服務(wù)service ntp restart4在其余板卡重復(fù)上述步驟CUDB:序號詳細(xì)實施步驟（細(xì)化到指令級）實施人配合人實施時間結(jié)果確認(rèn)1登錄CUDB的備用SC2更改NTP工作模式，修改ntp配置文件/etc/sysconfig/ntp，將“NTPD_OPTIONS="-g -u ntp:ntp"”改為“NTPD_OPTIONS="-x -g -u ntp:ntp"sed -i 's/-g -u ntp:ntp /

11、-x -g -u ntp:ntp/g' /etc/sysconfig/ntp3然后重啟板卡上的ntp服務(wù)rcntp restart4等2分鐘5登錄CUDB的主用SC6更改NTP工作模式，修改ntp配置文件/etc/sysconfig/ntp，將“NTPD_OPTIONS="-g -u ntp:ntp"”改為“NTPD_OPTIONS="-x -g -u ntp:ntp"sed -i 's/-g -u ntp:ntp /-x -g -u ntp:ntp/g' /etc/sysconfig/ntp7然后重啟板卡上的ntp服務(wù)rcntp

12、 restart五、測試與驗證方案5.1 驗證方案（定期撥測）序號撥測內(nèi)容網(wǎng)元驗證頻度驗證結(jié)果驗證人5.1 驗證方案（定期指標(biāo)）PG：序號指標(biāo)網(wǎng)元驗證頻度驗證結(jié)果驗證人1檢查PG的proclog，確認(rèn)各類指令的發(fā)放成功率指標(biāo)檢查，精確到指令級別（ADD_SUB、LST_SUB等）cat /var/proclog/raw/proclog* | grep ADD_SUB | grep FAILED | wc -lcat /var/proclog/raw/proclog* | grep ADD_SUB | grep SUCC | wc lPG一次CUDB：序號指標(biāo)網(wǎng)元驗證頻度驗證結(jié)果驗證人1檢查CU

13、DB狀態(tài) cudbSystemStatusCUDB一次六、應(yīng)急方案6.1 本次變更可能出現(xiàn)的問題系統(tǒng)有可能出現(xiàn)未知的軟件錯誤，對業(yè)務(wù)發(fā)放造成影響。6.2 應(yīng)急與倒回方案PG:使用備份的NTP配置文件倒回cp /etc/init.d/ntp.bak /etc/init.d/ntp然后重啟NTP服務(wù)service ntp restartCUDB:使用備份的NTP配置文件倒回cp /etc/sysconfig/ntp.bak /etc/sysconfig/ntp然后重啟NTP服務(wù)rcntp restart七、BIBYSITTING監(jiān)測要求PG:設(shè)備廠家網(wǎng)元類型序號核查項核查及判定標(biāo)準(zhǔn)備注說明愛立信P

14、G1dvemon status n allCAI3G ports should be E/E on all PL.MML ports should be E/E on all PL.CLI ports should be D/D on all PL.OAM ports should be E/E on all PL.Application server should be E/E on all PL.Deployment error number should be 0 on all PL.ESA processes should be UP on all the nodes.License

15、server should be UP on both SC.Block proxy should be UP on PL3.Reporter should be UP on all PL.Zookeeper should be UP on both SC and PL-3.EMA愛立信PG2fmacivealarmsThere should be no critical alarms.EMA愛立信PG15cat /var/proclog/raw/proclog* | grep ADD_SUB | grep FAILED | wc -l確保所有指令的成功率都在正常范圍EMAcat /var/p

16、roclog/raw/proclog* | grep ADD_SUB | grep SUCC | wc lCUDB:設(shè)備廠家網(wǎng)元類型序號核查項核查及判定標(biāo)準(zhǔn)備注說明愛立信CUDB1cudbSystemStatus(1) The time and date is right,(2)CUDB software version is right, (3)11B:System Monitor threads status on each system controller board are OK; 13B:System Monitor leader and BC Server status are

17、OK.(4)Clusters status on each PL and DS is OK, the memory usage lower than 80% (5) There are 4 NDBs process are running on PL Unit and 2 NDBs process are running on each DS Unit, (6) Replication Channels are OK, 'M': Master, 'Sn': Slave replication channel #n is active. (7)There is no new alarm is printed, more information refe