NVMe SSD在存儲系統(tǒng)中的應用與技術創(chuàng)新

1、NVMe SSD在存儲系統(tǒng)中的應用與技術創(chuàng)新技術創(chuàng)新，變革未來存儲陣列 NVMe SSD 的必備功能核心功能介紹-雙端口-VSS與端到端數(shù)據(jù)保護-在線固件升級提升運維效率及負載按需均衡的新技術新功能介紹-High Latency Log & Latency Statistics-Telemetry-提高運維效率(NVMe1.3)-Weighted Round Robin (WRR)PBlaze 系列在全閃存陣列中的應用關于 Memblaze1234PBlaze 系列在全閃存陣列中的應用1宏杉 NVMe 全閃存陣列 SPC-1 測試SPC-1測試模型與流程苛刻而復雜，SSD 如果有任何性能抖動都

2、會被逐級放大，嚴 重影響業(yè)務的性能一致性。16 控 IOPS 1100萬單控 IOPS 68.75萬總體延時 0.264ms最高壓力延時 0.337ms*數(shù)據(jù)時間2020-11-06PBlaze 系列在全閃存陣列中的應用288 塊 Memblaze PBlaze 系列NVMe SSDSUSTAIN Response Time GraphResponse Time vs. Throughput GraphMemblaze 助 MS7000G2-Mach 在 SPC-1 測試達到千萬 IOPSMS7000G2-Mach的時延抖動在最高壓力測試中不超過 7s，抖動范圍在2.07%之內(nèi)，是極少數(shù)SPC

3、-1測試中時 延抖動小于3%的產(chǎn)品。MS7000G2-Mach 最 高 壓 力 時 延 為 0.337ms ， 在 SPC-1 TOP10榜單中最低。在總體時延方面MS7000G2-Mach也有著十分優(yōu)異的表現(xiàn)，0.264ms的總體時延意味著可以應用于對 時延要求極為苛刻的場景，比如證券交易。288 塊 Memblaze PBlaze系列 NVMe SSDPBlaze系列有很好的穩(wěn)定性PBlaze系列高性能和低延時優(yōu)勢存儲陣列NVMe SSD 的必備功能雙端口、端到端數(shù)據(jù)保護、在線固件升級2高可用 | 存儲陣列必備：Dual-Port & Reservations雙端口NVMe SSD支持單個

4、主機內(nèi)存在的兩條數(shù)據(jù)路徑，同時通 過兩個控制器訪問相同的存儲設備實現(xiàn)冗余。在另一種常見的應用 場景中，兩個主機系統(tǒng)也可以同時使用雙端口訪問同一存儲系統(tǒng)。如果在一條數(shù)據(jù)路徑丟失時發(fā)生系統(tǒng)故障或電源丟失，可用的數(shù)據(jù) 路徑將繼續(xù)運行，對業(yè)務連續(xù)無影響，就好像沒有發(fā)生故障，對 QoS的影響達到最小。PCIeSwitchPCIeSwitchServer AServer BActive - ActivePCIeSwitchPCIeSwitchServer AServer BHeart beatActive - PassivePCIeSwitchPCIeSwitchLink RedundancyDual P

5、ort Application Models雙端口：提供了多個host訪問NVMe SSD共享命名空間的能力，提供鏈路冗余以及基于端口的PCIe和NVMe寄存器管理。Reservations：雙端口模式下的仲裁機制。用于存儲控制器Reserve仲裁盤。Server AServer BDual PortPCIe X 4 Server ASingle Port高可用 | 存儲陣列必備：Dual-Port & Reservations# nvme id-ns /dev/nvme0n1 -Hmc: 0 x31:1 : 0 x1 Metadata Pointer Supported0:0 : 0 x1

6、Metadata as Part of Extended Data LBA Supporteddpc: 0 xf4:4 : 0Protection Information Transferred as Last 8 Bytes of Metadata Not Supported 3:3 : 0 x1 Protection Information Transferred as First 8 Bytes of Metadata Supported 2:2 : 0 x1 Protection Information Type 3 Supported1:1 : 0 x1 Protection Inf

7、ormation Type 2 Supported0:0 : 0 x1 Protection Information Type 1 Supported-LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 x2 Good LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 BestLBA Format 2 : Metadata Size: 8 bytes

8、 - Data Size: 512 bytes - Relative Performance: 0 x2 Good (in use) LBA Format 3 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0 BestLBA Format 4 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0 x2 Good端到端數(shù)據(jù)保護原理及類型介質損壞？電磁噪聲？ 供電問題、固件/軟件問題、宇宙射線都有可能產(chǎn)

9、生靜默錯誤？加上PI信息，保障數(shù)據(jù)正確的寫下去和讀上來。LBA n Data (eg.512 byte)LBA nMetadataLBA n+1 DataLBA n+1MetadataMetadata contiguous with LBAGRDAPPREF8 byte PI (Protection Information)16bit guard tag (CRC of data portion)16bit application tag 32bit reference tag8bytes PI信息寫的PRINFOPRACT（指明了PI信息生成的機制）=1時， controller生成PI并將

10、其寫入NAND=0時， controller獲取上層應用下發(fā)的PI信息，將檢查PI信息并寫入NANDPRCHK（指明了控制器收到包時檢查的PI信息） Bit2=1，SSD 在 收 到 packet 時 ， 檢 查 CRC Bit1=1，SSD在收到packet時，檢查App TagBit0=1，SSD在收到packet時，檢查Reference TagVSS與端到端數(shù)據(jù)保護意義？Figure 257:Write Command Protection Infotmation Processinga) MD=8,PI,PRACT=0:Metadata remains same size in NV

11、M and host bufferHOSTCTLRLB Data8PILB Data8PINVM8B MD8B MDFigure 258: Read Command Protection Infotmation Processinga) MD=8,PI,PRACT=0:Metadata remains same size in NVM and host bufferHOSTCTLRLB Data8PILB Data8PINVM8B MD8B MDLBA n DataLBA nMetadataLBA n+1DataLBA n+1MetadataData Buffer (PRP1 & PRP2)H

12、ostDIFDIXLBA nMetadataLBA n+1MetadataLBA n+2MetadataMetadata Buffer (MD)LBA n DataLBA n+1 DataLBA n+2 DataData Buffer (PRP1 & PRP2)Host新一代 PBlaze5 920 系列支持在線固件升級功能， 無需重啟服務器。在線固件升級激活：主機端發(fā)送Firmware Commit 指令，參數(shù)指定 -a 3（即Commit Action 為011b），無需 Reset，即可 完成固件激活。21固件1正常運行時，通過Download 命 令將新固件鏡像文件下載到 SSD 的

13、DRAM 中；Figure 77：Firmware Commit Command Dword 10在線固件升級PBlaze5 520 及 PBlaze6 系列同樣 支持在線固件升級在此過程中上層應用會停頓12s，隨后恢復（實測）提升運維效率及負載按需均衡的新技術Latency tracking、Telemetry、Weighted Round Robin3Latency Statistics打 開 / 關 閉 Latency Statistics logging enable：nvme memblaze lat-stats /dev/nvme0 -edisable：nvme memblaze

14、lat-stats /dev/nvme0 dRetrieve and show IO Latency Statistics log read：nvme memblaze lat-stats-print /dev/nvme0 write: nvme memblaze lat-stats-print /dev/nvme0 wRangeDescription0-1 ms每32us一個level, 32個level1-32 ms每1ms一個level, 31個level32ms-1s每32ms一個level, 31個level1-2s1個level2-3s1個level3-4s1個level4-+IN

15、Fs1個levelMemblaze IO Read Command Latency HistogramMajor Revision : 1Minor Revision : 0-BucketStartEndValue-00us32us39084675132us64us38955560264us96us23634505396us128us239515444128us160us258333465160us192us290301396192us224us333979437224us256us369467088256us288us466026839288us320us12077027710320us35

16、2us105197121611352us384us128601928712384us416us20584069313416us448us15652548014448us480us29864894015480us512us24704314216512us544us42949672917544us576us124711750nvme memblaze lat-stats-print /dev/nvme021從測試結果中定位高延時的I/O：High Latency Logging :有13條 13 128KB 寫IO，時間是20201006 - 16:19:09 (UTC)；Latency Stat

17、istics Log: 2020-10-07 00:19:49的Log中有13個寫IO處于 96128ms區(qū)間中；Analyze the Reason of High Latency:根據(jù)log中QID, CID, StartLBA 以及其他的應用和系統(tǒng) Log可以綜合判定業(yè)務的狀態(tài)。High Latency Log Output-BucketStartEndValue-Latency Statistics Output123470000600005000040000300002000005010015030035040045090% latency(sec)new order(tpmc)My

18、SQL-TPCC（10000warehouse，7 hours）200250Time(minutes)count90% lat01230500010000150002000025000Latency(ms)iostattime(s)r_awaitw_awaitHigh Latency Log & Latency Statistics用戶需要一個標準接口獲取 NVMe SSD 的 Log。當發(fā)生問題時，通過標準化的 NVMe 命令獲取系統(tǒng)和 NVMe SSD 的 Log。需要特殊的工具收集 SSD 和系統(tǒng)中 SSD 的 Log。不同廠商、不同品牌的 NVMe SSD 接口和 Log 格式不盡相同

19、。每 512bytes 為1個 Telemetry Block，根據(jù)具體的需求，獲取不同規(guī)模的 Log信息。Telemetry-提高運維效率Data Area 1Data Area 2Data Area 3Data Area1First phase, a small size payload,designed for operational periodic data pulls (health monitoring,performance) during operation,contains critical drive data.Data Area2Second phase,medium

20、size payload,scaleup for additional content.Data Area3Third phase,a large size payload,designed to be comprehensive for failure triage and root cause analysis.Weighted Round Robin (WRR)User scenario: differentiated I/O service when applications with diverse performance-needs share a storage-device.關

21、于 Memblaze41.Engineer sample performance.2.Measurement is performed at Steady State follow SNIA SSS-PTS-E test specification.3.Average latency measured with 4KB random I/O pattern.Performance Consistency Data ReliabilityVSS, T10 PI, Quota by NamespacePBlaze5 920 Series 1D920C920D926C926User Capacity

22、 (TB)3.847.683.847.683.26.43.26.4InterfacePCIe 3.0 x 4PCIe 3.0 x 8PCIe 3.0 x 4PCIe 3.0 x 8Form Factor2.5-inch U.2HHHL AIC2.5-inch U.2HHHL AIC128KB Sequential Read (GB/s)3.53.55.65.93.53.55.65.9128KB Sequential Write (GB/s)3.33.53.33.73.33.53.33.7Sustained Random Read(4KB) IOPS825K840K825K970K825K835

23、K825K970KSustained Random Write (4KB)IOPS (Steady State) 2130K155K130K155K280K305K280K305KLatency Read/Write (s) 392 / 1292 / 12Lifetime Endurance1 DWPD3 DWPDUncorrectable Bit Error Rate 10-17Mean Time Between Failures2 million hoursProtocolNVMe 1.2aNAND Flash Memory96L 3D NANDOperation SystemRHEL,

24、SLES, CentOS, Ubuntu, Windows Server, VMware ESXiPower Consumption725 WBasic Feature SupportPower Failure Protection, Hot Pluggable,Full Data Path Protection, S.M.A.R.T, Flexible Power ManagementAdvanced Feature SupportTRIM, Multi-namespace, AES 256 Data Encryption & Crypto Erase,Dual Port & Reserva

25、tion (U.2 only), EUI64/NGUID,Variable Sector Size Management & T10 PI (DIF/DIX), Firmware Upgrade without Reset, Quota by NamespaceSoftware SupportOpen source management tool, CLI debug toolOS in-box driver (Easy system integration)PBlaze5 920 Series Product OverviewNOTES:1. Performance may vary due

26、 to different system configurations and firmware version.2. Measurement is performed at Steady State follow SNIA SSS-PTS-E test specification.3. Average latency measured with 4KB random I/O pattern. 4. DWPD, Drive Writes Per Day for 5 years.PBlaze5 520/526 Series 1C/D520C/D526User Capacity (TB)1.923.84