IntelMemory關(guān)鍵技術(shù)解析

1、Intel Memory關(guān)鍵技術(shù)解析Independent Channel ModeChannels can be populated in any order in Independent Channel Mode. All fourchannels may be populated in any order and have no matching requirements. Allchannels must run at the same interface frequency but individual channels may run atdifferent DIMM timing

2、s (RAS latency, CAS latency, and so forth).Lockstep Channel ModeIn Lockstep Channel Mode, each memory access is a 128-bit data access that spansChannel 0 and Channel 1, and Channel 2 and Channel 3. Lockstep Channel mode is theonly RAS mode that allows SDDC for x8 devices. Lockstep Channel Mode requi

3、res thatChannel 0 and Channel 1, and Channel 2 and Channel 3 must be populated identicallywith regards to size and organization. DIMM slot populations within a channel do nothave to be identical but the same DIMM slot location across Channel 0 and Channel 1and across Channel 2 and Channel 3 must be

4、populated the same.Mirrored Channel ModeIn Mirrored Channel Mode, the memory contents are mirrored between Channel 0 andChannel 2 and also between Channel 1 and Channel 3. As a result of the mirroring, thetotal physical memory available to the system is half of what is populated. MirroredChannel Mod

5、e requires that Channel 0 and Channel 2, and Channel 1 and Channel 3must be populated identically with regards to size and organization. DIMM slotpopulations within a channel do not have to be identical but the same DIMM slotlocation across Channel 0 and Channel 2 and across Channel 1 and Channel 3

6、must bepopulated the same.Rank Sparing ModeIn Rank Sparing Mode, one rank is a spare of the other ranks on the same channel. Thespare rank is held in reserve and is not available as system memory. The spare rankmust have identical or larger memory capacity than all the other ranks (sparing sourceran

7、ks) on the same channel. After sparing, the sparing source rank will be lost.    進行內(nèi)存熱備時，做熱備份的內(nèi)存在正常情況下是不使用的，也就是說系統(tǒng)是看不到這部分內(nèi)存容量的。每個內(nèi)存通道中有一個DIMM不被使用，預(yù)留為熱備內(nèi)存。芯片組中設(shè)置有內(nèi)存校驗錯誤次數(shù)的閾值, 即每單位時間發(fā)生錯誤的次數(shù)。當(dāng)工作內(nèi)存的故障次數(shù)達到這個“容錯閾值”，系統(tǒng)開始進行雙重寫動作，一個寫入主內(nèi)存，一個寫入熱備內(nèi)存，當(dāng)系統(tǒng)檢測到兩個內(nèi)存數(shù)據(jù)一致后，熱備內(nèi)存就代替主內(nèi)存工作，故障內(nèi)存被禁用，這樣就完成了熱備內(nèi)存接替故障內(nèi)

8、存工作的任務(wù)，有效避免了系統(tǒng)由于內(nèi)存故障而導(dǎo)致數(shù)據(jù)丟失或系統(tǒng)宕機。這個做熱備的內(nèi)存容量應(yīng)大于等于所在通道的最大內(nèi)存條的容量，以滿足內(nèi)存數(shù)據(jù)遷移的最大容量需求。    內(nèi)存刷洗（Memory Scrubbing）It is important to check each memory location periodically, frequently enough, before multiple bit errors within the same word are too likely to occur, because the one bit errors can be

9、 corrected, but the multiple bit errors are not correctable, in the case of usual (as of 2008) ECC memory modules.In order to not disturb regular memory requests from the CPU and thus prevent decreasing performance, scrubbing is usually only done during idle periods. As the scrubbing consists of nor

10、mal read and write operations, it may increase power consumption for the memory compared to non-scrubbing operation. Therefore, scrubbing is not performed continuously but periodically. For many servers, the scrub period can be configured in the BIOS setup program.The normal memory reads issued by t

11、he CPU or DMA devices are checked for ECC errors, but due to data locality reasons they can be confined to a small range of addresses and keeping other memory locations untouched for a very long time. These locations can become vulnerable to more than one soft error, while scrubbing ensures the chec

12、king of the whole memory within a guaranteed time.Key Info：1）Soft error, an important reason for doing memory scrubbing2）Error detection and correction, a general theory used for memory scrubbingECC技術(shù)90年代初，內(nèi)存體系采用奇偶性校驗（Parity Verifying）技術(shù)。奇偶校驗內(nèi)存在每一字節(jié)（8位）外又額外增加了一位作為錯誤檢測之用，BIOS中的監(jiān)控程序會將存入內(nèi)存中的數(shù)據(jù)位相加，并將結(jié)果存

13、于校驗位中。比如一個字節(jié)中存儲了某一數(shù)值10011110，每一位加起來的結(jié)果為奇數(shù)（100111105），校驗位存入1。當(dāng)CPU讀取儲存的數(shù)據(jù)時，監(jiān)控程序再次相加存儲的8位數(shù)據(jù)，并將計算結(jié)果與校驗位相比較。如果發(fā)現(xiàn)二者不同，系統(tǒng)就會產(chǎn)生出錯信息。奇偶校驗技術(shù)僅能粗略地檢查內(nèi)存錯誤，并不具備糾錯能力。另一種內(nèi)存糾錯技術(shù)叫做ECC（Error Correct Code，糾錯碼），它也是在原來的數(shù)據(jù)位上外加位來實現(xiàn)的，增加的位用來重建錯誤數(shù)據(jù)。在ECC糾錯體系中，如果數(shù)據(jù)為N個字節(jié)，則外加的ECC位為log2N + 5。例如對于64位數(shù)據(jù)，需要外加log28 + 5 = 8個ECC位。當(dāng)出現(xiàn)一個存儲

14、位錯誤時，ECC體系可以自動進行糾錯。當(dāng)出現(xiàn)2個數(shù)據(jù)位錯誤時，可以檢測出來，但不能糾錯，這種行為通常稱作“單錯糾正雙錯檢測（Single Error Correction/Double Error Detection ，簡稱SEC/DED）。一次存取中有2個以上的數(shù)據(jù)位出錯時，由于SEC/DED體系檢測不出來了，致使數(shù)據(jù)的完整性受損。采用這種結(jié)構(gòu)的存儲器，當(dāng)檢測出多位錯誤時，系統(tǒng)就會報告出現(xiàn)了致命故障（Fatal fault），之后系統(tǒng)崩潰。X4/X8 SDDC (Single Device Data Correction)隨著RAM芯片的集成度的提高和內(nèi)存容量的增大，內(nèi)存發(fā)生錯誤的概率也隨之

15、增加。幾年前被認為很可靠的SECDED內(nèi)存體系，今天已經(jīng)力不從心了，尋求具有多位糾錯能力的內(nèi)存體系結(jié)構(gòu)一直是眾多廠商追求的目標(biāo)。RAM器件失效最為嚴(yán)重的情形是其全部數(shù)據(jù)位全部發(fā)生錯誤，糾正這種錯誤的基本思路應(yīng)該著眼于芯片和系統(tǒng)的硬件結(jié)構(gòu)，而不可能通過軟件升級的方式來達到目的。存儲器中的每個字節(jié)外加一個ECC位構(gòu)成ECC字。如果存儲器系統(tǒng)的數(shù)據(jù)寬度為32個字節(jié)（或256位），實際的存儲器數(shù)據(jù)的寬度是25632288位。同時，每一個數(shù)據(jù)位都被置于分離的ECC字中。圖1描述了這種方法工作的原理。存儲系統(tǒng)由4個DIMM模塊構(gòu)成，32個字節(jié)（256位）的數(shù)據(jù)被分成4個ECC字，每個ECC字含有8個字節(jié)（

16、64位）的數(shù)據(jù)位和8個ECC位。這樣，一個ECC字的實際長度為64872位，存儲數(shù)據(jù)總長度為72×4288位。圖1 Chipkill內(nèi)存糾錯原理存儲器控制器（Memory Controller）把每個ECC字被分成4個長度為18位的段，分別存儲于4個DIMM中。同時，每個DIMM中也存儲了4個來自不同的ECC字的段。然后，每個段的18個位再被存儲在不同的RAM芯片中。經(jīng)過上述處理，每個DRAM芯片中只保存了ECC字的一位。如果RAM芯片失效，導(dǎo)致某個芯片中的全部18個位都出錯，也只是造成ECC字的一位錯誤。因為每個ECC字具有SECDED能力，可以自動糾錯，所以可以恢復(fù)所有的數(shù)據(jù)。W

17、hat is LR-DIMM or LRDIMM ?Today, using RDIMMs, a typical server system can accommodate up to three quad-rank 16GB RDIMMS per processor. However, that same system can support up to nine quad-rank 16GB LRDIMMS per processor, pushing the memory capacity from 48GB to 144GB. Load reduced DIMM (LRDIMM) is

18、 another new memory technology in development. Designed with a buffer chip (or chips) to replace the register to help minimize loading, the LRDIMM is targeted to increase overall server system memory capacity and speed using a memory buffer chip or chips as opposed to a register.( Large rectangular

19、memory buffer)Before we dive deep into LR-DIMM, lets refresh some key features on DIMM memory which is a Dual Inline Memory Module. DIMM stands for Dual Inline Memory Module. It is the RAM memory we found in our desktop computer. It consists of a few black chips (IC) on a small P

20、CB. It stores our file and data temporally when we turn on our computer. Refers to pins on both side of the module. We generally call them gold fingers? I used to put 4 of these sticks into my old 486 computer to reach the maximum allowable memory of 16MB. With the change of time and technology, I n

21、ow found that I still have 4 sticks of memory in my new computer but the total memory is 16GB instead. Not only the memory capacity has increased, I also found that my memory now is 200 times faster than the memory in my 486 computer.Memory loading in a consumer computerWith the 16GB of DDR3-1333Mhz

22、 memory in my computer, I now can play online games, stream movie and draw my 3D graphic pictures on the screen. It is just slick! Dont need more? Yes, I would like to share my movies with 3 of my friends. I would like to pass picture files to my cousin in China and I also want to watch the Space Sh

23、uttle Launch. I will never run out of memory appetite. My next question is: Why cant I put 6 sticks of memory into my computer?Wait a minute! You just cannot keep adding memory into your computer without any penalty. At 1333Mhz (1.3GHz), noise gets involved. It generally is something called the sign

24、al integrity or signal reflection issue. At a point, the accumulated noise in the system would render the system not operable.At high frequency, there is also something called loading factor? Each memory chip (IC) has input capacitance that tends to suppress the high frequency signal. Generally, eac

25、h chip has about 3 to 5 pf of input capacitance. The more chips on the module, the more accumulated capacitance will weaken the signal to an in-operable state. (Graphical illustration: Input capacitance increases the loading factor at high frequency) To solve the loading factor?problem, the PC desig

26、ner introduced multiple memory channels? Some are dual channel and some are triple channel just to let you maintain the large number of memory in your computer.Memory loading in a server computerServer computer further complicates the issue. Since server runs multiple applications simultaneously, it

27、 is best to have all the necessary data running on the active background all the time. That calls for tremendous amount of memory running dynamically at high speed and high band width. But the question is how to achieve that?Examining a regular memory module (unbuffered DIMM), we realize different g

28、roups of input lines on the same module has different loading. For example, those go to the address and control instruction lines are connected to more chips in parallel in comparison to the data bus lines. The data bus lines are usually only connected to either 1 chip or 2 chips versus the other li

29、nes can be connected to as many as 16 chips. Therefore, the question is if we can add a logical driver (buffer) chip in between the input and the address and control bus. It would even be better if this chip can also be used to line up all the address and control line signals. A register chip is, th

30、erefore, used to deliver the proper function. It is to increase drive power and keep the bus signals lined up.Registered DIMM. Bandwidth and scalabilityThis register chip really does wonders. It keeps the signal strong and also synchronized the timing between lines. Since the clock signal is repetit

31、ive, it can also be strengthened using a phase lock loop re-driver. A phase lock loop re-driver is also called a zero delay clock driver. It re-generates clock signal in time synchronization to the original clock signal. Using this method, several identical clock signals can be generated from the or

32、iginal clock source and thus multiplies the drive power of the origin repetitive clock source.(Graphical illustration: Register DIMM Block Diagram)Speed evolution limits the number of modules in a server systemSince the invention of the registered module, it kept the server industry going for years

33、until once again the increase in operational frequency had hindered the system memory capacity (number of modules per system) again.There comes the FB-DIMM with serial input and parallel outputIntel had invented the Fully Buffered DIMM to solve the above problem. It put a big driver chip in the midd

34、le of the DIMM module. This buffer chip accepts a high frequency serial signal input. Inside the chip, it converts this serial signal to parallel signal and re-drive the memory chips (DRAM) from there. Ideally, this approach reduces the physical number of signal lines at the input of the DIMM and th

35、erefore un-cluttered the physical system wiring. At the same time, it increases the number of module per system.When FB-DIMM run out of steam, LRDIMM comes into the picture.While Fully-Buffered DIMM was originally a good idea, the industry soon found that it has implementation problems. First, the s

36、erial input frequency has to be 4 times higher than the memory clock frequency. This puts it into the microwave frequency range and is a whole new page of technical difficulties. The signal weakening issue at high frequency is amplified to a difficult to control stage. Besides, the higher serial inp

37、ut frequency also increases the heat generation to an unacceptable point. Smart engineers soon announced the alternative approach, the LRDIMM. (Graphical illustration: Block Diagram FB-DIMM vs LRDIMM)LRDIMM is Load Reduced with high fan-out and bi-directionally buffers All lines are buffered. T

38、he LRDIMM (Load Reduced DIMM) works very similar to aRegistered DIMM. It buffers the address and control signals through register logic. It re-drives the clock through Phase Lock Loop. The difference is that it also buffers the data lines through bi-directional drivers. This way, all the signal line

39、s are truly fully buffered?in the parallel fashion. Pros and Cons of LRDIMM, technical pointThrough the full buffering of all signals, you can double the number of DIMM in a system using the LRDIMM re-drive method. With the addition of the new 4 rank modules and dual die chip, you can reach up to 16GB per channel with LRDIMM in todays system. Together with the new 4 channel system construction, 8 modules and 64GB memory population total can be achieved. Since the serial approached is abandoned, the heat and power dissipation problem no longer exist