PVE使用INTEL82576網(wǎng)卡開啟IOMMO和 SRIOV

PVE使用INTEL82576網(wǎng)卡開啟IOMMO和SRIOV1.建議先安裝基本的軟件aptupdateapt-getinstallnet-toolsvimethtool-y2.打開IOMMU在/etc/default/grub文件里，GRUB_CMDLINE_LINUX_DEFAULT行添加以下內(nèi)容如果intelCPU，則intel_iommu=onpcie_acs_override=downstream。sed-i'/GRUB_CMDLINE_LINUX_DEFAULT/cGRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=nouveauintel_iommu=on"'/etc/default/grub如果AMDCPU，則amd_iommu=onpcie_acs_override=downstream。sed-i'/GRUB_CMDLINE_LINUX_DEFAULT/cGRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=nouveauamd_iommu=on"'/etc/default/grub#更新grubupdate-grub#在/etc/modules文件里添加IOMMUvfio>>/etc/modulesvfio_iommu_type1>>/etc/modulesvfio_pci>>/etc/modulesvfio_virqfd>>/etc/modules#更新并重啟update-initramfs-u-kallreboot#檢查IOMMO功能是否開啟dmesg|grep-eDMAR-eIOMMU-eAMD-Vifind/sys/kernel/iommu_groups/-typellspci-vv|lspci-vv|

grepEth如果沒有開啟，則在/etc/modprobe.d/的任意一個(gè).conf下增加不安全中斷：optionsvfio_iommu_type1allow_unsafe_interrupts=13.啟用SRIOVlspci-nn|grepEth最前面的數(shù)字是設(shè)備的BDF號(hào)lspci-sBDF號(hào)-vvv|grepCapabilities查看是否支持SR-IOV#查看網(wǎng)口最大支持的VF網(wǎng)口數(shù)量cat/sys/bus/pci/devices/0000:BDF號(hào)/sriov_totalvfs返回結(jié)果就是最大支持的VF網(wǎng)口數(shù)量echoVF網(wǎng)口數(shù)量>/sys/class/net/enp1s0f0/device/sriov_numvfs#添加VF網(wǎng)口成功顯示lspci|grepEth首先確定你的intel網(wǎng)卡支持SR-IOV，如果支持，那么繼續(xù)root@pve:~#echo"optionsigbmax_vfs=6,6,6,6,6,6">/etc/modprobe.d/igb.confroot@pve:~#depmod-aroot@pve:~#update-initramfs-uroot@pve:~#reboot解釋一下，max_vfs=6,6,6,6,6,6：這條語句說明最大的虛擬網(wǎng)卡是6個(gè)，也就是虛擬6個(gè)echo"optionsigbmax_vfs=6,6,6,6,6,6">/etc/modprobe.d/igb.confdepmod-aupdate-initramfs-ureboot解釋一下，max_vfs=6,6,6,6,6,6：這條語句說明最大的虛擬網(wǎng)卡是6個(gè)，也就是1個(gè)物理網(wǎng)卡虛擬6個(gè)3.4檢查虛擬網(wǎng)卡root@pve:~#dmesg|grep82576[2.527369]igbvf0000:07:10.1:Intel(R)82576VirtualFunction[2.532109]igbvf0000:07:10.3:Intel(R)82576VirtualFunction[2.535583]igbvf0000:07:10.5:Intel(R)82576VirtualFunction[2.539074]igbvf0000:07:10.7:Intel(R)82576VirtualFunction[2.542565]igbvf0000:07:11.1:Intel(R)82576VirtualFunction[2.546055]igbvf0000:07:11.3:Intel(R)82576VirtualFunction[2.768637]igbvf0000:07:10.0:Intel(R)82576VirtualFunction[2.771872]igbvf0000:07:10.2:Intel(R)82576VirtualFunction[2.775080]igbvf0000:07:10.4:Intel(R)82576VirtualFunction[2.778273]igbvf0000:07:10.6:Intel(R)82576VirtualFunction[2.781481]igbvf0000:07:11.0:Intel(R)82576VirtualFunction[2.784682]igbvf0000:07:11.2:Intel(R)82576VirtualFunction[2.920650]igbvf0000:05:10.0:Intel(R)82576VirtualFunction[3.128489]igbvf0000:05:10.2:Intel(R)82576VirtualFunction[3.412422]igbvf0000:05:10.4:Intel(R)82576VirtualFunction[3.540328]igbvf0000:05:10.6:Intel(R)82576VirtualFunction[3.580426]igbvf0000:05:11.0:Intel(R)82576VirtualFunction[3.608433]igbvf0000:05:11.2:Intel(R)82576VirtualFunction[3.940652]igbvf0000:05:10.1:Intel(R)82576VirtualFunction[3.960418]igbvf0000:05:10.3:Intel(R)82576VirtualFunction[3.976422]igbvf0000:05:10.5:Intel(R)82576VirtualFunction[3.992427]igbvf0000:05:10.7:Intel(R)82576VirtualFunction[4.008435]igbvf0000:05:11.1:Intel(R)82576VirtualFunction[4.024426]igbvf0000:05:11.3:Intel(R)82576VirtualFunction[4.359456]igbvf0000:04:10.0:Intel(R)82576VirtualFunction[4.380383]igbvf0000:04:10.2:Intel(R)82576VirtualFunction[4.400395]igbvf0000:04:10.4:Intel(R)82576VirtualFunction[4.424404]igbvf0000:04:10.6:Intel(R)82576VirtualFunction[4.440398]igbvf0000:04:11.0:Intel(R)82576VirtualFunction[4.464396]igbvf0000:04:11.2:Intel(R)82576VirtualFunction[4.795365]igbvf0000:04:10.1:Intel(R)82576VirtualFunction[4.820409]igbvf0000:04:10.3:Intel(R)82576VirtualFunction[4.836453]igbvf0000:04:10.5:Intel(R)82576VirtualFunction[4.856446]igbvf0000:04:10.7:Intel(R)82576VirtualFunction[4.872448]igbvf0000:04:11.1:Intel(R)82576VirtualFunction[4.888444]igbvf0000:04:11.3:Intel(R)82576VirtualFunction出現(xiàn)VF代表成功了，那么在你的ProxmoxVE控制面板上，網(wǎng)絡(luò)一欄就會(huì)出現(xiàn)很多網(wǎng)卡注意把原來網(wǎng)卡標(biāo)記一下，否則會(huì)亂的二、測(cè)試SR-IOV網(wǎng)卡生成可以先進(jìn)行測(cè)試譬如在enp2s0f0網(wǎng)卡上開4個(gè)VFecho4>/sys/class/net/enp2s0f0/device/sriov_numvfs成功不會(huì)有任何反饋，回到PVE節(jié)點(diǎn)網(wǎng)絡(luò)查看網(wǎng)絡(luò)列表是否有VF網(wǎng)卡出現(xiàn)，也可以使用lspci查看pci設(shè)備中是否有vf網(wǎng)卡三、注冊(cè)系統(tǒng)服務(wù)首先創(chuàng)建啟動(dòng)腳本cd/etc/systemd/systemvisriov.service示例是在i350-T4(系統(tǒng)名稱enp2s0)的0號(hào)和1號(hào)口上分別開5個(gè)vf網(wǎng)卡、固化MAC地址防止重啟MAC隨機(jī)變動(dòng)并且設(shè)置所有PF和VF為UP狀態(tài)[Unit]Description=ScripttoenableSR-IOVonboot[Service]Type=simple#startSR-IOVExecStartPre=/usr/bin/bash-c'/usr/bin/echo5>/sys/class/net/enp2s0f0/device/sriov_numvfs'ExecStartPre=/usr/bin/bash-c'/usr/bin/echo5>/sys/class/net/enp2s0f1/device/sriov_numvfs'#setVFMACExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf0mac00:54:00:00:00:00'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf1mac00:54:00:00:00:01'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf2mac00:54:00:00:00:02'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf3mac00:54:00:00:00:03'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf4mac00:54:00:00:00:04'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf0mac00:54:00:00:01:00'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf1mac00:54:00:00:01:01'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf2mac00:54:00:00:01:02'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf3mac00:54:00:00:01:03'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf4mac00:54:00:00:01:04'#setPFupExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0up'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f1up'#setVFupExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0v0up'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0v1up'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0v2up