For Storage Spaces Direct it’s very important that we have RDMA adapters but there is additional configuration needed to get it work with RoCE adapters. We need to configure PFC.

Here are the steps to get that working:

  1. Install Data Center Bridging (if only Storage Spaces Direct without Hyper-V is planned):
    Install-WindowsFeature -Name "Data-Center-Bridging"
  2. If you also install Hyper-V (for Hyper converged infrastructure HCI), you can directly install all features needed for that:
    Install-WindowsFeature -Name "Hyper-V", "Failover-Clustering", "Data-Center-Bridging", "RSAT-Clustering-PowerShell", "Hyper-V-PowerShell", "FS-FileServer"
  3. Removing preconfigured settings if there are any:
    Remove-NetQosPolicy -Confirm:$False
Remove-NetQosTrafficClass -Confirm:$False
Disable-NetQosFlowControl -Priority 0,1,2,3,4,5,6,7
  4. Disable DCBx:
    Set-NetQosDcbxSetting -InterfaceAlias StorageA –Willing $false -Confirm:$False
Set-NetQosDcbxSetting -InterfaceAlias StorageB –Willing $false -Confirm:$False
  5. Create QoS Policies
    New-NetQosPolicy "SMBDirect" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0
  6. Enable Priority Flow Control (PFC) for priority 3:
    Enable-NetQosFlowControl -Priority 3,7
  7. Create QoS Traffic Classes:
    New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 50 -Algorithm ETS
New-NetQosTrafficClass "Cluster" -Priority 7 -BandwidthPercentage 1 -Algorithm ETS
  8. Disable Flow Control (optional, PFC overrides this anyway):
    Set-NetAdapterAdvancedProperty -Name "StorageA" -RegistryKeyword "*FlowControl" -RegistryValue 0
Set-NetAdapterAdvancedProperty -Name "StorageB" -RegistryKeyword "*FlowControl" -RegistryValue 0
  9. Enable QoS on adapters:
    Get-NetAdapterQos -Name "StorageA","StorageB" | Enable-NetAdapterQos
  10. Enable RDMA on adapters:
    Get-NetAdapterRDMA -Name "StorageA","StorageB" | Enable-NetAdapterRDMA
  11. Now we need to check if the settings are enabled:
    Get-NetAdapterRdma StorageA,StorageB
  12. You should get back:Name           InterfaceDescription                     Enabled     PFC       ETS
    —-           ——————–                     ——-    —       —
    StorageA       HPE Ethernet 10/25Gb 2-port 640SFP28 … True       True       True
    StorageB       HPE Ethernet 10/25Gb 2-port 640SFP28 … True       True       True
  13. Test the RDMA with the Script Test-RDMA.ps1:
    https://github.com/Microsoft/SDN/blob/master/Diagnostics/Test-Rdma.ps1  

     

    .\Test-RDMA.ps1 -IfIndex 1 -IsRoCE $true -RemoteIpAddress "10.10.10.11"
.\Test-RDMA.ps1 -IfIndex 2 -IsRoCE $true -RemoteIpAddress "10.10.11.11"