wire / wire-format / roce
RoCE v2 encapsulates InfiniBand transport (BTH) over UDP/IP/Ethernet. The packet: Ethernet header, IP header, UDP header (port 4791), then BTH, then payload. RoCE v2 uses standard IP routing, unlike RoCE v1 which uses Ethernet (not routable).
aka: RoCE (RDMA over Converged Ethernet) v2
spec: RoCE (InfiniBand Trade Association)
RoCE v2 encapsulates InfiniBand transport (BTH) over UDP/IP/Ethernet. The packet: Ethernet header, IP header, UDP header (port 4791), then BTH, then payload. RoCE v2 uses standard IP routing, unlike RoCE v1 which uses Ethernet (not routable).
| field | size | meaning |
|---|---|---|
| Ethernet header | 14 bytes | Dst MAC, Src MAC, EtherType 0x0800 (IPv4) or 0x86DD (IPv6). |
| IP header | 20 bytes (IPv4) or 40 bytes (IPv6) | Standard IP header. |
| UDP header | 8 bytes | Src port (ephemeral), Dst port 4791, length, checksum. |
| BTH | 12 bytes | InfiniBand Base Transport Header. |
| Payload | variable | RDMA operation payload. |
| ICRC | 4 bytes | Invariant CRC (over BTH + payload). |
| FCS | 4 bytes | Frame Check Sequence (Ethernet CRC). |
example:
ff ff ff ff ff ff 00 00 00 00 00 00 08 00
RoCE v2: Ethernet header with EtherType 0x0800 (IPv4)
agent: curl -H 'accept: application/json' wire.phall.io/wire-format/roce
or /wire-format/roce.json