wire / wire-format / zstd-format

Zstandard (zstd) frame format

Zstandard is a fast lossless compression algorithm developed by Facebook. A zstd frame: 4-byte magic (0x28B52FFD), frame header (window size, dictionary ID, frame content size), compressed blocks (literals + sequences), and optional checksum. Blocks are either compressed (literals Huffman + FSE sequences) or raw/RLE.

wire-format kind wire-format status standard verification verified tier B wire-format@1

aka: Zstandard (zstd) frame format

wire format

spec: RFC 8878

Zstandard is a fast lossless compression algorithm developed by Facebook. A zstd frame: 4-byte magic (0x28B52FFD), frame header (window size, dictionary ID, frame content size), compressed blocks (literals + sequences), and optional checksum. Blocks are either compressed (literals Huffman + FSE sequences) or raw/RLE.

fieldsizemeaning
Magic number4 bytes0x28B52FFD (little-endian: FD 2F B5 28).
Frame header descriptor1 byteBit 7-6: Frame_Content_Size flag, bit 5: Single_Segment, bit 4: unused, bit 3-2: reserved, bit 1-0: Dictionary_ID flag.
Window descriptor0 or 1 byteWindow size (if not single segment): exponent (5 bits) + mantissa (3 bits) -> window size = (1 << (exp+10)) + (mantissa << exp).
Dictionary ID0/1/2/4 bytesID of the dictionary used (if any).
Frame content size0/1/2/4/8 bytesSize of the uncompressed data (if present).
Block (repeated)variableBlock header (3 bytes): last_block (1 bit) + block_type (2 bits: 0 compressed, 1 raw, 2 RLE, 3 reserved) + block_size (29 bits). Then block content.
Compressed blockvariableLiterals section (Huffman-coded) + sequences section (FSE-coded: literal lengths, offsets, match lengths).
Content checksum4 bytesOptional XXH64 checksum of the uncompressed data (if checksum flag set).

example:

28 b5 2f fd 20

zstd magic + frame header descriptor (single segment, FCS 1 byte)


provenance

agent: curl -H 'accept: application/json' wire.phall.io/wire-format/zstd-format or /wire-format/zstd-format.json