Phase 4: Advanced Compression

Goal

Complete the compression story — add Zstandard, pre-compressed asset serving, per-route compression levels, and content-type-aware negotiation.

Current State

• Gzip compression: bext-core/src/compress/gzip.rs
• Brotli compression: bext-core/src/compress/brotli.rs
• Accept-Encoding negotiation in handler
• Streaming compression for SSR responses
• No Zstandard, no pre-compressed asset detection, no per-route tuning

What We're Adding

1. Zstandard (zstd) Compression

Zstd offers better compression ratio than gzip at similar CPU cost, and much faster decompression. Caddy already supports it; Chrome 123+ and Firefox 126+ support Accept-Encoding: zstd.

Compression ratio comparison (typical HTML):
  gzip level 6:  3.2:1  @ 150 MB/s compress, 400 MB/s decompress
  brotli level 4: 3.8:1  @ 100 MB/s compress, 500 MB/s decompress
  zstd level 3:   3.5:1  @ 400 MB/s compress, 1500 MB/s decompress

Zstd's fast compression makes it ideal for dynamic SSR content where Brotli's higher ratio doesn't justify its CPU cost.

2. Pre-Compressed Asset Detection

For static assets, skip runtime compression entirely. During build or on first request, write .gz, .br, .zst sidecar files. On subsequent requests, serve the pre-compressed version directly.

public/
  css/main.css        (120KB)
  css/main.css.br     (28KB, pre-compressed Brotli)
  css/main.css.gz     (35KB, pre-compressed gzip)
  css/main.css.zst    (31KB, pre-compressed zstd)

Handler logic:

1. Check if public/css/main.css.br exists and client accepts br
2. If yes, serve pre-compressed file with Content-Encoding: br — zero CPU
3. If no pre-compressed variant, fall back to runtime compression

3. Per-Route Compression Levels

Different content needs different compression strategies:

[[route_rules]]
pattern = "/api/**"
compression = "fast"          # zstd level 1, low latency

[[route_rules]]
pattern = "/blog/*"
compression = "balanced"      # brotli level 4, good ratio

[[route_rules]]
pattern = "/assets/**"
compression = "off"           # Already compressed (images, video)

4. Content-Type-Aware Compression

Don't compress already-compressed content:

Compress:     text/html, text/css, application/javascript, application/json,
              image/svg+xml, application/xml, text/plain
Skip:         image/jpeg, image/png, image/webp, video/*, audio/*,
              application/zip, application/gzip, font/woff2

5. Negotiation Priority

When client sends Accept-Encoding: zstd, br, gzip:

Implementation

New Module: bext-core/src/compress/zstd.rs

use zstd::stream::encode_all;
use zstd::stream::Encoder;

pub struct ZstdCompressor {
    level: i32,  // 1-22, default 3
}

impl ZstdCompressor {
    pub fn compress(&self, input: &[u8]) -> Result<Vec<u8>> {
        encode_all(input, self.level)
    }

    pub fn stream_compress(&self) -> ZstdStreamEncoder {
        // For streaming SSR responses
        ZstdStreamEncoder::new(self.level)
    }
}

Pre-Compression Module: bext-core/src/compress/precompressed.rs

/// Check for pre-compressed sidecar files
pub fn find_precompressed(
    path: &Path,
    accepted: &AcceptEncoding,
) -> Option<(PathBuf, ContentEncoding)> {
    // Priority: brotli > zstd > gzip (best ratio first for static)
    for (ext, encoding) in &[
        (".br", ContentEncoding::Brotli),
        (".zst", ContentEncoding::Zstd),
        (".gz", ContentEncoding::Gzip),
    ] {
        if accepted.supports(encoding) {
            let compressed_path = path.with_extension(
                format!("{}{}", path.extension().unwrap_or_default().to_str().unwrap(), ext)
            );
            if compressed_path.exists() {
                return Some((compressed_path, *encoding));
            }
        }
    }
    None
}

CLI: Pre-Compress Static Assets

bext compress ./public              # Pre-compress all static assets
bext compress ./public --format br  # Only Brotli
bext compress ./public --min-size 1024  # Skip files < 1KB

Updated Negotiation in Handler

fn select_encoding(
    accept: &AcceptEncoding,
    content_type: &ContentType,
    route_config: &RouteCompression,
    is_streaming: bool,
) -> Option {
    if route_config == RouteCompression::Off {
        return None;
    }
    if !should_compress(content_type) {
        return None;
    }

    match (is_streaming, route_config) {
        // Streaming: prefer gzip (best flush behavior)
        (true, _) => {
            if accept.supports_gzip() { Some(Gzip) }
            else if accept.supports_zstd() { Some(Zstd) }
            else { None }
        }
        // Fast mode: prefer zstd (fastest compress)
        (false, RouteCompression::Fast) => {
            if accept.supports_zstd() { Some(Zstd) }
            else if accept.supports_gzip() { Some(Gzip) }
            else { None }
        }
        // Balanced: prefer brotli (best ratio)
        (false, RouteCompression::Balanced | RouteCompression::Default) => {
            if accept.supports_brotli() { Some(Brotli) }
            else if accept.supports_zstd() { Some(Zstd) }
            else if accept.supports_gzip() { Some(Gzip) }
            else { None }
        }
    }
}

Config Reference

[server.compression]
enabled = true                    # Global toggle (default: true)
min_size = 256                    # Don't compress below N bytes
level = "balanced"                # Default: balanced | fast | max
precompressed = true              # Serve .br/.gz/.zst sidecar files

# Per-route override (in route_rules)
[[route_rules]]
pattern = "/api/**"
compression = "fast"

[[route_rules]]
pattern = "/assets/**"
compression = "off"

Vary Header

All compressed responses include Vary: Accept-Encoding to prevent CDN/proxy cache poisoning:

Vary: Accept-Encoding
Content-Encoding: zstd

Testing Plan

Dependencies

Done Criteria

• Zstd compression supported (Accept-Encoding: zstd)
• Pre-compressed asset detection (.br, .gz, .zst sidecar files)
• bext compress CLI command for pre-compressing static assets
• Per-route compression config (fast, balanced, off)
• Content-type-aware skip list (don't compress images/video)
• Vary: Accept-Encoding on all compressed responses
• Streaming SSR prefers gzip for flush behavior
• All tests passing