main
Raw Download raw file

Feature Parity: Python warcprox vs Go gowarcprox

Last Updated: 2026-01-07 Go Version: 0.1.0 (Post Phase 4) Python Version: 2.x (from internetarchive/warcprox)

Implementation Status Legend

  • ✅ Implemented: Feature complete and tested
  • 🚧 In Progress: Partially implemented
  • 📋 Planned: Scheduled for implementation
  • ❌ Not Planned: Explicitly excluded (plugins, RethinkDB)
  • ⚠️ Differs: Implemented differently in Go

1. Core Proxy Features

Feature Python Go Status Notes
HTTP proxy ✓ ✓ ✅ Phase 1 - Full HTTP/1.1 support
HTTPS MITM proxy ✓ ✓ ✅ Phase 2 - Certificate interception
CONNECT method handling ✓ ✓ ✅ Phase 2 - TLS tunneling
Certificate authority ✓ ✓ ✅ Phase 2 - Auto-generates CA
Dynamic cert generation ✓ ✓ ✅ Phase 2 - On-demand per host
Wildcard certificates ✓ ✓ ✅ Phase 2 - *.example.com support
Certificate caching ✓ ✓ ✅ Phase 2 - In-memory with sync.Map
Persistent cert storage ✓ ✓ ✅ Phase 2 - Saved to –certs-dir
Socket timeout config ✓ ✓ ✅ Phase 1 - –socket-timeout flag
Max threads/concurrency ✓ ✓ ✅ Phase 1 - Goroutines, no hard limit
Request queue ✓ ✓ ✅ Phase 3 - Buffered channels
Connection pooling ✓ ? 📋 Phase 8 - Reuse backend connections
Bad hostname cache ✓ ? 📋 Phase 8 - TTL cache for failed hosts
Thread pool executor ✓ N/A ⚠️ Go uses goroutines, not thread pools

2. WARC Writing Features

Feature Python Go Status Notes
WARC 1.1 format ✓ ✓ ✅ Via gowarc library
Request records ✓ ✓ ✅ Phase 4 - application/http; msgtype=request
Response records ✓ ✓ ✅ Phase 4 - application/http; msgtype=response
Warcinfo records ✓ ✓ ✅ Phase 4 - Via gowarc rotator
Metadata records ✓ ? 📋 Phase 8 - WARCPROX_WRITE_RECORD method
Resource records ✓ ? 📋 Phase 8 - Custom WARC-Type
Revisit records ✓ ? 📋 Phase 5 - Deduplication required
WARC file rotation ✓ ✓ ✅ Phase 4 - Via gowarc rotator
Size-based rollover ✓ ✓ ✅ Phase 4 - –size flag (default 1GB)
Time-based rollover ✓ ? 📋 Phase 8 - –rollover-idle-time
GZIP compression ✓ ✓ ✅ Phase 4 - –gzip flag
ZSTD compression ? ✓ ⚠️ gowarc supports, not tested
Custom WARC prefix ✓ ✓ ✅ Phase 4 - –prefix flag
Custom WARC filename ✓ ? 📋 Phase 8 - Template with variables
Subdir by prefix ✓ ? 📋 Phase 8 - –subdir-prefix
.open suffix ✓ ✓ ✅ Phase 4 - Handled by gowarc
File locking ✓ ? 📋 Phase 8 - When –no-warc-open-suffix
Multi-prefix support ✓ ? 📋 Phase 7 - WARC writer pool
Special prefix “-” ✓ ? 📋 Phase 7 - Disable archiving
Serial number tracking ✓ ✓ ✅ Phase 4 - Via gowarc
Random token ✓ ✓ ✅ Phase 4 - Via gowarc
WARC-Record-ID (UUID) ✓ ✓ ✅ Phase 4 - UUID v4
WARC-Date (RFC3339) ✓ ✓ ✅ Phase 4 - Proper timestamp
WARC-Target-URI ✓ ✓ ✅ Phase 4 - Full URL
WARC-IP-Address ✓ ✓ ✅ Phase 4 - Remote server IP
WARC-Payload-Digest ✓ ✓ ✅ Phase 3 - SHA1/SHA256/BLAKE3
WARC-Block-Digest ✓ ? 📋 Phase 8 - Full HTTP block digest
WARC-Concurrent-To ✓ ✓ ✅ Phase 4 - Links request to response
WARC-Refers-To ✓ ? 📋 Phase 5 - For revisit records
WARC-Truncated ✓ ? 📋 Phase 8 - When size exceeded

3. Digest Calculation

Feature Python Go Status Notes
SHA1 digest ✓ ✓ ✅ Phase 3 - Default algorithm
SHA256 digest ✓ ✓ ✅ Phase 3 - –digest sha256
BLAKE3 digest ? ✓ ⚠️ Go addition via gowarc
MD5 digest ✓ ? 📋 Python supports via hashlib
Other hash algorithms ✓ ? 📋 Python supports all hashlib
Base32 encoding ✓ ✓ ✅ Phase 3 - For SHA digests
Hex encoding ✓ ✓ ✅ Phase 3 - Default for BLAKE3
Payload digest ✓ ✓ ✅ Phase 3 - Response body only
Block digest ✓ ? 📋 Phase 8 - Full HTTP block
Digest format ✓ ✓ ✅ Phase 3 - “algorithm:hash”
Configurable algorithm ✓ ✓ ✅ Phase 3 - –digest flag

4. Deduplication

Feature Python Go Status Notes
SQLite dedup DB ✓ ? 📋 Phase 5 - warcprox.sqlite
Dedup table schema ✓ ? 📋 Phase 5 - key/value pairs
Dedup lookup ✓ ? 📋 Phase 5 - Before writing
Dedup storage ✓ ? 📋 Phase 5 - After writing
Revisit record creation ✓ ? 📋 Phase 5 - identical-payload-digest
Dedup buckets (RW mode) ✓ ? 📋 Phase 7 - Read-write buckets
Dedup buckets (RO mode) ✓ ? 📋 Phase 7 - Read-only buckets
Multiple buckets ✓ ? 📋 Phase 7 - Per-request config
Default bucket ✓ ? 📋 Phase 5 - unspecified
Bucket key format ✓ ? 📋 Phase 5 - “digest|bucket”
Min text size threshold ✓ ? 📋 Phase 8 - –dedup-min-text-size
Min binary size threshold ✓ ? 📋 Phase 8 - –dedup-min-binary-size
Blackout period ✓ ? 📋 Phase 8 - –blackout-period
Bucket requirement mode ✓ ? 📋 Phase 8 - –dedup-only-with-bucket
Disable deduplication ✓ ? 📋 Phase 5 - –dedup-db=/dev/null
WAL mode ✓ ? 📋 Phase 5 - Better concurrency
CDX server dedup ✓ - ❌ External service
RethinkDB dedup ✓ - ❌ Excluded per CLAUDE.md
Trough dedup ✓ - ❌ External system
RethinkDB big table ✓ - ❌ Excluded per CLAUDE.md

5. Statistics Tracking

Feature Python Go Status Notes
SQLite stats DB ✓ ? 📋 Phase 6 - warcprox.sqlite
Stats table schema ✓ ? 📋 Phase 6 - buckets_of_stats
Basic URL stats ✓ ? 📋 Phase 6 - Count, bytes
Stats categories ✓ ? 📋 Phase 6 - total/new/revisit
Wire bytes tracking ✓ ? 📋 Phase 6 - Including HTTP headers
Stats buckets ✓ ? 📋 Phase 7 - Custom bucket names
Default buckets ✓ ? 📋 Phase 6 - all, unspecified
Domain tallying ✓ ? 📋 Phase 7 - Per-domain sub-buckets
Batch updates ✓ ? 📋 Phase 6 - Efficiency
JSON storage ✓ ? 📋 Phase 6 - Stats as JSON blob
Disable statistics ✓ ? 📋 Phase 6 - –stats-db=/dev/null
Running stats ✓ ? 📋 Phase 8 - In-memory snapshots
Stats API query ✓ ? 📋 Phase 8 - Via /status endpoint
RethinkDB stats ✓ - ❌ Excluded per CLAUDE.md

6. Warcprox-Meta Header

Feature Python Go Status Notes
Header detection ✓ ✓ 🚧 Phase 3 - Detected but not parsed
JSON header parsing ✓ ? 📋 Phase 7 - Full implementation
Custom warc-prefix ✓ ? 📋 Phase 7 - Per-request override
Prefix validation ✓ ? 📋 Phase 7 - No slashes allowed
Dedup bucket override ✓ ? 📋 Phase 7 - dedup-buckets field
Stats bucket assignment ✓ ? 📋 Phase 7 - stats.buckets field
Stats domain tallying ✓ ? 📋 Phase 7 - tally-domains list
Hard limits (420) ✓ ? 📋 Phase 8 - limits field
Soft limits (430) ✓ ? 📋 Phase 8 - soft-limits field
URL blocking rules ✓ ? 📋 Phase 8 - blocks array
Compressed blocks ✓ ? 📋 Phase 8 - Zlib + base64
Metadata inclusion ✓ ? 📋 Phase 8 - metadata field
Accept flags ✓ ? 📋 Phase 8 - accept array
Capture metadata ✓ ? 📋 Phase 8 - Timestamp in response
Backward compatibility ✓ ? 📋 Phase 7 - captures-bucket, dedup-bucket
MIME type filters ✓ ? 📋 Phase 8 - mime-type-filters
Dedup-ok flag ✓ - ❌ RethinkDB big table only
Captures table extras ✓ - ❌ RethinkDB big table only

7. Filtering and Limits

Feature Python Go Status Notes
HTTP method filtering ✓ ? 📋 Phase 8 - –method-filter
Max resource size ✓ ✓ 🚧 Phase 4 - Config exists, not enforced
Resource truncation ✓ ? 📋 Phase 8 - When size exceeded
WARC-Truncated header ✓ ? 📋 Phase 8 - “length” or “time”
Time-based truncation ✓ ? 📋 Phase 8 - 3 hour timeout
MIME type filtering ✓ ? 📋 Phase 8 - Via Warcprox-Meta
MIME type REJECT ✓ ? 📋 Phase 8 - Block matching types
MIME type LIMIT ✓ ? 📋 Phase 8 - Only allow matching
Do-not-archive flag ✓ ? 📋 Phase 8 - Skip WARC writing

8. Advanced Features

Feature Python Go Status Notes
WARCPROX_WRITE_RECORD ✓ ? 📋 Phase 8 - Custom record injection
PUTMETA method ✓ - ❌ Deprecated in Python
/status endpoint ✓ ? 📋 Phase 8 - JSON metrics
Status: version/host/port ✓ ? 📋 Phase 8 - Basic info
Status: thread counts ✓ ? 📋 Phase 8 - Active requests
Status: queue status ✓ ? 📋 Phase 8 - Queued URLs
Status: rates ✓ ? 📋 Phase 8 - URLs/sec, bytes/sec
Status: postfetch chain ✓ ? 📋 Phase 8 - Pipeline status
Crawl logging ✓ ? 📋 Phase 8 - Heritrix-style logs
Crawl log format ✓ ? 📋 Phase 8 - Tab-separated
Crawl log per prefix ✓ ? 📋 Phase 8 - Separate files
Artificial status codes ✓ ? 📋 Phase 8 - -2, -6, -8, -404
Playback proxy ✓ - ❌ Separate feature, not planned
Playback index DB ✓ - ❌ For playback proxy
SOCKS proxy support ✓ ? 📋 Phase 8 - –socks-proxy
Tor .onion support ✓ ? 📋 Phase 8 - –onion-tor-socks-proxy
TLS fingerprinting ✓ ? 📋 Phase 8 - –ssl-context chrome/firefox
Unsafe SSL renegotiation ✓ ? 📋 Phase 8 - –unsafe-legacy-renegotiation
Plugin system ✓ - ❌ Explicitly excluded per CLAUDE.md
Listener plugins ✓ - ❌ Use Processor interface instead
Batch processors ✓ ✓ ⚠️ Via Processor interface
Service registry ✓ - ❌ RethinkDB-based
Service heartbeat ✓ - ❌ RethinkDB-based

9. Logging and Monitoring

Feature Python Go Status Notes
Verbose logging ✓ ✓ ✅ Phase 1 - -v flag
Log levels ✓ ✓ ✅ Phase 1 - debug/info/warn/error
Structured logging ✓ ✓ ✅ Phase 1 - log/slog package
Quiet mode ✓ ? 📋 Phase 8 - -q flag
Trace logging ✓ ? 📋 Phase 8 - –trace
Custom log levels ✓ ? 📋 Phase 8 - TRACE, NOTICE
Logging config file ✓ ? 📋 Phase 8 - YAML format
Sentry integration ✓ - ❌ External monitoring
Sentry DSN ✓ - ❌ –sentry-dsn
Sentry traces ✓ - ❌ Performance monitoring
Sentry profiles ✓ - ❌ Profiling
Deploy environment ✓ - ❌ –deploy-environment
Performance profiling ✓ ? 📋 Phase 8 - –profile cProfile
Thread stack traces ✓ ? 📋 Phase 8 - SIGQUIT handler

10. Configuration and CLI

Feature Python Go Status Notes
CLI framework argparse cobra ⚠️ Different libraries
Address binding ✓ ✓ ✅ Phase 1 - -b, –address
Port binding ✓ ✓ ✅ Phase 1 - -p, –port
WARC directory ✓ ✓ ✅ Phase 4 - -d, –directory
WARC prefix ✓ ✓ ✅ Phase 4 - –prefix
WARC size limit ✓ ✓ ✅ Phase 4 - –size
GZIP compression flag ✓ ✓ ✅ Phase 4 - -z, –gzip
Digest algorithm ✓ ✓ ✅ Phase 3 - -g, –digest
CA certificate file ✓ ✓ ✅ Phase 2 - -c, –cacert
Certs directory ✓ ✓ ✅ Phase 2 - –certs-dir
Dedup DB file ✓ ✓ 🚧 Phase 4 - Flag exists, not functional
Stats DB file ✓ ✓ 🚧 Phase 4 - Flag exists, not functional
Socket timeout ✓ ✓ ✅ Phase 1 - –socket-timeout
Max threads ✓ ✓ ✅ Phase 1 - –max-threads
Queue size ✓ ✓ ✅ Phase 3 - –queue-size
Tmp file memory ✓ ✓ ✅ Phase 3 - –tmp-file-max-memory
Max resource size ✓ ✓ 🚧 Phase 4 - Flag exists, not enforced
WARC writer threads ✓ ✓ ✅ Phase 4 - –warc-writer-threads
Version display ✓ ✓ ✅ Phase 1 - –version
Help display ✓ ✓ ✅ Phase 1 - –help

11. Error Handling and Resilience

Feature Python Go Status Notes
Graceful shutdown ✓ ✓ ✅ Phase 1 - SIGINT/SIGTERM
Signal handling ✓ ✓ ✅ Phase 1 - os/signal package
Pipeline drain ✓ ✓ ✅ Phase 3 - Wait for queue
WARC flush on shutdown ✓ ✓ ✅ Phase 4 - Close rotator
Failed URL handling ✓ ? 📋 Phase 8 - Error recording
Connection drop detection ✓ ? 📋 Phase 8 - Keep-alive
Panic recovery ✓ ? 📋 Phase 8 - Goroutine defer
Resource cleanup ✓ ✓ ✅ Phase 1 - Defer, WaitGroups
Error wrapping ✓ ✓ ✅ Phase 1 - fmt.Errorf with %w
Context cancellation ✓ ✓ ✅ Phase 3 - context.Context

Summary Statistics

Total Features Analyzed: ~120

By Status:

  • ✅ Implemented: 47 (39%)
  • 🚧 In Progress: 5 (4%)
  • 📋 Planned: 54 (45%)
  • ❌ Not Planned: 14 (12%)
  • ⚠️ Differs: 4 (<1%)

By Phase:

  • Phase 1-4 (Complete): 47 features ✅
  • Phase 5 (Deduplication): ~12 features 📋
  • Phase 6 (Statistics): ~10 features 📋
  • Phase 7 (Warcprox-Meta): ~15 features 📋
  • Phase 8 (Advanced): ~25 features 📋

Core Strengths:

  • HTTP/HTTPS proxy fully functional
  • WARC writing complete with gowarc integration
  • Digest calculation supports multiple algorithms
  • Pipeline architecture clean and extensible
  • Graceful shutdown properly implemented

Major Gaps:

  1. Deduplication - Critical feature, completely missing (Phase 5)
  2. Statistics - Important for monitoring, completely missing (Phase 6)
  3. Warcprox-Meta - Advanced feature, model exists but parsing missing (Phase 7)
  4. Advanced features - Many quality-of-life features pending (Phase 8)

Implementation Notes

Go-Specific Advantages

  • Goroutines: More efficient than Python thread pools
  • Channels: Clean producer-consumer pattern
  • Type Safety: Compile-time error detection
  • Performance: Generally faster execution
  • gowarc Library: Well-maintained WARC implementation

Python-Specific Features Not Planned

  • RethinkDB Integration: Excluded per project requirements
  • Plugin System: Different architecture approach in Go
  • External Service Integration: CDX server, Trough, etc.
  • Sentry Monitoring: Not core functionality

Testing Strategy

  • Unit tests for all implemented features
  • Functional tests comparing Python and Go output
  • WARC file byte-for-byte comparison (where applicable)
  • Payload digest validation (must match exactly)
  • Coverage target: 70% overall, 75%+ for critical packages

Next Steps

Before Phase 5:

  1. Complete unit test suite for Phases 1-4
  2. Build WARC comparison tools
  3. Write functional parity tests
  4. Achieve 70% test coverage
  5. Validate all tests pass

Phase 5 Priority:

  • SQLite deduplication database
  • Revisit record generation
  • Dedup bucket support (basic)

Phase 6 Priority:

  • SQLite statistics database
  • Basic stats tracking (total/new/revisit)
  • Stats bucket support

Phase 7 Priority:

  • Warcprox-Meta JSON parsing
  • Custom WARC prefix support
  • Bucket configuration (dedup + stats)

Phase 8 Priority:

  • /status endpoint
  • Limits enforcement
  • WARCPROX_WRITE_RECORD
  • Resource size truncation