Commit 9f54afc

@@ -0,0 +1,362 @@
+# Feature Parity: Python warcprox vs Go gowarcprox
+
+**Last Updated**: 2026-01-07
+**Go Version**: 0.1.0 (Post Phase 4)
+**Python Version**: 2.x (from internetarchive/warcprox)
+
+## Implementation Status Legend
+
+- ✅ **Implemented**: Feature complete and tested
+- 🚧 **In Progress**: Partially implemented
+- 📋 **Planned**: Scheduled for implementation
+- ❌ **Not Planned**: Explicitly excluded (plugins, RethinkDB)
+- ⚠️  **Differs**: Implemented differently in Go
+
+---
+
+## 1. Core Proxy Features
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| HTTP proxy | ✓ | ✓ | ✅ | Phase 1 - Full HTTP/1.1 support |
+| HTTPS MITM proxy | ✓ | ✓ | ✅ | Phase 2 - Certificate interception |
+| CONNECT method handling | ✓ | ✓ | ✅ | Phase 2 - TLS tunneling |
+| Certificate authority | ✓ | ✓ | ✅ | Phase 2 - Auto-generates CA |
+| Dynamic cert generation | ✓ | ✓ | ✅ | Phase 2 - On-demand per host |
+| Wildcard certificates | ✓ | ✓ | ✅ | Phase 2 - *.example.com support |
+| Certificate caching | ✓ | ✓ | ✅ | Phase 2 - In-memory with sync.Map |
+| Persistent cert storage | ✓ | ✓ | ✅ | Phase 2 - Saved to --certs-dir |
+| Socket timeout config | ✓ | ✓ | ✅ | Phase 1 - --socket-timeout flag |
+| Max threads/concurrency | ✓ | ✓ | ✅ | Phase 1 - Goroutines, no hard limit |
+| Request queue | ✓ | ✓ | ✅ | Phase 3 - Buffered channels |
+| Connection pooling | ✓ | ? | 📋 | Phase 8 - Reuse backend connections |
+| Bad hostname cache | ✓ | ? | 📋 | Phase 8 - TTL cache for failed hosts |
+| Thread pool executor | ✓ | N/A | ⚠️  | Go uses goroutines, not thread pools |
+
+---
+
+## 2. WARC Writing Features
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| WARC 1.1 format | ✓ | ✓ | ✅ | Via gowarc library |
+| Request records | ✓ | ✓ | ✅ | Phase 4 - application/http; msgtype=request |
+| Response records | ✓ | ✓ | ✅ | Phase 4 - application/http; msgtype=response |
+| Warcinfo records | ✓ | ✓ | ✅ | Phase 4 - Via gowarc rotator |
+| Metadata records | ✓ | ? | 📋 | Phase 8 - WARCPROX_WRITE_RECORD method |
+| Resource records | ✓ | ? | 📋 | Phase 8 - Custom WARC-Type |
+| Revisit records | ✓ | ? | 📋 | Phase 5 - Deduplication required |
+| WARC file rotation | ✓ | ✓ | ✅ | Phase 4 - Via gowarc rotator |
+| Size-based rollover | ✓ | ✓ | ✅ | Phase 4 - --size flag (default 1GB) |
+| Time-based rollover | ✓ | ? | 📋 | Phase 8 - --rollover-idle-time |
+| GZIP compression | ✓ | ✓ | ✅ | Phase 4 - --gzip flag |
+| ZSTD compression | ? | ✓ | ⚠️  | gowarc supports, not tested |
+| Custom WARC prefix | ✓ | ✓ | ✅ | Phase 4 - --prefix flag |
+| Custom WARC filename | ✓ | ? | 📋 | Phase 8 - Template with variables |
+| Subdir by prefix | ✓ | ? | 📋 | Phase 8 - --subdir-prefix |
+| .open suffix | ✓ | ✓ | ✅ | Phase 4 - Handled by gowarc |
+| File locking | ✓ | ? | 📋 | Phase 8 - When --no-warc-open-suffix |
+| Multi-prefix support | ✓ | ? | 📋 | Phase 7 - WARC writer pool |
+| Special prefix "-" | ✓ | ? | 📋 | Phase 7 - Disable archiving |
+| Serial number tracking | ✓ | ✓ | ✅ | Phase 4 - Via gowarc |
+| Random token | ✓ | ✓ | ✅ | Phase 4 - Via gowarc |
+| WARC-Record-ID (UUID) | ✓ | ✓ | ✅ | Phase 4 - UUID v4 |
+| WARC-Date (RFC3339) | ✓ | ✓ | ✅ | Phase 4 - Proper timestamp |
+| WARC-Target-URI | ✓ | ✓ | ✅ | Phase 4 - Full URL |
+| WARC-IP-Address | ✓ | ✓ | ✅ | Phase 4 - Remote server IP |
+| WARC-Payload-Digest | ✓ | ✓ | ✅ | Phase 3 - SHA1/SHA256/BLAKE3 |
+| WARC-Block-Digest | ✓ | ? | 📋 | Phase 8 - Full HTTP block digest |
+| WARC-Concurrent-To | ✓ | ✓ | ✅ | Phase 4 - Links request to response |
+| WARC-Refers-To | ✓ | ? | 📋 | Phase 5 - For revisit records |
+| WARC-Truncated | ✓ | ? | 📋 | Phase 8 - When size exceeded |
+
+---
+
+## 3. Digest Calculation
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| SHA1 digest | ✓ | ✓ | ✅ | Phase 3 - Default algorithm |
+| SHA256 digest | ✓ | ✓ | ✅ | Phase 3 - --digest sha256 |
+| BLAKE3 digest | ? | ✓ | ⚠️  | Go addition via gowarc |
+| MD5 digest | ✓ | ? | 📋 | Python supports via hashlib |
+| Other hash algorithms | ✓ | ? | 📋 | Python supports all hashlib |
+| Base32 encoding | ✓ | ✓ | ✅ | Phase 3 - For SHA digests |
+| Hex encoding | ✓ | ✓ | ✅ | Phase 3 - Default for BLAKE3 |
+| Payload digest | ✓ | ✓ | ✅ | Phase 3 - Response body only |
+| Block digest | ✓ | ? | 📋 | Phase 8 - Full HTTP block |
+| Digest format | ✓ | ✓ | ✅ | Phase 3 - "algorithm:hash" |
+| Configurable algorithm | ✓ | ✓ | ✅ | Phase 3 - --digest flag |
+
+---
+
+## 4. Deduplication
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| SQLite dedup DB | ✓ | ? | 📋 | Phase 5 - warcprox.sqlite |
+| Dedup table schema | ✓ | ? | 📋 | Phase 5 - key/value pairs |
+| Dedup lookup | ✓ | ? | 📋 | Phase 5 - Before writing |
+| Dedup storage | ✓ | ? | 📋 | Phase 5 - After writing |
+| Revisit record creation | ✓ | ? | 📋 | Phase 5 - identical-payload-digest |
+| Dedup buckets (RW mode) | ✓ | ? | 📋 | Phase 7 - Read-write buckets |
+| Dedup buckets (RO mode) | ✓ | ? | 📋 | Phase 7 - Read-only buckets |
+| Multiple buckets | ✓ | ? | 📋 | Phase 7 - Per-request config |
+| Default bucket | ✓ | ? | 📋 | Phase 5 - __unspecified__ |
+| Bucket key format | ✓ | ? | 📋 | Phase 5 - "digest\|bucket" |
+| Min text size threshold | ✓ | ? | 📋 | Phase 8 - --dedup-min-text-size |
+| Min binary size threshold | ✓ | ? | 📋 | Phase 8 - --dedup-min-binary-size |
+| Blackout period | ✓ | ? | 📋 | Phase 8 - --blackout-period |
+| Bucket requirement mode | ✓ | ? | 📋 | Phase 8 - --dedup-only-with-bucket |
+| Disable deduplication | ✓ | ? | 📋 | Phase 5 - --dedup-db=/dev/null |
+| WAL mode | ✓ | ? | 📋 | Phase 5 - Better concurrency |
+| CDX server dedup | ✓ | - | ❌ | External service |
+| RethinkDB dedup | ✓ | - | ❌ | Excluded per CLAUDE.md |
+| Trough dedup | ✓ | - | ❌ | External system |
+| RethinkDB big table | ✓ | - | ❌ | Excluded per CLAUDE.md |
+
+---
+
+## 5. Statistics Tracking
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| SQLite stats DB | ✓ | ? | 📋 | Phase 6 - warcprox.sqlite |
+| Stats table schema | ✓ | ? | 📋 | Phase 6 - buckets_of_stats |
+| Basic URL stats | ✓ | ? | 📋 | Phase 6 - Count, bytes |
+| Stats categories | ✓ | ? | 📋 | Phase 6 - total/new/revisit |
+| Wire bytes tracking | ✓ | ? | 📋 | Phase 6 - Including HTTP headers |
+| Stats buckets | ✓ | ? | 📋 | Phase 7 - Custom bucket names |
+| Default buckets | ✓ | ? | 📋 | Phase 6 - __all__, __unspecified__ |
+| Domain tallying | ✓ | ? | 📋 | Phase 7 - Per-domain sub-buckets |
+| Batch updates | ✓ | ? | 📋 | Phase 6 - Efficiency |
+| JSON storage | ✓ | ? | 📋 | Phase 6 - Stats as JSON blob |
+| Disable statistics | ✓ | ? | 📋 | Phase 6 - --stats-db=/dev/null |
+| Running stats | ✓ | ? | 📋 | Phase 8 - In-memory snapshots |
+| Stats API query | ✓ | ? | 📋 | Phase 8 - Via /status endpoint |
+| RethinkDB stats | ✓ | - | ❌ | Excluded per CLAUDE.md |
+
+---
+
+## 6. Warcprox-Meta Header
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| Header detection | ✓ | ✓ | 🚧 | Phase 3 - Detected but not parsed |
+| JSON header parsing | ✓ | ? | 📋 | Phase 7 - Full implementation |
+| Custom warc-prefix | ✓ | ? | 📋 | Phase 7 - Per-request override |
+| Prefix validation | ✓ | ? | 📋 | Phase 7 - No slashes allowed |
+| Dedup bucket override | ✓ | ? | 📋 | Phase 7 - dedup-buckets field |
+| Stats bucket assignment | ✓ | ? | 📋 | Phase 7 - stats.buckets field |
+| Stats domain tallying | ✓ | ? | 📋 | Phase 7 - tally-domains list |
+| Hard limits (420) | ✓ | ? | 📋 | Phase 8 - limits field |
+| Soft limits (430) | ✓ | ? | 📋 | Phase 8 - soft-limits field |
+| URL blocking rules | ✓ | ? | 📋 | Phase 8 - blocks array |
+| Compressed blocks | ✓ | ? | 📋 | Phase 8 - Zlib + base64 |
+| Metadata inclusion | ✓ | ? | 📋 | Phase 8 - metadata field |
+| Accept flags | ✓ | ? | 📋 | Phase 8 - accept array |
+| Capture metadata | ✓ | ? | 📋 | Phase 8 - Timestamp in response |
+| Backward compatibility | ✓ | ? | 📋 | Phase 7 - captures-bucket, dedup-bucket |
+| MIME type filters | ✓ | ? | 📋 | Phase 8 - mime-type-filters |
+| Dedup-ok flag | ✓ | - | ❌ | RethinkDB big table only |
+| Captures table extras | ✓ | - | ❌ | RethinkDB big table only |
+
+---
+
+## 7. Filtering and Limits
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| HTTP method filtering | ✓ | ? | 📋 | Phase 8 - --method-filter |
+| Max resource size | ✓ | ✓ | 🚧 | Phase 4 - Config exists, not enforced |
+| Resource truncation | ✓ | ? | 📋 | Phase 8 - When size exceeded |
+| WARC-Truncated header | ✓ | ? | 📋 | Phase 8 - "length" or "time" |
+| Time-based truncation | ✓ | ? | 📋 | Phase 8 - 3 hour timeout |
+| MIME type filtering | ✓ | ? | 📋 | Phase 8 - Via Warcprox-Meta |
+| MIME type REJECT | ✓ | ? | 📋 | Phase 8 - Block matching types |
+| MIME type LIMIT | ✓ | ? | 📋 | Phase 8 - Only allow matching |
+| Do-not-archive flag | ✓ | ? | 📋 | Phase 8 - Skip WARC writing |
+
+---
+
+## 8. Advanced Features
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| WARCPROX_WRITE_RECORD | ✓ | ? | 📋 | Phase 8 - Custom record injection |
+| PUTMETA method | ✓ | - | ❌ | Deprecated in Python |
+| /status endpoint | ✓ | ? | 📋 | Phase 8 - JSON metrics |
+| Status: version/host/port | ✓ | ? | 📋 | Phase 8 - Basic info |
+| Status: thread counts | ✓ | ? | 📋 | Phase 8 - Active requests |
+| Status: queue status | ✓ | ? | 📋 | Phase 8 - Queued URLs |
+| Status: rates | ✓ | ? | 📋 | Phase 8 - URLs/sec, bytes/sec |
+| Status: postfetch chain | ✓ | ? | 📋 | Phase 8 - Pipeline status |
+| Crawl logging | ✓ | ? | 📋 | Phase 8 - Heritrix-style logs |
+| Crawl log format | ✓ | ? | 📋 | Phase 8 - Tab-separated |
+| Crawl log per prefix | ✓ | ? | 📋 | Phase 8 - Separate files |
+| Artificial status codes | ✓ | ? | 📋 | Phase 8 - -2, -6, -8, -404 |
+| Playback proxy | ✓ | - | ❌ | Separate feature, not planned |
+| Playback index DB | ✓ | - | ❌ | For playback proxy |
+| SOCKS proxy support | ✓ | ? | 📋 | Phase 8 - --socks-proxy |
+| Tor .onion support | ✓ | ? | 📋 | Phase 8 - --onion-tor-socks-proxy |
+| TLS fingerprinting | ✓ | ? | 📋 | Phase 8 - --ssl-context chrome/firefox |
+| Unsafe SSL renegotiation | ✓ | ? | 📋 | Phase 8 - --unsafe-legacy-renegotiation |
+| Plugin system | ✓ | - | ❌ | Explicitly excluded per CLAUDE.md |
+| Listener plugins | ✓ | - | ❌ | Use Processor interface instead |
+| Batch processors | ✓ | ✓ | ⚠️  | Via Processor interface |
+| Service registry | ✓ | - | ❌ | RethinkDB-based |
+| Service heartbeat | ✓ | - | ❌ | RethinkDB-based |
+
+---
+
+## 9. Logging and Monitoring
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| Verbose logging | ✓ | ✓ | ✅ | Phase 1 - -v flag |
+| Log levels | ✓ | ✓ | ✅ | Phase 1 - debug/info/warn/error |
+| Structured logging | ✓ | ✓ | ✅ | Phase 1 - log/slog package |
+| Quiet mode | ✓ | ? | 📋 | Phase 8 - -q flag |
+| Trace logging | ✓ | ? | 📋 | Phase 8 - --trace |
+| Custom log levels | ✓ | ? | 📋 | Phase 8 - TRACE, NOTICE |
+| Logging config file | ✓ | ? | 📋 | Phase 8 - YAML format |
+| Sentry integration | ✓ | - | ❌ | External monitoring |
+| Sentry DSN | ✓ | - | ❌ | --sentry-dsn |
+| Sentry traces | ✓ | - | ❌ | Performance monitoring |
+| Sentry profiles | ✓ | - | ❌ | Profiling |
+| Deploy environment | ✓ | - | ❌ | --deploy-environment |
+| Performance profiling | ✓ | ? | 📋 | Phase 8 - --profile cProfile |
+| Thread stack traces | ✓ | ? | 📋 | Phase 8 - SIGQUIT handler |
+
+---
+
+## 10. Configuration and CLI
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| CLI framework | argparse | cobra | ⚠️  | Different libraries |
+| Address binding | ✓ | ✓ | ✅ | Phase 1 - -b, --address |
+| Port binding | ✓ | ✓ | ✅ | Phase 1 - -p, --port |
+| WARC directory | ✓ | ✓ | ✅ | Phase 4 - -d, --directory |
+| WARC prefix | ✓ | ✓ | ✅ | Phase 4 - --prefix |
+| WARC size limit | ✓ | ✓ | ✅ | Phase 4 - --size |
+| GZIP compression flag | ✓ | ✓ | ✅ | Phase 4 - -z, --gzip |
+| Digest algorithm | ✓ | ✓ | ✅ | Phase 3 - -g, --digest |
+| CA certificate file | ✓ | ✓ | ✅ | Phase 2 - -c, --cacert |
+| Certs directory | ✓ | ✓ | ✅ | Phase 2 - --certs-dir |
+| Dedup DB file | ✓ | ✓ | 🚧 | Phase 4 - Flag exists, not functional |
+| Stats DB file | ✓ | ✓ | 🚧 | Phase 4 - Flag exists, not functional |
+| Socket timeout | ✓ | ✓ | ✅ | Phase 1 - --socket-timeout |
+| Max threads | ✓ | ✓ | ✅ | Phase 1 - --max-threads |
+| Queue size | ✓ | ✓ | ✅ | Phase 3 - --queue-size |
+| Tmp file memory | ✓ | ✓ | ✅ | Phase 3 - --tmp-file-max-memory |
+| Max resource size | ✓ | ✓ | 🚧 | Phase 4 - Flag exists, not enforced |
+| WARC writer threads | ✓ | ✓ | ✅ | Phase 4 - --warc-writer-threads |
+| Version display | ✓ | ✓ | ✅ | Phase 1 - --version |
+| Help display | ✓ | ✓ | ✅ | Phase 1 - --help |
+
+---
+
+## 11. Error Handling and Resilience
+
+| Feature | Python | Go | Status | Notes |
+|---------|--------|----|---------| ------|
+| Graceful shutdown | ✓ | ✓ | ✅ | Phase 1 - SIGINT/SIGTERM |
+| Signal handling | ✓ | ✓ | ✅ | Phase 1 - os/signal package |
+| Pipeline drain | ✓ | ✓ | ✅ | Phase 3 - Wait for queue |
+| WARC flush on shutdown | ✓ | ✓ | ✅ | Phase 4 - Close rotator |
+| Failed URL handling | ✓ | ? | 📋 | Phase 8 - Error recording |
+| Connection drop detection | ✓ | ? | 📋 | Phase 8 - Keep-alive |
+| Panic recovery | ✓ | ? | 📋 | Phase 8 - Goroutine defer |
+| Resource cleanup | ✓ | ✓ | ✅ | Phase 1 - Defer, WaitGroups |
+| Error wrapping | ✓ | ✓ | ✅ | Phase 1 - fmt.Errorf with %w |
+| Context cancellation | ✓ | ✓ | ✅ | Phase 3 - context.Context |
+
+---
+
+## Summary Statistics
+
+**Total Features Analyzed**: ~120
+
+**By Status**:
+- ✅ Implemented: 47 (39%)
+- 🚧 In Progress: 5 (4%)
+- 📋 Planned: 54 (45%)
+- ❌ Not Planned: 14 (12%)
+- ⚠️  Differs: 4 (<1%)
+
+**By Phase**:
+- Phase 1-4 (Complete): 47 features ✅
+- Phase 5 (Deduplication): ~12 features 📋
+- Phase 6 (Statistics): ~10 features 📋
+- Phase 7 (Warcprox-Meta): ~15 features 📋
+- Phase 8 (Advanced): ~25 features 📋
+
+**Core Strengths**:
+- HTTP/HTTPS proxy fully functional
+- WARC writing complete with gowarc integration
+- Digest calculation supports multiple algorithms
+- Pipeline architecture clean and extensible
+- Graceful shutdown properly implemented
+
+**Major Gaps**:
+1. Deduplication - Critical feature, completely missing (Phase 5)
+2. Statistics - Important for monitoring, completely missing (Phase 6)
+3. Warcprox-Meta - Advanced feature, model exists but parsing missing (Phase 7)
+4. Advanced features - Many quality-of-life features pending (Phase 8)
+
+---
+
+## Implementation Notes
+
+### Go-Specific Advantages
+- **Goroutines**: More efficient than Python thread pools
+- **Channels**: Clean producer-consumer pattern
+- **Type Safety**: Compile-time error detection
+- **Performance**: Generally faster execution
+- **gowarc Library**: Well-maintained WARC implementation
+
+### Python-Specific Features Not Planned
+- **RethinkDB Integration**: Excluded per project requirements
+- **Plugin System**: Different architecture approach in Go
+- **External Service Integration**: CDX server, Trough, etc.
+- **Sentry Monitoring**: Not core functionality
+
+### Testing Strategy
+- Unit tests for all implemented features
+- Functional tests comparing Python and Go output
+- WARC file byte-for-byte comparison (where applicable)
+- Payload digest validation (must match exactly)
+- Coverage target: 70% overall, 75%+ for critical packages
+
+---
+
+## Next Steps
+
+**Before Phase 5**:
+1. Complete unit test suite for Phases 1-4
+2. Build WARC comparison tools
+3. Write functional parity tests
+4. Achieve 70% test coverage
+5. Validate all tests pass
+
+**Phase 5 Priority**:
+- SQLite deduplication database
+- Revisit record generation
+- Dedup bucket support (basic)
+
+**Phase 6 Priority**:
+- SQLite statistics database
+- Basic stats tracking (total/new/revisit)
+- Stats bucket support
+
+**Phase 7 Priority**:
+- Warcprox-Meta JSON parsing
+- Custom WARC prefix support
+- Bucket configuration (dedup + stats)
+
+**Phase 8 Priority**:
+- /status endpoint
+- Limits enforcement
+- WARCPROX_WRITE_RECORD
+- Resource size truncation