main
Raw Download raw file

gowarcprox

gowarcprox is a rewrite of internetarchive/warcprox in golang heavily relying on the features provided by internetarchive/gowarc. The rewritten software should emulate all of the observable features and interfaces.

Exceptions:

  • support only sqlite (omit all support for rethinkdb)
  • python plugins (no plugin support)

git

  • The source repos, venv and scratch directories (src, scratch, venv) should not be commited to this repo
    • They are added to the .gitignore for this project
  • Planned implementations should utilize git commits and branches as appropriate for keeping track of work accomplished, ready for testing, review, and merging.
  • An remote origin has been setup, however remote fetch and push will require a physical key touch (yubikey) and shouldn’t be attempted by agents.

source repos

internetarchive/warcprox

A git checkout of https://github.com/internetarchive/warcprox is provided as a source of inspection and functional information for the original project.

internetarchive/gowarc

A git checkout of https://github.com/internetarchive/gowarc is provided as a source of inspection and documenation information for the gowarc library.

uber-go/guide

A git checkout of https://github.com/uber-go/guide.git is provided as a source of inspection and documenation information for golang style guide. All reccomendations and guidelines from this repo should be followed.

venv

A python virtual environment is provided at ./venv and has the warxprox installed.

  • Use with ./venv/bin/warcprox for testing functionality of the original repo.

scratch

Use the ./scratch folder for testing. For example:

./venv/bin/warcprox --dir ./scratch/test_001 \
    --certs-dir ./scratch/test_001 \
    --stats-db-file  ./scratch/test_001/warcprox.sqlite \
    --dedup-db-file  ./scratch/test_001/warcprox.sqlite \
    --cacert ./scratch/test_001/ca.pem

Nothing from this directory should be stored or retained except while testing.

golang: prefered libraries