Crawl ethics4 min read
Responsible crawling starts before the first request
A practical design note on robots.txt, failure states, crawler identity, and the decisions that should precede a crawl.
Field note · 01 October 2026WEBKRAB FIELD NOTES
The crawl
starts here.
Five practical reads on the foundations of a capable web crawler. Open standards, sensible limits, and results worth understanding.
These articles explain design principles informing WebKrab. They describe what a crawler should do, not features already implemented in WebKrab.
Architecture4 min read
Designing a frontier that knows when to stop
Why URL parsing, explicit scope, cautious normalization, and trap detection belong together in a crawler frontier.
Field note · 01 October 2026HTTP field notes4 min read
Fetch less, learn more
Conditional requests, careful retries, and explicit budgets can make HTTP fetching both more useful and less wasteful.
Field note · 01 October 2026Discovery4 min read
Sitemaps and canonicals are evidence
A careful approach to discovering URLs, interpreting canonical declarations, and reporting duplicates without losing the original observations.
Field note · 01 October 2026Data practice4 min read
Make the export explain the crawl
A proposal for reproducible crawl records, useful diagnostics, and responsible boundaries around the data an export contains.
Field note · 01 October 2026