Responsible crawling starts before the first request

Write down the job before collecting URLs

Imagine a small documentation team checking its public help center for broken links. Its useful result is a bounded report about those pages. Downloading the company blog, customer portal, and every filtered search result would add traffic without serving that goal. A responsible crawl begins with an operator who can explain the purpose, identify the intended site, and name a stopping condition.

For WebKrab, the proposed starting point is a visible run summary: approved origins, allowed paths, a request ceiling, a time budget, and the information that may be retained. These are design intentions rather than shipped controls. Making the choices visible would help an operator catch a mistaken target before the first content request, when the mistake is easiest to prevent.

Give robots.txt a precise, limited role

The Robots Exclusion Protocol is standardized in RFC 9309. Its rules express crawling preferences; they do not grant access authorization. A successful fetch requires the crawler to obey the parseable rules. Network failures and server errors require assuming complete disallow, with separate provisions for cached policies and prolonged failure.

That distinction should stay visible in the product. An allowed path would mean that the evaluated robots rules did not exclude it. The operator would still need an appropriate purpose and authority for the work. A public URL can lead to material that should never enter a particular dataset, and the crawler should not turn a narrow technical decision into a broad claim of permission.

Make failure states understandable

RFC 9309 treats HTTP 400–499 responses for robots.txt as unavailable and permits crawling in that case. It treats HTTP 500–599 and network errors as unreachable. Cached rules generally should not be used beyond 24 hours unless the file is unreachable. Those distinctions matter because a missing file and an overloaded server describe different situations.

WebKrab's proposed policy is deliberately conservative around uncertainty. A 429 rate-limit response should pause requests even though it belongs to the broader 4xx class. An unexpected authentication challenge should prompt inspection. A timeout should produce a deferred origin, not an empty ruleset. These would be explicit product choices, with a reason shown to the operator, rather than undocumented exceptions hidden inside a retry loop.

A useful diagnostic might say that content requests were deferred because the robots endpoint failed, when another attempt becomes eligible, and whether a cached policy exists. It should avoid implying that deferred pages were fetched or inspected.

Identity and pacing need their own decisions

The planned crawler identity should be stable, truthful, and connected to a page that explains its purpose. Changing identities to obtain a different response would undermine the point of publishing one. A site operator should be able to understand the traffic without reverse-engineering a changing collection of browser-like headers.

The Google robots documentation provides a useful interoperability reminder: Google does not support crawl-delay. That field is outside RFC 9309's standardized rules. Any future WebKrab support would need a documented interpretation rather than a claim that every crawler shares the same behavior.

Meanwhile, independent pacing controls should exist even when no delay instruction is present. A short queue is no excuse to burst through every URL at once. We would rather expose a modest per-origin pace and let the operator understand the trade-off than present throughput as the only measure of quality.

A review checklist for the proposed first run

• State the purpose in one sentence and exclude work that does not support it.
• Confirm the exact origins and paths; treat subdomains as deliberate additions.
• Record which robots response and policy informed each scheduling decision.
• Set request, elapsed-time, response-size, and concurrency limits.
• Explain how rate limits, authentication challenges, and unreachable policy files stop or defer work.
• Choose an operator contact and a clear way to end the run.

These checks would make a small first crawl more informative than an ambitious unbounded one. The result we want is a report someone can trust, including a clear explanation of what was intentionally left untouched. That is a useful foundation to establish before writing the scheduler.