Fetch less, learn more

Start with the information the server supplies

RFC 9110 defines conditional requests. For a previously stored representation, an ETag can be sent in If-None-Match; Last-Modified can support If-Modified-Since. A matching condition on GET or HEAD can produce 304 Not Modified instead of another representation body. If-None-Match takes precedence when both conditions are present.

The proposed WebKrab record would connect a validator to the specific response it describes. An ETag should not become a universal content identifier for an entire site. The aim is a reliable relationship between a previous observation, the next request, and the server's answer.

For example, a team reviewing the same public guide every week might need to know that the server validated its earlier representation. The report should say when the body was originally collected and when it was revalidated. That preserves the difference between a new download and evidence that an earlier observation can still be used.

Keep caching semantics attached to the record

RFC 9111 distinguishes freshness, validation, and storage constraints. An unqualified no-cache response directive requires validation before reuse; it does not simply mean never store. A no-store directive restricts HTTP cache storage. Vary identifies request-header dimensions that matter when matching a stored response to a later request.

For WebKrab, the practical proposal is to use a well-tested cache component and make its decisions inspectable. A run should not reuse a representation just because its URL matches. If a language preference changed between requests, the tool should retain that context rather than quietly present one language as evidence about another.

We also want retention choices to be visible separately from cache behavior. An operator should decide whether a particular audit needs response bodies at all. Many link-health tasks can work with a smaller set of observations. Avoiding unnecessary retention would reduce both storage costs and the amount of material someone must review before sharing a result.

Retries need a reason and an exit

Retry-After can be an HTTP date or a delay in seconds. RFC 9110 describes its use with 503 responses and redirects. RFC 6585 defines 429 Too Many Requests and permits that response to include Retry-After. A missing header does not establish that an immediate retry is welcome.

The proposed scheduler would distinguish a transient failure from an outcome that needs operator attention. A rate limit should defer the affected work. Repeated timeouts should reduce activity and eventually stop attempts within the run. An authentication challenge should not trigger identity changes, alternate credentials, or attempts to work around the restriction.

Consider a response asking for a ten-minute wait when only two minutes remain in the run budget. The sensible result is deferred work with a recorded reason. Cutting the wait down to fit the deadline would erase the server's signal. Sleeping beyond the deadline without telling the operator would erase the run's own boundary.

Any retry proposal should include an attempt cap, a delay policy, and accounting that counts retries as requests. Jitter can spread retry timing, but it should never move an attempt earlier than a valid server-requested wait.

Budget more than requests per second

A single response can be expensive even at a low request rate. We want separate limits for concurrent requests, connection and response time, received bytes, decompressed bytes, redirect hops, and total run duration. These are planned controls that still need implementation and testing.

Budgets should have straightforward outcomes. If a response exceeds its size ceiling, the report should mark it incomplete. If a redirect leads outside the approved scope, the chain should end with that reason. If a connection stalls, its timeout should free capacity without quietly spawning unlimited replacement work.

These details become easier to evaluate with a small test site containing deliberate slow responses, redirects, validators, and oversized documents. Such fixtures would let contributors inspect behavior without directing experimental traffic at someone else's production service.

The first useful fetch report

• How many requests were initial attempts, retries, or conditional validations?
• How many bytes were transferred, and which responses were incomplete?
• Which origins were deferred, and what signal caused the pause?
• Which representation and request context supported each reused result?
• Which budget ended the run or prevented additional work?

Those answers would make optimization concrete. The goal is enough evidence to complete the operator's task with predictable resource use. A lower byte count is valuable when the report remains honest about what the crawler actually learned.