Make the export explain the crawl

Begin with the reader's next question

Imagine opening a crawl report three months after it was created. A page is marked missing. Was there a 404 response, a timeout, a robots exclusion, or simply no request before the run ended? Those outcomes require different follow-up work. An export that compresses all of them into one empty field leaves the next reader to guess.

The proposed WebKrab record would use explicit outcomes and attach a reason to incomplete work. A discovered address would remain distinguishable from an attempted request. A successful response would remain distinguishable from successful parsing. A partial body would never quietly stand in for a complete page.

This is as much an editorial decision as a schema decision. Labels should say what was observed in plain language. An operator should not need access to the scheduler's implementation to understand the limits of a report.

Package the context that changes interpretation

A useful run manifest would record the run identifier, start and finish times, configuration version, approved scope, relevant budgets, parser version, and reason for stopping. Individual observations would connect to that manifest and carry their own request and completion times.

We would also want the relationship between a result and its evidence to survive export. If a record was validated using an earlier representation, the earlier observation should remain identifiable. If a duplicate candidate was created by text extraction, the extraction method should be named. If a URL was excluded by a parameter rule, the rule should be visible.

The HTTP caching specification offers a relevant reminder: stored-response selection can depend on request-header values identified by Vary. That is one reason a URL alone is insufficient context for interpreting every observation. The export design should preserve necessary request context while excluding credentials and unrelated identifiers.

Reproducibility has two useful meanings

Reprocessing the same permitted inputs with the same configuration should ideally produce the same analysis. Repeating network requests later cannot promise an identical website. Separating those goals would keep WebKrab's claims realistic from the start.

For offline analysis, deterministic ordering and stable identifiers would make comparisons easier. A future exporter could sort records by documented keys, declare its schema version, and produce a checksum for the output. Volatile run metadata should be clearly separated from content comparisons so a changed timestamp does not overwhelm a meaningful diff.

For a later live run, the useful comparison is a documented change report. It should identify pages newly discovered, responses changed, outcomes unresolved, and configuration differences that affect coverage. If the second run excludes an entire path, a smaller result set should not be described as proof that the website lost those pages.

Collect less and redact before sharing

For the initial design, we favor purpose-specific exports over a default archive of everything received. A broken-link audit may need addresses, status outcomes, selected relationships, and timing. It may have no reason to retain page bodies, cookies, or the full set of response headers.

URLs themselves deserve review. Query strings can contain email addresses, session identifiers, private search terms, or tokens. A planned export pipeline should apply redaction before values reach human-readable logs and shared files. Any internal reference needed to join records should be designed so it does not require publishing the original sensitive value.

Redaction also needs honest boundaries. A detector cannot guarantee that every sensitive value has been recognized. The operator should review the intended audience and the actual export fields before distribution. Hashing a value should not be presented as automatic anonymization, especially when a small set of likely inputs could reveal what it represents.

These are proposed product safeguards, not a claim of legal compliance. Decisions about a dataset's lawful use, retention, or redistribution require their own review for the actual project and jurisdiction.

A practical acceptance test for the first exporter

• Can another reader distinguish excluded, deferred, failed, partial, and complete observations?
• Can they identify the configuration and source behind a surprising record?
• Does a repeated analysis of the same permitted input produce a stable comparison?
• Are credentials, cookies, and unnecessary personal values absent from the export?
• Does the report explain its coverage and stopping condition without implying completeness?

We want the first WebKrab export to pass those questions before it accumulates more formats or dashboard features. A modest, well-explained dataset is easier to inspect, safer to share, and more useful to the next person who needs to make a decision from it.