Designing a frontier that knows when to stop

Keep discovery separate from admission

A link found on a page is evidence that another address exists. It is only a candidate for this run. The proposed frontier would preserve that distinction: discovered URLs pass through parsing, scheme checks, scope rules, duplicate checks, and scheduling constraints before they become eligible requests.

Consider a documentation page that links to a payment provider, a customer support portal, and a downloadable guide. All three links may be useful to report. None should silently expand a documentation crawl into a new service. A rejected candidate can still contribute to a link inventory without becoming a network request.

Keeping discovery provenance would also make surprises easier to investigate. A report should be able to identify the page or sitemap that introduced a candidate, the rule that excluded it, and the run configuration in effect. That information is far more actionable than a counter labeled skipped.

Let a URL parser handle URL syntax

The WHATWG URL Standard defines parsing and serialization, including resolving relative references against a base URL. Its URL comparison procedure uses serialized forms and can exclude fragments. Those are useful foundations for a fetch key; they do not establish that different addresses serve identical content.

Our proposed approach is to keep the discovered spelling for diagnosis and use the parser's output for decisions. A relative reference such as ../guide needs the correct base. A hostname should be evaluated as a hostname, not matched by a convenient substring. An address containing the intended domain in its query or path should not qualify as an in-scope origin.

The first implementation should have a reviewable collection of fixtures: relative links, internationalized hostnames, explicit ports, encoded characters, empty queries, and malformed input. Each fixture should state the expected parsed address and admission outcome. Small fixtures make disagreements visible before they become a large crawl.

Normalize cautiously and preserve evidence

We should resist application-level rewriting without evidence about the target site. Removing every query parameter, lowercasing every path, sorting repeated query keys, or merging trailing-slash variants can discard distinctions an application uses. An apparently untidy URL is not sufficient evidence that it is redundant.

For a documentation audit, an operator might explicitly approve ignoring a known campaign parameter. That should become a named, narrow rule, accompanied by examples and a count of affected candidates. The original addresses should remain available in a suitably redacted diagnostic view so the decision can be reviewed.

It would also help to separate three concepts in the proposed data model: the discovered address, the fetch key used to avoid repeat work, and a later content relationship. A URL normalization decision should not masquerade as proof that two pages are duplicates. That proof requires different observations and may remain uncertain.

Treat growth as a signal to inspect

Google's faceted navigation guidance describes how combinations of filters can create enormous or effectively unbounded URL spaces. Calendar navigation, accumulating path segments, and repeated search refinements create similar planning problems for a general crawler.

The WebKrab design should make growth observable. Suppose a run has visited 80 useful guide pages but discovered 12,000 combinations under one filter path. A rising queue alone would look like progress. A breakdown by path family, parameter shape, and discovery source would show where attention is needed.

We are considering several independent brakes: an overall candidate ceiling, a per-pattern ceiling, a maximum traversal depth, and a deadline. These would be operator-selected limits, not claims that a page beyond the limit is invalid. When a brake engages, the report should identify the unfinished region and preserve enough context to choose a smaller, better-targeted second run.

Make the stopping condition part of the result

• Use an explicit origin allowlist and deliberate path boundaries.
• Apply scope checks again after redirects and newly discovered references.
• Record duplicate suppression separately from policy exclusions.
• Show which URL families dominate the pending queue.
• End with a reason: exhausted candidates, operator stop, deadline, or a specific budget.

A finite run cannot promise complete knowledge of a changing website. It can provide a clear account of the area it covered and the limits it encountered. That is the frontier we want to design: a queue whose behavior remains understandable as the site becomes more complicated.