Sitemaps and canonicals are evidence

Accept a map without assuming it is complete

Imagine auditing a publication that has a sitemap, a section index, and years of internal links. Each source may reveal a different set of addresses. The proposed WebKrab inventory would retain those discovery channels so an operator could distinguish sitemap-listed pages from pages found only through links.

That distinction supports useful questions. Which listed URLs redirect? Which discovered pages never appear in the supplied sitemap? Which pages were eligible but remained pending when the request budget ended? A single flattened list would make those questions harder to answer.

Discovery should still pass through the same admission checks. A sitemap can supply a candidate outside the approved origin or path. That candidate should be recorded as out of scope rather than silently expanding the job. A source of URLs is not an instruction to fetch every address it contains.

Parse the protocol and limit the work

The Sitemaps protocol defines URL entries, sitemap indexes, and optional metadata. A sitemap is limited to 50,000 URLs and 50 MB uncompressed. Page lastmod describes the page's modification date; an index entry's lastmod describes the sitemap file. Neither should be confused with the moment an inventory was downloaded.

Our proposed parser should report structural errors with enough location information to fix them. It should also bound decompression, entry counts, and the total number of sitemap documents considered. Protocol limits are useful interoperability boundaries; a particular audit may reasonably choose smaller operational budgets and report that it stopped early.

We would avoid fetching arbitrary XML external resources while parsing a document. We would also preserve the sitemap source and observation time for every accepted entry. These defensive design choices keep the discovery stage understandable and help ensure that a supposedly simple index cannot quietly create unrelated network activity.

Separate declarations from observed behavior

Google's canonicalization guidance describes redirects and rel=canonical annotations as strong signals, with sitemap inclusion a weaker signal. These are statements about Google's selection process, not a universal instruction that another crawler must merge records immediately.

For WebKrab, a useful record would distinguish the requested URL, the redirect destination, a declared canonical, and any content comparison. Each answers a different question. If a page declares a canonical elsewhere, that is something the publisher asserted. If the response redirected, that is behavior observed during the run.

Suppose /handbook-old declares /handbook as canonical while its content still describes a retired feature. Collapsing both records at discovery time would hide precisely the inconsistency an audit might need to find. Preserving the relationship lets the report show the declaration and the observed difference side by side.

A canonical target outside the approved origin would remain a recorded relationship unless the operator explicitly expanded the crawl. It should not become an automatic bridge to another site.

Define what duplicate means for the task

There are several useful comparison layers. Byte-identical response bodies are a narrower claim than matching extracted text. Matching main text is different again from pages that merely share a layout. A report should name the method it used and keep the underlying observations separate.

We are considering exact content hashes as one possible diagnostic, with the byte representation and preprocessing steps documented. If text extraction is added later, its version and options would need to travel with the result. Otherwise a parser improvement could appear as a content change even when the source page stayed the same.

Approximate matches deserve especially careful language. Similar headings, repeated navigation, or a shared error template can make unrelated URLs look close. We would prefer reviewable candidate groups over a confident duplicate label that erases useful differences. Authentication pages and soft error pages would also need their own diagnostic context.

Review conflicts before consolidating

• Retain whether each URL came from a sitemap, a link, or an explicit seed.
• Keep redirect observations separate from canonical declarations.
• Flag conflicting targets, loops, missing targets, and out-of-scope relationships.
• Label the exact comparison method behind a duplicate suggestion.
• Report unvisited targets as unknown rather than assuming they are healthy.

This approach should produce an inventory that remains useful even when a site is inconsistent. The planned crawler's job is to make those inconsistencies inspectable. Any later consolidation should be a documented decision based on the task, with enough original evidence left to understand how the result was reached.