WebKrab.

THE BLUEPRINT

Useful by design.
Responsible by default.

What we want WebKrab to do, why it matters, and what still has to be built.

Planned capabilities · development begins soon

CORE CAPABILITIES

A focused first foundation.

The first implementation should make a small, bounded crawl understandable from start to finish. These are proposed priorities; availability and exact behavior will be confirmed with a release.

PLANNED / 01

Deliberate discovery

Find useful pages without letting every link become a new expedition.

  • Seed URLs and configurable scope
  • Sitemap discovery as a source of candidates
  • URL normalization and duplicate-aware scheduling
PLANNED / 02

A polite request engine

Treat every destination as a shared resource with finite capacity.

  • Robots Exclusion Protocol support
  • Per-origin pacing and concurrent-request limits
  • Bounded retries and Retry-After handling
PLANNED / 03

Clear crawl boundaries

Know what the crawler may visit and how much work it may do.

  • Explicit host and path rules
  • Page, duration, and response-size budgets
  • Redirect checks and crawler-trap defenses
PLANNED / 04

HTTP-aware fetching

Preserve the evidence needed to understand what the server actually returned.

  • Status codes and response headers
  • Timeouts, content-type checks, and redirect history
  • Validator-aware revisits where appropriate
PLANNED / 05

Inspectable extraction

Start with well-defined information and keep provenance attached.

  • Links, page titles, and selected metadata
  • Source URL and discovery context
  • Visible parsing failures instead of silent omissions
PLANNED / 06

Portable results

Make it straightforward to examine, compare, and reuse a bounded crawl.

  • Machine-readable export formats
  • Per-URL outcomes and skip reasons
  • Run configuration and summary diagnostics

A PROPOSED CRAWL LIFECYCLE

Every step should be explainable.

  1. 01

    Define

    Set seeds, scope, permissions, and budgets.

  2. 02

    Check

    Apply robots rules, scope rules, and origin limits.

  3. 03

    Fetch

    Request deliberately and record the response.

  4. 04

    Learn

    Extract, deduplicate, and consider new URLs.

  5. 05

    Explain

    Export results, decisions, and run limits.

Conceptual workflow. No running crawler or product interface is included on this website.

PROJECT DIRECTION

A roadmap,
with room to learn.

The target is useful software with honest boundaries. A date is a planning aid, not a reason to skip the basics.

  1. NOW

    Define the foundation

    Publish the project direction and educational field notes. Code development will begin soon.

  2. NEXT · PLANNED

    Build a bounded prototype

    Establish URL scope, robots handling, request pacing, and a small inspectable result set.

  3. TARGET · 10 NOVEMBER 2026, 00:00 UTC

    Prepare an initial open-source release

    Confirm the repository, license, documentation, and supported features. Release timing and scope may change.

  4. LATER · SUBJECT TO FEEDBACK

    Extend what proves useful

    Improve extraction, revisit strategies, and integrations based on real evidence. JavaScript rendering and large distributed crawls are not promised for the initial release.

Good crawlers start with good questions.

Read the field notes