Understand the machinery.
Our intention is to release the crawler as open source. The source repository and license will be announced when confirmed, so people can inspect the implementation and understand the terms.
ABOUT WEBKRAB
Curiosity.
With a sense of direction.
An upcoming open-source web crawler for understanding the web, one considered request at a time.
THE IDEA
Follow the links.
Keep the context.
The web is a network of relationships, not just a pile of pages. A good crawler should preserve enough context to help people make sense of it.
WebKrab is being planned around that idea. The aim is a practical crawler that makes scope, request behavior, and results easy to inspect. It is intended for developers, researchers, and site owners who want a careful starting point for their own authorized crawling projects.
The blue crab is our mark: a small explorer with a distinctive shape and a deliberate way of moving. Its colors connect the project to the wide, interconnected ocean of the web.
PROJECT PRINCIPLES
What we want to get right.
Make limits a feature.
Scope and resource budgets should be explicit. Requests should be paced. The tool should make it easy to stop, review the evidence, and adjust deliberately.
Show the whole outcome.
A skipped URL is different from a failed request. A planned feature is different from a working one. Those distinctions should remain visible in the product and in the way we describe it.
WHERE THINGS STAND
The project,
today.
Code development begins soon
This website introduces the project and shares practical crawler design notes. There is no public crawler release, source repository, or download available here yet.
A tentative release target
We are aiming for 10 November 2026 at 00:00 UTC. Timing, license selection, and release scope are not final. The countdown is a planning marker; reaching zero does not mean a release has shipped.
Future participation
A contribution process and project discussion channel will be shared once the repository and maintainership details are ready. For now, explore the planned features and the field notes to understand the direction.