Crawl4AI

Point this Python library at a URL and it drives a real browser through Playwright, renders the page, and hands back Markdown tidied for language models, with optional filters that strip navigation and boilerplate or keep only passages relevant to a query. Beyond single pages it offers breadth-first, depth-first and best-first deep crawls that can pick up again after a crash, URL discovery from sitemaps and Common Crawl, sessions with saved logins, proxy rotation, a stealth mode, and screenshot or PDF capture. Structured output comes either from CSS or XPath schemas with no model involved, or from an LLM of your choice routed through LiteLLM, local Ollama included. Install with pip and run the setup command to fetch a browser; a Docker image adds a token-protected REST server, an MCP endpoint and a monitoring dashboard, and the crwl CLI handles quick jobs. Apache 2.0 governs the code, but the license file adds a clause requiring visible credit to the project in any public use. A paid hosted API exists, but the library is complete without it. Good for RAG builders and agent developers who want web data on their own hardware.