Firecrawl

Firecrawl pulls web content into forms that language models and agents can use directly. Give it a URL and the scrape endpoint returns Markdown, HTML, structured JSON, or a screenshot, rendering JavaScript-heavy pages in a headless browser and optionally clicking or typing first. Crawl walks a whole site, map lists its URLs, batch scrape handles thousands of addresses asynchronously, search returns full page text for web results, and an agent endpoint answers plain-language data requests, optionally into a Pydantic schema. Web-hosted PDFs and DOCX files are parsed too. SDKs cover Python, Node, Go, Rust, Java, Ruby, .NET, and PHP, alongside a CLI and an MCP server for coding agents. Self-hosting uses Docker Compose to run the API, workers, Playwright, Redis, RabbitMQ, and PostgreSQL; AI features need an OpenAI-compatible endpoint or Ollama, and the default API ships without authentication. The hosted firecrawl.dev service adds features missing from the open build. The core is AGPL-3.0 (SDKs are MIT), so a network service running a modified copy must publish its source. Suited to RAG builders and agent developers.