Web data APIs for AI agents
Agents need the live web as clean text, not raw HTML. Firecrawl and Context.dev scrape and crawl any URL into Markdown. Tavily and Exa are search APIs built for LLMs. Apify runs ready-made scrapers for specific sites. Pomerado turns a website, logins included, into a hosted API. 6 tools, in the order they appear in the directory. This is not a ranking.
Grouped by job: MCP Servers & Tools · AI Agent Infrastructure
Pomerado
Turns any website, from consumer sites to legacy portals, into a hosted API or MCP server for your agents, faster and more reliable than browser agents; open source and backed by Y Combinator.
Context.dev
Web data API for AI agents: scrape any URL into LLM-ready Markdown, search the live web, crawl and map sites, monitor pages for changes and pull brand data, with SDKs, a CLI and an MCP server; free credits to start.
Firecrawl
Web data API for AI agents: scrape any URL into clean Markdown or JSON, crawl and map whole sites, search the web, run browser sessions and monitor pages, with an MCP server. Open source under AGPL-3.0 with a self-host guide, or use the cloud with 1,000 free credits a month.
Tavily
Search API built for AI agents: search, extract, map, crawl and research endpoints that return clean, LLM-ready results, with SDKs, an MCP server and a CLI. 1,000 free credits a month, no card.
Exa
Web search API for AI: real-time search with page contents, deep multi-step search, answers with citations, monitors and an async research agent, plus a hosted MCP server. Pay as you go, with $10 in free credits every month.
Apify
Web scraping and automation platform: thousands of ready-made scrapers (Actors) in the Apify Store plus your own, with proxies, storage and scheduling, and an MCP server so agents can run any Actor. Apify also maintains the open-source Crawlee library. Free plan with $5 of usage a month.
What it returns, open source, what it costs.
Straight from each vendor's own pages. Plans move often, so the check date is below.
| Tool | What you get | Open source | Free option | Price |
|---|---|---|---|---|
| Pomerado | A hosted API or MCP tool for one task on one site, built on request, with sign-in and 2FA handled. | Core harness is MIT on GitHub. The service is hosted. | Free: 100 tool calls a day, 10 new integrations a month. | Premium $50/mo: 1,000 tool calls a day, 100 new integrations a month. Business on request. |
| Context.dev | Scrape, crawl, map, web and news search, page monitors, brand data and file parsing, one credit balance. | No. | Free: 1,000 credits a month, no card. A standard scrape is 1 credit. | Developer $20/mo for 10,000 credits. Pro from $90/mo for 100,000. Growth, Scale and Enterprise above that. |
| Firecrawl | Scrape, crawl, map, search, browser sessions and monitors. Markdown or JSON out. | Yes, AGPL-3.0, with a self-host guide. | Free: 1,000 credits a month, no card. A scrape is 1 credit per page. | Hobby $19/mo for 5,000 credits. Standard $99/mo for 100,000. Growth $399/mo. Scale $749/mo. |
| Tavily | Search, extract, map, crawl and research endpoints. Results ready for an LLM. | No. | Researcher: 1,000 credits a month, no card. A basic search is 1 credit. | Project $30/mo for 4,000 credits, up to Growth $500/mo for 100,000. Pay as you go $0.008 a credit. |
| Exa | Web search with page contents, deep search, cited answers, monitors and a research agent. | No. | $10 in credits that resets monthly, no card. | Pay as you go, no subscription. Search from $4 per 1,000 requests. Contents $1 per 1,000 pages. |
| Apify | Ready-made scrapers (Actors) from the Apify Store or your own, with proxies and storage. | Its Crawlee library is Apache-2.0. The Apify platform is hosted. | Free: $5 of usage a month. | Starter $19/mo, Scale $199/mo, Business $999/mo, each as prepaid usage, then pay as you go. |
Checked 2026-10-30. Sources: Firecrawl pricing, Firecrawl on GitHub, Tavily credits and pricing, Exa pricing, Apify pricing, Crawlee on GitHub, Context.dev pricing, Pomerado, Pomerado on GitHub.
Pick it if.
- FirecrawlYou want URLs in, Markdown out, and the option to self-host the whole thing.
- Context.devYou want scraping, search and brand data on one credit balance, with JS rendering and proxies included.
- TavilyYour agent mostly needs to search and read the results, priced per credit.
- ExaYou want real-time search with full page contents, billed per request with no plan to pick.
- ApifySomeone has already built a scraper for the site you need, and you'd rather run it than write one.
- PomeradoThe data sits behind a login or a form, and a browser agent is too slow or too flaky.
Each listing.
The directory page, alternatives in the same category, and a head-to-head where we have one.
- PomeradoAlternatives
- Context.devAlternatives
- FirecrawlAlternatives
- TavilyAlternatives
- ExaAlternatives
- ApifyAlternatives
Category pages list every tool in that category, including ones left off this page. Built one of these? Get it featured in its category.
The 2 biggest AI stories, every morning.
One short email: the two stories that matter, and why.
You're in.
The Daily AI Brief will land in your inbox soon. If you don't see it, check Promotions or Spam and move it to your main inbox.