Bot protection
Many valuable sites sit behind a WAF (web application firewall) — Cloudflare, Akamai, PerimeterX and friends — built specifically to tell bots from humans. This page covers how the crawler detects those blocks, how the auto transport escalates past them, and the free fallback when nothing else works.
The fetching machinery itself (what each transport is, how to configure browserless, FlareSolverr and the paid API) lives in HTTP transports.
The escalation ladder (auto)
guzzle ─► browser ─► flaresolverr ─► scraping_api
fast JS Cloudflare hardest WAFs (paid)
The default auto transport tries the cheapest rung first. On a block, it doesn't retry blindly — it is vendor-aware: Cloudflare → flaresolverr; Akamai / PerimeterX / DataDome / Kasada → straight to scraping_api (it skips the transports it knows can't help). The winning transport is baked into the generated robot, so subsequent runs go straight to it — no re-escalation.
auto only escalates to the paid scraping_api rung when a key is configured; otherwise it stops with an honest message instead of producing a broken robot.
The rungs at a glance
| Transport | What it does | Cost | Use for |
|---|---|---|---|
auto (default) | Tries the cheapest transport, and on a detected block identifies the WAF vendor and escalates to the right stronger transport — then bakes the one that worked into the blueprint | free* | leave it on; most sites just work |
guzzle | Plain HTTP client (no JS, no anti-bot) | free | normal sites, APIs |
browser | Real headless Chrome via the browserless container (runs JS, real browser fingerprint) | free | JS/SPA sites, soft protections |
flaresolverr | Stealth Chromium tuned to solve Cloudflare / DDoS-Guard challenges, via the FlareSolverr container | free | Cloudflare-protected sites |
scraping_api | A managed provider (ScrapingBee/ZenRows/ScraperAPI/…) that brings residential proxies + anti-bot solving | paid (your key) | the hardest WAFs (Akamai, PerimeterX) |
* auto only escalates to scraping_api when a key is configured; otherwise it stops with an honest message instead of producing a broken robot.
The free transports, in the simplest terms
| What it does | Speed | When you need it | |
|---|---|---|---|
guzzle | Plain HTTP request — just downloads the HTML | ⚡ Fastest, lightest | Normal sites where the data is already in the HTML |
browser | Opens the page in a real headless Chrome, runs the JavaScript | 🐢 Slower (boots Chromium) | Sites where content only appears after JS runs (React/Vue/SPA), or soft bot-checks |
flaresolverr | A stealth Chrome that specifically waits out Cloudflare "checking your browser…" pages | 🐢🐢 Slowest | Sites stuck behind a Cloudflare challenge screen |
How to configure each one (browserless, FlareSolverr, the paid API) lives in HTTP transports.
Bot-protection detection
Every transport's response is inspected for WAF fingerprints (in headers and body, including challenge pages served with HTTP 200). When blocked, you get a clear message naming the vendor instead of a raw HTML dump, e.g.:
Blocked by Akamai Bot Manager (HTTP 403) — couldn't read https://…
… Even the headless browser was blocked — this firewall also scores IP
reputation, so a datacenter IP is flagged regardless of the browser. Options:
route through residential proxies; or capture the JSON API from your browser's
Network tab and re-run with --api-endpoint=<url>.
Detected vendors: Akamai, Cloudflare, PerimeterX/HUMAN, DataDome, Imperva/Incapsula, AWS WAF, Sucuri, Kasada.
Free fallback for hard WAFs: replay your own session
When no free transport beats a WAF (Akamai, PerimeterX) and you don't want a paid key, you can reuse the session your own browser already passed — for a one-off run (cookies expire in hours):
- Open the page in your browser; in DevTools copy the relevant cookies (
_px3,_abck, session…) and key headers (User-Agent, Accept-Language). - Generate with
guzzle(exact replay) + the captured values:
php artisan datahelm:scrap:generate "<url>" \
--transport=guzzle \
--cookie="_px3=…; _pxhd=…" \
--header="user-agent: Mozilla/5.0 …" \
--header="accept-language: pt-BR,pt;q=0.9" \
--robot-name=example
The cookies/headers are baked into the blueprint and sent on every request.
The same mechanism handles login-gated sites — log in through your browser, copy the session cookie, and pass it via --cookie. It even lets auto's headless-render step discover the JSON API behind a login-gated SPA, since it authenticates with the same cookie. See Scraping behind a login for the full walkthrough, and the OrangeHRM tutorial for a real login, a real session-expiry failure, and the fix.
Which rung should you be on?
- Data already in the HTML? →
guzzle(or just leaveautoon). - Empty shell, content appears after JS? →
browser— but check the Network tab for a JSON API first; API mode is faster and needs no Chromium. - Cloudflare "checking your browser…" page? →
flaresolverr. - Akamai / PerimeterX at scale? →
scraping_api, or session replay (above) for one-offs.