JavaScript sites & JSON APIs (API mode) #

Many modern sites (e.g. quotes.toscrape.com/scroll) render their listing with JavaScript: the server returns an HTML shell and the items are fetched by the browser from a JSON API in the background. Scraping the server HTML finds nothing, so the generator falls back to API mode — it calls that JSON endpoint directly and reads fields by dot-path instead of CSS selectors. This is faster and far more reliable than rendering the page in a headless browser, and needs no Chromium.

How detection works #

When HTML list detection fails, the generator:

1. Checks for SPA markers (id="root", __NEXT_DATA__, ng-app, mostly-script pages, …).
2. Scans the HTML/inline scripts for candidate data endpoints (/api/…, /public/…search, fetch("…"), …).
3. Fetches each candidate, auto-detects the items array (the largest list of objects in the JSON), and scaffolds a type: json field for every scalar key in the first record.

When candidates are found, the generator asks which endpoint to build from. The prompt defaults to "None of these" — discovered candidates are unverified (the first one is often an action endpoint like add_to_cart), so a default answer never silently commits to one. And if a chosen endpoint yields no recognisable list of records, the command fails instead of scaffolding an empty robot that would run "successfully" and output zero items.

Auto-discovery is best-effort — it works for endpoints reachable with a simple GET. When you already know the endpoint (or it needs a POST body / is signed, so discovery can't probe it), pass it explicitly:

php artisan datahelm:scrap:generate "https://quotes.toscrape.com/scroll" \
  --api-endpoint="https://quotes.toscrape.com/api/quotes" \
  --api-items-path=quotes \
  --robot-name=Quotes

For a POST-only endpoint, add --api-method=POST alongside --api-endpoint.

The generator prints diagnostic notes (SPA detected, endpoint probed, records found, fields scaffolded) so you can see what it decided.

The api blueprint block #

API-mode blueprints carry "mode": "api" and an api block:

{
  "mode": "api",
  "api": {
    "endpoint": "https://quotes.toscrape.com/api/quotes",
    "method": "GET",
    "headers": {},
    "body": {},
    "query": {},
    "items_path": "quotes",
    "total_path": null,
    "page_param": "page",
    "page_size_param": null,
    "page_size": 10,
    "start_page": 1,
    "page_in_body": false,
    "detail": {
      "enabled": false,
      "endpoint": "https://api.example.com/items/{id}",
      "method": "GET",
      "items_path": "data"
    }
  },
  "fields": [
    { "name": "text",   "css": "text",        "type": "json" },
    { "name": "author", "css": "author.name", "type": "json" },
    { "name": "tags",   "css": "tags",        "type": "json", "multiple": true }
  ]
}

JSON fields #

In API mode every field has "type": "json" and its css holds the dot-path into each record:

• "text" → top-level key
• "author.name" → nested key
• "tags.0" → first element of a list
• "multiple": true returns the whole array at the path (e.g. an image gallery)
• regex still post-processes the extracted string

Mixed css/xpath/json types are allowed on the same FieldSelector, but in API mode only json fields resolve (others yield null).

Per-item detail requests #

Set api.detail.enabled = true to fetch a second JSON document per item. The endpoint is a template — {field} placeholders are replaced with the item's already-extracted values (URL-encoded):

"detail": {
  "enabled": true,
  "endpoint": "https://api.example.com/items/{id}",
  "method": "GET",
  "items_path": "data"
}

Then add type: json entries to the blueprint's detail_fields[]; their paths are resolved against the object at detail.items_path.

Form-encoded & cookie-protected endpoints #

Some production "APIs" aren't clean JSON-in/JSON-out. A common pattern is a DataTables endpoint: a POST with an application/x-www-form-urlencoded body (draw, start, length, filter[…], query, page, size) that returns { "draw":…, "recordsTotal":…, "recordsFiltered":…, "data":[…] }. Such endpoints are often also guarded by anti-bot cookies (Imperva/Incapsula and similar). To handle this:

• set api.body_format to "form" (nested objects encode as filter[MISC]=…);
• write booleans as strings ("true"/"false") so they survive form-encoding;
• set items_path to data and total_path to recordsFiltered;
• use page_in_body: true with page_param: page, page_size_param: size (the engine also advances DataTables start/length automatically when those keys exist in the body);
• paste the session/anti-bot cookies into http_config.cookies (reese84, incap_ses_*, visid_incap_*, the member/session ids).

The fastest way to build such a blueprint is to copy the request from DevTools (Network → the search XHR → Copy as cURL) and translate its URL, headers, body and cookies into the blueprint.

Getting zero items? #

The most common API-mode failure is a crawl that "succeeds" with zero items and no error. Nine times out of ten the cause is start_page: the API is 1-indexed but the blueprint says "start_page": 0, so the very first request returns an empty list and the crawler concludes the data ran out. Diagnose it in ten seconds:

curl "https://quotes.toscrape.com/api/quotes?page=0"   # → "quotes": []  ← empty!
curl "https://quotes.toscrape.com/api/quotes?page=1"   # → 10 quotes

Whenever an API-mode crawl returns nothing, curl the endpoint with the first two page values and check which one actually has data — then set start_page accordingly. The other usual suspect is a wrong items_path; curl the endpoint and follow the dot-path by eye.

What carries over #

Everything else works the same in API mode: get_*_images / hash_names (the image / images fields just hold URLs from the JSON), dedup, conditional result_filters, output_config (json/jsonl/csv + streaming), crawl delays, max_items / --limit, and the crawl-stats summary. max_pages caps how many API pages are requested.

Next: Infinite scroll →