Troubleshooting #

The answers to most failures already live somewhere in these docs — this page indexes them by the symptom you actually see, so you can search by error message.

Quick index #

Could not detect a repeating item list #

The generator requires a repeating pattern; an article, profile or dashboard page has none. Pass --single-page to treat the whole page as one record, optionally with --main-content to skip nav/footer/sidebar. See Single-page mode.

If the page is a listing, it's probably JavaScript-rendered — see Empty shell below.

API mode returns zero items #

Almost always start_page: the blueprint says 0 but the API is 1-indexed, so the first request returns an empty list and the crawler stops. Diagnose with two curls against the first two page values; whichever has data is your start_page. The other suspect is a wrong items_path. Full walkthrough: Getting zero items? and the quotes tutorial.

API: response is not JSON #

The endpoint answered with HTML (usually a challenge page or 403) instead of JSON — on cookie-protected APIs this means the anti-bot cookies expired (e.g. reese84). Re-capture the request from DevTools and refresh http_config.cookies. See Form-encoded & cookie-protected endpoints.

Blocked by <vendor> #

The response was fingerprinted as a WAF block (the message names the vendor). The auto transport escalates for you — Cloudflare goes to flaresolverr, Akamai/PerimeterX-class vendors need scraping_api (paid) or a replayed browser session for one-off runs. Start at Which rung should you be on?.

Fields are null / the page is an empty shell #

The content is rendered by JavaScript after load. Options, cheapest first:

1. Check the Network tab for a JSON API and use API mode — no browser needed.
2. Use the browser transport, adding --browser-wait-for with a selector that only exists once the data has loaded (a row, not the container).

The generator usually handles this itself: it detects the empty shell, re-renders in a headless browser, and even sniffs the JSON endpoint the page calls — see Part 2 of the quotes tutorial.

The detector picked the wrong list #

A sidebar or nav can be a "cleaner" repeating pattern than the real content, so the heuristics score it higher. Confirm the right item_selector in the selector shell (the match count should equal the items you see), then fix the blueprint — it's just JSON. Worked example: Part 1 of the quotes tutorial.

The infinite-scroll endpoint returns 500 #

Some endpoints reject parameters they don't expect — even empty ones (e.g. an empty order=). Send exactly what the real "Load more" request in the Network tab contains, no more. See Infinite scroll.

primary_image is an icon or tracking pixel #

The image-quality heuristic normally down-scores icons, badges and pixels — but it can only choose among the URLs your selectors produce. Check the image field's attribute (lazy-loaded sites put the real URL in data-src, not src), and in single-page mode rely on --get-primary-image to let the heuristic pick from all page images. See Images and Single-page mode.

Where did my output go? #

By default items are saved to storage/app/scrapes/<name>.json — the path is printed at the end of the run, on stderr (stdout is kept clean for piping). --output=PATH chooses the destination, --output=- prints to stdout. Crawl progress and the stats summary are stderr too. See Where output goes.

Still stuck? Open an issue at github.com/data-helm/crawler with the command, the blueprint, and the stderr output.