Generate a blueprint #
Step 1 of the two-step workflow. You point the generator at a URL; it fetches the page and auto-detects the item list, pagination, and field selectors (link, title, image, price, rating, address, description), printing a blueprint as JSON.
php artisan datahelm:scrap:generate <url> [--get-detail=true] [--max-pages=0] [--blueprint]
The three output modes
datahelm:scrap:generate does one of three things with the blueprint it builds:
- default (no flag) — scaffolds a
Robot{Name}PHP command (see Robots). --blueprint— saves a reusable JSON file under the host name, fordatahelm:scrap:run <host>.--json— prints the raw blueprint JSON to stdout (inspect it, pipe it, save it yourself).
Field detectors are configurable in config/crawler.php (detectors) — add your own to teach the generator domain-specific fields.
TIP
In the Docker stack, prefix with docker compose run --rm: docker compose run --rm artisan datahelm:scrap:generate <url> …
Worked example #
php artisan datahelm:scrap:generate \
https://www.exampleauctions.com/real-estate/apartments \
--get-detail=true --get-primary-image=true --hash-names=true --robot-name=ExampleAuctions
Prefer a hands-on walkthrough with a live site? Follow the books.toscrape.com tutorial.
The blueprint is plain data — open the JSON and fix any selector the heuristics got wrong before running it.
Options #
| Option | Effect |
|---|---|
--get-detail=true | Also visit a sample item to detect detail-page fields (incl. image gallery) |
--max-pages= | Page cap stored in the blueprint (default 0 = all pages) |
--get-all-images=true | Put every image URL into all_images in the JSON (default false) |
--get-primary-image=true | Put only the primary (most relevant) image URL per item into primary_image (default false) |
--get-gallery-images=true | Detect the detail-page gallery and put its image URLs into gallery_images; implies --get-detail (default false) |
--hash-names=true | Rename stored images to a unique content hash when you download them (default false) |
--http-delay= | Milliseconds to wait between page requests (default 0) |
--preset= | Detection heuristics profile: generic (default), ecommerce, auctions, properties. See Presets |
--single-page | Treat the URL as one record instead of a list — skips list detection. See Single-page mode |
--main-content | With --single-page, scope detection to the page's primary content region (skips nav/footer/sidebar) |
--browser-wait-for= | CSS selector (or networkidle / domcontentloaded / load) the headless browser waits for before capturing; implies the browser transport. See Transports |
--user-agent= | Custom User-Agent baked into http_config |
--stream | Set output_config.stream=true — write items to disk as they are scraped |
--http-timeout= | Request timeout in seconds (default 60) |
--http-retries= | Retry count on transient failures (default 3) |
--output-format= | Output format: json, jsonl, csv, markdown (default json). See Markdown output |
--dedup | Enable deduplication (drop items with a repeated key-field value) |
--dedup-key= | Field used as the dedup uniqueness key (default link) |
--page-delay= | Milliseconds to wait after each pagination page (default 0) |
--item-delay= | Milliseconds to wait after each item (default 0) |
--max-items= | Blueprint-level item cap (0 = unlimited) |
--api-endpoint= | Force API mode: the JSON endpoint a JS site calls (skips HTML detection) |
--api-method= | HTTP method for the API endpoint: GET (default) or POST |
--api-items-path= | Dot-path to the items array in the JSON (e.g. data.results.content) |
--search-filters= | JSON array of category pages to crawl with the same blueprint. See Multiple categories |
--transport= | HTTP transport, also baked into the blueprint: auto, guzzle, browser, flaresolverr, scraping_api. See Transports |
--render-js | Shortcut to set render_js=true (use a headless browser; needs the browser/flaresolverr transport) |
--header="K: V" | Extra request header (repeatable), baked into http_config.headers. Replay captured browser headers |
--cookie="a=1; b=2" | Cookies (e.g. captured _px3/_abck/session) baked into http_config.cookies — the free way past hard WAFs for a one-off run |
--json | Print the raw blueprint JSON to stdout instead of saving it |
--blueprint | Save a reusable blueprint JSON under the host name (for datahelm:scrap:run <host>) instead of scaffolding a robot |
--robot-name= | Class base name for the scaffolded robot (default: derived from the host). Scaffolding a robot is the default action |
--force | Overwrite an existing robot command file |
--image-disk= | Filesystem disk for downloaded images: storage, public, s3, … (default config('crawler.images.disk')) |
--no-dedup | Disable the automatic link-based deduplication |
--fields= | Comma-separated whitelist of field names to keep (e.g. title,link,price,image). Empty = keep all |
--no-labeled-fields | Skip the labeled-fields heuristic (avoids nav labels / breadcrumbs becoming fields) |
--resumable | Mark the blueprint resumable (persists dedup state; use --resume on the robot to skip already-scraped items). See Resumable crawls |
What detection produces #
The generator suggests fields generously — as many as it can find in both the list and the detail page — so you keep what you want and delete the rest. A field that appears in both sections uses the same name in each: keep it under fields or detail_fields, whichever you prefer, and remove the other.
The key blueprint keys it fills in:
| Key | Meaning |
|---|---|
item_selector | CSS selector matching each row in the list |
scrape_detail | false = scrape the list only; true = also visit each item's detail page |
fields[] | Selectors read from each list row |
detail_link_field | Which list field holds the detail-page URL (when scrape_detail is true) |
detail_fields[] | Selectors read from the detail page |
pagination | strategy (none / link_list / next_link / infinite_scroll) + css |
See the full Blueprint format reference for every key.
Label-based fields #
Auction pages express facts as label: value pairs whose value elements share CSS classes (so CSS alone can't tell them apart). For those, a field carries a label instead of relying on css, and the value is looked up by that label at run time. The field name is derived from the label:
1ª Praça: 22/06/2026 às 10:01→"1_praca": "22/06/2026 às 10:01"Comitente→comitenteValor de Avaliação→valor_de_avaliacao
XPath selectors #
Any field in fields[] or detail_fields[] can use XPath instead of CSS by adding "type": "xpath":
{
"name": "price",
"css": "//span[contains(@class,'price')][1]",
"type": "xpath",
"attribute": null
}
"type" defaults to "css", so existing blueprints are unaffected. XPath is more expressive than CSS for attribute-based conditions, text matching, and ancestor/sibling traversal.
Validate a blueprint #
After hand-editing, check it parses and is structurally sound:
php artisan datahelm:scrap:validate path/to/blueprint.json
Next: Run a scrape →