Scaffold a robot

A robot is a self-contained Artisan command with the blueprint JSON embedded directly in the file — so it needs no external storage. It is also where per-item logic lives: image downloading, image processing, and persistence (Eloquent, queue, webhook).

Scaffolding a robot is the default action of datahelm:scrap:generate — you don't need a --robot flag. Just give it a name:

bash

php artisan datahelm:scrap:generate \
  https://www.exampleauctions.com/real-estate/apartments --get-detail=true --robot-name=ExampleAuctions
# -> creates app/Console/Commands/RobotsCommand/RobotExampleAuctions.php

php artisan datahelm:robot:exampleauctions --limit=20

The robot name defaults to the host (exampleauctions → RobotExampleauctions); pass --robot-name= for exact casing, and --force to overwrite an existing file. Edit the embedded BLUEPRINT JSON in the generated command to refine selectors.

TIP

--blueprint (save a JSON file) and --json (print to stdout) are the two alternatives to scaffolding a robot — pick one when you don't want a PHP command. See the three output modes.

Anatomy of a robot

The generated handle() method loads the embedded blueprint and calls crawlEach(), which runs your closure once per scraped item as it streams in (nothing is buffered):

php

class RobotExampleMarket extends Command
{
    use ScrapesToConsole;

    /** Any Laravel filesystem disk: 'storage' (local), 'public', 's3', 'gcs', … */
    protected string $imageDisk = 'storage';

    /** Subfolder inside the disk where images for this site will be stored. */
    protected string $imageFolder = 'scrapes/images/www.example-market.com';

    private const BLUEPRINT = <<<'JSON'
    { ...the whole blueprint is embedded here... }
    JSON;

    public function handle(CrawlEngine $engine, ImageStore $images): int
    {
        $blueprint      = ScrapeBlueprint::fromJson(self::BLUEPRINT);
        $hashNames      = $blueprint->hashNames;
        $downloadImages = ! $this->option('no-images');

        $this->crawlEach(
            $engine,
            $blueprint,
            function (ScrapedItem $item) use ($images, $hashNames, $downloadImages): void {
                $imagePath = null;

                if ($downloadImages) {
                    // "primary_image" is the URL the engine resolved (falls back to "image").
                    $imageUrl = $item->get('primary_image') ?? $item->get('image');
                    if (is_array($imageUrl)) {
                        $imageUrl = $imageUrl[0] ?? null;
                    }
                    $imagePath = is_string($imageUrl) && $imageUrl !== ''
                        ? $images->store($imageUrl, $this->imageDisk, $this->imageFolder, $hashNames)
                        : null;

                    $this->processImage($imagePath);  // optional resize/watermark

                    // Uncomment to also store the full gallery from "gallery_images":
                    // foreach ((array) $item->get('gallery_images') as $url) { ... }
                }

                // DEFAULT — append to storage/app/scrapes/<name>.json (or --output).
                $this->saveJson([...$item->toArray(), 'image_path' => $imagePath]);

                // SAVE TO A MODEL — comment saveJson() above, uncomment:
                // Product::updateOrCreate(
                //     ['id' => $item->get('id')],
                //     ['title' => $item->get('title'), 'price' => $item->get('price'),
                //      'image_path' => $imagePath, 'raw' => $item->toArray()],
                // );
            },
            (int) $this->option('limit'),
        );

        return self::SUCCESS;
    }
}

The closure is the heart of the robot — swap the saveJson() line for an Eloquent updateOrCreate(), a queued job, or a webhook POST to send records straight into your app.

The two lines you usually change

Property	Purpose
`$imageDisk`	Any Laravel filesystem disk — `'storage'`, `'public'`, `'s3'`, `'gcs'`, …
`$imageFolder`	Subfolder inside that disk where this site's images go

Cloud disks just need their Flysystem adapter installed and configured in config/filesystems.php. See RobotExampleMarket in the reference project for a complete worked example.

Per-item persistence

Inside the CallbackSink closure you have the full ScrapedItem. This is where you:

download images ($images->store(...)) — see Images;
run processImage() to resize / watermark / convert;
build your record and persist it however you like — Eloquent model, dispatched job, webhook POST, etc.

Because it is plain PHP inside a Laravel command, anything your app can do, a robot can do per item.

Built-in sinks

When the per-item logic is a standard pattern — upsert to a table, dispatch a job, POST a webhook — you can skip the closure entirely and hand a ready-made sink to crawlToSink() (available on the same ScrapesToConsole trait as crawlEach()):

php

use DataHelm\Crawler\Output\DatabaseSink;
use DataHelm\Crawler\Output\QueueSink;
use DataHelm\Crawler\Output\WebhookSink;

// Upsert each item into an Eloquent model
$this->crawlToSink($engine, $blueprint, new DatabaseSink(
    model:      Product::class,
    uniqueBy:   ['link'],               // upsert key(s)
    updateOnly: ['title', 'price'],     // restrict which columns are updated (null = all)
    fieldMap:   ['link' => 'url'],      // rename scraped fields before insert
    exclude:    ['image'],              // drop scraped fields you don't store
), (int) $this->option('limit'));

// Dispatch a queued job per item
$this->crawlToSink($engine, $blueprint, new QueueSink(
    job:       ProcessProduct::class,   // constructor receives the ScrapedItem
    queue:     'scrapes',
    passArray: true,                    // …or an array, when true
));

// POST each item (or batches) to an HTTP endpoint
$this->crawlToSink($engine, $blueprint, new WebhookSink(
    url:       'https://api.example.com/items',
    headers:   ['Authorization' => 'Bearer token123'],
    batchSize: 25,                      // 1 = one POST per item
));

Notes on behaviour:

DatabaseSink upserts one row at a time — fine for typical crawls; buffer in batches yourself for very high volume.
WebhookSink errors are non-fatal by default: they're written to stderr and the crawl continues (throwOnError: true to abort instead).
CallbackSink (what crawlEach() uses under the hood) remains the escape hatch for anything custom — and the only place image downloading is wired up.

Run options

Every generated robot supports the same run-time flags as datahelm:scrap:run:

bash

php artisan datahelm:robot:exampleauctions --limit=20                       # cap items
php artisan datahelm:robot:exampleauctions --output=storage/app/out.json    # custom path
php artisan datahelm:robot:exampleauctions --output=- > out.json            # stdout

Schedule it

Because a robot is a normal Artisan command, a recurring crawl is one line in routes/console.php (or app/Console/Kernel.php on older Laravel versions):

php

Schedule::command('datahelm:robot:exampleauctions')->dailyAt('03:00');

Combine with --resumable / --resume so each scheduled run only processes items it hasn't seen before.

Robot vs. blueprint file — which to use?

	Blueprint file (`--blueprint`)	Robot (default)
Storage	`storage/app/blueprints/*.json`	Committed PHP file
Per-item logic	None — output goes to a file	Full PHP closure
Scheduling	Wrap `scrap:run` yourself	Native Artisan command
Best for	Exploration, one-offs	Anything recurring or feeding your app

Rule of thumb: explore with --json, iterate with --blueprint, ship a robot.

Next: Selector shell →

Scaffold a robot #

Anatomy of a robot #

The two lines you usually change #

Per-item persistence #

Built-in sinks #

Run options #

Schedule it #

Robot vs. blueprint file — which to use? #