Scraping behind a login #

Some data only exists after you sign in — an HR system, a CRM, an admin dashboard. DataHelm Crawler has no automated login flow (no form-filling, no OAuth). Instead, you log in once with your own browser and hand the crawler your session cookie: every request it makes then looks, to the target site, exactly like a request from your already-authenticated browser.

Step 1 — Capture the session cookie #

1. Log in to the site normally, in your own browser.
2. Open DevTools (F12) → Application tab → Cookies → the site's origin.
3. Find the cookie that identifies your session (common names: sessionid, orangehrm, PHPSESSID, laravel_session) and copy its value.

Step 2 — Pass it with --cookie #

php artisan datahelm:scrap:generate "<url>" \
  --cookie="orangehrm=<session cookie value>" \
  --transport=auto \
  --robot-name=example

The cookie is baked into the blueprint's http_config.cookies and sent on every request the robot makes, from then on:

"http_config": {
  "cookies": [
    { "name": "orangehrm", "value": "abc123", "domain": "opensource-demo.orangehrmlive.com" }
  ]
}

domain is optional. A cookie captured via --cookie="orangehrm=abc123" (no domain) is automatically scoped to whichever host each request actually goes to — across every transport (guzzle, browser, auto). That's what lets one copy-pasted session cookie authenticate both the initial page load and any JSON API the page calls in the background, including through the headless-browser transport.

--header="K: V" works the same way for session identifiers carried in a header instead of a cookie.

Authenticated SPAs: finding the API behind a login #

This is where it gets useful rather than just functional. Many login-gated dashboards are JavaScript SPAs: the employee list, the CRM records, the admin table — all fetched from a JSON API after the page loads, and invisible to a plain HTML fetch.

Recall how API-mode auto-discovery works: when static detection finds nothing, --transport=auto renders the page once in a headless browser and listens to its network traffic, picking the JSON response with the richest records as the data API. That render step uses the same --cookie / --header / user-agent as every other request — so it authenticates too. A login-gated list that only appears after auth renders correctly, and the real (now-populated) data endpoint is discovered exactly like a public one:

php artisan datahelm:scrap:generate \
  "https://opensource-demo.orangehrmlive.com/web/index.php/pim/viewEmployeeList" \
  --get-detail=true --get-primary-image=true --hash-names=true \
  --robot-name=opensource-demo-pim \
  --cookie="orangehrm=<session cookie value>" \
  --transport=auto

· Rendered in a headless browser and captured 5 JSON response(s) from its network activity.
· Auto-detected the site's data API from its network activity:
  https://opensource-demo.orangehrmlive.com/web/index.php/api/v2/pim/employees?limit=50&…
  (50 records) — building an API-mode robot.

The generated blueprint bakes in the cookie and the discovered limit/offset-style pagination, so the resulting robot is a fast, no-browser API-mode robot from then on — see JavaScript sites & JSON APIs for how page_param/items_path work. It re-runs unattended, for as long as the session stays valid.

When the session expires #

Session cookies are short-lived — sometimes minutes on public demo instances, hours on real apps. A run against an expired session fails one of two ways:

• API: response is not JSON — the API answered with an HTML login/error page instead of data. This is what you'll typically see in API mode.
• 401 Session expired (or a similar auth error) — some sites are explicit about it.

Either message means the same fix: log in again, copy the fresh cookie value, then either

• edit http_config.cookies directly in the blueprint / robot and re-run — no regeneration needed, since the blueprint is just data; or
• re-run datahelm:scrap:generate with the new --cookie and --force to overwrite the robot.

Not for unattended scheduled robots #

Because there's no automated login, a captured session is inherently temporary. This technique is ideal for a one-off scrape or an interactive investigation — it is the wrong tool for a robot you Schedule::command(...) to run nightly, since it will start failing the moment the session lapses with nobody there to refresh it. For unattended, long-lived access, look for an API token or an officially supported auth flow the site provides instead.

Next: JavaScript sites & JSON APIs →