How to scrape job postings from company career pages

The short answer: first look for the board’s free job API. If there is none, fetch each job page with response_types: "page_data" over plain HTTP. Most career pages carry a JobPosting JSON-LD block for search engines, so you get title, location, employment type, date posted and company with no selectors and no browser. That worked on three of the four boards we measured, including two built as single-page apps. On the fourth, two CSS rules read the title and location.

The free way first: the board’s own API

Career pages are usually hosted by an applicant tracking system, and many of those publish a free, public job-board API: one call returns every open job on a board as JSON. Look for it before you pay anyone, us included. On the Greenhouse-hosted board we measured, one call to its free job-board API returned 210 jobs. The Workday board has a free JSON endpoint too.

A free API is not enough when the board turned it off, when it blocks your script with a 403 or a bot check, or when you need many boards in one shape. Then read the job pages themselves.

Check a board first

Before a big run, ask about the host. It is free. Over MCP, your agent calls check_site:

{"name": "check_site", "arguments": {"host": "jobs.lever.co"}}

From cURL, the same answer is two calls: /v1/tier for the price, /v1/stats/domain for how our routes have done there lately.

curl -s -H "Authorization: Bearer YOUR_KEY" \
  "https://get.megalania.io/v1/tier?host=jobs.lever.co"
curl -s -H "Authorization: Bearer YOUR_KEY" \
  "https://get.megalania.io/v1/stats/domain?host=jobs.lever.co&days=30"

This is what /v1/tier answered us:

{"host": "jobs.lever.co", "tier": null, "temporary": null, "price": null, "pending": null,
 "quotes": {"residential": 15}, "capabilities": {"http": ["residential"], "browser": []}}

tier: null means we have not priced this host yet, so a page pays the price of the route that delivers it, and quotes shows that number. Our test key had the datacenter routes off, so it shows residential only. Yours also shows datacenter, which is cheaper. All four boards answered tier: null. Check a site first explains every field.

One request: page_data over HTTP (cURL)

The body is the exact one we ran:

curl -sS https://get.megalania.io/v1 \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -D headers.txt \
  -d '{"url": "https://jobs.lever.co/palantir/ac978161-6f46-4f6b-ad9e-a258e642751c", "engine": "http", "response_types": "page_data", "max_credits": "64"}'

page_data is the data the page carries for its own scripts: {"json_ld": [...], "state": {...}}. json_ld is every JSON-LD object on the page. It adds 0 credits. engine: "http" keeps it on plain HTTP, which is the cheap route. max_credits is the most this page may cost; we skip any route above it.

What JobPosting gives you

We read six fields from the JobPosting block: title, location, employmentType, datePosted, hiringOrganization and validThrough.

The missing field was validThrough, the closing date. These boards do not publish one.

The Ashby and Workday boards are single-page apps: the visible page builds itself with JavaScript. Their JSON-LD is in the plain HTML anyway, so you do not need a browser for these fields. Try page_data over HTTP before you pay for a browser. Our post on JavaScript pages has the full test.

Python: one JSON record per job

The posting() parser is the one our measurement ran. It walks every JSON-LD block, including @graph wrappers, and returns the first JobPosting.

# pip install requests
# Run: MEG_API_KEY=YOUR_KEY python jobs.py https://jobs.lever.co/palantir/ac978161-6f46-4f6b-ad9e-a258e642751c
import json
import os
import sys
import uuid

import requests


def walk(node):
    if isinstance(node, list):
        for x in node:
            yield from walk(x)
    elif isinstance(node, dict):
        yield node
        yield from walk(node.get("@graph"))


def posting(pd):
    for b in pd.get("json_ld", []):
        for n in walk(b):
            if n.get("@type") == "JobPosting":
                loc = n.get("jobLocation") or {}
                loc = loc[0] if isinstance(loc, list) and loc else loc
                addr = loc.get("address", {}) if isinstance(loc, dict) else {}
                org = n.get("hiringOrganization") or {}
                return {"title": n.get("title"), "location": (addr.get("addressLocality") if isinstance(addr, dict) else addr) or n.get("jobLocationType"),
                        "employmentType": n.get("employmentType"), "datePosted": n.get("datePosted"),
                        "hiringOrganization": org.get("name") if isinstance(org, dict) else org, "validThrough": n.get("validThrough")}
    return None


for url in sys.argv[1:]:
    resp = requests.post(
        "https://get.megalania.io/v1",
        headers={
            "Authorization": f"Bearer {os.environ['MEG_API_KEY']}",
            "Idempotency-Key": str(uuid.uuid4()),
        },
        json={"url": url, "engine": "http", "response_types": "page_data", "max_credits": "64"},
        timeout=90,
    )
    if resp.status_code != 200:
        err = resp.json()["error"]
        print(f"{url}: {resp.status_code} {err['code']}: {err['message']}")
        continue
        continue
    job = posting(resp.json())
    if job is None:
        print(f"{url}: no JobPosting JSON-LD, credits {resp.headers['X-Credits-Charged']}")
        continue
    print(json.dumps({"url": url, **job, "credits": resp.headers["X-Credits-Charged"]}))

A block is HTTP 422 with error.code blocked and X-Billing: blocked, and it costs 0 credits, so the status check catches it. X-Hops on that answer lists every route we tried. Locations differ by board: some give a city in jobLocation, remote jobs may give only jobLocationType, so the parser falls back to it.

No JSON-LD? Two CSS rules

The Greenhouse-hosted board had no JobPosting block. Its job pages use stable class names, so we sent response_types: "json" with extract rules. This is the body we ran:

curl -sS https://get.megalania.io/v1 \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -D headers.txt \
  -d '{"url": "https://job-boards.greenhouse.io/gitlab/jobs/8860302002", "response_types": "json", "extract": "title:.job__title h1|location:.job__location|pay:.job__pay-ranges|tags:.job__tags", "max_credits": "64"}'

Title and location came back on all 10 pages. pay and tags are there when the board fills them in; a rule that matches nothing comes back as null. json output adds 1 credit.

What it costs

We fetch on the cheapest route first: plain HTTP on a datacenter exit (h:dc). If the board blocks it, we climb to a residential exit (h:res). A sample answer’s headers:

x-credits-charged: 15
x-engine: http
x-hops: h:res:chrome=1495
x-identity: chrome
x-route: http/residential
x-target-status: 200

Measured on our dev cluster with the datacenter routes off, so this climb starts at h:res.

A page like this costs about 1-2 credits on h:dc (15-104 KiB, the boards in the table below), worked out from our price table on 4 Oct 2026, because that route was off in our test; run check_site (free) to see the price for your site.

A page like this was billed 15-23 credits on h:res (15-104 KiB, the boards in the table below) in our test on 3 and 4 Oct 2026; run check_site (free) to see the price for your site.

A page up to 64 KiB on the wire costs the base price; each started 128 KiB above it adds 1 on h:dc and 8 on h:res. On h:res, 15 is the price of a host we have not priced yet.

So 1,000 job pages cost 1,000 to 2,000 credits on h:dc, or 15,000 to 23,000 on h:res. The 76,000-credit beta grant covers at least 3,300 job pages, even if every one needs a residential exit.

How we measured: 39 job pages with page_data over HTTP (9 on jobs.ashbyhq.com, 10 each on jobs.lever.co, nvidia.wd5.myworkdayjobs.com and job-boards.greenhouse.io), plus 10 Greenhouse pages with the CSS rules. 3 Oct 2026, 00:11 UTC, to 4 Oct 2026, 04:47 UTC, with the bodies above.

Many boards, every day

For hundreds of job pages, send the same body as one batch with a urls list: Batches. Nothing is charged at submit; each URL is charged when we fetch it. Our measurement ran one request per page.

Terms and personal data

• Blocks we detect are free. The answer carries X-Billing: blocked and costs 0 credits. Each try shows in X-Hops.
• Errors are free. Every error has a code and a fix in Error codes.
• Personal data: job pages can name a recruiter or a hiring manager. Do not collect names, emails or phone numbers. We read only the job fields above.

This is not legal advice. Read each board’s terms and robots.txt, and follow them. Keep your rate low: a few seconds between pages on one board.