How to scrape a website without writing a single selector

Every scraper starts the same way. You open dev tools, right-click an element, copy the selector, and paste div.product-card > span:nth-child(3) into your code. It works. You ship it.

Then the site ships a redesign and your selector points at nothing.

The problem was never your code. The problem is that selectors couple your scraper to the layout of a page, and layout is the thing site owners change most often. So let’s not do that.

Describe the data, not the DOM

The idea is simple: instead of telling the scraper where the title is, you tell it what a title is. You hand over a URL and a JSON schema, and you get back validated JSON that matches the schema.

const article = await client.extract({
  url: "https://example.com/blog/post",
  schema: {
    headline: z.string(),
    author: z.string(),
    published: z.string(),
    wordCount: z.number(),
  },
});

There is no nth-child anywhere in that snippet, and there never will be. If the site moves the headline into a different container next month, the extraction still works, because “the headline” is a concept, not a position in the DOM.

Why this holds up

A reasonable objection: doesn’t an LLM in the loop just trade one kind of flakiness for another? It would, if we let the model return whatever it felt like. We don’t.

Three things make this reliable instead of clever:

The schema is enforced at the boundary. If the model returns a price as a string when you asked for a number, the response is rejected and retried. You never receive data that does not match what you asked for.
Missing is allowed, wrong is not. If a field genuinely is not on the page, you get null, not a hallucinated value. That distinction matters more than people expect.
The page is rendered first. JavaScript-heavy pages get a real headless browser, so client-rendered content is visible before extraction happens.

When selectors still win

To be honest about it: if you are scraping one static page that never changes and you need it done in the next ten minutes, a hand-written selector is fine. This approach earns its keep when you are pulling the same shape of data from many pages, or from pages you do not control, or from sites that redesign on their own schedule.

That is most real scraping jobs.

Try it

Point it at any public URL and describe the fields you want. No selectors, no setup. Run the live demo, or start free when you want your own key.

Describe the data, not the DOM

Why this holds up

When selectors still win

Try it

Related