Dataset schema for web extraction

Dataset pages on platforms like Kaggle, Hugging Face, data.gov and academic repositories all describe the same core thing, a published collection of data with a license, a format, a size and a download, but every site lays it out differently. One uses a sidebar metadata card, another a schema.org Dataset JSON-LD block, a third a plain table under the title. With Extracto you skip all of that. You describe the fields you want in a JSON schema, send a single dataset URL, and Extracto runs the page through a real headless browser and returns typed JSON validated against your shape. There are no CSS selectors to write and nothing to re-tune when a repository redesigns its layout, because you are describing meaning ("the SPDX license identifier", "the total file size") rather than DOM position. When a field genuinely is not present on the page, for example a dataset that publishes no explicit license or hides its row count, Extracto returns null for that field instead of guessing a plausible value. That distinction matters a lot for dataset metadata, where a fabricated license string or an invented size could send a downstream pipeline, or a compliance review, in the wrong direction. This reference entry gives you a copy-paste schema for the Dataset shape, an example of the typed values you get back, and answers to the questions that come up most often when extracting dataset listings at modest, Terms-of-Service-respecting volume.

Fields in a Dataset schema

The fields teams most often extract for a dataset, what type each one is, and whether it is usually present.

Field	Type	Required	Notes
name	string	yes	The dataset's title as shown on the page, for example its Kaggle slug heading or schema.org name. Returning it typed as a clean string (not the surrounding HTML heading markup) means you can key records, dedupe and display them without parsing tags out yourself.
description	string	optional	The human-written summary of what the data contains and how it was collected. Captured as plain text so you can index it for search or feed it to a model, with null returned when a dataset ships no description rather than echoing back boilerplate from the page chrome.
license	string	optional	The usage license, ideally the SPDX identifier or license name (CC0-1.0, MIT, CC-BY-4.0). This is the field where null-not-guess matters most: an unlicensed dataset returns null instead of a hallucinated permissive license, so your compliance checks stay honest.
format	string	optional	The primary file format or media type of the data, such as CSV, Parquet, JSON or SQLite. Returned as a normalized string so you can filter a catalog by what your loader actually supports, instead of scraping a format badge whose CSS class changes between layouts.
size	number	optional	The total size of the dataset in bytes, parsed from human strings like '2.3 GB' or '450 MB' into a real number. Typed as a number so you can sort, sum and threshold storage without re-parsing units, and null when a page states no size rather than inventing one.
lastUpdated	string	optional	The date the dataset was last updated or its version refreshed, normalized to an ISO 8601 date string. Typed consistently so you can detect stale data and trigger re-downloads, even when the source page writes dates as 'Updated 3 months ago' or a localized format.
publisher	string	optional	The organization or user who published the dataset, for example the Kaggle account, the Hugging Face org, or the agency on a data portal. Returned as a string for attribution and provenance tracking, with null when authorship is genuinely not stated on the page.
downloadUrl	string	optional	The direct link to the dataset's downloadable distribution or archive when the page exposes one. Returned as an absolute URL string so a pipeline can fetch it, and null (not a guessed path) when the page only offers an authenticated or button-driven download with no visible href.

The schema

Copy this and send it with any URL.

{
  "name": "string",
  "description": "string",
  "license": "string",
  "format": "string",
  "size": "number",
  "lastUpdated": "string",
  "publisher": "string",
  "downloadUrl": "string"
}

Example output

Validated JSON back, matching the schema.

{
  "name": "Wine Reviews",
  "description": "130k wine reviews with variety, location, winery, price, and description scraped from WineEnthusiast.",
  "license": "CC-BY-NC-SA-4.0",
  "format": "CSV",
  "size": 51200000,
  "lastUpdated": "2017-11-27",
  "publisher": "zackthoutt",
  "downloadUrl": "https://www.kaggle.com/datasets/zynicide/wine-reviews/download"
}

Try this dataset schema against a real page in the live demo, or read the docs to use it via the API.

Dataset schema: FAQ

How is this different from writing CSS selectors for each dataset platform?

With selectors you target DOM nodes, so a Kaggle sidebar refactor or a Hugging Face redesign silently breaks your scraper and you maintain a separate selector set per site. With Extracto you describe the meaning of each field once (the license, the byte size, the publisher) and the same Dataset schema works across Kaggle, Hugging Face, data.gov and academic repositories, because the model reads the rendered page rather than a fixed element path.

What happens when a dataset has no stated license or no listed size?

Extracto returns null for that field instead of guessing. This is deliberate and especially important for dataset metadata: a fabricated license could expose you to a compliance problem and an invented size could mislead a storage or cost estimate. A null tells you the page genuinely did not state the value, which is information you can act on, rather than a plausible-looking value you cannot trust. Every field is validated against the JSON schema before it is returned to you.

Can it handle dataset pages behind Cloudflare or other anti-bot protection?

Yes. Every request runs through a managed anti-bot bypass layer that handles protected sites such as Cloudflare, DataDome and PerimeterX, with proxies managed for you and no setup on your side. It works on any public HTTPS dataset page, including JavaScript-rendered listings that build their metadata cards client-side. The only thing it cannot reach is login-gated content that needs your session cookies, for example a private repository visible only when signed in, which is available on the Enterprise plan.

Does it crawl an entire dataset catalog or one page at a time?

Extracto extracts one URL per call, it is not a crawler. You point it at a specific dataset page and get back the typed JSON for that page. If you want to process a catalog, you collect the dataset URLs yourself (from a sitemap, a search results page you also extract, or an API) and call Extracto once per URL. Keeping volume modest and respecting each site's Terms of Service and robots.txt is part of using it responsibly.

Why is size returned as a number when the page shows '2.3 GB'?

Because typed output is the whole point. Extracto parses the human-readable size string and returns the value in bytes as a real JSON number, so you can sort datasets by size, sum storage across a catalog, or apply a threshold without writing unit-parsing code on your end. The schema declares size as a number, and the value is validated against that type before it reaches you, so you never receive a string like '2.3 GB' where your code expects an integer.