How to scrape Wikipedia
Extract Wikipedia article title, summary, infobox key-values, section headings, references count and last-edited date as typed, schema-validated JSON.
- Site:
- en.wikipedia.org
- Difficulty:
- Easy
- Rendering:
- Static HTML
- Anti-bot:
- None in practice on article pages, Server-side rendered HTML, no JS or anti-bot challenge, Polite-crawler rate expectations (honor Crawl-delay and back off on 429)
What you can extract from Wikipedia
These are the fields teams most often pull from en.wikipedia.org, and why a hand-written selector for each one tends to break over time.
| Field | Type | Why selectors break |
|---|---|---|
| title | string | The article title lives in the h1#firstHeading element, but it is rendered with nested span markup for italics, superscripts and disambiguation parentheticals, so a naive selector grabs stray spans or duplicated text instead of the clean canonical title. |
| summary | string | The lead summary has no dedicated container; it is simply the first one to three p tags before the first h2, often preceded by hatnotes, coordinate boxes and the infobox, so any nth-child selector silently captures a disambiguation note or an empty paragraph instead of the real intro. |
| infobox | object | Infobox rows use table.infobox with th/td pairs, but the class suffix and row layout differ per template (infobox vcard, infobox biography, geography boxes), so a selector tuned to one article returns nulls or misaligned label-value pairs on the next. |
| section_headings | array | Section titles sit inside span.mw-headline within h2 and h3, and the surrounding edit-section links and anchor spans mean a bare h2 selector pulls in bracketed [edit] text and citation needed markers along with the heading. |
| references_count | number | References render as li elements under ol.references, but cite-error placeholders, named-reference reuse and grouped footnote lists inflate or deflate a raw li count, so the visible numbered total rarely matches a simple length query. |
| last_edited | string | The last-modified date appears only in the footer li#footer-info-lastmod as free prose like 'This page was last edited on...', so a selector must parse a localized sentence rather than read a clean machine date attribute. |
The schema-first approach
Instead of maintaining selectors, you hand Extracto a JSON schema describing the shape you want. Here is a schema that works for Wikipedia:
{
"title": "string",
"summary": "string",
"infobox": "object",
"section_headings": "array",
"references_count": "number",
"last_edited": "string"
}- 1
Describe the Wikipedia fields you want
Write a small JSON schema listing the fields to pull from en.wikipedia.org (for example title, summary, infobox). No CSS selectors or XPath needed.
- 2
Send the URL and schema to Extracto
POST the Wikipedia page URL and your schema to the Extracto API. Extracto renders the page in a real browser, handling the HTML automatically.
- 3
Get schema-validated JSON back
Extracto returns JSON that matches your schema exactly, validated before it leaves the API, so your pipeline never receives broken or partial data.
Common pitfalls when scraping Wikipedia
- Treating the lead summary as a fixed paragraph index breaks constantly, because hatnotes, italic-title notices and coordinate boxes shift the real intro down, so anchor the summary to the prose before the first h2 rather than to nth-child positions.
- Infobox shape is template-driven, not fixed: a biography box, a software box and a country box expose entirely different keys, so model the infobox as an open key-value object and let unmatched fields return null instead of expecting a rigid set of labels.
- The references count visible on the page is not the same as counting ol.references list items, because named references reused many times, grouped footnotes and citation-error stubs distort the raw total, so decide whether you want distinct citations or total inline markers and be consistent.
- Some heavily edited articles carry maintenance banners, 'citation needed' tags and bracketed [edit] links inside headings, which leak into naively extracted text; the section headings should be the clean human titles, not the surrounding editorial chrome.
- Mass-extracting Wikipedia over live HTML is the wrong tool for bulk work and risks crawler blocks; for whole-corpus needs use the Wikimedia dumps or REST API, and reserve URL-level extraction for targeted, low-volume enrichment.
Is it legal to scrape Wikipedia?
Wikipedia article prose and the structured data within it are published under the Creative Commons Attribution-ShareAlike (CC BY-SA) license, with some media under separate terms, so reuse is broadly permitted as long as you attribute Wikipedia and the contributing authors and preserve the share-alike condition. The robots.txt at en.wikipedia.org does NOT disallow regular /wiki/ article pages, but it explicitly blocks site-mirroring tools (HTTrack, wget, WebReaper and similar) and the /w/ and /api/ internal paths, and the file warns that "irresponsible" high-volume crawlers may be blocked. The Wikimedia Foundation Terms of Use and the official robots policy ask that bulk reusers prefer the public dumps and the REST API over scraping rendered HTML, identify themselves with a descriptive User-Agent, and keep request rates modest. Extracting a handful of article pages at a polite cadence for analysis or enrichment is well within accepted use; mass mirroring of the site is not. As of 2026 the canonical large-scale path remains the Wikimedia dumps, so treat live extraction as a targeted, low-volume complement, not a substitute.
Extracto follows robots.txt by default and rate-limits politely. You are responsible for compliance with Wikipedia's terms and applicable law (GDPR, CCPA) in your use case.