How to scrape npm

Extract npm package name, version, weekly downloads, license and dependents from npmjs.com pages as schema-validated JSON, no selectors.

Site:: npmjs.com
Difficulty:: Easy
Rendering:: JavaScript-rendered
Anti-bot:: Cloudflare, rate-limiting on unauthenticated registry endpoints

What you can extract from npm

These are the fields teams most often pull from npmjs.com, and why a hand-written selector for each one tends to break over time.

Field	Type	Why selectors break
weeklyDownloads	number	The weekly download count is fetched client-side from api.npmjs.org after the page shell loads, then injected into a div whose class is a hashed CSS-module name like _702d723c that changes every deploy, so any selector you pin today silently returns null next week.
dependents	number	The dependents total renders as a tab label such as 'Dependents (4,812)' inside a JS-built nav, and the number is formatted with locale commas, so a selector must both survive the hashed tab markup and strip thousands separators before the string will parse as an integer.
latestVersion	string	The version sits in a sidebar that React re-renders after hydration; before JS runs the node is empty, so a selector against the initial HTML grabs a blank string and a selector against the hydrated DOM depends on a class hash that npm rotates between builds.
lastPublish	string	Last publish shows as relative text like 'a month ago' driven by a JS time library, with the absolute ISO date hidden in a title attribute, so a selector returns the fuzzy human string rather than a usable timestamp and breaks the moment the relative wording changes.
license	string	License appears in a definition-list 'Details' block where the label and value are sibling nodes with no stable id, and some packages omit the field entirely, so a positional selector points at the wrong sibling for any package that is missing a license.

The schema-first approach

Instead of maintaining selectors, you hand Extracto a JSON schema describing the shape you want. Here is a schema that works for npm:

{
  "packageName": "string",
  "latestVersion": "string",
  "weeklyDownloads": "number",
  "description": "string",
  "license": "string",
  "lastPublish": "string",
  "dependents": "number"
}

1

Describe the npm fields you want

Write a small JSON schema listing the fields to pull from npmjs.com (for example weeklyDownloads, dependents, latestVersion). No CSS selectors or XPath needed.
2

Send the URL and schema to Extracto

POST the npm page URL and your schema to the Extracto API. Extracto renders the page in a real browser, handling JavaScript and dynamic content automatically.
3

Get schema-validated JSON back

Extracto returns JSON that matches your schema exactly, validated before it leaves the API, so your pipeline never receives broken or partial data.

Common pitfalls when scraping npm

Hitting registry.npmjs.org gives you name, dist-tags.latest, description and license, but it does NOT carry weekly downloads or the dependents count, so people assume one endpoint covers the whole page and end up with null for two of the most-wanted fields.
Weekly downloads come from a separate host (api.npmjs.org/downloads/point/last-week/<pkg>) and dependents only exist as a rendered number on the HTML page, so any single-source approach quietly drops a field; Extracto reads the rendered page and returns the field as null rather than guessing a count.
Scoped packages like @types/node must be URL-encoded as @types%2Fnode in API paths but appear as @types/node on the website, so a scraper that reuses the same string for both the API and the page URL 404s on one of them.
The latest dist-tag is not always the highest semver: packages publish 'next', 'beta' and 'canary' tags, so reading the top version row off the page can capture a prerelease instead of the stable latest unless you specifically target the latest tag.
npmjs.com is a client-rendered Next.js app behind Cloudflare, so a plain HTTP GET returns a near-empty shell or a 403 challenge; the download number and dependents tab simply are not in that initial HTML, which is why a real headless browser render is required.

Is it legal to scrape npm?

npm is operated by GitHub (Microsoft) and the public registry data is intended to be consumed: the npm Terms of Use and Open-Source Terms describe the registry as a free public service, and registry.npmjs.org plus api.npmjs.org are documented, CORS-enabled endpoints meant for programmatic access. Scraping the rendered npmjs.com package page is more sensitive than hitting those APIs, because the website is fronted by Cloudflare and the HTML is a client-rendered Next.js app rather than a data product. npm's robots.txt allows crawling of /package/ paths but disallows search and internal routes, so stay on canonical package URLs. Treat the human-facing site as a fallback for fields the APIs do not expose (such as the dependents count and the rendered weekly-downloads sparkline), keep request volume modest, identify yourself honestly, and prefer the official registry and downloads APIs whenever they carry the field you need. Do not republish package metadata in a way that misrepresents authorship, and respect each package author's own license.

Extracto follows robots.txt by default and rate-limits politely. You are responsible for compliance with npm's terms and applicable law (GDPR, CCPA) in your use case.

Scraping npm: FAQ

Should I scrape the npm website or use the registry API?

Use the registry API for everything it exposes: registry.npmjs.org/<package> returns the name, dist-tags.latest, description, license and per-version publish times as clean JSON, and it is a documented public endpoint. Scrape the rendered npmjs.com page only for fields the APIs do not return well, such as the dependents count and the human-facing weekly-downloads display. Extracto lets you point at the page once and get all seven fields back as typed JSON without juggling three endpoints.

Where does the weekly downloads number actually come from?

On the website it is loaded client-side from api.npmjs.org/downloads/point/last-week/<package>, then formatted and injected into the page after hydration. That is why a static HTML fetch never sees it and why a CSS selector against the page is so fragile. Extracto renders the page in a real headless browser so the number is present in the DOM by the time the schema is applied, and returns weeklyDownloads as a number or null if the package is too new to have data.

How do I handle scoped packages such as @scope/name?

On npmjs.com the URL keeps the slash, for example /package/@types/node, while the registry and downloads APIs require the slash to be percent-encoded as @types%2Fnode. Mixing the two formats causes 404s. With Extracto you pass the normal website URL with the slash intact and let the rendering layer fetch the page, so you do not maintain two encodings yourself. Just keep the schema field names stable across packages.

Why is the dependents field sometimes null?

The dependents count is only computed and rendered on the HTML page, not returned by registry.npmjs.org, and npm hides or omits it for some packages and during partial loads. Rather than emit a fabricated number, Extracto validates against your JSON schema and returns dependents as null when the value is genuinely absent, so a missing count never silently becomes a made-up integer that pollutes your dataset.

Does npm block automated requests to its package pages?

npmjs.com is fronted by Cloudflare and the package page is a JavaScript-rendered Next.js app, so an unconfigured HTTP client commonly receives a 403 challenge or an empty shell with none of the data fields. Extracto includes a managed anti-bot bypass layer and renders each page in a real browser, so protected, client-rendered pages resolve to complete typed JSON. Keep volume modest and prefer the official APIs where they carry the field.