How to scrape github

Extract GitHub repository data (stars, forks, issues, language, topics, latest release) into schema-validated JSON from a single repo URL.

Site:: github.com
Difficulty:: Easy
Rendering:: JavaScript-rendered
Anti-bot:: None on public repo pages: github.com serves repository pages over plain HTTPS without a JavaScript challenge or interstitial., Light rate-limiting on aggressive anonymous browsing, which Extracto's managed proxy rotation absorbs at modest volume., Extracto's managed rendering and proxy layer is included on every request, so the occasional throttle is handled for you with no setup.

What you can extract from github

These are the fields teams most often pull from github.com, and why a hand-written selector for each one tends to break over time.

Field	Type	Why selectors break
stars	number	The star count lives in an anchor whose id is dynamically suffixed and whose visible text is abbreviated to '246k' once the value crosses a thousand, so a CSS selector tends to grab a rounded string instead of the exact integer, which in pages observed today is typically carried in a title attribute, though that exact location is GitHub's own rendering detail and can change between releases.
forks	number	Forks share the same repeated social-count component markup as stars and watchers with no stable distinguishing class, so positional selectors like nth-child silently return the wrong counter whenever GitHub reorders or A/B tests that header row.
open_issues	number	The Issues tab counter typically appears in a small counter element that often seems to settle slightly after initial render rather than being present in the first HTML, and the surrounding utility class names look auto-generated and have been observed to differ across deploys, so a hardcoded selector is fragile and a hardcoded one may read a number before the page has finished updating.
primary_language	string	The primary language is not a labeled field; it must be inferred from the first colored dot in the languages bar, whose only signal is an inline style background color and an aria label, defeating any plain text CSS selector.
latest_release	string	The latest release appears in a sidebar widget that is injected after hydration and is omitted entirely on repos with no releases, so a fixed selector either misses the value on JS-rendered pages or throws on repos that never tagged a version.
topics	array	Topics render as a variable-length list of pill links with click handlers and truncation behind a 'show more' control, so a selector capturing visible pills drops hidden topics and mixes in the toggle button text.

The schema-first approach

Instead of maintaining selectors, you hand Extracto a JSON schema describing the shape you want. Here is a schema that works for github:

{
  "repo_name": "string",
  "description": "string",
  "primary_language": "string",
  "stars": "number",
  "forks": "number",
  "open_issues": "number",
  "latest_release": "string",
  "topics": "array"
}

1

Describe the github fields you want

Write a small JSON schema listing the fields to pull from github.com (for example stars, forks, open_issues). No CSS selectors or XPath needed.
2

Send the URL and schema to Extracto

POST the github page URL and your schema to the Extracto API. Extracto renders the page in a real browser, handling JavaScript and dynamic content automatically.
3

Get schema-validated JSON back

Extracto returns JSON that matches your schema exactly, validated before it leaves the API, so your pipeline never receives broken or partial data.

Common pitfalls when scraping github

Star and fork counts are displayed abbreviated (for example '246k' on facebook/react) while the exact integer hides in a title attribute, so naive text scraping loses precision; describe the field as a number in your schema and Extracto returns the resolved value rather than the rounded label.
The latest release widget is absent on repositories that have never tagged a release, and many active repos publish code without GitHub Releases, so treat latest_release as optional. Extracto returns null for it instead of guessing a version, which matters because a fabricated tag could silently corrupt a downstream dependency check.
The open issues counter on the Issues tab counts only open issues and excludes pull requests, and GitHub hydrates that number after initial render. Pulling it from raw HTML before the page settles can yield a stale figure, so rely on the rendered DOM that the headless browser produces.
Primary language is an inference from the languages bar, not a labeled value, and forks or vendored code can skew it. A repo dominated by generated files may report an unexpected primary_language, so validate it against expectations rather than trusting it as canonical project intent.

Is it legal to scrape github?

GitHub's Acceptable Use Policies permit accessing public repository pages, and its robots.txt has historically allowed indexing of repository routes while disallowing various high-churn paths such as search, blame and raw views and many query-string endpoints, though the exact set of disallowed paths is defined by GitHub and can change, so treat the live robots.txt as the source of truth. Scraping public, non-gated repo metadata (name, description, stars, forks, topics, releases) is widely treated as low-risk because the same data is openly published and also exposed through GitHub's official REST and GraphQL APIs, which are the sanctioned path for high-volume access and come with documented rate limits. GitHub's terms prohibit using scrapers to harvest personal information of users or to circumvent rate limits at scale. Keep request volume modest, respect the disallowed paths in robots.txt, prefer one repo URL per call rather than crawling an org's entire tree, and consider the authenticated API when you need thousands of repos. Always review GitHub's current Terms of Service and Acceptable Use Policies before any sustained extraction, since these reflect 2026 policy and can change.

Extracto follows robots.txt by default and rate-limits politely. You are responsible for compliance with github's terms and applicable law (GDPR, CCPA) in your use case.

Scraping github: FAQ

Can I scrape GitHub repos without using the GitHub API token?

Yes. Extracto renders the public repository page with a real headless browser and returns the repo name, description, stars, forks, open issues, primary language, topics and latest release as typed JSON, with no GitHub token required. For very high volume across thousands of repositories the official REST or GraphQL API is the sanctioned path with documented rate limits, but for targeted, modest extraction of a known repo URL the page-rendering approach works reliably today and needs no auth setup.

Will the abbreviated star count like '246k' break my numeric field?

No. GitHub shows a rounded label such as '246k' for display, but the exact integer is typically present elsewhere in the page (on pages observed today it commonly surfaces in the counter's title attribute, though that placement is GitHub's own rendering choice and may shift over time). When your schema declares stars as a number, Extracto works from the rendered page to resolve the precise value rather than returning the truncated string, so you get a clean integer you can compare, sort or store without parsing 'k' and 'm' suffixes yourself in downstream code.

What happens when a repository has no releases or no description?

Extracto returns null for any field it cannot find on the page, instead of inventing a plausible value. Many repos ship without GitHub Releases or leave the description blank, so latest_release and description will simply be null on those targets. This null-not-guess behavior is deliberate: a fabricated release tag or made-up description would silently poison dashboards, dependency audits or alerting built on the data.

Does GitHub block automated rendering of repository pages?

Public repository pages on github.com are served over plain HTTPS with no JavaScript challenge or interstitial, so they render cleanly. GitHub does apply light rate-limiting to aggressive anonymous browsing, and Extracto's managed proxy and rendering layer absorbs that at modest volume with no configuration on your side. Keep request rates reasonable, honor the disallowed paths in robots.txt, and prefer one repo URL per call over crawling an entire organization.

How do I get the list of repository topics as a clean array?

Declare topics as an array in your JSON schema. GitHub renders topics as a variable-length set of pill links, sometimes truncated behind a 'show more' control, which is awkward for CSS selectors. Extracto reads the rendered page and returns the topics as a typed array of strings, so a repo with several tags comes back as a simple list of string values that you can iterate or filter directly, with no need to handle truncation toggles or pill markup yourself.