How to scrape Hacker News
Scrape Hacker News front-page stories into typed JSON: title, points, author, comment count, submission URL and submission time. No CSS selectors.
- Site:
- news.ycombinator.com
- Difficulty:
- Easy
- Rendering:
- Static HTML
- Anti-bot:
- none
What you can extract from Hacker News
These are the fields teams most often pull from news.ycombinator.com, and why a hand-written selector for each one tends to break over time.
| Field | Type | Why selectors break |
|---|---|---|
| title | string | The headline lives in a span.titleline > a, but HN reuses the same .titleline markup for stories, Ask HN text posts, and job listings, so a selector tuned to one row shape silently grabs the wrong anchor or the trailing (domain) sitelink span on others. |
| points | number | Points sit in span.score with an id like score_38123456 and text such as '190 points', but job posts and brand-new submissions render no .score node at all, so a positional selector either throws or shifts every later field up by one when the element is absent. |
| author | string | The author is an a.hnuser inside the .subtext row, yet that same subtext also contains 'hide', 'past' and 'discuss' anchors, so an index-based or generic 'a' selector frequently returns a navigation link instead of the actual username for job and dead posts. |
| comment_count | number | Comment count is the last anchor in .subtext, but its text varies between '80 comments', 'discuss' (zero comments) and is simply missing on job posts, so a selector expecting a number breaks on the words and on absent nodes across the same listing. |
| submission_url | string | The outbound link is the href of span.titleline > a, but for Ask HN and text submissions that href is a relative item?id= link rather than an external URL, so a selector assuming an absolute http(s) target captures an on-site path and conflates self-posts with link posts. |
| submission_time | string | Time is rendered as relative text ('3 hours ago') inside span.age, with the absolute timestamp hidden only in the title attribute, so a selector reading the visible text yields a value that drifts every minute and is unparseable without resolving the attribute. |
The schema-first approach
Instead of maintaining selectors, you hand Extracto a JSON schema describing the shape you want. Here is a schema that works for Hacker News:
{
"title": "string",
"points": "number",
"author": "string",
"comment_count": "number",
"submission_url": "string",
"submission_time": "string"
}- 1
Describe the Hacker News fields you want
Write a small JSON schema listing the fields to pull from news.ycombinator.com (for example title, points, author). No CSS selectors or XPath needed.
- 2
Send the URL and schema to Extracto
POST the Hacker News page URL and your schema to the Extracto API. Extracto renders the page in a real browser, handling the HTML automatically.
- 3
Get schema-validated JSON back
Extracto returns JSON that matches your schema exactly, validated before it leaves the API, so your pipeline never receives broken or partial data.
Common pitfalls when scraping Hacker News
- Job posts and very fresh submissions have no points and no comment count. A scraper that assumes every row carries those values will either crash or misalign columns. Extracto returns null for the missing fields instead of guessing, so a job listing comes back with points and comment_count as null rather than a borrowed neighbor value.
- HN renders the submission age as relative text like '3 hours ago' and only keeps the absolute UTC timestamp in the age element's title attribute. If you store the visible text you capture a moving target that is worthless for sorting or dedup. Define submission_time precisely in your schema prompt so you get the stable timestamp, not the relative phrase.
- The front page is paginated with a 'More' link pointing to ?p=2, ?p=3 and so on, and each request is one URL for Extracto, not a crawl. Walk pages explicitly and respect the 30-second Crawl-delay from robots.txt between hits so you stay well within polite-use bounds rather than firing rapid sequential calls.
- Ask HN, Show HN and text posts put a relative item?id= link in the title anchor instead of an external domain, so submission_url is sometimes on-site. Treat an internal href as a self-post signal rather than discarding it, and do not assume submission_url is always an external website.
Is it legal to scrape Hacker News?
Hacker News is operated by Y Combinator and served from news.ycombinator.com as plain server-rendered HTML. Its robots.txt allows crawling of listing and item pages but disallows interaction endpoints such as /vote, /reply, /submitlink, /login, /hide, /fave, /flag and /x, and it publishes a Crawl-delay of 30 seconds, so automated reads of the same path should be spaced accordingly and kept to a modest volume. The HN Guidelines and Y Combinator's terms govern acceptable use; scraping public listings for research or personal tooling is generally tolerated, but reselling bulk content or hammering the site is not. Note that Y Combinator also offers an official, free Firebase API (hacker-news.firebaseio.com) and an Algolia-powered search API (hn.algolia.com), which are the documented, ToS-friendly way to pull HN data at scale. Use Extracto for HTML-shaped reads of the live front page; prefer those official APIs for high-volume historical pulls, and always honor robots.txt and the 30-second crawl delay.
Extracto follows robots.txt by default and rate-limits politely. You are responsible for compliance with Hacker News's terms and applicable law (GDPR, CCPA) in your use case.