Aggregate content into a single feed
Pull articles and posts from many public sources into one uniform feed, with a single JSON schema instead of one scraper per site.
- Industries:
- Media, Publishing, Developer tools
The problem
Building a feed, a newsletter, or an internal digest means pulling from many sources, and every source presents its content differently. Some publish a clean RSS feed, many do not, and the ones that matter most often bury the article inside a custom layout full of related-post widgets, share buttons, and ad slots. Stitching all of that into one consistent stream is the hard part.
The traditional fix is a scraper per source, and that ages badly. Each parser is glued to one site's structure, so a redesign anywhere in your source list silently drops that publication from the feed, and you only notice when the gap becomes obvious. Maintaining a dozen of these is a part-time job nobody wants, and the feed quality drifts as parsers quietly rot.
An aggregator really needs one thing: the same fields, title, author, date, body, and link, from every source, in the same shape. Define that once and the feed stops being a maintenance project.
The workflow
- 1
Define one feed-item schema
Write a single JSON schema for a feed item: title, author, published date, summary, body text, and canonical link. Every source, however it is laid out, will be normalized into this one shape before it reaches your feed.
- 2
List your source URLs
Collect the public article or post URLs you want to aggregate from news sites, blogs, documentation, and JavaScript-rendered or anti-bot-protected pages. Extracto works on any public HTTPS URL, so the feed can draw from the sources you actually want rather than only the simplest ones.
- 3
Extract each item through Extracto
Send each URL with the shared schema. Extracto runs every request through a real headless browser with a managed anti-bot bypass layer that handles protected sites like Cloudflare, DataDome, and PerimeterX, pulls just your fields, and validates them, so widgets, ads, and related-post clutter never make it into the feed item. Proxies and anti-bot handling are managed for you, with no setup.
- 4
Normalize and de-duplicate
Because every item already shares one shape, normalizing dates and de-duplicating by canonical link is straightforward. Null fields flag items that are incomplete so you can hold them back rather than publish broken entries.
- 5
Merge, sort, and publish
Combine the validated items into one stream, sort by date, and publish to your feed, newsletter, or app. Re-running the same URLs and schema refreshes the feed without touching any per-source parsing code.
A schema to start from
Hand Extracto a schema like this and a URL. You get back validated JSON with exactly these fields.
{
"title": "string",
"author": "string",
"publishedDate": "string",
"summary": "string",
"bodyText": "string",
"canonicalUrl": "string"
}Is it worth it?
One schema across many sources gives you a uniform feed without a scraper per site, and validation keeps clutter and broken items out. The trade-off is a per-page cost per refresh. Extracto works on any public HTTPS URL, including JavaScript-rendered and anti-bot-protected pages, because every request runs through a real headless browser with a managed anti-bot bypass layer, so it fits news, blogs, docs, and anti-bot-protected sites alike. The one boundary is login-gated content like LinkedIn, X, or Instagram, which needs session cookies available on Enterprise plans. The free plan covers 100 pages with no credit card, and paid plans start at $14/month.