Substack Scraper: Why It Breaks, the Rules, and Alternatives

"Substack scraper" is a popular search, and plenty of tools promise one. Before you build or buy, it helps to know two things: what Substack's terms say, and why scrapers built on Substack's pages keep failing. This page covers both, then lists the official routes that usually get you the same data. It contains no scraping instructions and we don't recommend scraping Substack. For the full list of Substack's developer surfaces, see our Substack API guide.

newsletterapi is an independent project, not affiliated with Substack. We are not lawyers, and this is not legal advice.

The problem

People usually reach for a scraper because they think there's no other way in. A Substack user asked on Notes: "Does substack have developer API keys? I need to make an automated scraper" (Note). Then the scraper hits a wall. One open-source exporter's issue tracker reads: "Running this grabs the first few posts, but then substack complains of too many requests." (issue #28). A team trying to show their own posts on a corporate site wrote: "Between the Substack CORS blocks and Substack blocking Azure addresses, where the corporate website is hosted, there isn't a decent way of share Substack post information with the corporate website." (Reddit).

Most of these jobs have an official answer. The rest of this page maps them.

What do Substack's terms say about scraping?

Substack's Terms of Use (last updated October 6, 2026) list uses you agree not to make. Four clauses matter here. You may not use Substack in a manner that:

• "Crawls," "scrapes," or "spiders" "any page, data, or portion of Substack (through use of manual or automated means)".
• "Copies or stores any significant portion of the content on Substack".
• Runs "any processes that run or are activated while you are not logged into Substack, or that otherwise interferes with the proper working of Substack (including placing an unreasonable load on Substack's infrastructure)".
• "Jeopardizes the security of your Substack account", "such as allowing someone else to log into Substack as you, or sharing your account or password with someone".

The account section adds that you will use Substack "for your own personal or organizational use, and not on behalf of or for the benefit of any third party."

The separate Developer API Terms (last updated January 8, 2026) cover data from Substack's API. They forbid redistributing that data "as a standalone product or dataset" and using the API to "replicate, substitute for, or materially compete with core features of the Substack platform", and they add that you agree "not to circumvent or attempt to bypass" rate limits.

Doesn't robots.txt allow it?

Substack's robots.txt disallows some paths for all crawlers, such as /sign-in, /subscribe and /embed, and blocks one named bot entirely. It does not list everything else as permitted for scraping, and robots.txt is a crawler convention, not a licence. The Terms of Use apply whatever it says.

Why do Substack scrapers keep breaking?

Even setting the terms aside, scrapers are a poor foundation:

Failure	What it looks like	Likely cause
Rate limiting	"too many requests" after a few pages (issue #28)	Substack appears to throttle rapid automated requests
Blocked hosts	Requests from cloud servers refused (Reddit)	Server traffic appears to be treated differently from readers' browsers
Account flags	"That is 1000s of calls in automated way with potential of flagging." (comment)	Bulk calls tied to a login look like abuse
Format drift	Parsers fail after a site update	Page markup and internal JSON have no version contract
Unexpected body format	"A Substack post body isn't HTML, it's a ProseMirror document" (LinkedIn)	Internal data is shaped for Substack's editor, not for you
Paywalls	Paid posts cut short	Full text needs a subscription, and sharing a login is barred

Hosted scraper services carry the same problems, plus one more: if a service asks for your Substack password or session to reach paywalled or dashboard data, it is asking you to let "someone else log into Substack as you".

What are the legitimate alternatives to a Substack scraper?

You want…	Use this instead	Who can use it
Your own recent posts on a website or in a script	Your RSS feed at https://your.substack.com/feed (help)	Anyone, for public posts
Your full archive, including paid posts	Settings, Exports, "Create new export" (help)	The writer
Your post stats and subscriber counts in code	The read-only Publisher API, per the substack-publisher-mcp wrapper	Publications where Substack has enabled it
Your numbers inside an AI assistant	Substack's official MCP server (help)	Admins of Bestseller publications
To follow other newsletters	Their RSS feeds in a feed reader, or subscribing	Anyone
To find a creator's public profile	The Developer API (LinkedIn-handle lookup), under its terms	Approved applicants
Research on another publication	Ask the writer, or quote a few public facts by hand with a link	Anyone

Three of those deserve a closer look:

• RSS is the intended machine-readable surface. It is documented by Substack, needs no login and covers recent public posts. Our Substack RSS feed guide shows what it contains, and embedding Substack posts on your website covers the corporate-site case above.
• The export is your data. It's manual, but it's complete and yours. Our Python tutorial reads the zip with the standard library, and how to back up your Substack covers keeping copies.
• Official keys beat sessions. The Publisher API and the official MCP server are read-only and don't need your password in anyone else's hands.

What if I need data about many publications?

This is where scrapers are most tempting and the terms are clearest. Building a dataset of other writers' posts, or a history of their numbers, means copying or storing content at scale, which the Terms of Use forbid; and Developer API data can't be resold "as a standalone product or dataset". If you are doing research, contact the writers or Substack directly. If you only need a handful of public facts, read them yourself and link to the source.

How does newsletterapi fit in?

newsletterapi works only with a writer's own publication: their public feed plus the export they choose to upload. The Posts API serves that archive as JSON with search, and our MCP server gives AI agents the same read-only view. It never asks for your Substack password. To get access, see pricing or email hello@newsletterapi.io.