Scraping SEC EDGAR Yourself vs. a Maintained API

What scraping SEC EDGAR yourself actually involves

The filings are free and public. The work is everything downstream of "download the XML."

SEC's fair-access rate limits

SEC.gov's developer resources page states it directly: "Current guidelines limit each user to a total of no more than 10 requests per second, regardless of the number of machines used to submit requests," and adds that SEC "reserve[s] the right to block IP addresses that submit excessive requests" (verified live, September 2026). The site also does not allow "unclassified" bots — its webmaster FAQ requires you to declare a User-Agent header in the form Company Name AdminContact@yourdomain.com, or requests get flagged as an "Undeclared Automated Tool." None of this is hard to comply with for a single script — but it is a real constraint the moment you want to backfill years of history or keep a poller running continuously, and getting it wrong risks your IP being throttled or blocked outright.

Form 4 XML has real parsing complexity

A Form 4 is an XML document with separate tables for non-derivative and derivative transactions, a transaction code (P, S, A, F, M, G, and others) on every row that changes what the row means, an indirect vs. direct ownership flag, and footnotes that can materially change how a transaction should be read — for example, a footnote clarifying that a sale (code S) was to cover tax withholding on a vesting grant, not a discretionary decision. Reading the raw fields without the footnotes and codes attached gets you data that looks structured and is quietly wrong.

Amendments can revise or void an earlier filing

When an insider needs to correct a Form 4 — a wrong share count, the wrong code, a transaction left off entirely — they file a Form 4/A referencing the original filing's accession number. A pipeline that ingests every filing as an independent row either double-counts the transaction (the original and the correction both counted) or keeps the stale original and never applies the fix. Handling this correctly means detecting the amendment relationship and reconciling the two records into one — logic that has nothing to do with downloading XML and everything to do with getting the resulting dataset right.

Ongoing maintenance is not a one-time cost

None of the above is a one-time build. SEC periodically changes EDGAR's schema versions and site structure, and a scraper built against today's markup or field layout can break silently against tomorrow's — you find out when your data goes stale or malformed, not when the change happens. A robust pipeline is commonly estimated at a few hundred lines of parsing logic before amendments are even handled, and that number only grows as you add monitoring for silent breakage.

The cheap-scraper marketplace: Apify and friends

You don't have to write the scraper yourself. Marketplaces like Apify host actors — hosted, pay-per-call scrapers — that already do the EDGAR fetching for you. A search of Apify's public store for Form 4 insider scrapers turned up more than a dozen live, independently priced listings (checked live, September 2026), with names like "SEC Form 4 Insider Trading Scraper" and "SEC Form 4 Insider Monitor," plus a separate set of listings covering congressional STOCK Act trading disclosures. Pricing follows the same shape across most of them: a small one-time run fee plus a per-result charge, which across the listings checked ranged from roughly $0.002 to $0.05 per row — call it $2 to $50 per 1,000 filings, varying a lot listing to listing.

These are a genuinely cheap way to solve "get me the raw rows" without writing a scraper. What they don't uniformly solve: whether footnotes and transaction-code meaning carry through correctly, whether a Form 4/A amendment is reconciled against the original filing or handed back as a second, double-counted row, and who maintains the actor when EDGAR changes something — that depends entirely on an independent operator you don't control, and listings on these marketplaces get created, repriced, and abandoned on their own schedule. Read each listing's description carefully; "one row per SEC Form 4 filing returned" is not the same guarantee as "one row per real-world transaction, amendments reconciled."

When scraping EDGAR yourself is the right call

Being honest about this is the point of the page. Scrape it yourself when:

• It's a one-off research pull. Checking what insiders at one company did last quarter doesn't justify a subscription or even an account.
• You're learning. Parsing a handful of real Form 4 filings by hand is a genuinely good way to understand the data model — codes, tables, footnotes and all.
• You want full control and don't mind the upkeep. If owning the fair-access limits, the parsing, and the amendment logic — and keeping all of it working when SEC changes something — is a feature to you, not a cost, that's a legitimate choice.

When a maintained API is the right call

• Production systems. Something other people or other services depend on staying up shouldn't depend on a scraper nobody is watching.
• Amendment-correctness matters. A naive scrape can show a trade that was later revised or voided by a Form 4/A — wrong in exactly the way that's hardest to notice, because the bad row looks perfectly well-formed.
• You need real-time delivery. Webhooks push a new filing to you within minutes of it landing on EDGAR; a scraper only knows something new exists on its next scheduled poll.
• Total cost of ownership, not just data cost. Free data isn't free engineering time — a scraper that breaks silently once a quarter and needs an afternoon to fix costs more, over a year, than most subscription plans.

Comparison at a glance

Pricing and policy details checked directly from sec.gov and the Apify store (September 2026); both change over time, so verify current terms before deciding.

What a maintained option looks like

To make the trade-off concrete rather than abstract, here is what Form4API specifically gives you instead of the scraping work above: parsed Form 4 (plus Form 144 and 13F-HR) as JSON with amendments reconciled — one record per real-world transaction, not per filing — real-time webhooks that fire within a minute or two of SEC acceptance, and a free tier of 500 requests/day (15,000/month), no credit card, covering every core endpoint and post-trade return field. For AI agents rather than application code, the same data is reachable through an MCP server (form4api-mcp on npm) exposing 35 tools and 6 ready-made research prompts, 22 of which work on the free plan.

This is one example among several maintained providers — see the buyer's guide for how to evaluate the category, not just this one option.

Frequently asked questions