250,000 FDA pages, a Claude summary pipeline, and a cache bug that hid for a month

I built DeviceClearance.com solo, a site over the FDA’s public 510(k) clearance and recall data, live since April 23, 2026 and indexed by Google and Bing. Astro on Cloudflare Workers + D1: roughly 175,000 device clearance pages, 38,000-plus recalls, 34,000-plus companies, 4,700-plus product codes, over 250,000 pages total. I have professional FDA regulatory-affairs experience; the bet is that credentialed structure plus real prose beats “cleaner spreadsheet” as a reason to visit a wrapper around data anyone can already pull from openFDA directly.

What I built

Every 510(k) record is machine data - device name, product code, regulation number, decision date. Accurate, and unreadable as a page. I built two paths to turn it into prose: generate-summaries.mjs, a resumable script calling Claude Haiku 4.5 with a cached-ready system prompt at roughly $0.001 a record, and a second path where I hand-wrote 50 summaries in the same coding session at Opus quality for zero marginal cost. I shipped the 50 hand-written ones - they lead the 50 most recent /510k/ pages - and left the Haiku pipeline built but unrun against the rest of the corpus: I deliberately declined the API spend until the format proved itself. I wanted the D1 table, the renderer, and the page layout proven before committing API budget to a format I hadn’t validated.

The rest of production hardening: stateless HMAC-SHA256 unsubscribe tokens verified with a hand-rolled constant-time compare instead of ===, so a timing attack can’t fish out a valid token; RFC 8058 one-click unsubscribe headers so Gmail and Outlook show a native button; a weekly digest firing every Monday since April 27.

The architecture decision that mattered

The site started on Cloudflare Pages and hit its 20K-files-per-deploy limit on day one of scaling past a few thousand pages. Static generation was structurally impossible at over 250,000 pages, so I pivoted the stack to Workers + D1, rendering every page from a request instead of pre-building it. That solved deploy size and created a new problem: every request now hits D1 and re-renders on a platform billed per invocation. Uncached isn’t a viable steady state at that count, so the edge cache design mattered as much as the SSR pivot itself: caches.default in Astro middleware, try/catch around match and put so a cache failure degrades silently instead of 500ing the page, ctx.waitUntil() so a cache write never blocks the response, and an X-Cache: HIT|MISS|BYPASS|SKIP|ERROR header on every response so I could verify behavior with curl instead of trusting a dashboard. Same instinct behind the rate limiter added later: per-IP counters in isolate memory rather than KV (the free tier’s ~1,000 writes/day would be gone in one scraper burst), scoped only to the paths actually expensive to serve.

The incident

Getting the cache working took three failed PRs before it even ran without crashing, and a fourth before it cached anything. PR #3 shipped the initial caches.default wiring; PR #5 re-added it alongside the weekly-digest and unsubscribe flow after an earlier revert; PR #9 was a diagnostic patch meant to explain why the first two hadn’t worked. All three are named in middleware.ts’s own code comment as the ones that crashed: sharing Astro’s SSR TransformStream body between a cache.put() and the client response creates a streaming deadlock, and the sync await tore the isolate down hard enough that a try/catch couldn’t save the request. PR #10 fixed that crash by buffering the response body to an ArrayBuffer first, so the write and the client response each get an independent copy.

The site stopped 500ing. It still wasn’t caching. Every request was still a MISS, with no error to chase - the cache block ran clean, set its headers, and did nothing. It took over a month to find why: Astro 6 had quietly removed Astro.locals.runtime.ctx, the property the cache block’s waitUntil call depended on. Reading it threw, and the block’s own try/catch, added for safety, silently swallowed that exception, so cache.put() never executed. PR #32 (May 28) fixed it by switching to Astro.locals.cfContext, verified live: /510k/ went MISS → HIT → HIT with CF-Cache-Status: HIT. From first attempt to confirmed working: five weeks, four PRs, and a monitoring gap the whole time, because the failure mode produced zero errors to alert on.

The other production incident had a cleaner signature. On May 31, one datacenter IP pulled roughly 56,000 catalog pages in an hour - about 16 requests a second, nearly all cache misses on unique URLs - and drove close to a third of that burst to 500s and 504s. The in-Worker rate limiter, 120 requests per IP per 60-second window, shipped the same day.

What it proves

The AI content layer covers 50 of roughly 175,000 device pages right now, by design - I proved the format earned its place before spending a cent summarizing records I hadn’t validated. The Haiku pipeline is built, at a cost cheap enough that running it against the full corpus is a decision, not an engineering project. Same posture behind the cache work: cheap, reversible checks before committing spend or scope, and honesty about what actually shipped versus what merely deployed. The cache bug is the sharper version of that discipline - a try/catch added specifically to make failures safe was the exact thing that hid a real bug for five weeks. Fixing it took the same instinct as the summary pipeline: don’t trust something works because it didn’t crash, verify it with a header you can curl.

Repo: web/deviceclearance, live at deviceclearance.com. PR history is 50+ deep on GitHub as of this writing.