Back to Seatbelt

Check your AI-built app before you show anyone

The short version

  • No secret keys in your public build. Anything that leaked gets rotated, not just removed.
  • Row level security on every table, with policies that actually check who is asking.
  • Authorization enforced on the server. Every API route checks the caller, not just the UI.
  • No admin, debug, export, or delete routes left public.

Seatbelt reads all seven surfaces in one pass and returns a plain English Ship Read, or verify each one by hand below.

Run a Ship ReadSet up /seatbelt in your agent

Your agent says it is done. Run these four checks before you let another human near it: a cofounder, a beta tester, or your first customer. Showing someone is not the same as publishing a URL, but an exposed key or open database rule can leak just as fast. Veracode measured what done means: its 2025 report found 45% of AI-generated code introduced a known vulnerability, and its Spring 2026 update found the security pass rate stuck near 55% even after a year of newer, more capable models. In May 2026, security firm RedAccess scanned about 380,000 apps built on Lovable, Base44, Replit, and Netlify and found roughly 5,000 leaking sensitive data. Axios verified named cases. The pace is accelerating too: Georgia Tech's Vibe Security Radar tracked AI-attributed CVEs in public advisories at 6 in January 2026, 15 in February, and 35 in March alone (74 cumulative through March, a confirmed lower bound). Georgia Tech research news publishes the monthly counts. AI writes code that runs. It does not always write code that is safe. The four checks below catch the failures that actually hurt launches. Do each one by hand with the steps on this page, or run them all in one pass.

Reviewed July 2026

  1. 01

    No secret keys in your public build

    When an agent needs an API to work, the fastest path is pasting the key right where the call happens. That is how live payment keys end up in browser code, readable by anyone who opens dev tools. A leaked key can mean real charges on your account and a database that answers to strangers. Exposed secrets are the most common real world failure in AI-built apps.

    Check it yourself

    • Open your live site, open the browser dev tools, and search the loaded JavaScript for key shaped strings: sk_live, service_role, AKIA, api_key.
    • Search your repo for committed .env files and for env values copied into the build output.
    • If a real key is exposed, removing it from the code is not enough. Bots scan public repos for fresh keys around the clock, so treat any key that was public even for a few seconds as compromised. Rotate it with the provider before the bots find it. Old commits and old copies of the build still hold the value.

    What Seatbelt reads: the secrets surface. Key patterns and env looking names anywhere in the build you hand it. A secret in the public build is a must fix, and with the push gate wired that push parks on a pull request instead of main until the key is out and rotated. Publishable keys, like Stripe pk_ values, are expected in front end code and stay noted.

  2. 02

    Row level security on every table

    App platforms hand the browser a public database key and trust the database's own rules to keep users apart. Row level security is that rule set. When it is off, or the policy never checks who is asking, every row in the table is one API call away from any visitor. RLS can also be toggled on while a policy still allows everything, or while INSERT and UPDATE policies omit WITH CHECK: reads look scoped in the dashboard, but a signed-in user can still write rows as someone else. Real apps have leaked every user's private messages exactly this way.

    Check it yourself

    • Supabase anon key: the key in your client bundle is public by design. Hiding it is not the fix. Row level security is. If you searched can supabase anon key be exposed, the bug is usually missing RLS, not the key itself.
    • Supabase: check each table in the public schema. Row level security on for every one, with a policy that filters by the signed in user's id rather than allowing everything.
    • Grep supabase/migrations/ (or your SQL files) for CREATE TABLE. Every app table needs a matching ALTER TABLE … ENABLE ROW LEVEL SECURITY (or equivalent in the same migration). Lovable often creates tables without turning RLS on. A Jul 2026 audit of 50 deployed Lovable repos found 89% missing RLS on app tables (DEV tgoldi, Jul 2026).
    • On INSERT and UPDATE policies, open the SQL. A USING clause alone is not enough: confirm the same policy has WITH CHECK so a user cannot write a row with another user's id.
    • Firebase: read your Firestore and Storage rules. allow read, write: if true means open to anyone. It is the classic paste-over for expired test mode, and it ships to production more often than anyone admits.
    • The blunt test: create a second account and use it to request the first account's data through your app's API. If it answers, the rules are not doing their job.

    What Seatbelt reads: the databases surface. Code cues only: a service key shipped to the browser, rules files that allow everything, database URLs in public code, raw SQL built by string glue, database files sitting in public folders, and migrations with tables but no ENABLE ROW LEVEL SECURITY when anon key material is present (RLS-never tail on the detector roadmap). The careless ones are must fix. Seatbelt never connects to your live database. It reads the build you hand it.

  3. 03

    Authorization on the server, not just the screen

    Agents are good at hiding buttons from signed out users. Hiding is not checking. If the API route behind the screen never asks who is calling, anyone can call it directly and skip your UI entirely. Next.js server actions are a second gap: the page can verify the session at render while the action file deletes or updates with no ownership check. The page check runs once. The action runs on every direct POST. A common AI auth pattern decodes a JWT payload without verifying the signature first, so a forged token can pass the middleware. Login handlers that return different messages for unknown email vs wrong password leak which addresses are registered. Forgot-password flows often ship plain reset tokens in the database with no expiry or single-use guard. A separate miss is unlimited login attempts: a 2026 study of AI-built apps found roughly 80% lacked rate limiting on at least one auth endpoint, so brute-force and account-enumeration scripts can run as fast as the network allows. A 2026 study of 100 AI-built apps found roughly one in five had API endpoints with no authentication at all.

    Check it yourself

    • Sign out, then call your API routes directly with curl or the address bar. Anything that returns user data while signed out is open.
    • Sign in as a test user, find a request with an id in it, and swap the id for another user's. If their data comes back, the route checks that you are signed in but not what belongs to you.
    • Read each API handler. The auth check belongs at the top of the handler, before the database read. A check in the page component protects nothing.
    • Next.js: open each file with 'use server' that calls .delete(, .update(, or Prisma writes. The action should import your session helper and verify the caller owns the row before the mutation. Page protected does not mean action protected.
    • JWT expiry: search auth routes for jwt.sign( or SignJWT. Each call needs expiresIn (for example '7d') or setExpirationTime. A bare jwt.sign({ userId }, secret) with no expiry means a stolen token works forever. Practitioner Simon Roses reports this in roughly half the AI-built apps he reviews (May 2026). Founders often Google jwt token that never expires before they know the fix name.
    • In JWT middleware, search for jwt.decode or decode( on the token. The same function block should call jwt.verify (or jwtVerify) with an explicit allowed-algorithms list before you trust the payload. Decode-then-use-then-verify order is still a miss.
    • Login enumeration: try a random email and a known email with a wrong password. If the API returns different messages, you are leaking which addresses are registered. Both paths should return the same generic invalid-credentials response. See auth account enumeration for why it matters.
    • Password reset: open your forgot-password handler and user schema. A plain resetToken or passwordResetToken column without a hash, an expiresAt window, and a used or invalidated flag is account takeover if the row leaks. Practitioner audits in 2026 found reset tokens with no time bound in several of ten AI-built codebases reviewed.
    • Rate limiting: open each login, register, and forgot-password route (or app/api/auth/ handler). Search the file and middleware.ts for rateLimit, @upstash/ratelimit, express-rate-limit, or an edge limiter. If bcrypt compare or JWT sign runs with no throttle before it, attackers can brute-force passwords or spray sign-ups. A common baseline is about 5 attempts per minute per IP on auth routes.

    What Seatbelt reads: the login surface. Seatbelt notes where sign in lives and flags identity APIs and data handlers that read a store with no auth guard in the handler body. It also soft-flags distinct login error strings in the same handler, plain reset-token storage without expiry or single-use fields, and jwt.sign or SignJWT without an expiry option in the same handler. Auth rate-limit cues are on the detector roadmap: use the hand-check above until repo read ships for that class. Open looking routes are marked worth a look, with the file and line, so your agent knows exactly where the check goes.

  4. 04

    No admin, debug, or export routes left public

    Builds accumulate scaffolding: an admin page for testing, a debug route that dumps state, an export endpoint meant for one internal use, a delete helper from an early reset flow. When an agent adds "export user data as CSV" late in a long chat, it often ships a handler that streams the whole table with no session check. The feature looked scoped in the prompt. The endpoint is public. Baudr (Mar 2026) is the first named AI-built breach: streamer Grenbaud launched an AI-built social network live on Twitch (~€40, no security review). Within hours a visitor typed /admin and reached the admin panel with no authentication. Bulk account deletes, PII download, and fraudulent messages followed (DEV Community, Mar 2026). A second class hides in link-preview and image-proxy features: a URL from the query string or form flows straight into fetch() on the server with no host allowlist, so your app can be tricked into calling 169.254.169.254, localhost, or internal VPC addresses. Georgia Tech's Vibe Security Radar lists SSRF among top AI-attributed CWE classes in 2026 advisories. Agents rarely clean these up on their own, and one public delete route can end a project in an afternoon.

    Check it yourself

    • Search your routes for admin, dashboard, panel, debug, test, export, dump, backup, reset, and delete.
    • While signed out, try /admin, /dashboard, and /panel on your staging URL. If any loads without a login wall, treat it as urgent. Baudr (Mar 2026) fell exactly this way.
    • Open each export, dump, backup, or download-all route while signed out. If it returns rows, the handler needs getServerSession, getToken, or your auth helper before the query.
    • SSRF: in server actions and route handlers, search for fetch( or axios.get( where the target URL comes from searchParams, formData, or the JSON body (url, target, webhookUrl, and similar). The handler should allowlist hosts, block private IPs and link-local ranges, and set redirect: "error" on outbound fetches. A UI that never shows a URL field is not a defense: server actions are direct POST endpoints.
    • Open next.config (or next.config.mjs). images.remotePatterns with hostname: "**" turns image optimization into an open proxy. Tighten patterns to hosts you actually serve.
    • Anything that changes or exports data goes behind auth. Anything destructive should not ship at all.
    • Include the neighbors: a database admin UI deployed next to the app counts as an open admin route.

    What Seatbelt reads: the risky shortcuts surface. Unguarded /admin paths, open admin pages, debug leftovers, and export or delete routes get flagged, and destructive ones are a hard stop that parks the push on a pull request until they are gone. Outbound-fetch SSRF cues are on the detector roadmap: use the hand-checks above until repo read ships for that class.

Two more that bite after launch day

The four checks above cause the loudest failures. These two cost real money and trust a little later, so Seatbelt reads them in the same pass.

  • Payments and money

    Checkout that half works is worse than no checkout. Stripe shaped pieces, amounts set in the browser, and unverified webhooks get flagged worth a look, so you run a real test charge before a customer does. A separate cost-abuse miss: AI apps that gate paid generation with a browser-only quota counter while the server route calls OpenAI with no throttle.

    Check paid generation yourself

    • Search client components for localStorage.getItem or sessionStorage reads of dailyUsage, generationQuota, or similar counters that run immediately before fetch("/api/generate"), fetch("/api/chat"), or another paid LLM route.
    • Open the matching API route (app/api/generate/route.ts or equivalent). Confirm a rate limiter, usage row, Stripe meter, or session auth runs before the model SDK call. If the handler only checks process.env.OPENAI_API_KEY and posts, anyone can POST directly and burn your bill.
    • Quick test: call the generate endpoint with curl while signed out. If it returns model output with no auth header, the client counter was theater. Enrichlead (Mar 2026) learned this the hard way: subscription enforcement lived only in the browser; users cleared devtools and maxed API keys (Autonoma failure taxonomy, Mar 2026). Authorization is enforced server-side, not in the browser.

    What Seatbelt reads: the payments surface. Distinct from Stripe amounts set in the browser (client-priced-checkout) and from auth rate limits on login routes. Client-only storage gates before paid LLM routes are on the detector roadmap: use the hand-check above until repo read ships for that class.

  • Customer data

    The report notes where real people's data lives: orders, profiles, uploads. Loose upload rules, and customer emails or passwords printed to the browser console, get flagged worth a look.

    Check uploads yourself

    • Firebase: search your repo for getDownloadURL imported from firebase/storage (or .getDownloadURL( calls) in upload, profile, or verification handlers. A VibeWrench study of 100 Firebase apps found 52% used it. The helper mints a permanent public signed URL that bypasses Storage security rules even when rules require auth. ID photos and verification selfies are the highest-stakes case.
    • Prefer server-side signed URLs with a short expiry, or upload through a Cloud Function that returns a time-limited link. Rules that look locked down in the dashboard do not protect a link that was already minted and shared.
    • Quick grep: grep -rn getDownloadURL src/ (or your app folder). Every hit in an upload flow deserves a read before you show anyone.

    What Seatbelt reads: the customer data surface. Repo read flags getDownloadURL in upload and profile paths without a nearby server-signed-URL pattern. Distinct from open Storage rules (allow read, write: if true) and from console-log leaks.

Questions

Do I need to check before I show a cofounder or beta tester?

Yes, if anything sensitive could leak from what you show them. Show is not publish: a screen share, a staging link, or a local demo can still expose secret keys in the build or open database rules that anyone with access can reach. Run the four checks above, or one Ship Read, before you let another human near it, whether or not you have published yet.

How do I check my AI-built app before launch?

Lock down four things: no secret keys in the public build, row level security on every table, authorization enforced on the server, and no admin or debug routes left public. Each section above shows how to verify one by hand. A Seatbelt Ship Read checks all of them in one pass and explains what it found in plain English.

What security checks should I run before Product Hunt?

Checklist sellers already publish PH-specific lists worth respecting. VibeDoctor's Product Hunt checklist names 10 manual checks: secrets in the build, auth on every API route, input validation, rate limiting, security headers, NEXT_PUBLIC_ and VITE_ misuse, SSL, error handling, npm audit, and basic tests. SableOffensive (Sable) runs a credit-based agent pentest console with 150 free credits, five specialists (pen-scout through pen-compliance), and a fixed-scope Pre-Launch $29 tier (Jul 2026). The legacy five-minute curl audit blog and sable/agent scan terminal still automate public API routes, cross-user access, headers, and a Supabase anon-key table probe. Indie guides often add curl recipes before you post. Those are real pre-launch hygiene. Seatbelt is a different moment: seven surfaces in plain English at the git push boundary, optional deny hooks, and a shareable Ship Read link at /r/:id that anyone you send it to can verify for themselves, without trusting your scan vendor or a local PDF only you have seen.

How common are leaks in AI-built apps?

Common enough that a third party measured it. In May 2026, the security firm RedAccess scanned about 380,000 apps built on Lovable, Base44, Replit, and Netlify and found roughly 5,000 leaking sensitive data. About 40% of the vulnerable apps exposed medical records, financial data, corporate strategy documents, or customer service chat transcripts. Axios independently verified named cases, including a UK clinical trials tracker and a Brazilian bank's internal financial records. RedAccess traced the root cause to defaults: these platforms start new projects publicly accessible, and many builders never change that. The checks on this page are how you verify your own app by hand, or a Ship Read reads them in one pass.

How fast is AI-generated code producing new vulnerabilities?

Accelerating. Georgia Tech's Vibe Security Radar counted 74 AI-attributed CVEs through March 2026 (6 in January, 15 in February, 35 in March alone), drawn from public advisories. That is a confirmed lower bound: researcher Hanqing Zhao notes Copilot inline suggestions leave no metadata, so the real count may be 5 to 10 times higher. Seatbelt reads the seven surfaces that hurt launches in your build before you share the link. It is not a CVE dependency audit. Tools like Snyk own that lane. See Georgia Tech's Vibe Security Radar for the monthly breakdown.

Row level security is on. What can still go wrong?

Three subtle misses that look safe in the dashboard: a policy that allows everything (USING (true) on Supabase or allow read, write: if true on Firebase) even though the RLS toggle is on; an INSERT or UPDATE policy with USING but no WITH CHECK, so reads look scoped but a signed-in user can write rows as another user; and auth middleware that calls jwt.decode without jwt.verify in the same handler, so a forged token can pass the check. Run check 02 and check 03 above by hand, or one Ship Read on your repo export.

I'm building a dating or safety app. What should I check first?

Apps that collect verification selfies, government IDs, or private safety reports sit in the highest-stakes Firebase bucket class. Security.org summarized the Tea App breach: "The promise was privacy and safety. Then, in July 2025, that promise collapsed." Tea left Firebase Storage open. About 72,000 images leaked, including roughly 13,000 verification IDs. Read your Firestore and Storage rules for allow read, write: if true. Chat & Ask AI (Jan 2026) was the same open-rules class on Firebase Realtime Database at chat scale: roughly 300 million private messages exposed before patch (404 Media). Seatbelt reads rules files in your repo export, a static read, not a pentest. Run check 02 above by hand or one Ship Read before you let another human near it.

Do I get the full report?

Yes. Every flag, the file and line, and the suggested fix, not a locked score. If the scan finds almost nothing, the report says so honestly.

Does Seatbelt scan my live site?

No. Seatbelt reads the build you hand it, a project folder or a ZIP export. It is a static read of your code, not a pentest, and it never touches your live site or your customers' data.

I build on Lovable, Bolt, or Replit. Does this apply to me?

Yes. Seatbelt is not tied to one platform. It reads the code itself, so it works wherever you build. On Lovable, clone the GitHub repo your project syncs to (all plans). ZIP export needs Business. On Bolt, use the Bolt.new security checklist before Netlify publish. On Replit, sync to private GitHub and scan the clone, never a Repl URL. Replit always runs a pre-publish scan, but Block publishing of critical vulnerabilities is opt-in (Deploy → Advanced, off by default), and dismissed findings unblock publish (Replit docs, July 2026). That GitHub export path is still ungated. See Get started for the day-to-day setup in your agent.

My login page looks fine. Can a server action still be open?

Yes. A common Next.js miss: the page calls verifySession at render, but a 'use server' action file deletes or updates with no ownership check. The page check runs once at render. The action runs on every direct POST. Open each actions.ts file that mutates data and confirm your session helper runs inside the action before the database call. Run check 03 above by hand, or one Ship Read on your repo export.

Do different login error messages leak which emails exist?

Yes. When an unknown email gets a different message than a wrong password, attackers can script a harvest list of registered addresses before they spray passwords. The fix is one generic invalid-credentials response for both paths, same status code and JSON shape. Password-reset flows that reveal whether an email is registered vs whether a token is wrong create the same harvesting shape. Try a random email and a known email with a wrong password: if the messages differ, fix the handler. Run check 03 above by hand, read auth account enumeration, or one Ship Read on your repo export.

Does my login route need rate limiting?

Yes, if attackers can try unlimited passwords or spam sign-up on your auth endpoints. A 2026 study of AI-built apps found roughly 80% lacked rate limiting on at least one auth route. AI scaffolds often ship bcrypt and JWT sign-in with no throttle middleware. Open each login, register, and forgot-password handler and confirm a limiter runs before credential checks: Upstash Ratelimit, express-rate-limit, or an edge rule on Cloudflare or Vercel. A common baseline is about 5 attempts per minute per IP on auth. Run check 03 above by hand. Auth rate-limit cues are on the detector roadmap.

Can a link preview or image proxy fetch internal URLs?

Yes. AI scaffolds often ship link-preview, image-proxy, or import-from-URL features where a url query param or form field flows straight into fetch() on the server with no host allowlist. That turns your app into a confused deputy to cloud metadata endpoints, localhost, and internal VPC addresses. In Next.js server actions and route handlers, confirm user-controlled URLs pass through validateUrl, an ALLOWED_HOSTS list, private-IP blocking, and redirect: "error" before the outbound call. Also open next.config and reject images.remotePatterns with hostname: "**". Run check 04 above by hand. Repo SSRF cues are on the detector roadmap.

Can someone open /admin on my app without logging in?

Yes, and it has already happened in production. Baudr (Mar 2026): Italian streamer Grenbaud launched an AI-built social network live on Twitch (~€40, no security review). Within hours a visitor typed /admin in the browser and reached the admin panel with no authentication. Bulk account deletes, PII download, and fraudulent messages followed (DEV Community). Search your codebase for routes containing admin, dashboard, or panel. Open each in the browser while signed out. Server-side session or middleware checks must run before any admin UI or API. Run check 04 above by hand, or one Ship Read on your repo export. Seatbelt soft-flags unguarded /admin paths on repo read.

Firebase Storage rules look fine. Can uploaded files still leak?

Yes. A VibeWrench study of 100 Firebase apps found 52% called getDownloadURL() after upload. That helper mints a permanent public signed URL that bypasses Storage security rules even when rules require auth. ID verification photos, health records, and profile uploads are the highest-stakes case. Search your repo for getDownloadURL from firebase/storage in upload handlers. Prefer server-side signed URLs with a short expiry or a Cloud Function upload path instead. Seatbelt flags getDownloadURL in customer-data paths on repo read. Run the customer data hand-check above by hand, or one Ship Read on your repo export.

What about slopsquatting or malicious npm packages?

Real risk when AI agents write install commands or fetch skills. Typosquatting is a human typo on a real name. Slopsquatting is when the model invents a package at codegen (the Jan 2026 react-codeshift incident hit 237 repos). HalluSquatting is when an agent hallucinates a repo or skill at fetch time with no paste (Jul 2026 research: up to 85% on repos and 100% on skills for trending names; Cursor was among the agents tested). Run Socket or Snyk on dependencies. Seatbelt reads your repo export on six launch surfaces before you share. It does not registry-lookup packages. See the full supply-chain ladder on Compare tools for dual-namespace collisions and complementary layers.

Does localStorage stop users from maxing my AI API bill?

No. A client-only dailyUsage or generationQuota counter in localStorage or sessionStorage is not authorization. Anyone can clear storage, edit the counter in devtools, or call your /api/generate route directly. The server handler must enforce quota, rate limits, or session auth before the OpenAI or Anthropic call runs. Enrichlead (Mar 2026) shipped subscription checks only in the browser; users bypassed paywalls with devtools and maxed API keys (Autonoma failure taxonomy, Mar 2026). Authorization is enforced server-side, not in the browser. Search your repo for localStorage before fetch to /api/generate or /api/chat, then open the matching API route. Run the payments hand-check above by hand. Payments quota cues are on the detector roadmap.

Can my Supabase anon key be exposed? Is that the bug?

Usually no. The Supabase anon key is public by design. It is meant to ship in your browser bundle. Row level security is the protection layer, not hiding the key. Founders search can supabase anon key be exposed or supabase rls disabled when the real failure is RLS never turned on: Lovable and other stacks create tables in migrations but skip ENABLE ROW LEVEL SECURITY, so any logged-in user can read every row through the public API. That is distinct from a wide-open USING (true) policy when RLS is toggled on, and from having no policy files at all. Run check 02 above: confirm RLS is enabled on every table in your migration SQL, not just in the dashboard toggle. Seatbelt reads migration files in your repo export for missing RLS cues. It does not probe your live Supabase project.

Did I create a JWT token that never expires?

Maybe. A common AI login scaffold calls jwt.sign or SignJWT without expiresIn or setExpirationTime, so a stolen token stays valid forever. Practitioner Simon Roses reports seeing this in roughly half the AI-built applications he reviews (May 2026). Founders Google jwt token that never expires or create jwt token that never expires before they know the fix name. Open your login or session handler on check 03: confirm every jwt.sign includes an expiry (for example expiresIn: '7d') or that SignJWT calls setExpirationTime. Seatbelt flags jwt.sign and SignJWT without expiry in auth routes on repo read.

Verified by Seatbelt badge

After a cleared Ship Read, embed the badge on your site. Visitors who want their own check land on the app with ?ref=badge tracked.

Verified by Seatbelt
Copy embed HTML
<a href="https://app.withseatbelt.com/app?ref=badge" target="_blank" rel="nofollow noopener noreferrer" title="Verified by Seatbelt: independent scan report"><img src="https://www.withseatbelt.com/badge.svg" width="200" height="51" alt="Verified by Seatbelt" loading="lazy" decoding="async" /></a>

Earned badges on cleared reports link to your live proof URL with ?ref=badge. See a sample report for the full pattern.

For agent fleets

Seatbelt is an enforced human-in-the-loop layer between agents: one agent's output is evidence, not authorization. The next agent builds only after a deterministic seven-surface read. You still approve blocks and accept risks. Wire it once with /seatbelt init on Get started.