Crawler contact · Undertow
Foretop Undertow crawler
This crawler reads a small reviewed catalogue of public subprocessor disclosure pages so Foretop can detect additions, removals, and disclosed region or purpose changes with dated evidence.
Deployed · not currently scheduled
Public channel: Do not include secrets, credentials, or customer data. Describe the public page and the correction, access concern, or opt-out request only.
How requests are bounded
The contact URL in the crawler User-Agent resolves to this page. Its collection boundary is deliberately narrower than a general web crawler.
Reviewed URLs only
It requests the exact disclosure URL in Undertow's reviewed catalogue. It does not discover links, search a domain, submit forms, or crawl unrelated paths.
Honor the publisher's boundary
Before a page request it checks robots.txt. It validates redirect targets, rejects private or unsafe network addresses, uses bounded timeouts, and spaces vendor requests within an execution.
Avoid downloading unchanged pages
When a prior extraction is still compatible, the crawler sends an ETag condition. A confirmed 304 response reuses the earlier immutable evidence instead of uploading or extracting the same page again.
What is retained
Raw public evidence
A successful public response may be stored in private Supabase Storage under a content-addressed path. Postgres retains the source URL, retrieval time, HTTP state, content hash, storage locator, and bounded error state.
Structured history
Undertow records the extraction model plus prompt and schema hashes, validated subprocessor rows, and structural changes linked to both the previous and current snapshots. A collection or extraction failure remains unknown; it is never treated as a clear result.
Logs and credentials
The job never logs page bodies, credentials, storage tokens, signed URLs, or prompts containing page content. Credentials are supplied to the runtime through secret bindings and are not part of the retained evidence.
Request a correction or opt out
If you operate a listed page and want a URL corrected, request behavior investigated, or future requests stopped, open a public GitHub issue. Include the public URL and the requested action. Do not include secrets, credentials, customer data, private logs, or security-sensitive detail.