The site-wide files

Three files, three lines each. All three are written out below on purpose: the shape is easy to get subtly wrong when it is left as “repeat for the other two”.

src/pages/llms.txt.ts

import { createGeoffyTextEndpoint } from "@geoffy/headless/astro";
export const GET = createGeoffyTextEndpoint(
  { siteKey: import.meta.env.GEOFFY_SITE_KEY },
  "llms.txt",
);

src/pages/llms-full.txt.ts

import { createGeoffyTextEndpoint } from "@geoffy/headless/astro";
export const GET = createGeoffyTextEndpoint(
  { siteKey: import.meta.env.GEOFFY_SITE_KEY },
  "llms-full.txt",
);

src/pages/agents.md.ts

import { createGeoffyTextEndpoint } from "@geoffy/headless/astro";
export const GET = createGeoffyTextEndpoint(
  { siteKey: import.meta.env.GEOFFY_SITE_KEY },
  "agents.md",
);

All three serve the same document

One body, three names. Three conventions exist for “the file an AI agent should read first”, so we answer to all three rather than making you pick. Mounting only one means agents looking for the other names find nothing.

A real file beats a dynamic route

src/pages/llms.txt.ts is a real file, so a catch-all such as src/pages/[...lang].astro cannot swallow it. That matters more than it sounds: a catch-all answering /llms.txt with your home page returns HTTP 200, which a crawler reads as a successful answer rather than a missing file.

Also check public/

A static file in public/ wins over everything. If you have a public/llms.txt from an earlier attempt, delete it. Otherwise your endpoint is never reached and the stale file is served forever, with no error anywhere.

robots-rules.txt is deliberately not accepted here

createGeoffyTextEndpoint refuses it. Every other name serves a whole file; the robots rules are a fragment you append to your own. Serving them as a whole file would replace your rules with ours. See crawler rules.

Caching

The endpoint answers with Cache-Control: max-age=300, so a browser or CDN in front of your site can keep a copy for 5 minutes. revalidateSeconds has no effect in Astro. On a prerendered site the file is read once, at build time, so your next build is the refresh; see how updates arrive.

Telling Geoffy which AI agent asked

A third argument makes the endpoint tell Geoffy which AI agent fetched the file, by its agent name and never a raw user agent:

export const prerender = false;
export const GET = createGeoffyTextEndpoint(
  { siteKey: import.meta.env.GEOFFY_SITE_KEY },
  "llms.txt",
  { forwardCrawler: true },
);

It works only on an endpoint rendered on request, so it needs prerender = false. It is off by default. Turn it on if you want these reads counted without the AI visit middleware.

If Geoffy is unreachable

The endpoint answers 503 with a Retry-After header, never a fabricated 404. A crawler retries a 503; a 404 tells it to drop the page from its index.