Crawler rules

Merge the Geoffy lines into your existing robots.txt. Do not replace your file — yours contains your own rules.

Serve robots.txt from a route handler, so your file is text you control:

app/robots.txt/route.ts

import { fetchGeoffyText } from "@geoffy/headless";

const YOUR_RULES = `User-Agent: *
Disallow: /admin
`;

const YOUR_GLOBAL_RECORDS = `Host: https://yourdomain.com
Sitemap: https://yourdomain.com/sitemap.xml
`;

export async function GET() {
  const geoffyRules = await fetchGeoffyText(
    { siteKey: process.env.GEOFFY_SITE_KEY! },
    "robots-rules.txt",
  );

  // `null` means Geoffy was unreachable. Serve YOUR rules anyway — dropping your own
  // disallows because our API had a bad minute is the worse failure.
  const body = [YOUR_RULES, geoffyRules ?? "", YOUR_GLOBAL_RECORDS]
    .filter(Boolean)
    .join("\n");

  return new Response(body, {
    headers: {
      "content-type": "text/plain; charset=utf-8",
      "cache-control": "public, max-age=300, stale-while-revalidate=86400",
    },
  });
}

Delete app/robots.ts if you have one

Two files cannot both answer /robots.txt.

Why not app/robots.ts, the typed metadata convention

It cannot carry what we send. That file must return a MetadataRoute.Robots object, and what we serve is robots.txt text; there is no string field to append it to.

Parsing our text into rule objects loses two things that matter:

• the # BEGIN Geoffy / # END Geoffy comments, which are how the block stays identifiable and removable — the typed API cannot express a comment at all;
• any directive we add later that the type does not model, which would be dropped in silence.

The route handler passes our bytes through unchanged, which is the only version of this that stays correct without you re-reading this guide.

Where the block goes, and why it matters

robots.txt has two kinds of record and they are not interchangeable.

A group is a User-agent: line plus the rules under it, and a group must stay contiguous — a rule separated from its User-agent: by an unrelated line belongs to nothing. A global record — Host:, Sitemap: — applies to the whole file wherever it sits.

Our block contains a group. So it belongs with the other groups, and your own global records go last:

robots.txt

User-Agent: *            ← your groups
Disallow: /admin

# BEGIN Geoffy …         ← our block, including its own Sitemap: line
User-agent: Bytespider
# END Geoffy …

Host: https://yourdomain.com    ← your global records, at the end
Sitemap: https://yourdomain.com/sitemap.xml

Appending our block after your own Host: and Sitemap: lines produces a file that still parses, but not the way you meant it to.

What is in the block

• No User-agent: * group. Yours stays the only global policy, so appending can never override the disallows you wrote.
• # BEGIN / # END markers. If you ever paste it in statically instead of fetching it, replace everything between those markers on an update — never merge line by line.
• A Sitemap: line, once your namespace is mounted. That line is what makes the guides and product twins reachable to Googlebot and Bingbot, which read robots.txt and do not read llms.txt. Before the namespace is mounted the line is withheld, because until then the URL it would name does not answer.

It is cached for an hour, and that will confuse you at least once

fetchGeoffyText caches its fetch for revalidateSeconds, 3600 by default. So a change on the Geoffy side — mounting the namespace, publishing your first guide — does not reach your robots.txt until the window turns over.

Mounting the revalidate route purges the tag and makes it seconds. While you are integrating, pass a short one:

await fetchGeoffyText({ siteKey, revalidateSeconds: 10 }, "robots-rules.txt");

If your robots.txt looks stale in next dev after you already fixed something, it is this — not your route.