Crawler rules

Merge the Geoffy lines into your existing robots.txt. Do not replace your file — yours contains your own rules.

src/pages/robots.txt.ts

import type { APIRoute } from "astro";
import { fetchGeoffyText } from "@geoffy/headless";

const YOUR_RULES = `User-Agent: *
Disallow: /admin
`;

const YOUR_GLOBAL_RECORDS = `Host: https://yourdomain.com
Sitemap: https://yourdomain.com/sitemap.xml
`;

export const GET: APIRoute = async () => {
  const geoffyRules = await fetchGeoffyText(
    { siteKey: import.meta.env.GEOFFY_SITE_KEY },
    "robots-rules.txt",
  );

  // Your own rules survive an unreachable Geoffy. Never the other way round.
  const body = [YOUR_RULES, geoffyRules ?? "", YOUR_GLOBAL_RECORDS]
    .filter(Boolean)
    .join("\n");

  return new Response(body, {
    headers: { "content-type": "text/plain; charset=utf-8" },
  });
};

Delete public/robots.txt if you have one

A real file in public/ wins, so your endpoint would never run and your rules would silently stay as they were. This is the single most common reason “I added the route and nothing changed”.

Why createGeoffyTextEndpoint refuses robots-rules.txt

It deliberately does not accept that name. Every other name serves a whole file; this one is a fragment you append to yours. Serving it as a whole file would replace your rules with ours.

Where the block goes, and why it matters

robots.txt has two kinds of record and they are not interchangeable.

A group is a User-agent: line plus the rules under it, and a group must stay contiguous — a rule separated from its User-agent: by an unrelated line belongs to nothing. A global record — Host:, Sitemap: — applies to the whole file wherever it sits.

Our block contains a group. So it belongs with the other groups, and your own global records go last:

robots.txt

User-Agent: *            ← your groups
Disallow: /admin

# BEGIN Geoffy …         ← our block, including its own Sitemap: line
User-agent: Bytespider
# END Geoffy …

Host: https://yourdomain.com    ← your global records, at the end
Sitemap: https://yourdomain.com/sitemap.xml

Appending our block after your own Host: and Sitemap: lines produces a file that still parses, but not the way you meant it to.

What is in the block

• No User-agent: * group. Yours stays the only global policy, so appending can never override the disallows you wrote.
• # BEGIN / # END markers. If you ever paste it in statically instead of fetching it, replace everything between those markers on an update — never merge line by line.
• A Sitemap: line, once your namespace is mounted. That line is what makes the guides and product twins reachable to Googlebot and Bingbot, which read robots.txt and do not read llms.txt. Before the namespace is mounted the line is withheld, because until then the URL it would name does not answer.

On a static build this runs once

At build time. Your deployed robots.txt carries whatever we served during that build.

That is usually fine — the rules change rarely — but it has one consequence worth planning around: the Sitemap: line appears only if the namespace was already mounted when you built. Mount first, then deploy, or your robots.txt will be missing that line until the build after next.