robots.txt

Publish crawl rules and point search engines to the generated sitemap.

Included in starter templates

This functionality is already configured in the starter templates. Follow the complete setup in the Quickstart.

Publish robots.txt at the site root to tell crawlers that documentation pages may be indexed while keeping Heyo Docs' internal Markdown resource route out of search results. Every generated template serves the same response at /robots.txt:

text

User-agent: *Allow: /Disallow: /__heyo-docs/Sitemap: https://docs.example.com/sitemap.xml

Set siteUrl in heyo-docs.config.ts before deployment. It is used for the absolute sitemap URL. During a preview, the route falls back to the origin of the incoming request.

heyo-docs.config.ts

export default heyoDocs({
  siteUrl: "https://docs.example.com",
  // ...the rest of the configuration
});

robots.txt is advisory: it does not protect a route or remove an already public document from the web. Keep private content outside the published documentation directory and protect private application routes separately.

React Router

Create a project

The React Router template includes the resource route and prerenders it for static deployments.

bun create @heyo-sh/heyo-docs my-docs --template react-router

Manual, minimal configuration

app/routes.ts

route("robots.txt", "routes/robots.ts"),
route("*", "routes/docs.tsx"),

Return a plain-text response from the route. The configured site URL is used when available, so the same implementation works in local previews.

app/routes/robots.ts

import type { LoaderFunctionArgs } from "react-router";

import config from "../../heyo-docs.config";

export function loader({ request }: LoaderFunctionArgs) {
  const siteUrl = config.siteUrl ?? new URL(request.url).origin;

  return new Response(
    [
      "User-agent: *",
      "Allow: /",
      "Disallow: /__heyo-docs/",
      "",
      `Sitemap: ${siteUrl}/sitemap.xml`,
      "",
    ].join("\n"),
    { headers: { "content-type": "text/plain; charset=utf-8" } },
  );
}

For static hosting, include /robots.txt in the array returned by prerender() in react-router.config.ts.

Next.js

Create a project

The Next.js template includes an App Router Route Handler at app/robots.txt/route.ts.

bun create @heyo-sh/heyo-docs my-docs --template next

Manual, minimal configuration

After exposing config from the server-only docs helper, add this handler:

app/robots.txt/route.ts

import { config } from "../lib/docs";

export function GET(request: Request) {
  const siteUrl = config.siteUrl ?? new URL(request.url).origin;

  return new Response(
    [
      "User-agent: *",
      "Allow: /",
      "Disallow: /__heyo-docs/",
      "",
      `Sitemap: ${siteUrl}/sitemap.xml`,
      "",
    ].join("\n"),
    { headers: { "content-type": "text/plain; charset=utf-8" } },
  );
}

Astro

Create a project

The Astro template emits the root endpoint from src/pages/robots.txt.ts.

bun create @heyo-sh/heyo-docs my-docs --template astro

Manual, minimal configuration

Add an API route under src/pages:

src/pages/robots.txt.ts

import type { APIRoute } from "astro";

import config from "../../heyo-docs.config";

export const GET: APIRoute = ({ request }) => {
  const siteUrl = config.siteUrl ?? new URL(request.url).origin;

  return new Response(
    [
      "User-agent: *",
      "Allow: /",
      "Disallow: /__heyo-docs/",
      "",
      `Sitemap: ${siteUrl}/sitemap.xml`,
      "",
    ].join("\n"),
    { headers: { "content-type": "text/plain; charset=utf-8" } },
  );
};