In the modern web, your audience isn't just human readers clicking links on Google. Your content is being actively crawled by LLM agents (like ChatGPT, Claude, and Perplexity), RSS aggregators, and automated summarizers.
To ensure your "Digital Garden" is fully discoverable, you must implement machine-readable endpoints at the root of your application.
In the Next.js App Router, special files like robots.ts, sitemap.ts, and API routes (feed.ts) are treated as top-level SEO endpoints. They should live directly inside your app directory.
Here is the ideal architectural setup for maximum indexing:
Instead of writing static .xml or .txt files, Next.js allows us to generate these files dynamically using TypeScript. This ensures that every time you write a new note, search engines and AI agents are immediately notified.
Use the tabs below to explore the implementation details of each critical file.
The robots.ts file is the first thing any crawler checks. It dictates which parts of your site are public and provides the absolute URL to your sitemap.
import type { MetadataRoute } from "next"
export default function robots(): MetadataRoute.Robots {
const baseUrl = process.env.NEXT_PUBLIC_BASE_URL || "[https://vaultscript.com](https://vaultscript.com)"
return {
rules: {
userAgent: "*",
allow: "/",
// Prevent bots from crawling draft previews or internal API routes
disallow: ["/api/private/", "/drafts/"],
},
sitemap: `${baseUrl}/sitemap.xml`,
}
}By utilizing TypeScript for these SEO endpoints, you unlock several architectural advantages:
MetadataRoute.Sitemap, MetadataRoute.Robots) ensuring you never ship a malformed XML file that could penalize your SEO ranking.process.env.NEXT_PUBLIC_BASE_URL, your sitemap paths are always correct, whether you are testing on localhost:3000 or deployed to production.Once these three files are in place, your next priority should be adding an llms.txt to your public directory. This provides strict synthesis instructions to AI models crawling your site, ensuring they quote you correctly and retain your code block formatting.