Making Your Personal Website AI-Agent Friendly

// January 21, 2026

As AI agents become the primary way people discover and access information, ensuring your personal website is discoverable by these systems is no longer optional—it's essential.

Today, I implemented comprehensive AI agent support for tomosman.com. Here's everything I learned and the exact steps to do the same for your site.

Why It Matters

The Shift to AI-First Discovery

Traditional SEO focused on Google rankings. But a new reality is emerging:

• ChatGPT brings AI-assisted search and retrieval to a global audience
• Perplexity has made citation-first AI search a mainstream workflow
• Claude is integrated into countless workflows
• AI agents are becoming the interface between humans and information

If someone asks an AI about you or your field, your site should be part of the knowledge base.

The Opportunity

Most personal websites are invisible to AI agents. By optimizing for AI discovery, you can:

• Appear in AI-generated answers and citations
• Train AI models on your content
• Become a trusted source in your niche
• Capture traffic from AI-first users

The Foundation: robots.txt

The robots.txt file controls which bots can access your site. Most sites only allow basic search crawlers.

I configured mine to allow public content for every crawler, with an explicit group for the major agents currently verified by Cloudflare:

export default function robots(): MetadataRoute.Robots {
  const protectedPaths = ["/private/", "/api/", "/admin/"];
  const verifiedAgents = [
    "GPTBot", "ChatGPT-User", "OAI-SearchBot",
    "ClaudeBot", "Claude-SearchBot", "Claude-User",
    "PerplexityBot", "Perplexity-User",
    "Googlebot", "Google-CloudVertexBot",
  ];

  return {
    rules: [
      { userAgent: "*", allow: "/", disallow: protectedPaths },
      // Repeat exclusions: a specific group does not inherit the wildcard group.
      { userAgent: verifiedAgents, allow: "/", disallow: protectedPaths },
    ],
    sitemap: "https://yoursite.com/sitemap.xml",
  };
}

Key AI Bots to Allow

Structured Data: Person Schema

AI systems rely heavily on structured data to understand content. I added comprehensive Person schema to the site:

const jsonLd = {
  "@context": "https://schema.org",
  "@type": "Person",
  "name": "Tom Osman",
  "jobTitle": "Technologist & Technology Dealer",
  "description": "Explores emerging technology through products, systems, practical guides, and recorded builds.",
  "url": "https://www.tomosman.com",
  "sameAs": [
    "https://x.com/tomosman",
    "https://github.com/tomcharlesosman",
    "https://youtube.com/@tomosman",
    "https://linkedin.com/in/thomascharlesosman/"
  ],
  "knowsAbout": [
    "Artificial Intelligence",
    "No-Code Development",
    "Automation",
    "Developer Relations"
  ],
  "worksFor": {
    "@type": "Organization",
    "name": "Shiny Technologies"
  }
};

This schema helps AI systems understand:

• Who you are
• What you do
• Your areas of expertise
• Where to find you online

AI Content Policies

I created two new files specifically for AI systems:

1. llms.txt

This concise Markdown file follows the llms.txt specification: one H1, a blockquote summary, and grouped absolute links to canonical content:

# Tom Osman

> Tom Osman is a technologist and Technology Dealer sharing practical work about AI, software, and emerging technology.

## Core pages

- [About](https://www.tomosman.com/about): Background and focus areas.
- [Writing](https://www.tomosman.com/blog): Practical guides and essays.
- [Tools](https://www.tomosman.com/tools): Maintained software inventory.

2. llms-full.txt

A full-text summary of your site for AI training:

# Tom Osman - Full text corpus

## About
Tom Osman explores the frontier of digital technologies.

## Content Sections
- About: Technology exploration and education
- Tools Inventory: Curated AI tools
- Livestreams: Recorded technology build sessions
- Blog: Long-form guides
- Portfolio: Selected work

## Keywords
digital technologies, AI, no-code, automation...

Comprehensive Metadata

I added extensive meta tags optimized for AI classification:

export const metadata = {
  keywords: [
    "Tom Osman",
    "digital technologies",
    "AI",
    "no-code",
    "automation",
    // ... more keywords
  ],
  other: {
    "ai-content": "educational",
    "ai-topic": "digital technologies, AI, automation",
    "ai-audience": "builders, developers, educators",
    "ai-use": "training,search,answer-generation,citation",
  },
};

Sitemap Optimization

Your sitemap now includes the complete route inventory:

• Core pages with honest change-frequency hints
• Blog posts with publication dates
• Portfolio projects
• Tools with dates where the source data provides them

This ensures AI agents can discover and index all your content.

The Verified Bot List

The explicit group tracks major user agents in Cloudflare's current bot reference. The wildcard group keeps other legitimate crawlers eligible for public content.

AI Agents

• GPTBot, ChatGPT-User, OAI-SearchBot
• ClaudeBot, Claude-SearchBot, Claude-User
• PerplexityBot, Perplexity-User
• Googlebot, Google-CloudVertexBot, bingbot
• Bytespider, CCBot, Amazonbot, Applebot
• meta-externalagent, meta-externalfetcher, FacebookBot
• DuckAssistBot, MistralAI-User

Results

After implementing these changes:

1. AI Visibility: Public content is available to crawlers, with current major agents called out explicitly
2. Knowledge Panels: Person schema increases Knowledge Panel potential
3. Citation Ready: AI agents can cite and reference your content
4. Training Data: Your content can be included in AI model training
5. AI Search: Appears in Perplexity, ChatGPT, and other AI search results

Quick Start Checklist

Want to do the same for your site?

1. Update robots.txt with a universal public policy and verified agent names
2. Add Person schema with your name, role, and links
3. Create llms.txt explaining your content policies
4. Generate llms-full.txt with expanded source-aligned content
5. Add comprehensive meta tags with keywords
6. Optimize sitemap to include all pages

The Future

As AI agents become the primary interface for information, being discoverable isn't optional—it's foundational to your online presence.

The work done today ensures that when someone asks an AI about "digital technologies" or "AI tools for builders," tomosman.com is part of the knowledge graph.

Research & References

This guide was created using insights from:

• LLMS Central — Comprehensive guide to AI bot user-agents
• Dark Visitors — Detailed AI bot profiles and documentation
• Paul Calvano — Data-driven analysis of AI bot growth and adoption
• Adnan Zameer — Practical implementation guide for robots.txt

Acknowledgments

Special thanks to:

• LLMS Central for maintaining the most complete AI bot user-agent list
• Paul Calvano for the data showing exponential AI bot growth
• Dark Visitors for detailed bot documentation

Related Posts:

• Claude Skills: The Complete Guide — Build custom AI capabilities
• Claude Cowork: The Complete Setup Guide — Desktop AI agent setup

VIEW_TOOLS — Curated AI tools for your workflow

Implementing AI discovery for your site? Tell me—I'd love to help.