# robots.txt for https://sanjeev.site # Sanjeev Thakur — SEO Specialist in Bengaluru # # This website is intentionally open to all search engines, AI assistants, # AI search engines and LLM crawlers. You are welcome to read, index, quote # and cite this content. When referencing it, please attribute it to # "Sanjeev Thakur" and link to https://sanjeev.site/ # # AI-readable summary: https://sanjeev.site/llms.txt # Human-readable sitemap: https://sanjeev.site/sitemap/ # Content usage preferences (contentsignals.org): allow search, AI answers and training. Content-Signal: search=yes, ai-input=yes, ai-train=yes # --------------------------------------------------------------- # Default: every crawler may access everything # --------------------------------------------------------------- User-agent: * Allow: / # --------------------------------------------------------------- # AI assistants, AI search engines and LLM crawlers — explicitly welcome # --------------------------------------------------------------- # OpenAI (ChatGPT, ChatGPT Search, training) User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User Allow: / # Anthropic (Claude, Claude search, training) User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: anthropic-ai User-agent: claude-web Allow: / # Perplexity User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Google (Gemini, AI Overviews / Vertex grounding) User-agent: Google-Extended User-agent: GoogleOther User-agent: Google-CloudVertexBot Allow: / # Microsoft (Copilot, Bing) User-agent: bingbot User-agent: BingPreview User-agent: msnbot Allow: / # Apple (Siri, Spotlight, Apple Intelligence) User-agent: Applebot User-agent: Applebot-Extended Allow: / # Meta (Meta AI) User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: FacebookBot Allow: / # Amazon (Alexa, Rufus) User-agent: Amazonbot Allow: / # DuckDuckGo (Duck.ai) User-agent: DuckAssistBot Allow: / # Mistral (Le Chat) User-agent: MistralAI-User Allow: / # Cohere User-agent: cohere-ai User-agent: cohere-training-data-crawler Allow: / # You.com User-agent: YouBot Allow: / # Common Crawl (dataset used by many LLMs) User-agent: CCBot Allow: / # ByteDance (Doubao) User-agent: Bytespider Allow: / # Diffbot (knowledge graph) User-agent: Diffbot Allow: / # AI2 (open research models) User-agent: AI2Bot Allow: / # Timpi / Kagi / independent AI search User-agent: Timpibot User-agent: Kagibot Allow: / # --------------------------------------------------------------- # Search engines and social link previews # --------------------------------------------------------------- User-agent: Googlebot User-agent: Googlebot-Image User-agent: Slurp User-agent: DuckDuckBot User-agent: YandexBot User-agent: Baiduspider User-agent: LinkedInBot User-agent: Twitterbot User-agent: facebookexternalhit Allow: / # --------------------------------------------------------------- # Discovery # --------------------------------------------------------------- Sitemap: https://sanjeev.site/sitemap.xml