# robots.txt — www.authentikconsulting.ca
#
# Last reviewed: 2026-08-01        Next review due: 2026-11-01
#
# ⚠️ REVIEW THIS FILE EVERY QUARTER, and put the new date on the line above.
# The AI crawler landscape moved materially three times in the first half of
# 2026 alone: tokens are retired, split in two, or added, and a file written to
# last year's list quietly stops meaning what it says. Two names below are here
# ONLY as a warning not to re-add them.
#
# THE POLICY, IN ONE LINE: everything is allowed, training included. That is
# Craig's decision of 2026-08-01, and /aipolicy states it in plain English and
# points readers here to check it. If one changes, change the other.
#
# WHY ALLOW EVERYTHING. Three different jobs hide behind the phrase "AI
# crawler", they use INDEPENDENT tokens, and they cannot be grouped:
#
#   training       feeds model weights.     Blocking it costs nothing citation-wise.
#   search index   the index an assistant queries when answering.
#                  Blocking it costs EVERYTHING — you vanish from cited answers.
#   live fetch     one URL, because a human pasted a link.
#                  Blocking it costs the paste-a-link and deep-research paths.
#
# The search and live-fetch tokens were never really a choice. The training
# tokens were, and allowing them matches a published AI policy that is openly
# pro-AI, on a site with no content moat to defend — the Sunday Stories ARE the
# marketing. Blocking Applebot-Extended was offered as a free "we limit training
# use" line and declined on purpose: the policy should be simple and true rather
# than nuanced and unmaintained.
#
# NOTE ON WHAT THE GROUPS BELOW ACTUALLY DO. The wildcard group is what allows
# every crawler, including any that does not exist yet. The named groups are
# documentation — a person checking whether the site does what /aipolicy says
# can read them. A bot matching a named group ignores the wildcard group
# entirely, which is fine here because both say the same thing.

User-agent: *
Allow: /

# ---------------------------------------------------------------------------
# AI and assistant crawlers — allowed, all three jobs.
# ---------------------------------------------------------------------------

# OpenAI. GPTBot is training; blocking it does NOT stop ChatGPT citing the site.
# OAI-SearchBot is what controls search appearance, and ChatGPT-User is a live
# fetch that OpenAI documents as not used for search inclusion.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic.
#
# ⚠️ `anthropic-ai` and `claude-web` are RETIRED tokens. They are named here so
# that nobody copies them back in off an out-of-date blog post: a rule for a
# token no crawler sends does nothing at all, and it makes the file read as if
# it were maintained when it is not. Do not add them.
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Google. Note the split, which is the most commonly misconfigured thing here:
# Google-Extended governs Gemini training and Gemini/Vertex grounding, and does
# NOT control AI Overviews. Googlebot is the AI Overviews / AI Mode control, so
# blocking Googlebot to "opt out of AI" removes the site from Search as well.
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

# Perplexity.
User-agent: PerplexityBot
Allow: /

# Apple. Applebot-Extended is training-only; Applebot is Siri and Spotlight.
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# Meta.
User-agent: Meta-ExternalAgent
Allow: /

# ByteDance.
User-agent: Bytespider
Allow: /

# ---------------------------------------------------------------------------
# Carried over from the Wix robots.txt — deliberate blocks that still apply.
# ---------------------------------------------------------------------------

# PetalBot returns nothing: no traffic, no citations, no index anyone here uses.
User-agent: PetalBot
Disallow: /

# Commercial SEO crawlers. Slowed rather than blocked, as they were on Wix.
# Crawl-delay is honoured by these two and ignored by Google, which is fine —
# Google is not the problem this line is for.
User-agent: dotbot
Crawl-delay: 10

User-agent: AhrefsBot
Crawl-delay: 10

# ---------------------------------------------------------------------------
# NOT carried over from Wix, on purpose:
#
#   Disallow: *?lightbox=                   a Wix gallery URL parameter. This
#                                           site emits no such URL.
#   Disallow: /_partials*                   Wix internals. Neither path exists
#   Disallow: /pro-gallery-webapp/v1/...    here, so both rules are dead weight.
#
# Also deliberately absent: any rule for /admin. It is noindex, nofollow in its
# own markup, which is the directive that actually keeps it out of an index —
# a Disallow here would only advertise the path while PREVENTING a crawler from
# reading the noindex it is meant to obey.
# ---------------------------------------------------------------------------

Sitemap: https://www.authentikconsulting.ca/sitemap.xml