# https://optimalwellness.shop/robots.txt # # The site is live and indexed. Crawling stays open so every crawler can read the page # and its meta directives; only the scroll-frame sequences and internal pages are # excluded. Every group repeats those exclusions, because robots.txt groups do NOT # inherit from "User-agent: *" - a named bot obeys only its own block, so a group with # a bare "Allow: /" was free to pull all 2,400 frame files. # --- AI answer engines: explicitly welcome --- User-agent: GPTBot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: OAI-SearchBot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: ChatGPT-User Allow: / Disallow: /frames/ Disallow: /source/ User-agent: ClaudeBot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Claude-User Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Claude-SearchBot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: PerplexityBot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Perplexity-User Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Google-Extended Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Applebot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Applebot-Extended Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Amazonbot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: Bingbot Allow: / Disallow: /frames/ Disallow: /source/ User-agent: meta-externalagent Allow: / Disallow: /frames/ Disallow: /source/ User-agent: CCBot Allow: / Disallow: /frames/ Disallow: /source/ # --- Everyone else --- User-agent: * Allow: / # 2,400 scroll frames — crawlable but pointless to index, and a crawl-budget sink. Disallow: /frames/ Disallow: /source/ Sitemap: https://optimalwellness.shop/sitemap.xml