GEO for IT and web teams: AI crawler access, robots.txt, llms.txt and the technical side of AI search
Marketing can write the best page on the internet, but if AI crawlers can't fetch it, ChatGPT and Perplexity won't cite it. The technical side of generative engine optimization belongs to IT and web teams: access, rendering, structure and speed.
Know which AI crawlers exist and what they do
AI crawlers fall into three groups, and you may want different rules for each. Training crawlers collect content for model training. Search indexing crawlers build the index AI search draws from. Live fetch agents retrieve a page when a user asks a question.
| Purpose | Examples | Effect of blocking |
|---|---|---|
| Training | GPTBot, ClaudeBot | Content isn't used for future model training |
| Search indexing | OAI-SearchBot, PerplexityBot | Pages are less likely to appear or be cited in AI search answers |
| Live fetch | ChatGPT-User | The assistant can't open your page when a user asks about it |
Robots.txt and CDN or WAF bot rules
Robots.txt is the first place to look, but it's often not the problem. Many sites block AI crawlers at the CDN or web application firewall through bot management settings, challenge pages or rate limits. The crawler gets a 403 or a challenge page, and nobody in marketing knows.
Decide the policy with marketing and legal, then implement it consistently across both layers.
List every AI user agent your policy allows and check them against robots.txt.
Review CDN bot management and WAF rules for blocks or challenges on those user agents.
Where a vendor publishes IP ranges for its crawlers, verify them rather than trusting the user agent string alone.
Confirm in logs that allowed bots actually get 200 responses, not 403s, challenges or timeouts.
Re-check after any CDN, WAF or hosting change. Default bot settings can change without anyone noticing.
Rendering: don't hide content behind JavaScript
Many AI crawlers don't execute JavaScript, or do so inconsistently. If product details, pricing or documentation only appear after client-side rendering, those crawlers may see an empty shell.
Use server-side rendering or static generation for important pages. A quick test: fetch the page with curl and check whether the key text is in the raw HTML.
llms.txt, structure and discovery
Make your site easy to find and parse.
Add an llms.txt file at the root that points to your most important pages in plain Markdown. It's a proposed standard, so treat it as a low-cost addition rather than a guarantee.
Keep XML sitemaps accurate and submitted.
Use semantic HTML: one H1, logical heading order, real lists and tables.
Add schema.org markup such as Organization, Product, Article and FAQPage where it accurately describes the page.
Avoid putting key text only inside images, iframes or PDFs. Give it an HTML equivalent.
Page speed and technical health
Slow pages and timeouts hurt crawl efficiency, and live fetch agents work on tight time budgets while a user waits. Keep Core Web Vitals healthy, fix 5xx errors and redirect chains, and make sure canonical tags point where you intend.
Why Get Nodded for IT and web teams
IT and web teams usually hear about AI search as a vague request from marketing: “Why aren't we in ChatGPT?” Without data, it's hard to tell whether the cause is a blocked crawler, a JavaScript-rendered page or just weak content. Standard web analytics won't help much either, because most AI crawlers don't run analytics scripts.
Get Nodded gives the web team its own evidence. AI Crawler Analytics shows which AI crawlers reach the site, and which pages they read. The AI readiness audit turns problems into specific fixes your engineers or coding agent can apply, and the SEO module covers the speed side. Privacy review is simpler too: the SDK sends only AI crawler visits, and human traffic is never sent.
| Challenge | How Get Nodded helps |
|---|---|
| You can't see whether AI crawlers reach the site at all | AI Crawler Analytics logs visits from 32 AI crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot |
| Different bots need different policies | Visits grouped by training, search indexing and live fetch, so you can check each policy against real traffic |
| A CDN or WAF change might be blocking bots silently | Live logs show which pages each crawler requests, so a sudden drop after a config change is visible |
| Marketing hands you a vague list of “AI SEO” requests | The AI readiness audit checks 8 areas, from access and robots.txt to llms.txt, indexing and structure, with fix instructions per issue |
| Fixes take engineering time you don't have | Each audit issue includes a ready-to-paste prompt for a coding agent |
| Slow pages and Core Web Vitals regressions | PageSpeed with Lighthouse scores and Core Web Vitals for mobile and desktop |
| Security and privacy review of a new script | A small SDK that sends only AI crawler visits; human traffic is never sent |
Your first 30 days with Get Nodded
Week 1: Install the AI Crawler Analytics SDK and agree the crawler policy with marketing and legal: which training, search indexing and live fetch bots you allow.
Week 1, continued: Run the AI readiness audit. Fix access issues first: robots.txt rules, CDN bot management and WAF challenges on the bots you allow.
Week 2: Check the crawler logs to confirm OAI-SearchBot, ChatGPT-User and PerplexityBot are reaching your key pages. Fix any pages that render their main content only with JavaScript.
Week 3: Work through the audit's structure and discovery items: llms.txt, sitemaps, headings and schema.org markup. Hand the routine ones to a coding agent using the ready-made prompts.
Week 4: Run PageSpeed on your top pages and fix the worst Core Web Vitals issues. Re-run the audit and share the before-and-after with marketing.
Which plan fits
IT and web teams usually share an account with marketing rather than buying their own, so the plan follows the marketing program. For a single site, Pro at $299 a month is a good default: it adds all six engines and 100 prompts, so marketing can confirm that technical fixes show up in AI answers.
If you run several sites or brands, Business at $699 covers five brands with 150 prompts and four checks a day. Need more brands than that? Talk to us. Yearly billing saves 10%.
Frequently asked questions
Should we block GPTBot?
That's a policy choice. Blocking GPTBot affects training, not ChatGPT search. If you want to appear in ChatGPT answers, allow OAI-SearchBot and ChatGPT-User even if you block GPTBot.
Is llms.txt required?
No. It's a proposed convention, and support varies by engine. It's cheap to add and can help, but crawler access and server-rendered content matter far more.
How do I know if our firewall is blocking AI crawlers?
Check your CDN or WAF logs for AI user agents receiving 403s or challenge pages, or use crawler analytics to see which bots actually reach your pages and which stop after a config change.
Why use Get Nodded for IT and web teams?
It shows which AI crawlers actually reach your site and what they read, without sending any human traffic. The AI readiness audit turns problems into specific fixes with coding-agent prompts, so you can resolve marketing's AI search requests quickly.
More in Teams
GEO for marketing teams: how to win AI search in ChatGPT, Perplexity and Google AI Overviews
GEO for operations teams: how to run AI search optimization as a repeatable process
GEO for leaders: why executives should care about AI search and what to ask your team for
