# GEO Checklist > Technical, content and reputation steps to get your site found, described accurately and cited by AI assistants such as ChatGPT, Gemini, Google AI Mode, Perplexity, Copilot and Claude. Every item comes with priority, effort, evidence level and a way to verify it. This file is the Markdown version of . Open the page to tick items off and check your site automatically. Last verified: November 1, 2026 ## AI search bot access If the bots that index or fetch pages on behalf of users cannot read your site, AI answers cannot cite it. This section checks robots.txt rules bot by bot. ### Serve robots.txt with HTTP 200 as plain text Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Bots read robots.txt first. If it returns a 5xx error, many crawlers, Google included, temporarily treat the whole site as off limits. **How to do it:** 1. Check that /robots.txt at the root returns 200 as text/plain. 2. Keep any redirect to a single hop. 3. Confirm CDN and WAF rules do not block this file for bots. **How to verify:** curl -sI https://{site}/robots.txt should return HTTP 200 and Content-Type: text/plain. **AI prompt:** ```text Check https://{site}/robots.txt: what status code, Content-Type, redirects and file size does it return? List syntax errors and rules that accidentally block bots, and propose a corrected version. ``` **Sources:** ### Split bots by role: search, user fetch, training Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: SEO, Business. **Why it matters:** Providers use separate bot names for search, user-initiated fetches and model training. Blocking them all at once to opt out of training also removes you from AI search. **How to do it:** 1. List each provider’s official bot tokens. 2. Keep search and user-fetch bots allowed for visibility. 3. Make a separate business decision for training bots. 4. Implement the decisions as separate User-agent groups in robots.txt. **How to verify:** robots.txt explicitly allows OAI-SearchBot, Claude-SearchBot and PerplexityBot, and has a deliberate rule for GPTBot and ClaudeBot. **AI prompt:** ```text Read https://{site}/robots.txt and classify every AI bot rule by role (search index, user fetch, training, ads). Flag rules meant to control training that also hurt search visibility, and draft a role-based robots.txt. ``` **Sources:** , ### Keep Googlebot allowed: AI Overviews and AI Mode depend on it Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: SEO. **Why it matters:** Google uses no separate bot for its AI features. A page only needs to be crawled by Googlebot and indexed to be eligible for AI Overviews and AI Mode. **How to do it:** 1. Check that robots.txt allows Googlebot on important paths. 2. Do not block CSS and JS files. 3. Run a live test with the URL Inspection tool. **How to verify:** Search Console \> URL Inspection \> Test live URL reports “URL is available to Google”. **AI prompt:** ```text For {site}, check via robots.txt and meta tags that Googlebot can crawl key pages, category pages and CSS/JS assets. Explain any rule that would make pages ineligible for AI Overviews or AI Mode. ``` **Sources:** ### Allow OAI-SearchBot: required for ChatGPT search Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** OAI-SearchBot indexes pages for ChatGPT search results. Blocked sites can at most appear as navigational links, and robots.txt changes take about 24 hours to apply. **How to do it:** 1. Add a User-agent: OAI-SearchBot group to robots.txt. 2. Allow key paths and disallow only low-value areas. 3. Do not block OpenAI’s published IP ranges in your WAF. 4. Re-test 24 hours after a change. **How to verify:** curl -A "OAI-SearchBot" -sI https://{site}/ returns 200 and robots.txt does not disallow OAI-SearchBot. **AI prompt:** ```text Find the rule group that applies to OAI-SearchBot in https://{site}/robots.txt. Tell me whether the homepage, category and content pages are open to it, and write corrected rules if anything blocks it. ``` **Sources:** ### Do not block ChatGPT-User at the server Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** ChatGPT-User fetches a page when a user asks for it in a conversation. OpenAI says robots.txt rules may not apply to it, so it is not a search control, but a WAF block returns an empty answer to the user. **How to do it:** 1. Check WAF and rate-limit rules rather than robots.txt. 2. Allowlist OpenAI’s chatgpt-user.json IP ranges. 3. Test that real content is served instead of a block page. **How to verify:** Server logs show ChatGPT-User requests getting 200; the 403/429 share is low. **AI prompt:** ```text Explain step by step how to audit robots.txt, WAF and rate-limit settings on {site} that could stop ChatGPT-User from fetching pages, and give a sample grep command to check this in access logs. ``` **Sources:** ### Allow Claude-SearchBot for Claude search results Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Claude-SearchBot builds the index used by Claude’s web search. Blocking it reduces how often Claude can cite your pages. **How to do it:** 1. Add a User-agent: Claude-SearchBot group to robots.txt. 2. Allow important paths. 3. Use a reasonable Crawl-delay if needed; Anthropic supports it. **How to verify:** A robots.txt tester or parser reports key URLs as “allowed” for Claude-SearchBot. **AI prompt:** ```text Separate the rules in https://{site}/robots.txt that apply to Claude-SearchBot, Claude-User and ClaudeBot. If anything blocks visibility in Claude search, write the corrected rules. ``` **Sources:** ### Let Claude-User fetch pages on user request Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Claude-User fetches a page when a user asks Claude to read it, and it respects robots.txt. If it is blocked, users get your information from second-hand sources instead. **How to do it:** 1. Check that no robots.txt rule blocks Claude-User. 2. Keep Anthropic’s published bot IP list allowed in your WAF. 3. Test by asking Claude to read a sample URL. **How to verify:** Ask Claude to “read and summarize https://{site}/”; it should summarize the actual page. **AI prompt:** ```text Does any rule in https://{site}/robots.txt block Claude-User directly or through the * group? If so, show the exact line and how to fix it. ``` **Sources:** ### Allow PerplexityBot to enter Perplexity’s index Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** PerplexityBot builds the index Perplexity cites in answers and is not used for model training. Blocking it only costs visibility. **How to do it:** 1. Add a User-agent: PerplexityBot group to robots.txt. 2. Keep the perplexitybot.json IP ranges allowed in your WAF. 3. Test with a brand query in Perplexity after the change. **How to verify:** Ask Perplexity a question that includes “site:{site}”; your pages should appear in the sources. **AI prompt:** ```text Review the PerplexityBot and Perplexity-User rules in https://{site}/robots.txt. Report anything that would stop Perplexity from citing the site and suggest fixes. ``` **Sources:** ### Avoid blocking Perplexity-User in your WAF by accident Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Perplexity-User fetches a page when a user asks for it, and Perplexity says it generally ignores robots.txt. The control point is the WAF; unnecessary blocks give users incomplete answers. **How to do it:** 1. Check the perplexity-user.json IP list. 2. Keep these requests allowed in bot management rules. 3. If you want to block it, do it deliberately in the WAF and document it. **How to verify:** Count status codes for Perplexity-User requests in logs; the 403 share should be low. **AI prompt:** ```text Write a checklist for {site} to identify Perplexity-User traffic and confirm the WAF is not blocking it by mistake, including IP list checks and log analysis steps. ``` **Sources:** ### Allow Bingbot: Bing and Copilot answers rely on it Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Bingbot feeds Bing search and the index Microsoft Copilot grounds on. Because Bing’s index is used by other AI search products too, a block costs visibility on several surfaces. **How to do it:** 1. Check that no robots.txt rule blocks Bingbot. 2. Use URL Inspection in Bing Webmaster Tools to see crawl status. 3. If you use Crawl-delay, keep it low. **How to verify:** Bing Webmaster Tools \> URL Inspection reports the homepage as indexed. **AI prompt:** ```text Review the rules and Crawl-delay that apply to Bingbot in https://{site}/robots.txt. Report anything that limits visibility in Bing and Copilot. ``` **Sources:** ### Keep DuckAssistBot allowed for DuckDuckGo AI answers Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** DuckAssistBot fetches pages in real time for DuckDuckGo’s AI-assisted answers, which cite their sources; DuckDuckGo says the data is not used to train AI models. robots.txt changes take about 72 hours to apply. It is a small channel, but allowing it costs nothing. **How to do it:** 1. Check that no robots.txt rule blocks DuckAssistBot. 2. Confirm the generic \* group does not block it indirectly. **How to verify:** A robots.txt parser reports the homepage as “allowed” for DuckAssistBot. **AI prompt:** ```text According to https://{site}/robots.txt, can DuckAssistBot crawl the homepage and content pages? Give a short answer and a corrected rule if needed. ``` **Sources:** ### Do not block MistralAI-User, MistralAI-Index and similar bots Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Mistral’s assistant uses MistralAI-User, which follows robots.txt, to fetch pages on user request, and MistralAI-Index to build its search index. Neither is used for training; the training crawler is the separate MistralAI-Training. **How to do it:** 1. Check that no robots.txt or WAF rule blocks MistralAI-User or MistralAI-Index. 2. Apply your training decision to MistralAI-Training in its own group. 3. Review other user-fetch bots such as Meta-ExternalFetcher the same way. **How to verify:** Logs show MistralAI-User and MistralAI-Index requests getting 200. **AI prompt:** ```text Explain how to check whether bots such as MistralAI-User, MistralAI-Index and Meta-ExternalFetcher are blocked by robots.txt or the WAF on {site}, and write a separate training rule example for MistralAI-Training. ``` **Sources:** , ### Publish a robots.txt on every subdomain Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** robots.txt only applies to the host it is served on. If blog.{site} or shop.{site} has no file or returns errors, bots behave differently there. **How to do it:** 1. List all subdomains and protocols. 2. Check that /robots.txt returns 200 on each. 3. Apply the same bot policy consistently in every file. **How to verify:** curl -sI https://\.{site}/robots.txt returns 200 for each subdomain. **AI prompt:** ```text Guess the common subdomains of {site} (www, blog, shop, docs, help) and write a shell command that checks robots.txt on each. Explain how to fix any inconsistencies in the results. ``` **Sources:** , ### Account for bot-specific groups overriding the generic rules Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** When a bot has its own User-agent group, it ignores the \* group entirely. Disallow lines not repeated in the specific group no longer apply to that bot, and missing Allow lines can create accidental blocks. **How to do it:** 1. Read each specific group on its own. 2. Copy necessary Disallow lines from the generic group into specific groups. 3. Test changes with a robots.txt parser. **How to verify:** Test sample URLs for each bot with Google’s open-source robots.txt parser or a testing tool. **AI prompt:** ```text Interpret https://{site}/robots.txt according to RFC 9309. For each specific User-agent group, show in a table which URLs end up allowed or blocked differently from the * group. ``` **Sources:** ## Training bots and usage preferences Bots that collect data for model training are controlled separately from search visibility. The goal is not to allow everything but to make a deliberate, documented decision for each. ### Make a deliberate allow or block decision for GPTBot Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Business, SEO. **Why it matters:** GPTBot collects data to train OpenAI models. Blocking it does not affect ChatGPT search visibility, so the decision should rest only on content licensing and brand preference. **How to do it:** 1. Weigh the benefit of being represented in model knowledge. 2. Write the decision down. 3. Set the User-agent: GPTBot group in robots.txt accordingly. **How to verify:** robots.txt has an explicit GPTBot group, and the decision is recorded with a date. **AI prompt:** ```text Compare the pros and cons of allowing versus blocking GPTBot for {site} in my industry. Explain how it differs from OAI-SearchBot and write a robots.txt example for each option. ``` **Sources:** ### Decide on ClaudeBot training separately from Claude search Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Business, SEO. **Why it matters:** ClaudeBot collects data to train Anthropic models, while Claude search uses Claude-SearchBot. Managing them separately lets you apply a training preference without losing visibility. **How to do it:** 1. Decide whether to allow or block ClaudeBot. 2. Keep Claude-SearchBot and Claude-User allowed in their own groups. 3. Limit load with Crawl-delay if needed. **How to verify:** robots.txt has separate groups for ClaudeBot and Claude-SearchBot. **AI prompt:** ```text Check whether the ClaudeBot rule in https://{site}/robots.txt affects Claude-SearchBot, and write an example that blocks training while keeping search open. ``` **Sources:** ### Decide on Google-Extended knowing it does not affect Search Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Business, SEO. **Why it matters:** Google-Extended is not a crawler but a control token for training and grounding use in Gemini and Vertex AI. It does not change Search ranking or visibility in AI Overviews or AI Mode. **How to do it:** 1. Weigh the value of being used as a source in Gemini apps. 2. Implement the decision with User-agent: Google-Extended in robots.txt. 3. Explain to the team that it cannot remove you from AI Overviews. **How to verify:** The Google-Extended line in robots.txt matches the documented decision. **AI prompt:** ```text Explain what Google-Extended controls and does not control (Search, AI Overviews, AI Mode) for {site}, and give me a short decision tree. ``` **Sources:** , ### Decide on Applebot-Extended without blocking Applebot Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Business. **Why it matters:** Applebot-Extended is a token that controls use of your content in Apple’s AI features and does not crawl itself. Blocking Applebot instead also removes you from Siri and Spotlight suggestions. **How to do it:** 1. Keep Applebot allowed. 2. Decide whether to allow or block Applebot-Extended. 3. Record the decision. **How to verify:** robots.txt does not block Applebot and has a deliberate Applebot-Extended rule. **AI prompt:** ```text Review the Applebot and Applebot-Extended rules in https://{site}/robots.txt and flag anything that hurts Siri and Spotlight visibility. ``` **Sources:** ### Decide on CCBot (Common Crawl) Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Business. **Why it matters:** The Common Crawl archive is used in many model training pipelines. Blocking CCBot keeps your content out of these open datasets but does not affect search bots. **How to do it:** 1. Weigh the pros and cons of being in open datasets. 2. Set the User-agent: CCBot group in robots.txt. **How to verify:** robots.txt has a deliberate rule for CCBot. **AI prompt:** ```text Explain how allowing CCBot affects model knowledge and brand representation for a site like {site}, then suggest the right robots.txt lines. ``` **Sources:** ### Decide separately on Meta-ExternalAgent and Meta-ExternalFetcher Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Business. **Why it matters:** Meta-ExternalAgent crawls for Meta’s AI models, while Meta-ExternalFetcher fetches pages on user request. Keeping the latter allowed is enough for visibility to Meta AI users. **How to do it:** 1. Make your training decision for Meta-ExternalAgent. 2. Do not block Meta-ExternalFetcher. 3. Record the decisions. **How to verify:** robots.txt has separate, documented rules for both bots. **AI prompt:** ```text Review the Meta bot rules in https://{site}/robots.txt and write an example that blocks the training crawler while keeping the user-fetch bot allowed. ``` **Sources:** ### Write a deliberate rule for Bytespider Priority: Low priority. Effort: Quick win. Evidence: Emerging standard. Roles: Business. **Why it matters:** Bytespider is ByteDance’s crawler for model training and has been widely reported to crawl heavily and not always respect robots.txt. Without a rule, the default \* group applies. **How to do it:** 1. Measure Bytespider volume in server logs. 2. Decide to allow or block and write it into robots.txt. 3. Rate-limit it in the WAF if needed. **How to verify:** Bytespider volume in logs and its robots.txt rule match the documented decision. **AI prompt:** ```text Write a command to find and measure Bytespider traffic in the server logs of {site}, then recommend robots.txt and WAF settings. ``` ### Decide on Amazonbot separately and keep Amzn-SearchBot allowed Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Business. **Why it matters:** Amazon says content Amazonbot collects may be used to train Amazon AI models. Search experiences in Amazon products such as Alexa rely on Amzn-SearchBot instead, and user-requested fetches on Amzn-User. Blocking Amazonbot does not remove that visibility. **How to do it:** 1. Assess how important the Amazon channel and Alexa visibility are for you. 2. Implement your training decision for Amazonbot in its own robots.txt group. 3. Check that Amzn-SearchBot and Amzn-User are not blocked. **How to verify:** robots.txt has a deliberate rule for Amazonbot and does not block Amzn-SearchBot. **AI prompt:** ```text Explain the roles of Amazonbot, Amzn-SearchBot and Amzn-User for {site}, and suggest robots.txt lines that block training while keeping the site visible in Amazon search experiences such as Alexa. ``` **Sources:** ### Do not block OAI-AdsBot if you run ChatGPT ads Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Business. **Why it matters:** OAI-AdsBot only visits pages submitted as ads to validate them and is not used for training. If blocked, ad landing pages may not be approved. **How to do it:** 1. Check whether you run ChatGPT ad campaigns. 2. Keep landing page paths open to OAI-AdsBot. **How to verify:** robots.txt does not block OAI-AdsBot from ad landing pages. **AI prompt:** ```text According to https://{site}/robots.txt, can OAI-AdsBot reach the ad landing pages? Briefly explain what this bot does and that it is unrelated to training. ``` **Sources:** ### Know GoogleOther, Google-CloudVertexBot and Storebot-Google Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** GoogleOther runs research and development crawls, Storebot-Google serves shopping surfaces, and Google-CloudVertexBot only crawls when a site owner requests it for Vertex AI Agents. Wrong blocks can break shopping data. **How to do it:** 1. Separate these bots’ requests in your logs. 2. If you run an online store, do not block Storebot-Google. 3. If you do not use Vertex AI, do not expect CloudVertexBot traffic; verify its source if you see it. **How to verify:** Check status codes for these bots in server logs; Storebot-Google should get 200. **AI prompt:** ```text Explain how to identify GoogleOther, Storebot-Google and Google-CloudVertexBot requests in the logs of {site} and the right robots.txt approach for each. ``` **Sources:** ### Declare usage preferences with Content Signals Priority: Low priority. Effort: Quick win. Evidence: Emerging standard. Roles: Business, Developer. **Why it matters:** Cloudflare’s Content Signals policy expresses search, ai-input and ai-train preferences in robots.txt. It is not a standard and compliance is not guaranteed, but it records your intent clearly. **How to do it:** 1. Decide your preferences for search, ai-input and ai-train. 2. Add a Content-Signal line to robots.txt. 3. Note for the team that it is not an access control. **How to verify:** robots.txt contains a line such as “Content-Signal: search=yes, ai-input=yes, ai-train=no”. **AI prompt:** ```text Explain what the search, ai-input and ai-train signals mean for {site}, suggest a Content-Signal line based on my goals and state its limits. ``` **Sources:** , ### Track the IETF AIPREF draft and trial it lightly Priority: Low priority. Effort: Quick win. Evidence: Emerging standard. Roles: Developer. **Why it matters:** The IETF AIPREF working group is drafting a Content-Usage header and a content-usage robots.txt rule. Google documents a plan to support the AI-training preference through a \_\_usage\_ai-training user agent in early 2027. It is not a standard yet. **How to do it:** 1. Follow the latest draft version. 2. Express your preference experimentally in robots.txt and an HTTP header, if you wish. 3. Update the rule once the standard is final. **How to verify:** A tracking note exists and, if applied, curl -sI shows the Content-Usage header. **AI prompt:** ```text Summarize the current state of the IETF AIPREF draft and write an experimental Content-Usage header and content-usage robots.txt rule for {site}, noting the risks. ``` **Sources:** , ### Document the bot policy with an owner and review date Priority: Medium priority. Effort: Quick win. Evidence: Emerging standard. Roles: Business, SEO. **Why it matters:** New bots and rules keep appearing. A written policy stops a release from accidentally blocking search bots and keeps decisions from being forgotten. **How to do it:** 1. Create a table with decision, rationale and owner per bot. 2. Require code review for robots.txt changes. 3. Set a quarterly review date. **How to verify:** The team wiki has a dated bot policy table that matches robots.txt exactly. **AI prompt:** ```text Create a bot policy table template for {site} covering AI bots, with columns for bot name, provider, role, decision, rationale, owner and review date. ``` ## Indexing and snippet eligibility Google says AI features need no special markup: a page must be indexed and eligible to show a snippet. This section checks those basic conditions. ### Never leave noindex on important pages by mistake Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** A page with noindex appears neither in classic results nor in AI Overviews and AI Mode. A noindex left over from staging is the most common cause of lost visibility. **How to do it:** 1. Scan templates for meta robots and X-Robots-Tag headers. 2. Add a noindex check to the release process. 3. Review “Excluded by noindex tag” in Search Console. **How to verify:** Neither curl -sI https://{site}/ nor the page source contains noindex. **AI prompt:** ```text Check meta robots tags and X-Robots-Tag headers on the homepage and key templates of {site}. List any page with noindex or none and guess why. ``` **Sources:** , ### Do not restrict quoting with nosnippet or low max-snippet Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: SEO. **Why it matters:** Google applies nosnippet, max-snippet and data-nosnippet to its AI features too. Over-restricting stops your page from being used in answers. **How to do it:** 1. Search templates for nosnippet and max-snippet values. 2. Remove unnecessary limits or use max-snippet:-1. 3. Keep restrictions only where there is a legal reason. **How to verify:** The page source has no nosnippet; any max-snippet is -1 or high enough. **AI prompt:** ```text Review the use of nosnippet, max-snippet, max-image-preview and data-nosnippet on {site} pages, and report settings that needlessly limit quoting in AI answers. ``` **Sources:** , ### Hide sensitive parts with data-nosnippet, not the whole page Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** You do not need to restrict an entire page to hide small parts such as legal text. data-nosnippet removes only the marked element from snippets. **How to do it:** 1. Identify the parts that must be hidden. 2. Add the data-nosnippet attribute to those elements. 3. Remove page-level nosnippet. **How to verify:** data-nosnippet appears only on the targeted span, div or section elements in the source. **AI prompt:** ```text Which parts of {site} pages might make sense to hide with data-nosnippet? Give example HTML and explain the difference from page-level restrictions. ``` **Sources:** ### Allow large image previews with max-image-preview:large Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** AI answers and surfaces like Discover can use large image previews. A restricted setting makes your images small or absent there. **How to do it:** 1. Add \ to templates. 2. Handle pages without image usage rights separately. **How to verify:** The page source contains max-image-preview:large. **AI prompt:** ```text Check the max-image-preview value in {site} templates and briefly explain why it should be large. ``` **Sources:** ### Know and track how many pages Google has indexed Priority: High priority. Effort: A few days. Evidence: Official source. Roles: SEO. **Why it matters:** A page that is not indexed cannot be a source for AI Overviews or AI Mode. Sudden drops in indexed pages are usually the first sign of a technical problem. **How to do it:** 1. Record the Search Console Pages report monthly. 2. Use separate sitemaps for key templates to track index rates. 3. Review “Crawled - currently not indexed”. **How to verify:** In Search Console \> Indexing \> Pages, key sitemaps show an indexing rate above 90%. **AI prompt:** ```text Help me interpret the Search Console Pages report for {site}: which exclusion reasons matter, which are normal, and what should I fix first? Suggest an order of checks. ``` **Sources:** ### Check your coverage in the Bing index Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: SEO. **Why it matters:** Bing’s index grounds Copilot answers and is reportedly used by other AI search products. A site doing well in Google can still be under-indexed in Bing. **How to do it:** 1. Review Site Explorer and index reports in Bing Webmaster Tools. 2. Submit missing URLs or use IndexNow. 3. Fix crawl errors. **How to verify:** Bing Webmaster Tools \> Site Explorer shows indexed URLs across key directories. **AI prompt:** ```text Walk me through which Bing Webmaster Tools reports to use to diagnose indexing issues for {site}, and how to compare with Google. ``` **Sources:** ### Use a correct, absolute canonical tag on every page Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** The canonical tag says which of several duplicate URLs should be shown. A wrong canonical can make another URL get picked instead of the page you want cited. **How to do it:** 1. Add a self-referencing absolute canonical to each page. 2. Point variant and parameter URLs to the main version. 3. Check that canonical targets return 200 and are not noindex. **How to verify:** In Search Console URL Inspection, the user-declared and Google-selected canonicals match. **AI prompt:** ```text Check the canonical tags on the homepage, a category page and a content page of {site}. List errors such as relative URLs, redirected targets or noindex targets. ``` **Sources:** ### Consolidate http, https, www and trailing-slash variants Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Serving the same page at several addresses splits signals and lets bots cite different versions. **How to do it:** 1. Choose one preferred host and protocol. 2. Redirect other variants to it with a 301. 3. Fix internal links to use the preferred version. **How to verify:** curl -sI http://{site}/ and https://www.{site}/ return a single 301 hop to the preferred URL. **AI prompt:** ```text Write a set of curl commands that test http/https, www/non-www and trailing-slash variants of {site}, and explain the correct expected results. ``` **Sources:** ### Merge duplicate and thin pages Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: SEO, Content. **Why it matters:** Thin pages that repeat the same topic waste crawl budget and make it unclear which page a model should cite. One strong page is easier to quote. **How to do it:** 1. List pages sharing the same title or description. 2. Merge content into the strongest page. 3. Redirect the others with a 301. **How to verify:** A site crawler reports close to zero duplicate titles and meta descriptions. **AI prompt:** ```text Prepare a crawl plan for {site} to find duplicate or thin content: which metrics to use, which pages to merge and how to set up redirects. ``` **Sources:** ### Keep parameter and filter URLs out of the index Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** Sorting, filter and tracking parameters can create endless URLs. These duplicates use up crawl budget and lead to the wrong URL being cited. **How to do it:** 1. Decide which filter combinations deserve indexing. 2. Use canonical or robots.txt rules for the rest. 3. Do not use tracking parameters in internal links. **How to verify:** The Search Console Pages report shows few parameter URLs; site:{site} inurl:? returns few results. **AI prompt:** ```text Create a table for {site} that classifies URL parameters (sort, filter, utm, session) and recommends canonical, noindex or robots.txt handling for each. ``` **Sources:** ### Prevent staging sites and duplicate domains from being indexed Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** If test servers or old domains serve a copy of your site, AI answers may cite the wrong or outdated copy. **How to do it:** 1. Protect staging environments with a password. 2. Redirect old domains to the main site with a 301. 3. Search regularly for exact copies of your content elsewhere. **How to verify:** Search an exact sentence from your site in quotes; only {site} should appear. **AI prompt:** ```text Which searches and tools should I use to find copies of {site} content on other domains, staging environments or scraper sites? Give me a checklist. ``` ### Make sure your official site ranks first for your brand name Priority: High priority. Effort: Quick win. Evidence: Experimental. Roles: SEO. **Why it matters:** If your official site is not first for your brand query, search engines and models see your entity as weak or ambiguous. This carries over into AI answers to brand questions. **How to do it:** 1. Search your brand name in Google and Bing. 2. If you are not first, strengthen entity signals. 3. Use a descriptive site name that sets you apart from others with the same name. **How to verify:** https://{site}/ is the first organic result for the brand name in Google and Bing. **AI prompt:** ```text List the reasons (name collisions, missing entity signals, technical issues) that could stop the official site from ranking first for brand searches of {site}, and suggest fixes. ``` ## Crawl and server health When bots get errors, slow or misleading responses, they crawl less or skip your pages. This section checks status codes, redirects and consistency. ### Serve every page over HTTPS Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** HTTPS is a trust and ranking signal, and many bots and agents avoid insecure connections. Mixed protocols also create duplicate URLs. **How to do it:** 1. Install a valid TLS certificate with auto-renewal. 2. Redirect all HTTP requests to HTTPS with a 301. 3. Add an HSTS header. **How to verify:** curl -sI http://{site}/ returns a 301 to https and the certificate is valid. **AI prompt:** ```text Audit the HTTPS setup of {site}: certificate validity, HTTP-to-HTTPS redirects, HSTS header and any http links in internal navigation. ``` **Sources:** ### Do not load http resources on HTTPS pages Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Browsers block mixed content, so when images or scripts fail to load, agents and rendering bots see an incomplete page. **How to do it:** 1. Search the source for src and href values starting with http://. 2. Move resources to https or use relative paths. 3. Add Content-Security-Policy: upgrade-insecure-requests. **How to verify:** Chrome DevTools Console shows no “Mixed Content” warnings. **AI prompt:** ```text Find images, scripts or stylesheets loaded over http:// in the homepage source of {site} and explain how to fix them. ``` ### Keep 5xx server errors close to zero Priority: High priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Bots that hit frequent 5xx errors slow their crawling. Assistants fetching a page on user request answer without your content when they get an error. **How to do it:** 1. Track the 5xx rate in Search Console Crawl Stats and server logs. 2. Find the failing endpoints and time windows. 3. Fix capacity or caching issues. **How to verify:** Search Console \> Settings \> Crawl stats shows a server error rate below 1%. **AI prompt:** ```text Write a command that groups 5xx responses to bot requests by user agent in the server logs of {site}, and explain how to interpret the results. ``` **Sources:** , ### Return real 404 or 410 for removed pages, no soft 404s Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Pages that show “not found” but return 200 pollute the index as soft 404s, and broken internal links send bots to dead ends. **How to do it:** 1. Test that a non-existent URL returns 404. 2. Fix the soft 404 list in Search Console. 3. Update or redirect internal links to 404 targets. **How to verify:** curl -sI https://{site}/no-such-page-xyz returns 404 or 410. **AI prompt:** ```text Create an audit plan for {site} to find soft 404s, broken internal links and indexed 404 pages, naming the tools and report fields to use. ``` **Sources:** ### Use single-hop 301 redirects for permanent moves Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Long redirect chains make bots give up and lose signals. A temporary 302 can keep the old URL in the index. **How to do it:** 1. List redirect chains with a crawler. 2. Collapse each chain to one 301 straight to the final target. 3. Use 301 or 308, not 302, for permanent moves. **How to verify:** curl -sIL https://{site}/old-url shows at most one redirect. **AI prompt:** ```text Suggest a crawl method to find redirect chains and unnecessary 302s on {site}, and give examples of corrected redirect rules. ``` **Sources:** ### Use server redirects instead of meta refresh and JavaScript redirects Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Most AI bots do not run JavaScript and do not reliably follow meta refresh. Server-side redirects are understood by every client. **How to do it:** 1. Search the source for http-equiv="refresh" and window.location. 2. Replace them with server-side 301s. **How to verify:** No page source contains http-equiv="refresh". **AI prompt:** ```text Explain how to find meta refresh or JavaScript redirects on {site} and convert them to server-side 301s. ``` **Sources:** ### Do not serve critical content inside iframes Priority: Low priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Content inside an iframe is a separate document and often not credited to the host page, so information such as pricing tables or FAQs does not get quoted. **How to do it:** 1. List iframe usage across pages. 2. Move critical text into the main HTML document. 3. Back up embeds such as maps and videos with text. **How to verify:** No critical text on the homepage or product/service pages sits inside an iframe. **AI prompt:** ```text Is any important content on {site} served inside an iframe? Give a method to find it and explain how to move it into the main HTML. ``` ### Do not embed key information in images; write it as text Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Content, Developer. **Why it matters:** Models do not reliably read text inside images. Prices, specs or campaign terms that live only in images cannot make it into answers. **How to do it:** 1. List critical information in banners and infographics. 2. Add it to the page as HTML text. 3. Write descriptive alt text for the images. **How to verify:** Read the page with images hidden; the critical information should still be visible as text. **AI prompt:** ```text Could images on the {site} homepage contain critical information not written as text? Give a method to audit this and an example conversion. ``` **Sources:** ### Show bots and users the same content Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Serving different content by user agent is cloaking under search engine policies and damages trust. Special “optimized” pages for AI bots carry the same risk. **How to do it:** 1. Request the same URL with bot user agents and a normal browser. 2. Compare text and prices. 3. Remove code that changes content based on bot detection. **How to verify:** The main text and prices match between a curl -A "OAI-SearchBot" request and a normal browser. **AI prompt:** ```text Write a script that requests the same {site} URL with different user agents (Googlebot, OAI-SearchBot, ClaudeBot, a normal browser), compares the content, and explain how to interpret differences. ``` **Sources:** ### Monitor crawl stats and alert on sudden drops Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: SEO, Developer. **Why it matters:** A sudden drop in crawl requests usually follows a robots.txt, WAF or server change. Catching it early prevents weeks of lost visibility. **How to do it:** 1. Check the Search Console Crawl Stats report weekly. 2. Track daily requests per bot in server logs. 3. Alert when volume falls below a threshold. **How to verify:** Search Console \> Settings \> Crawl stats shows no unexplained drop in the last 90 days. **AI prompt:** ```text Design a simple monitor for {site} that extracts daily request counts for Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot from server logs and alerts on a 50% drop. ``` **Sources:** ### Keep low-value areas out of the crawl Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** Internal search results, cart, account pages and endless filter combinations consume crawl budget. Bot time should go to valuable pages. **How to do it:** 1. Identify low-value URL patterns. 2. Block them in robots.txt. 3. Test that the blocked patterns do not cover important pages. **How to verify:** Server logs show a low share of bot requests to /search, /cart or filter URLs. **AI prompt:** ```text Guess the low-value URL patterns on {site} that should not be crawled (internal search, cart, account, filters) and suggest robots.txt rules that leave important pages untouched. ``` **Sources:** ## Rendering and HTML delivery Many AI bots do not run JavaScript and read only the initial HTML response. Content, links and language information must be in that first response. ### Serve main content without JavaScript Priority: High priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Googlebot renders JavaScript, but most AI crawlers and user-fetch bots do not. Text loaded client-side is an empty page to them. **How to do it:** 1. Open the page with JavaScript disabled or fetch it with curl. 2. Check that headings, body text and prices are in the raw HTML. 3. Plan server-side rendering for templates that fail. **How to verify:** curl -s https://{site}/ | grep "\" returns a match. **AI prompt:** ```text Fetch the {site} homepage and a content page without running JavaScript. Are headings, body text, prices and links present in the raw HTML? List what is missing and suggest fixes. ``` **Sources:** ### Use server-side rendering or static generation for key templates Priority: High priority. Effort: Project. Evidence: Official source. Roles: Developer. **Why it matters:** SSR or static generation gives bots and agents the content in the first response. Client-only single-page apps stay invisible to AI bots. **How to do it:** 1. Prioritize templates that drive traffic and revenue. 2. Choose SSR, SSG or prerendering for them. 3. Check that content does not change after hydration. **How to verify:** HTML fetched with curl contains the same main text as the DOM rendered in a browser. **AI prompt:** ```text Create a prioritization table to decide which {site} templates need server-side rendering, and suggest SSR or SSG options that fit the tech stack. ``` **Sources:** ### Build menus and internal links with real \ elements Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Bots only follow links with an href. Menus opened with onclick or links generated by JavaScript leave the rest of the site undiscoverable. **How to do it:** 1. Check main menu, footer and pagination links in the raw HTML. 2. Replace onclick navigation with \. 3. Render mega menus in the initial HTML. **How to verify:** curl -s https://{site}/ | grep -c "\ ### Keep tab and accordion content in the initial HTML Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Tab content loaded from the server on click is invisible to bots. The same content hidden in the HTML can be read and quoted. **How to do it:** 1. List tab, accordion and “read more” components. 2. Render the content in the initial HTML and hide it with CSS. 3. Remove API calls triggered by clicks. **How to verify:** Text in collapsed tabs is searchable in the raw HTML. **AI prompt:** ```text Give a method to check whether tab and accordion content on {site} product or service pages is in the initial HTML, and describe the code change needed. ``` ### Declare \ with the correct language code Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** The language declaration helps models and search engines match the page to queries in the right language. lang="en" left on Turkish pages is a common mistake. **How to do it:** 1. Check the \ value in every template. 2. Use lang="tr" for Turkish pages. 3. Set the value dynamically per page language on multilingual sites. **How to verify:** curl -s https://{site}/ | grep -o "\\]\*lang=\[^ \>\]\*" shows the right language code. **AI prompt:** ```text Check the html lang value on {site} pages and list any that do not match the content language. ``` ### Declare UTF-8 and serve non-ASCII characters correctly Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** A wrong charset breaks letters such as ç, ğ, ı and ş; garbled text is hard to understand and gets quoted incorrectly. **How to do it:** 1. Send charset=utf-8 in the Content-Type header. 2. Add \ at the top of the HTML. 3. Check that the database and templates use UTF-8. **How to verify:** curl -sI https://{site}/ shows Content-Type: text/html; charset=utf-8. **AI prompt:** ```text Check the character encoding of {site} through the HTTP header, meta tag and content samples. If you see garbled characters, guess the cause. ``` ### Use semantic HTML elements: main, article, nav, header, footer Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Semantic elements separate main content from menus and ads, so bots and agents extract the right text and the accessibility tree is clearer. **How to do it:** 1. Wrap main content in \ and \. 2. Mark navigation with \ and page regions with \ and \. 3. Use \ and \ for tabular data. **How to verify:** The page source has one \ element and content pages use \. **AI prompt:** ```text Review the HTML structure of a {site} page template. Point out areas built with divs that should be semantic elements and propose a corrected skeleton. ``` ### Add the mobile viewport meta tag Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Google uses mobile-first indexing; a page without a viewport looks broken on mobile and weakens quality signals. **How to do it:** 1. Add \. 2. Remove settings that block zooming. **How to verify:** The page source contains a viewport tag with width=device-width. **AI prompt:** ```text Check the viewport meta tag on {site} pages and flag settings such as user-scalable=no that hurt accessibility. ``` ### Serve the same content on mobile and desktop Priority: High priority. Effort: A few days. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** Google indexes the mobile version. Shortened text, missing structured data or hidden sections on mobile are also missing from AI answers. **How to do it:** 1. Compare mobile and desktop HTML. 2. Add missing text, links and structured data to mobile. 3. Check that CSS and JS are not blocked on mobile. **How to verify:** In Search Console URL Inspection, the crawled mobile HTML contains the main text and JSON-LD of the desktop version. **AI prompt:** ```text Write a script that fetches the same {site} page with a mobile and a desktop user agent and compares text, links and JSON-LD. ``` **Sources:** ## Sitemaps and fast discovery Sitemaps and instant notification protocols help bots find new or updated pages quickly. ### Publish a valid XML sitemap Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** A sitemap tells bots which URLs matter and when they changed, speeding up discovery of deep pages. **How to do it:** 1. Generate an auto-updating sitemap.xml with your CMS or a plugin. 2. Stay within 50,000 URLs and 50 MB per file. 3. Use a sitemap index if needed. **How to verify:** https://{site}/sitemap.xml returns 200 and is valid XML. **AI prompt:** ```text Review https://{site}/sitemap.xml: is it valid XML, how many URLs does it hold, is it a sitemap index, and does it look auto-updated? ``` **Sources:** ### Reference the sitemap in robots.txt Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** AI bots that do not use Search Console usually find the sitemap through the Sitemap line in robots.txt. **How to do it:** 1. Add “Sitemap: https://{site}/sitemap.xml” to robots.txt. 2. For several sitemaps, use one line each or an index file. **How to verify:** robots.txt contains a Sitemap: line with an absolute URL. **AI prompt:** ```text Does https://{site}/robots.txt contain Sitemap lines with absolute URLs that actually return 200? Check it. ``` **Sources:** ### Update lastmod only for meaningful changes Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** An accurate lastmod lets bots recrawl updated pages first. A lastmod that changes daily on every URL is considered unreliable and ignored. **How to do it:** 1. Update lastmod only when content changes. 2. Use the W3C date format. 3. Do not refresh all dates on template changes. **How to verify:** Sitemap lastmod values vary and match the update dates shown on pages. **AI prompt:** ```text Review the lastmod values in the {site} sitemap: are they all identical, and do they match the update dates on the pages? Suggest an implementation if not. ``` **Sources:** ### List only canonical URLs that return 200 in the sitemap Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** Redirected, 404 or noindex pages reduce sitemap trust and waste bot time. **How to do it:** 1. Crawl all sitemap URLs with their status codes. 2. Remove 3xx, 4xx and noindex URLs. 3. Include only canonical URLs. **How to verify:** When a crawler checks the sitemap, over 99% of URLs return 200 and are indexable. **AI prompt:** ```text Write a script that checks the status codes of URLs in the {site} sitemap and reports URLs that are not 200, not canonical or contain noindex. ``` **Sources:** ### Split sitemaps by content type Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** Separate sitemaps for products, categories, blog and help pages show in Search Console which section has indexing problems. **How to do it:** 1. Create a sitemap file per content type. 2. Group them in a sitemap index. 3. Monitor each separately in Search Console. **How to verify:** Search Console \> Sitemaps shows submitted and indexed counts per type. **AI prompt:** ```text Propose a sitemap structure for {site} split by content type and write a sample sitemap index file. ``` **Sources:** ### Submit the sitemap to Search Console and Bing Webmaster Tools Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: SEO. **Why it matters:** Submitting a sitemap unlocks error and indexing reports, so you see problems before the bots do. **How to do it:** 1. Submit it under Search Console \> Sitemaps. 2. Add the same sitemap in Bing Webmaster Tools. 3. Check error warnings monthly. **How to verify:** Sitemap status is “Success” in both Search Console and Bing Webmaster Tools. **AI prompt:** ```text Write a step-by-step guide for submitting the {site} sitemap to Search Console and Bing Webmaster Tools and the reports to watch afterwards. ``` **Sources:** ### Notify Bing and other engines instantly with IndexNow Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** IndexNow instantly notifies participating engines such as Bing, Yandex, Seznam, Naver and Yep of changed URLs. Google does not use it, but because Bing grounds Copilot, it can help changes reach Copilot answers sooner. **How to do it:** 1. Create an IndexNow key and host it at the root. 2. Ping on publish, update and delete via your CMS or CDN integration. 3. Check the IndexNow report in Bing Webmaster Tools. **How to verify:** https://{site}/\.txt returns the key and Bing Webmaster Tools \> IndexNow shows submissions. **AI prompt:** ```text Explain IndexNow setup for {site} step by step: key file, a sample API request and which events should trigger a ping. Note that Google does not use IndexNow. ``` **Sources:** , ## Performance and Core Web Vitals Slow pages lose both users and bots that work with timeouts. This section covers speed, caching and resource weight. ### Meet Core Web Vitals thresholds: LCP, INP and CLS Priority: High priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Google uses page experience in ranking, and poor experience generally works against pages being chosen as sources for AI answers. Low CLS also stops agentic browsers from clicking the wrong element. **How to do it:** 1. Find poor URL groups in the Search Console Core Web Vitals report. 2. Preload the LCP element and speed up the server response. 3. Lower INP by breaking up long JavaScript tasks. 4. Lower CLS by reserving fixed space for images and ads. **How to verify:** Search Console \> Core Web Vitals shows no “Poor” URL groups on mobile or desktop. **AI prompt:** ```text Help me interpret PageSpeed Insights results for {site}: rank the likely causes of LCP, INP and CLS issues and build a prioritized fix list. ``` **Sources:** , ### Keep server response time (TTFB) under 800 ms Priority: High priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** AI assistants that fetch pages on user request work with timeouts of a few seconds. A page that responds late gets skipped instead of included in the answer. **How to do it:** 1. Measure TTFB from several regions. 2. Set up page or edge caching. 3. Reduce slow database queries and external calls. **How to verify:** curl -o /dev/null -s -w "%{time\_starttransfer}\\n" https://{site}/ returns under 0.8 seconds. **AI prompt:** ```text Write a curl script that measures TTFB across different page types on {site}, and list likely causes and fixes for results above 800 ms. ``` **Sources:** ### Serve images as WebP or AVIF at the right size Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Large images are the most common cause of slow LCP. Modern formats cut file size significantly at the same quality. **How to do it:** 1. Convert images to WebP or AVIF. 2. Serve the right size per screen with srcset. 3. Add width and height attributes. **How to verify:** PageSpeed Insights shows no savings under the “Improve image delivery” insight. **AI prompt:** ```text Review the format, size and srcset usage of images on the {site} homepage and suggest the three optimizations with the biggest payoff. ``` **Sources:** ### Lazy-load only offscreen images, never text Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** loading="lazy" saves bandwidth on offscreen images. Applied to the hero image or text content, it delays LCP and can hide content from bots. **How to do it:** 1. Add loading="lazy" to offscreen images. 2. Remove lazy loading from the above-the-fold hero image. 3. Do not put text content in scroll-triggered components. **How to verify:** The PageSpeed Insights “LCP request discovery” insight does not flag lazy loading on the LCP image. **AI prompt:** ```text Review loading="lazy" usage on {site} pages and flag misuse on above-the-fold images or text content. ``` **Sources:** ### Minify JS and CSS and enable Brotli or gzip compression Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Compression and minification reduce transferred bytes, so bots and users receive the page faster. **How to do it:** 1. Enable JS and CSS minification in the build. 2. Turn on Brotli, or gzip as a fallback, on the server or CDN. 3. Remove unused CSS and JS. **How to verify:** curl -sI -H "Accept-Encoding: br,gzip" https://{site}/ shows Content-Encoding: br or gzip. **AI prompt:** ```text Write commands to check compression and minification of HTML, CSS and JS responses on {site}, and suggest server settings for anything missing. ``` ### Use deliberate Cache-Control headers and a CDN Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Good caching rules speed up repeat bot and user requests and reduce server load. A CDN lowers latency for requests from different regions. **How to do it:** 1. Use long max-age and versioned file names for static assets. 2. Use short caching or stale-while-revalidate for HTML. 3. Serve through a CDN and monitor the cache hit ratio. **How to verify:** curl -sI https://{site}/ shows a deliberate Cache-Control value and a CDN cache header (e.g. cf-cache-status). **AI prompt:** ```text Check the Cache-Control headers on HTML, image, CSS and JS responses from {site} and give recommended values per type in a table. ``` ### Keep DOM size reasonable Priority: Low priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Very large DOM trees increase render time and memory use and make the accessibility tree harder for agentic browsers to navigate. **How to do it:** 1. Measure DOM element count with Lighthouse. 2. Remove unnecessary wrapper divs. 3. Paginate or virtualize long lists. **How to verify:** The Lighthouse “Optimize DOM size” insight does not flag the page. **AI prompt:** ```text Assess the DOM structure of the {site} homepage and give concrete ways to reduce the element count. ``` **Sources:** ### Load web fonts without blocking visible text Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Render-blocking font loading delays text and causes layout shift. **How to do it:** 1. Use font-display: swap. 2. Preload critical fonts. 3. Drop unused font weights and character sets, keeping the subsets your languages need. **How to verify:** The PageSpeed Insights “Font display” insight shows no savings. **AI prompt:** ```text Review the font loading strategy of {site}: are font-display, preload and subsetting used well? Suggest improvements that keep language-specific characters working. ``` **Sources:** ### Measure and improve mobile performance separately Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Google indexes the mobile version, and mobile scores are usually lower than desktop. Measuring only desktop hides problems. **How to do it:** 1. Review the mobile tab in PageSpeed Insights separately. 2. Assess field data (CrUX) separately from lab data. 3. Defer heavy third-party scripts on mobile. **How to verify:** PageSpeed Insights mobile field data shows “Passed” for the Core Web Vitals assessment. **AI prompt:** ```text List likely reasons for mobile vs desktop performance gaps on {site} and draft a mobile-first improvement plan. ``` **Sources:** ## Site architecture and internal linking A clear hierarchy and strong internal links help bots find pages and understand how topics relate. ### Organize categories around topics people search for Priority: Medium priority. Effort: Project. Evidence: Official source. Roles: SEO, Content. **Why it matters:** A logical category structure helps users and models understand which topics the site is an authority on. **How to do it:** 1. Build the list of topics and subtopics from search data. 2. Assign one category page per topic. 3. Merge overlapping or empty categories. **How to verify:** The menu and category pages cover all core topics within 2 levels, with no empty categories. **AI prompt:** ```text Review the menu and category structure of {site}. Point out overlapping, missing or unnecessary categories and propose a clearer hierarchy. ``` ### Use short, readable URLs that reflect the hierarchy Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer, SEO. **Why it matters:** Meaningful URLs help models and users understand a page’s topic from the link and look trustworthy in source lists. **How to do it:** 1. Use the category path and meaningful words in URLs. 2. Transliterate non-ASCII letters consistently or use UTF-8 throughout. 3. Remove IDs and unnecessary parameters. **How to verify:** All of 20 random URLs are lowercase, hyphen-separated and made of descriptive words. **AI prompt:** ```text Review the URL structure of {site} for readability, hierarchy, non-ASCII character handling and parameters, and propose a URL guideline. ``` **Sources:** ### Keep important pages within three clicks of the homepage Priority: Medium priority. Effort: A few days. Evidence: Experimental. Roles: SEO. **Why it matters:** Deep pages are crawled less often and receive weaker signals. Pages that drive revenue or that you want cited should sit near the surface. **How to do it:** 1. Get a click-depth report from a crawler. 2. Link deep important pages from category or hub pages. 3. Set up pagination and filters so they do not add depth. **How to verify:** The crawler reports a click depth of 3 or less for important pages. **AI prompt:** ```text Draft a crawl plan to measure the click depth of important pages on {site} and suggest links that reduce it. ``` ### Add visible breadcrumb navigation Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Breadcrumbs show users and bots where a page sits on the site and provide natural internal links to parent categories. **How to do it:** 1. Add breadcrumbs to all content and product pages. 2. Make each step a real \ link. 3. Match the breadcrumb to BreadcrumbList structured data. **How to verify:** The raw HTML of a product or article page contains breadcrumb links. **AI prompt:** ```text Check whether {site} pages have breadcrumb navigation with real HTML links that match the site hierarchy, and suggest example code. ``` **Sources:** ### Link topic clusters with descriptive anchor text Priority: High priority. Effort: A few days. Evidence: Official source. Roles: SEO, Content. **Why it matters:** Internal links show which page is the main source on a topic. Anchor text like “click here” throws that context away. **How to do it:** 1. Define the supporting pages that should link to each key page. 2. Use the target page’s topic in the anchor text. 3. Add links from related older pages when new content goes live. **How to verify:** The crawler shows inbound internal link counts and anchor texts for key pages that match their topics. **AI prompt:** ```text Propose a method to analyze inbound internal links and anchor texts for key pages on {site}, and find weak anchors like “click here” and missing links. ``` ### Link or remove orphan pages Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: SEO. **Why it matters:** Pages found only in the sitemap get weak signals and are crawled rarely. Link them if they are valuable, remove them if not. **How to do it:** 1. Compare sitemap URLs with linked URLs found in a crawl. 2. Link valuable orphan pages from related pages. 3. Remove or redirect low-value ones. **How to verify:** A sitemap vs crawl comparison shows close to zero orphan pages. **AI prompt:** ```text Write the steps to find orphan pages on {site} by comparing the sitemap with crawl data, and suggest decision criteria for each orphan page. ``` ### Serve the homepage at a single URL Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Variants such as /index.html, /home or /?ref= split homepage signals and lead to different addresses being cited. **How to do it:** 1. List homepage variants. 2. Redirect them all to the preferred root with a 301. 3. Point logo and menu links to the root. **How to verify:** curl -sI https://{site}/index.html returns a 301 to https://{site}/. **AI prompt:** ```text Write commands that test likely homepage URL variants of {site} (index.html, index.php, /home, with parameters) and explain the correct behavior. ``` **Sources:** ### Build pagination with crawlable links Priority: Low priority. Effort: A few days. Evidence: Official source. Roles: Developer. **Why it matters:** Lists that only use a “load more” button or infinite scroll hide products and posts beyond the first page from bots. **How to do it:** 1. Serve every pagination step at its own URL with \. 2. Back infinite scroll with paginated URLs. 3. Use self-referencing canonicals on paginated URLs. **How to verify:** The raw HTML of a category page contains next-page links such as ?page=2. **AI prompt:** ```text Check whether pagination on {site} category and blog lists is crawlable. If infinite scroll or a JavaScript button is used, explain the recommended fix. ``` **Sources:** ## Page metadata and headings Title tags, descriptions, heading hierarchy and sharing tags tell bots what a page is about at first glance. ### Write a unique, descriptive title tag for every page Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: SEO, Content. **Why it matters:** The title tag is shown in source lists and search results, and models infer the page topic from it first. **How to do it:** 1. Put the main topic first and the brand last. 2. Stay around 30–60 characters. 3. Eliminate duplicate titles. **How to verify:** A crawler reports close to zero empty, duplicate or over-60-character titles. **AI prompt:** ```text Assess the title tags of the {site} homepage and 5 key pages for length, uniqueness and clarity, and write improved alternatives. ``` **Sources:** ### Write a unique, summarizing meta description for every page Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: SEO, Content. **Why it matters:** Meta descriptions do not affect ranking directly, but they may be used as page summaries in results and some AI interfaces. **How to do it:** 1. Summarize the page’s promise in 120–160 characters. 2. Include a page-specific fact or number. 3. Fix template-driven duplicates. **How to verify:** A crawler reports close to zero missing or duplicate meta descriptions. **AI prompt:** ```text Assess the meta descriptions of 5 key {site} pages and write new page-specific suggestions under 160 characters. ``` **Sources:** ### Use one H1 per page that states its topic Priority: Medium priority. Effort: Quick win. Evidence: Experimental. Roles: Content, Developer. **Why it matters:** A single, clear H1 tells bots the page’s main topic. H1s on the logo or menu blur that signal. **How to do it:** 1. Remove H1s from the logo and menu in templates. 2. Mark the main heading on each page as H1. 3. Keep the H1 aligned with the title tag. **How to verify:** curl -s https://{site}/ | grep -c "\ Priority: Medium priority. Effort: Quick win. Evidence: Emerging standard. Roles: Content. **Why it matters:** Heading hierarchy splits a page into sections. AI systems use these sections when choosing passages for answers, and skipped levels break the structure. **How to do it:** 1. Use H2 after H1 and H3 under H2. 2. Pick heading levels for meaning, not visual size. 3. Keep each H2 section to one subtopic. **How to verify:** A heading outline tool (e.g. HeadingsMap) shows no skipped levels. **AI prompt:** ```text Extract the heading structure of a content page on {site}, flag skipped levels and empty or vague headings, and propose a corrected outline. ``` ### Use complete Open Graph tags on every page Priority: Medium priority. Effort: Quick win. Evidence: Official source. Roles: Developer, Content. **Why it matters:** og:title, og:description and og:image are used in link previews and in source cards of some AI interfaces. **How to do it:** 1. Add og:title, og:description, og:image, og:url and og:type. 2. Make the image about 1200x630 pixels. 3. Generate values per page. **How to verify:** An Open Graph preview tool shows the right title, description and image. **AI prompt:** ```text Check the Open Graph tags on the {site} homepage, flag missing or wrong fields and write a correct example. ``` ### Add X (Twitter) card tags Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** twitter:card tags control link previews on X and in other tools that read them. **How to do it:** 1. Set twitter:card to summary\_large\_image. 2. Remember missing fields fall back to Open Graph values. **How to verify:** The page source contains \. **AI prompt:** ```text Check the twitter:card tags on {site} and write the recommended minimal tag set alongside Open Graph. ``` ### Write alt text that describes image content Priority: Medium priority. Effort: A few days. Evidence: Official source. Roles: Content. **Why it matters:** Alt text tells bots and screen readers what an image shows, and multimodal models use it to tie the image to the page context. **How to do it:** 1. Write specific alt text for every informative image. 2. Use empty alt="" for decorative images. 3. Avoid keyword stuffing. **How to verify:** A crawler shows under 5% of informative images missing alt text. **AI prompt:** ```text List the alt texts of images on the {site} homepage and write better alternatives for missing, vague or keyword-stuffed ones. ``` **Sources:** ### Define a favicon and preferred site name Priority: Low priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** Google and some AI interfaces show sources by site name and favicon. Missing or inconsistent values make your brand harder to recognize. **How to do it:** 1. Publish a square favicon of at least 48x48 pixels. 2. Declare the site name with name and alternateName in WebSite structured data. 3. Keep it consistent with the title and og:site\_name. **How to verify:** Google search results show the correct site name and favicon. **AI prompt:** ```text Check the favicon definitions and site name signals (WebSite JSON-LD, og:site_name, title) for {site} and list inconsistencies. ``` **Sources:** , ## Structured data (Schema.org) Structured data tells machines explicitly which entities a page describes and how they relate. Google requires no special markup for AI features, but correct JSON-LD directly supports entity recognition and shopping surfaces. ### Publish structured data as valid JSON-LD Priority: High priority. Effort: Quick win. Evidence: Official source. Roles: Developer. **Why it matters:** JSON-LD is Google’s recommended format and is easy to maintain because it is separate from the HTML. A single syntax error makes the whole block ignored. **How to do it:** 1. Add structured data as \