Signal Observatory
One robots.txt rule against AI training, and brand AI Overviews move aside
Google and Bing back one robots.txt rule that says no to AI training, and Bing splits training from grounding. Brand AI Overviews move to the right column, and Reddit counts what AI Mode costs it.
Signal readout
Topic mix
- AI: 33%
- Google & Search: 36%
- PPC: 11%
- Analytics: 12%
- Brands: 8%
- AI33%
- Google & Search36%
- PPC11%
- Analytics12%
- Brands8%
Share of the issue's words about each topic, counted section by section.
Why it matters
Nothing here needs an edit today, and nothing is enforced yet. Two things are worth doing now: if your pages use NoArchive or NoCache to keep them out of AI answers, plan to replace them with Bing's grounding tag before 1 March 2027; and if you add the training token, check that you have not also blocked the search crawlers. A User-Agent: * group never matches the token: the draft says wildcard groups "MUST NOT" be read as matching it.
The short answer
- What
- A shared robots.txt token for AI training backed by Google and Microsoft; Bing's separate training and grounding controls; brand AI Overviews moved to the right column; Reddit's traffic loss; a working paper on ChatGPT ads; a one-site test of opting out of AI features; Lighthouse 13.6; publisher lawsuits over AI answers.
- Why
- Because for the first time two search companies agreed on one signal that separates training from everything else, and because the clearest numbers of the week, from Reddit and from ChatGPT's advertisers, both describe AI answers that keep the click.
- Who
- Site owners and publishers deciding what AI may do with their pages; brands watching their own search results; advertisers weighing ChatGPT ads against Google; anyone who reads PageSpeed Insights scores.
- Where
- Google Search, AI Overviews, AI Mode, Bing and Copilot, ChatGPT, Reddit, Brave, Discord, PageSpeed Insights and Google Tag Manager.
- When
- 30 October – 3 November 2026, plus a few items announced on 28–29 October and reported inside the window.
- How we know
- Google Search Central and the Bing blog, Reddit's shareholder letter, the working paper's own page, the Lighthouse release notes, and Search Engine Roundtable, Search Engine Journal and PPC Land for what had no primary page.
In plain words
AI companies read websites for two reasons: to teach their models, and to look things up when someone asks a question. This week Google and Microsoft agreed to respect one shared line in a website's robots.txt file that says "do not use my pages for teaching", and Microsoft also gave sites a separate switch for "do not use my pages to answer questions". Google started showing its AI summary for brand searches at the side of the page instead of on top. Reddit said it gets about a million fewer visits a week from Google because people read the AI answer and stop there. And a study found that shoppers who click ads in ChatGPT spend much less than shoppers who click ads in Google.
This week the arguments about AI and the open web turned into settings. Google and Microsoft both said they will read one shared robots.txt rule that opts a site out of AI training, and Bing went further: it now lets a page say no to training and no to grounding separately. Google moved the AI Overview for many brand searches into the right-hand column. And the clearest numbers of the week came from two companies that depend on clicks: Reddit said search sends it more than a million fewer referrals a week, and a working paper put a price on a ChatGPT ad click.
One robots.txt rule for AI training
On 30 October John Mueller announced on the Google Search Central blog that Google will support a shared "policy token" in robots.txt that lets a site "set broad preferences across participating AI providers". The token comes from an IETF draft, draft-kelley-aipref-training-preference-signal. Google's post does not print the syntax. Microsoft's post, published the same day, does.
User-Agent: __usage_ai-training
Disallow: /The rule is not a crawler. It does not stop anybody from fetching a page; it states what the page may be used for. Google says that a site which uses the token and says nothing about Google-Extended will be read "as if the publisher had disallowed Google-Extended", and that "Existing company-specific controls will continue to function alongside this new joint signal." There is no date yet: Google "aims" to support it in 2027. Search Engine Journal reports, on 1 November, that OpenAI intends to honour the signal in the next few months and that Anthropic has started building support; we could not open OpenAI's own post. Nate Hake, who takes part in the IETF work, warned on X that the standard "is not actually done at the ietf level".
Bing separates training from grounding
Krishna Madhavan's post on the Bing blog goes further than Google's. Bing now reads two different preferences. AI training decides whether a page may teach a model; it can be set in robots.txt with the shared token, or on a page. AI grounding decides whether a page may be fetched to support an answer in Copilot and in the grounding services Microsoft sells to other companies. Grounding is set only on the page, with a meta tag or an X-Robots-Tag header; there is no robots.txt rule for it.
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">The third value keeps a page in Microsoft's own products and out of the grounding Microsoft provides to third parties. The change also retires two old signals: from 1 March 2027 NoArchive and NoCache stop counting as training or grounding preferences. Bing is blunt about what the switches buy: "Neither choice guarantees inclusion, traffic, or payment."
| Date | What happens | Who |
|---|---|---|
| 30 Oct 2026 | Shared training token announced; Bing training and grounding controls published | Google, Microsoft |
| 15 Dec 2026 | Temporary grounding preferences open in Bing Webmaster Tools for sites using NoArchive or NoCache | Microsoft |
| 1 Feb 2027 | Bing starts applying the new preferences as it recrawls | Microsoft |
| 1 Mar 2027 | NoArchive and NoCache no longer read as training or grounding preferences | Microsoft |
| 2027, no date | Google aims to support the shared token | |
| 3 May 2027 | The IETF draft expires unless it is renewed or replaced | IETF |
| 30 Sep 2027 | Temporary grounding preferences expire | Microsoft |
Publishing a setting is not the same as activating it.
Krishna Madhavan, Microsoft Bing, "Your content, your choice", 30 October 2026. Bing has to recrawl a page before it sees the new preference.
Try it on your own page
See what Voicescope finds on a page of yours
Everything above is measurable on your own work, and the check takes about a minute. Paste one address and read what comes back.
Open VoicescopeFree · no account · one at a time
Brand AI Overviews move to the right column
On 2 November Search Engine Roundtable showed brand searches on desktop where the AI Overview no longer sits above the results. It sits in the right-hand column, in the place of the knowledge panel, and the brand's own site leads the main column. The examples were Apple, Zillow, Coca-Cola, Google and Search Engine Roundtable itself. Chris Long, who spotted it, wrote: "Whoa, Google is changing the way they're showing AI Overviews for brand searches." Barry Schwartz's reading was that Google is "essentially replacing brand knowledge panels with AI Overviews", and that "this works much better than having it in the main results". Google has not commented, so it is not known whether this is a test.
For a brand this is a better place and a riskier text. The site wins back the top of the page, but the panel that used to show checked facts now shows a generated summary. PPC Land, writing on 3 November, found the first error in its own examples: Zillow's AI text gives a founding year of 2006 while the fact row in the same panel says 2004.
Reddit: AI answers cost a million referrals a week
Reddit's third-quarter shareholder letter, written up by Search Engine Journal on 30 October, says search referral traffic was down meaningfully, and puts a number on October: "During October, we have seen that search traffic has dropped by more than 1 million referrals on average per week." Chief executive Steve Huffman named Google's AI Overviews and AI Mode as the biggest part of Reddit's explanation. He also said that "Google is crawling us as hard as ever": the content is still read, and the reader no longer arrives. Reddit will test a logged-in-only version in markets outside the US in the fourth quarter. Its chief operating officer, Jen Wong, said Reddit is "pretty skeptical of GEO".
The loss shows up exactly where search traffic lives, among people who are not logged in. Global daily active users stayed at 130.3 million.
| Reddit, US daily active uniques | Q2 2026 | Q3 2026 | Change |
|---|---|---|---|
| Logged out | 30.1 m | 27.3 m | −2.8 m |
| Logged in | 23.1 m | 23.4 m | +0.3 m |
| All US | 53.2 m | 50.7 m | −2.5 m |
AI visibility: what an AI impression is worth
Cyrus Shepard published a small test on 30 October, written up by Search Engine Roundtable the same day. He used Google's control to keep one site out of AI Overviews and AI Mode, then compared the period before and after. AI features had brought about 8% of the site's impressions, and he estimates their click-through rate at "~0.9%". The surprise was the other direction: "regular organic clicks ALSO DROPPED when AI features were disabled." His explanation is that "the more places you can get your brand to appear, the more likely a user will click at least one of them". It is one site, and its average position changed during the test, so treat it as a question, not an answer.
Search Console will not settle it soon. After a LinkedIn post from a Search Central event suggested that clicks and queries were coming to the AI report, Gary Illyes replied on 2 November: "The team is super interested in giving more data, data that makes sense, but we don't have anything to announce yet." Today the AI report shows impressions; AI clicks stay mixed into the ordinary Performance report. Search Engine Journal notes that the UK competition regulator's requirements for Google, which include clicks for AI Overviews and AI Mode, apply from 3 December.
Tracking tools have their own gaps. PPC Land reported on 29 October that Semrush will stop tracking Microsoft Copilot in its enterprise AI visibility product on 9 November, because "Recent changes to Microsoft Copilot's access requirements prevent AI visibility platforms" from reading it. Eight other platforms stay.
PPC: a ChatGPT ad visit is worth a third of a Google one
The most cited paper of the week is a working paper by Maximilian Kaiser (WU Vienna) and Christian Schulze (Frankfurt School), "Can Advertising Fund Free AI?", written up by PPC Land on 2 November. It reads Google Analytics 4 ecommerce data for nearly 10 billion sessions over 60 weeks, from 29 June 2025 to 22 August 2026: 1,898 sites for organic traffic and 387 that bought ChatGPT clicks. For US traffic, a paid ChatGPT visit brought $0.89 in revenue against $2.45 for a paid Google Search visit, and converted at 0.37% against 1.17%. Of 87 advertisers who bought both, 73 recorded no paid ChatGPT sale at all.
Revenue per paid visit, US, $ (Kaiser and Schulze, working paper, October 2026)
The paper's conclusion is blunt: "ChatGPT ads offer more than six times worse value for money than Google Search ads." At the revenue these visits bring, it puts a fair price for a ChatGPT click at about $0.26, against $1.72 paid by the largest account whose price is public. It also found that organic ChatGPT referrals fell 5.9% on average in the 13 weeks after ads launched, with an interval from −12.6% to +1.2%, so the fall is not certain.
OpenAI keeps building the product regardless. Search Engine Roundtable reported on 30 October, from OpenAI's weekly advertiser email, "a new, wider standard ad card that uses more of the conversation width on desktop", that multi-image ads "are now available in open beta to all advertisers", and that the Shopify integration now works outside the US. The rest of the week's ad changes:
| Platform | What changed | Status | Date |
|---|---|---|---|
| ChatGPT Ads | Wider standard ad card on desktop; multi-image ads in open beta | Rolling out | 30 Oct |
| Google Tag Manager | Server-side containers pass full IP addresses to Google Ads, DV360, SA360 and CM360 tags; Analytics stays redacted | From 11 Jan 2027, opt-out | 2 Nov |
| Google Ads | "Promotions offers" section inside Promotions | Spotted | 2 Nov |
| TikTok Search Ads | Impression share and top-1 impression share; brand and category keyword packs | Announced | 30 Oct |
| Bing | Sponsored results inside a Bing AI Overview | One screenshot, unconfirmed | 30 Oct |
| X | Lead Gen Ads for all advertisers | Generally available | 30 Oct |
Brand: who travel AI answers name
BrightEdge measured share of voice for holiday travel in AI Overviews and ChatGPT, written up by Search Engine Journal on 31 October. The two engines do not agree on the leader in every category: Hilton leads hotels in AI Overviews at 28.3%, Marriott leads them in ChatGPT at 29.4%. Booking platforms appear in 78% of AI Overviews answers to budget queries with words like "cheap" or "last-minute", against 17% of other answers; BrightEdge adds that this "is an observed association, not evidence that using these words causes a brand to appear."
Share of voice in Google AI Overviews, holiday travel, % (BrightEdge)
Publishers went to court in the same days. On 2 November The New York Times and three Tribune papers sued Brave in New York over its AI answers and its ad blocking. PPC Land reports that in the Times's test, Brave's LLM Context endpoint returned 1,291 of the 1,295 words of one story, and that the complaints say Brave Search answers more than 50 million queries a day. On 29 October the Wikimedia Foundation asked the court in the OpenAI case, in a proposed brief, to hold that copying sources into answers to user queries is not fair use, while it calls the training of the models "likely fair use". The line both cases draw is the one Bing drew this week: training is one question, answering with your page is another.
SEO: Lighthouse changes scores, and Mueller on technical SEO
PageSpeed Insights now runs Lighthouse 13.6.0, released on 27 October, and Search Engine Journal explained on 3 November why lab scores may move on pages nobody touched. Lighthouse's simulator now uses Chrome's own label for which requests block rendering, and estimates LCP differently when the largest element is text. Barry Pollard of Chrome says the change is "more likely to impact JS-heavy sites". Compare only reports made with the same Lighthouse version. The same release reorders its check for agent manifests: an Agentmap line in robots.txt, a rel="ard" link, then /.well-known/ard.json, and only then the older ai-catalog names.
In an interview with David Iwanow, written up by Search Engine Journal on 2 November, John Mueller said: "I think especially technical SEO is going to remain very relevant," and "I think you'll see that a lot of the AI systems, they build on SEO." AI crawlers, he said, crawl with different intents, for training or for fresh content, which he called "another flavor of technical SEO". On listicles: "I wouldn't necessarily say anything that is a listicle is spam," but a strategy made only of them will "at some point" switch off.
A new source of user-generated content is on the way. Discord said on 30 October that from 16 November every discoverable server gets a public web page with AI-written channel summaries, and that from mid-December some public-channel messages may appear on those pages and in web search, without usernames. Discord counts more than 63,000 discoverable servers. Members can turn their messages off; in the EU, the UK, Canada, Brazil, Australia and South Korea the setting starts off.
Updates and links: a quiet week
Google's Search Status Dashboard lists no update that started or finished between 30 October and 3 November; the last confirmed one is the September 2026 spam update. We found nothing on link building published in the window that is worth your time, so this issue has no link section. One sighting to watch: on 30 October Brodie Clark saw the num=100 parameter, which Google removed in September 2025, returning 99 organic results after a click from a knowledge panel. Search Engine Roundtable could not repeat it, and Google has said nothing. If it is real, rank trackers will notice first.
Our news: agent skills that ask before they act
Two stories of this week are about permission: who may train on a page, who may answer with it. The same question applies when you hand your own site to an AI agent. GLLmolfar, our desktop crawler, gives an agent 20 skills over MCP, and this week we rewrote them as checklists: what must be true before the agent starts, the steps, how it checks its own result, what it reports, and when it must stop and ask you. The agent asks before it changes code, opens issues or crawls a site other than yours, and it tells you the URL limit before any crawl.
Each skill is on the GLLmolfar page with the real answer of an agent that ran it, where we have one, and a button that copies the whole checklist for your own agent.
Also this week
- Connected apps in AI Mode. Google opened a public preview on 28 October for apps that run inside AI Mode and the Gemini app, through a remote MCP server or Android AppFunctions. In Search it is US and English only, with Instacart, Canva, YouTube Music, Yahoo Fantasy Football and Sleeper live. Search Engine Journal, 31 October.
- What AI agents do on websites. Microsoft Clarity counted more than 5.2 million sessions it classified as agents and found checkout in 25.21% of them, against 14.65% of human sessions; agents clicked out less often, 14.28% against 26.99%. Published 29 October; PPC Land, 1 November, notes that the counts imply a smaller base than 5.2 million.
- Publishers' numbers. A Digital Content Next court brief says its members lost 26% of unique visitors between 2023 and 2025, and quotes an internal Microsoft line about a "'doom loop'" for AI models. DCN gives no method. PPC Land, 1 November.
- 83 checks from Google's own guidance. Aleyda Solis turned Google's content quality guidance into 83 checks, 76 of them citing 11 Google documents, with a Claude skill. Not written or reviewed by Google. PPC Land, 3 November.
- Fake faces in Google Ads outreach. Joe Youngblood says a "Google Ads Strategist" who contacted him had an AI-generated photo and no LinkedIn profile. Google's reps use @xwf.google.com addresses; Google did not answer Search Engine Roundtable. 2 November.
- Ad business results. Apple said its advertising business set an all-time revenue record, without a figure, on 2 November. S&P moved Meta's outlook to negative on 30 October, citing AI spending of "more than $225bn" forecast for 2027.
Questions people ask
How do I opt my site out of AI training in robots.txt?
Add a group for the shared token: User-Agent: __usage_ai-training, then Disallow: /. Microsoft documents it now; Google aims to support it in 2027 and will treat it as a Google-Extended disallow if you do not name Google-Extended. The token does not stop crawling, and existing rules for individual bots keep working.
What is the difference between AI training and AI grounding?
Training uses a page to teach a model. Grounding fetches a page to support a specific answer, as Copilot does. Bing now takes them separately: training in robots.txt or on the page, grounding only on the page, with ai-grounding:disallow or ai-grounding:disallowextended.
Why did the AI Overview for my brand move to the right?
Since about 2 November Google has shown AI Overviews for many brand searches on desktop in the right column, where the knowledge panel was, with the brand's site first in the main results. Google has not confirmed whether it is a test.
Are ChatGPT ads worth it compared with Google Ads?
One working paper says not yet: for US traffic, $0.89 of revenue per paid ChatGPT visit against $2.45 for Google Search, and 73 of 87 advertisers that bought both saw no paid ChatGPT sale. It is not peer reviewed and uses one data provider's panel.
Will Search Console show clicks from AI Overviews?
Not announced. On 2 November Gary Illyes said the team wants to give more data but has nothing to announce. The AI report shows impressions; AI clicks are counted inside the ordinary Performance report.
Sources
Every claim above is dated and linked. Where a number comes from a measurement rather than an announcement, whoever measured it is named.
- Supporting a shared AI training control for websites — Google Search Central, 30 October 2026
- Your content, your choice: clearer preferences for training and grounding — Microsoft Bing, 30 October 2026
- AI training preference signal, IETF draft — IETF
- Google, Microsoft and OpenAI back a shared AI training opt-out — Search Engine Journal, 1 November 2026, and Search Engine Roundtable, 30 October 2026
- Google AI Overviews for brands moved to the right column — Search Engine Roundtable, 2 November 2026
- Google moves AI Overviews for brand searches out of the main results — PPC Land, 3 November 2026
- Reddit to test login-only mode as Google search traffic drops — Search Engine Journal, 30 October 2026, and Reddit's Q3 2026 shareholder letter
- Google AI Overviews and AI Mode 0.9% click-through rate, opt-out study — Search Engine Roundtable, 30 October 2026
- Click data in Search Console AI reporting — Search Engine Roundtable, 2 November 2026, and Search Engine Journal
- Semrush drops Copilot from AI visibility tracking on November 9 — PPC Land, 29 October 2026
- Can Advertising Fund Free AI? — Kaiser and Schulze, working paper, October 2026, and PPC Land's write-up, 2 November 2026
- ChatGPT gets wider ads, multi-image ads expands — Search Engine Roundtable, 30 October 2026
- Google ad tags gain full visitor IP addresses in server-side GTM in 2027 — PPC Land, 2 November 2026
- Google Ads promotion offers — Search Engine Roundtable, 2 November 2026
- TikTok Search Ads adds brand and category keyword packs — Search Engine Journal, November 2026, and TikTok's Q4 product preview
- Bing showing ads within AI Overviews — Search Engine Roundtable, 30 October 2026
- Big brands dominate AI search but leave openings for smaller sites — Search Engine Journal, 31 October 2026
- New York Times and three papers sue Brave over AI answers and ad blocking — PPC Land, 3 November 2026
- Wikimedia asks judge to rule ChatGPT's web retrieval is not fair use — PPC Land, 2 November 2026
- PageSpeed Insights scores and the Lighthouse update — Search Engine Journal, 3 November 2026, and Lighthouse 13.6.0
- Google says technical SEO remains key for AI SEO — Search Engine Journal, 2 November 2026
- Discord public server chats in search engines — Search Engine Journal, 30 October 2026, and Discord's post
- Google Search Status Dashboard and Google may be bringing back the 100 results parameter — Search Engine Roundtable, 30 October 2026
- Google opens connected apps preview for AI Mode and Gemini — Search Engine Journal, 31 October 2026
- Checkout appears in 25% of AI agent sessions vs 15% for humans — PPC Land, 1 November 2026
- Publisher group says AI cut visitors 26% — PPC Land, 1 November 2026
- Aleyda Solis turns Google's content quality guidance into 83 checks — PPC Land, 3 November 2026
- Google Ads strategists using AI-generated fake names and photos — Search Engine Roundtable, 2 November 2026
- Apple's ad business sets an all-time record and S&P turns negative on Meta — PPC Land, 30 October – 2 November 2026
- GLLmolfar — GetLoopLoop, desktop crawler, preview
The product
Watch it on every page, every day
One reading tells you where a page stands today. The product asks the same questions of the same assistants continuously, so a change is something you are told about rather than something you go looking for.
Request an invitationInvite-only while we keep the readings honest
On this topic
What changed in classic search?
Every day Search Console beside the assistants' answers. How the product works
Author
Olexandrauthor
15+ years in marketing and SEO. Dozens of products, mostly the US market. Research and measurement.
LinkedInPass it on
Read in your language
Opens a browser translation of this English article. The original source stays in English.