top of page

AI Search in Singapore: What the New Robots.txt Control Means

Writer: Bobby Harjani
Bobby Harjani
Nov 4
4 min read

Updated: Nov 7

Short answer: Google has announced plans to support a shared robots.txt signal that lets websites express an AI-training preference across participating providers. Google aims to support it in 2027. It concerns AI training preferences; it does not automatically tell search systems whether to crawl, index or show your pages. Singapore businesses should review the distinction before changing their crawler rules.

Google plans to support a shared AI-training preference in robots.txt in 2027. Learn what it means for Singapore businesses, SEO and AI search visibility.
Google plans to support a shared AI-training preference in robots.txt in 2027.

When businesses explore answer engine optimisation (AEO) or generative engine optimisation (GEO), a common question is whether they should let AI systems access their website. A new announcement from Google makes that question more specific.

On 30 October 2026, Google said it plans to support a shared AI-training control in robots.txt, based on work by the Internet Engineering Task Force’s AIPREF working group. The proposed signal would let website owners express a broad preference for participating AI providers. Google says it aims to support the control along with other AI providers in 2027. That means this is an announced plan, not a setting businesses should rush to add today.

Training preferences and search visibility are different

A crawler rule can have different purposes. Some crawlers collect information that may be used to train models. Other systems crawl pages to make them available in search or retrieve information to answer a user’s question.

Google’s existing Google-Extended token is a useful example. Google says it can be used to manage whether content it crawls may be used to train future Gemini models and for grounding in Gemini Apps. Google also says that using Google-Extended does not affect a site’s inclusion or ranking in Google Search.

This matters for SEO and GEO. A publisher might not want content used for model training while still wanting its pages to be discoverable through search. Those are separate preferences. Treating every AI-related crawler as one switch could lead to blocking access you intended to keep open.

What should a Singapore business do now?

For now, don’t add a new shared token just because the proposal was announced. Google’s stated target is support in 2027, and the underlying IETF work is a draft standard in development. Review your current directives and decide what you want each crawler to be able to do.

Start with the business purpose of each page. A tuition centre may want parents to find its programmes and class details in search. A café might want its menu and event pages discoverable. A B2B supplier may want product specifications accessible to buyers comparing options. Those pages should be crawlable if search visibility is a goal.

Then make a separate, deliberate decision about training preferences. If your business has a reason to restrict model training, document that preference and check the current instructions from each provider. Don’t assume that one setting will govern all providers before they announce support for it.

Check robots.txt before changing it

robots.txt gives crawler instructions, but it is not a way to hide a page from search results. Google advises using noindex or password protection to keep a page out of Search; blocking a URL in robots.txt can still leave its address visible in results.


Before editing the file, check which rules already exist, which user agents they target and whether key pages or resources are accidentally blocked. If the site is built on a platform such as Wix, confirm how the platform manages robots.txt and avoid adding directives you cannot explain.

What the announcement means for AEO and GEO

A training preference does not make a business more likely to be cited. Nor does allowing training guarantee that a page will appear in an AI answer. These controls express access preferences; visibility depends on other factors, including whether information is accessible, relevant and useful for the user’s question.

For an SME, the practical GEO and AEO work is still to explain clearly what the business offers, who it serves, where it operates and what evidence supports its claims. A Singapore service page should answer real customer questions and make the next step—such as booking, calling or requesting a quote—clear.

Siloam’s broader offer spans SEO, lead generation, paid advertising and social media as well as GEO. That integrated approach is useful because an AI mention is only one possible discovery point; customers still need clear information and a reason to enquire.

A simple review checklist

Before changing any crawler settings, ask:

  • Which pages do we want customers to discover through search?

  • Are those pages currently accessible to Googlebot?

  • Do our current rules distinguish search access from training preferences?

  • Can we explain what each rule does and which pages it affects?

  • Have we checked the latest guidance from the providers we care about?


Keep a dated record of changes. Then monitor Search Console and website enquiries to see whether your important pages remain available and useful to customers.


FAQs

Is the new shared AI-training control available now?

Google announced plans to support it, with a target of 2027. The announcement does not say that businesses can use the shared control today.

Does blocking AI training stop a page appearing in Google Search?

Not necessarily. Google says its existing Google-Extended control does not affect inclusion or ranking in Google Search. Search crawling and AI-training preferences are distinct controls.


Does robots.txt remove a page from search results?

No. Google recommends noindex or password protection to prevent a page from appearing in Search. robots.txt controls crawling and should not be treated as a removal tool.

Will allowing AI crawlers improve GEO results?

There is no guarantee. Access rules alone do not ensure that an AI system will use, cite or recommend a business. Clear, accessible and useful information remains the sensible foundation.

Make AI search decisions with a clear purpose

The new proposal is a reason to review your website’s crawler rules, not to copy a new directive without understanding it. Singapore businesses should separate the goal of being discoverable in search from their preferences about AI training, then make deliberate choices for each.

AI Search in Singapore


Want to review your website’s SEO and AI search readiness? Contact Siloam Technologies for a marketing audit and a practical discussion about your next steps.


Related Case Study & Research: Curious how AI search engines actually choose which local companies to cite? Read our teardown: We Tested 50 Singapore SME Prompts on ChatGPT, Gemini & Perplexity: Why Some Local Brands Get Recommended (And Others Don't).

Comments


bottom of page