Polski
Technical SEO · Guide

AI bots and robots.txt: who to let in and who to block

OpenAI does not have one bot. Its documentation lists four today, and each of them is a different transaction. Which one decides whether ChatGPT cites you, which one collects training data, why one of them cannot be blocked with robots.txt, and how to check that it really is an OpenAI bot and not someone hiding behind a spoofed header.

Sebastian Baryś
Sebastian Baryś
10 August 2026 17 min read

Introduction

English version of an article first published in Polish on 10 August 2026. Read the Polish original.

The conversation about AI bots usually comes down to one question: let them in or block them. It is the wrong question, and that is why it so often ends in the wrong decision.

OpenAI does not have one bot. Its documentation lists four today, each with a different role and each a completely different transaction. The list is growing: the most recent one joined the documentation in April 2026. One takes content for training and gives nothing immediate in return. The second decides whether ChatGPT can cite you. The third checks ad landing pages. The fourth is not bound by robots.txt at all.

I break this down into four separate decisions, show how to verify a bot by its address instead of trusting the header, and what to look for in the logs, because on the OpenAI side there is simply no equivalent of Search Console.

You will learn:

  • How the four OpenAI crawlers differ and why each one is a separate decision
  • Which bot decides whether ChatGPT can cite you
  • Why one of them cannot be blocked with robots.txt, and why that is not a bug
  • How to verify a bot by IP address instead of trusting a header that can be forged
  • What to look for in server logs, given that OpenAI offers no equivalent of Search Console
  • How OpenAI crawling compares with Google in hard numbers

Not one decision, but four

The most important sentence in the whole OpenAI documentation looks unremarkable: each setting is independent of the others. You can let one crawler in and block another, because this is not a single switch.

In practice it means you have four separate decisions to make, and each concerns a different exchange. With training, you hand over content and get in return a promise that the model "will know" one day that you exist. With search, you hand over access and get a chance of a citation with a link. These are two completely different deals and there is no reason to answer them the same way.

You hand over content for training

you get: a place in the model's "memory", one day

You hand over access for search

you get: a citation with a link, now

You let the ads bot in

you get: the ability to advertise

A fetch at the user's request

this decision is not made in robots.txt

Four crawlers are four different exchanges. The answer "we block everything" treats them as one.

The four OpenAI bots, one by one

Below are all four, with their exact names from the documentation. The name matters, because a robots.txt rule has to match the token, not a colloquial label. In the logs you will see them with a version number, for example GPTBot/1.4. The numbers go up over time, but in robots.txt you specify the token alone, without the version.

GPTBot Model training respects robots.txt

Collects content that may be used to train OpenAI models. It gives you nothing directly: you pay with content so that the model "will know" in the future that you exist.

OAI-SearchBot Search in ChatGPT respects robots.txt

This is the one that decides citations. It is responsible for surfacing websites in ChatGPT's search features. Blocking it means dropping out of the answers, even if the model knows your brand.

OAI-AdsBot Ads respects robots.txt

Checks the safety of ad landing pages and assesses their relevance. In Poland it affects hardly anyone today, but that will change on the day ChatGPT ads reach the Polish market.

ChatGPT-User Fetch on request outside robots.txt

Visits a page when a user or a Custom GPT explicitly asks for it. This is not crawling in the classic sense, it is carrying out a person's instruction, and that is where its special status comes from.

If you were to remember one thing from this list, let it be the difference between the first and the second. GPTBot is training, OAI-SearchBot is citations. Confusing them is the source of the most expensive mistake on this topic, which I come to in a moment.

What robots.txt will not do

Three of the four crawlers respect robots.txt. For the fourth, ChatGPT-User, the documentation states outright that because these actions are initiated by a user, "robots.txt rules may not apply". In practice that means you cannot count on robots.txt here.

This is not a loophole, it is a design decision

The logic is that when a person pastes a link into ChatGPT and asks for a summary, that is not crawling. It is carrying out their instruction, much like opening the page in a browser. You may disagree with that, but it helps to understand why the file does not cover it: robots.txt governs robots, not users.

The practical conclusion: if you really have to cut something off, you do it at the level of access, with a login, a server rule or an IP block. A robots.txt entry is a request, not a lock on the door, and that is how it should be treated for every crawler, not just this one.

Two small details that save your nerves

A change takes a day

OpenAI says its systems take about 24 hours to adjust to a new file. Do not judge the results that same afternoon.

One fetch instead of two

When both crawlers are allowed, OpenAI may use the result of one crawl for both purposes, so as not to fetch the same page twice.

How to check it really is OpenAI

The User-Agent header is plain text sent by the client. Forging it takes a second, and the logs of any larger site show traffic impersonating well-known crawlers, sometimes to get around protections, sometimes to collect content under someone else's name.

That is why the only reliable verification is by IP address. OpenAI publishes current ranges in separate JSON files: one each for the search crawler, the ads crawler, the training crawler and fetches on request. You take the addresses from there and compare them with what you see in the logs.

The log shows a header

GPTBot/1.4

that is the client's claim

You check the IP address

in OpenAI's JSON file

that is the only proof

It matches: it is OpenAI

no match: someone is impersonating it

The header says who the client claims to be. The address says who it really is.

What the server logs show

Here we get to something that has to be said plainly: OpenAI offers no tool along the lines of Search Console. There is no dashboard, no report, no notifications. The only place where you can see whether these crawlers have visited you at all is the server logs.

Three things are worth looking for in them. First, whether the crawlers come at all, because the absence of entries is an answer in itself. Second, which URLs they visit, because that shows which part of the site the system considers worth fetching. Third, which response codes they get, because a run of errors or redirects can quietly cut you out of circulation.

Putting this next to citation reports gives a fuller picture than either on its own: the logs tell you whether they were able to fetch you, the reports tell you whether anything came of it. I wrote about that other side when covering the Bing Webmaster Tools reports and citations in Microsoft Clarity.

The scale in numbers

The most solid data I know of on this subject comes from a Botify analysis published in April 2026 and based on around seven billion log files from November 2024 to March 2026. One caveat straight away: these are the logs of Botify's customers, not a cross-section of the whole internet, so the industry mix is skewed towards larger sites. I treat the direction as reliable and the specific percentages with caution.

3.5×

growth in OAI-SearchBot activity

2.9×

growth in GPTBot activity

~5%

OpenAI as a share of Google's volume

OpenAI's total volume has tripled since August 2025, that is, since the launch of GPT-5. In the last month studied it came out like this: 18.2 billion Google events against 887 million for OpenAI. The comparison is fair, because on both sides it covers indexing crawlers only: Googlebot desktop and mobile against OAI-SearchBot and GPTBot. Fetches at the user's request are counted separately and are not part of that figure. By industry, healthcare and media grew the most. In both, search crawler activity rose by more than 700%.

Read the direction, not the current level

Just under five per cent sounds marginal, and today it is. A year earlier, though, the same measurements gave 1.38%. At that pace the question is not "is it worth dealing with", but "do you want to make this decision now, or when it has already been made for you".

Four decision scenarios

There is no single right answer, because the answer depends on what you trade in. Below are four typical situations from practice.

Publisher or content site

The hardest case. Content is the product, so handing it over for training hurts the most, but dropping out of citations cuts off real traffic. The typical solution: training blocked, search open.

Online shop

Usually wants to be everywhere: being present in shopping answers is often worth more than protecting product descriptions. One caveat: if your descriptions are written by an editorial team and you treat them as an asset, the argument turns around, because you are also feeding price-comparison sites and aggregators with them. Basket and customer account closed in every variant.

Service business

The simplest situation and the most often neglected. You have no content to lose, and everything to gain in recognition. Blocking AI bots here is shooting yourself in the foot.

Paid or premium site

Here a block makes sense, but at the level of access, not robots.txt. If the content sits behind a login, bots will not see it anyway, and whatever is in front of the login can happily work for your visibility.

The common denominator is that blocking search is a decision to disappear, while blocking training is a decision not to give content away for nothing. The first costs visibility immediately, the second costs almost nothing in the short term. That is why the default setup I suggest starting from in most cases looks the same: training closed, search open.

The decision may be made without you

If you use Cloudflare, some of these settings may be decided at the level of the service, not your file. I cover the changes to the default rules and the split of bots into categories separately, in the piece on blocking AI bots in Cloudflare. Check the dashboard before you assume that robots.txt has the last word.

Myths that cost visibility

"I blocked GPTBot, so I am not in ChatGPT"

The most expensive mistake on this topic, only the other way round from what it seems. GPTBot is responsible for training, and a separate crawler, OAI-SearchBot, for citations. By blocking the first you have not disappeared. By blocking the second, you have. Closing off training has its price, but a deferred one: over time the model knows less about you than it could, and mentions you less often without reaching for the web. It is a real trade-off, just not the one people usually feared.

"An entry in robots.txt cuts everything off"

It does not cut off fetches at the user's request. The file governs robots, not people clicking a link.

"A block protects my content from AI"

It protects against one fetch in the future. It removes nothing from models already trained and does not work retroactively. It is access control, not content protection.

"AI bots will hammer my server"

At just under five per cent of Google's volume, if the server copes with Googlebot, it will cope with this too. Traffic impersonating bots is a separate matter, but that is a security problem, not a performance one.

Limitations

  • No tool on the OpenAI side. There is no dashboard and no report. Everything you know, you know from your own logs, and on some cheap hosting plans those are rudimentary or deleted after a few days.
  • The scale data is a sample. The Botify analysis is based on its customers' logs, so the industry mix is not a cross-section of the whole internet. The direction is reliable, the specific percentages come with a caveat.
  • Crawlers and their names change. Version numbers in the headers go up and new bots arrive: OAI-AdsBot appeared later than the others. It is worth looking at the documentation once a quarter.
  • This is only one provider. Google, Anthropic, Perplexity and the rest have their own crawlers and their own rules. This text describes OpenAI because that is where the questions usually start, but the decision then has to be repeated for the others.

FAQ: common questions

If I block GPTBot, will I disappear from ChatGPT?

No, and this is the most common mistake on this topic. GPTBot collects data for model training, while a separate crawler, OAI-SearchBot, is responsible for showing websites in ChatGPT search. Blocking the first does not remove you from the answers, blocking the second does. OpenAI states explicitly that the settings for individual bots are independent of one another.

Is robots.txt enough to cut OpenAI off from my site?

Not in every case. Three of the four crawlers respect robots.txt, but ChatGPT-User, the one that fetches a page at a user's explicit request, is not bound by it. OpenAI treats that as a person's action, not crawling. If you really want to cut something off, you do it at the level of access: a login, an IP block or a server rule, not an entry in the file.

How do I check whether it really is an OpenAI bot?

By IP address, never by the header alone. The User-Agent is plain text sent by the client and forging it takes a second. The logs of any larger site show traffic impersonating well-known crawlers. OpenAI publishes current address ranges in separate JSON files for each of its bots, and that is the only reliable way to verify.

How quickly does a robots.txt change take effect?

OpenAI says its systems take about a day to adjust to a changed file. That matters when you are diagnosing: if you changed a rule in the morning and check the logs that same afternoon, you will not see anything yet. Wait a day before you conclude that something did not work.

Will letting AI bots in put a load on my server?

Less than is commonly assumed. According to the Botify analysis, in the last period studied there were 887 million OpenAI events against 18.2 billion Google events, which is just under five per cent. If your server copes with Googlebot, it will cope with this too. OpenAI adds that when both of its crawlers are allowed, it may make one fetch instead of two.

Does blocking bots protect my content from AI?

It protects it from being fetched by that particular crawler from that particular address, and that is all. It does not remove content from models that were trained earlier, it does not protect against copying by people, and it does not work retroactively. It is an access control tool for the future, not protection of the content itself. It pays to have realistic expectations.

If OpenAI is just under 5% of Google's volume, is it worth dealing with at all?

It is worth looking at the direction, not the current level. A year earlier the same measurements gave 1.38%, and since August 2025 OpenAI's volume has tripled. In media and publishing, OAI-SearchBot growth exceeded 700%. Besides, the decision to let a bot in costs one line in a file. It is hard to find a cheaper investment in something growing at that pace.

Where can I check whether OpenAI bots have visited my site at all?

Only in the server logs. OpenAI offers nothing like Search Console. That is a real limitation, because on some cheap hosting plans the logs are rudimentary or deleted after a few days. If you plan to measure anything, start by making sure you have access to them and that they go back far enough.

Worth remembering

  • OpenAI documents four crawlers: GPTBot (training), OAI-SearchBot (search in ChatGPT), OAI-AdsBot (ad landing pages) and ChatGPT-User (a fetch at the user's request).
  • The settings are independent of one another. You can block training and leave search open. They are two different transactions.
  • ChatGPT-User is not bound by robots.txt. If a user asks ChatGPT for your page, the file will not hide it.
  • The User-Agent header can be forged in a second. OpenAI publishes IP ranges separately for each bot, so verify against those.
  • A robots.txt change takes about a day to take effect on the OpenAI side, so do not judge the results on the same day.
  • According to the Botify analysis, OpenAI has tripled its crawl since August 2025, but it is still just under 5% of Google's volume. A year earlier it was 1.38%. What counts is the direction, not the current level.

Summary

"Let them in or block them" is a question that cannot be answered well, because it assumes there is one party on the other side with one request. There are four, and they ask for four different things.

Blocking training is a decision not to hand over content. Blocking search is a decision to disappear. Confusing the two costs the most.

Open your robots.txt today and check one thing: whether OAI-SearchBot appears there, and on which side. If the file has a rule cutting off all unknown crawlers, it is probably cutting this one off too.

The check takes a moment and ends with one of two answers. The worse one, that you are blocking something you did not mean to block, is also the easiest to fix: it is one line. The harder part starts further on, with the question of whether you have content worth coming for at all.

Not sure who you are letting in today and who you are blocking?

I will review your robots.txt, server settings and logs, then tell you which decisions on your site were made by accident.

See Technical SEO

How this text was written

I will not pretend it was written entirely by hand, by candlelight. For the research, the proofreading and the code of this page I used a language model. It is a good tool for digging through documentation and catching the typos you no longer see on the third read of your own text. The English version was also prepared with a model, from my Polish original.

What the model did not do: it did not choose the topic, form the arguments or decide what this data means for your site. That stays with me, as does the last word on what goes into the text and what comes out of it.

Where the model helped

  • digging through sources and organising data
  • proofreading and the English version of the text
  • code, page layout and infographics

What stays with me

  • the choice of topic and what is worth saying about it
  • the arguments, the reading of the data and the conclusions
  • fact-checking and the decision on what goes into the text
Sebastian Baryś – freelance SEO specialist

Written by

Sebastian Baryś

Freelance SEO specialist · Częstochowa, Poland, and remote

I started when nobody had dreamt of Core Web Vitals and rankings were checked by hand. Today I work on making companies visible not only in Google, but wherever people ask questions. I write only about things I check myself in client projects.

SEOGoogle AdsTechnical SEOSEO in the AI era

Let's check who your site lets in.

I will review your robots.txt, server settings and logs, then tell you which decisions about AI bots were made on your site by accident.

Get in touch