This how to video tutorial shows you how to use Cloudflare to block Chatgpt, Gemini, Claude and the most popular AI and machine learning bots from Scraping and Crawling your website. It takes a few seconds.
Subtitles:
Hey everybody, this is Ori from One Scales.
In today's video, I'm going to cover
a really cool feature by Cloudflare called
Block AI scrapers and crawlers.
So blocking them.
What is an AI scraper and crawler?
Many of you are aware it's 2024 right now.
You know, Chat GPT, Gemini by Google, Claude all these
companies that are AI machine learning companies that they understand the
world at least like you and better.
You can actually ask chat GPT hey, what is ....
You know this information and it'll answer for you.
So how does it know that information?
It actually crawls around the web.
It visits a huge amount of websites and
understands how the world is and uses that
knowledge to understand you and information.
So some of you might want to actually block
them because they're actually reading and learning from you
guys and they're actually bypassing you in some cases
and your information and your website and giving the
result of your information that you worked hard, you
spent money on.
And you don't necessarily want to do it.
Some people don't want to do it
at one scales we don't block.
We actually like our information to spread and we're trying to
help people, so we like it to keep it this way.
But a lot of you want to block it.
A lot of news sites, things
like that, actually block it.
If you want to block it, all you have to do is
go to your Cloudflare account, make sure you click on your website,
go to security and bots, and you'll be able to see this
new section right here and just turn it on.
All you have to do is turn it on and it'll block it.
So I'll give you an example of
some of the bots that it blocks.
For example, GPT bot is the chat GPT one, a
bot that actually, it's basically code that goes and scrolls
your website and learns, you know, and there's Claude and
there's many, many different things, perplexity as well.
So this is just a small list of the bots that are blocked.
There's a lot more bots that they're going to block,
and I'm sure they're going to update that over time.
So you should always consider how your information
is used, if others can use it, if
they have the right to use it.
If you want to block it, you can use things like
robots txt manually to block it, or you can use Cloudflare
to just click on the on button and it'll work.
It's a really, really nice feature.
Just one small thing before we exit this
video is you can only use this if
you're actually using Cloudflare's routing system.
So how do you know if you do that?
If you only use Cloudflare for domain registration
and DNS, this is not going to work
because the traffic has to go through Cloudflare.
So if you go here to your DNS right
here, and you look at your websites, you need
to make sure that your traffic is proxied.
If you're using it only for DNS, for example, most
of our stuff is used just for DNS right here.
It will not work.
You have to have it proxied.
So, for example, we will only block bots in
what we call the tools dot onescales.com dot.
So anything you need to actually block, make
sure it's turned on via the security bots.
And you have to have it proxied, which means the
traffic goes through Cloudflare, which means they can actually block
it because the traffic is going through them. That's it.
If you have any questions about bots in general, Cloudflare,
anything we talk about on one scales, let us know.
We're trying to go very, very deep with you guys.
So if you have any specific question, no
matter if it's for mass or just
for your specific custom case, let us know.
This is what we do at one scales.
We try to help you guys.
That's it.
If you had any questions, let us know.
We have onescales.com, x/twitter
obviously YouTube in the comments.
And if this video helped you in any way,
just say hello, subscribe, like, share with a friend.
Let us know what you thought about this video.
Much, much appreciated. Thank you guys.
