As a Hostinger customer/user, I was pleasantly surprised to receive an email in my inbox a few weeks ago detailing that AI crawlers can be blocked from crawling your website. It was a no brainer for me, I jumped straight into my hPanel and blocked all AI crawlers from accessing the Sentient Blogs platform.
Quick Note
Now don’t get me wrong – Hostinger is, for the most part, jumping on the AI train just as much as the next company, but that doesn’t mean they don’t deserve credit for enabling and promoting this feature for which I am grateful.
What do the crawlers do & why block them?
Pre-AI crawlers were used by search engines such as Google and Bing (and many others) in order to index your site so that your site can show up on search engine result pages (SERPs). Of course, this is where SEO was derived – the idea of optimising your site and content for higher SERP ranking.
But once again, thanks to AI, crawlers mean a whole new thing. AI crawlers will access your site in the same way that traditional crawlers would – but now they can scrape the contents of your site and use it for their own training data/learning. Some believe that having your site’s content scraped (stolen) is worth it as it means your site can show up in AI results and be shown as a reference – but how much genuine, organic traffic is that going to produce? (hint: not much!)
How to Block AI Crawlers in Hostinger hPanel
Hostinger really have made this easy to do.
- Login in hPanel and navigate to your website dashboard
- Sidebar: Performance >> CDN
- AI Audit
Other Ways to Block AI Crawlers
If your hosting provider does not yet provide this capability, it is still very much possible to block AI crawlers and thankfully it is relatively easy to do and there are already a ton of resources out there explaining how to do it.
In a nutshell it involves adding a robots.txt to your sites root and/or adding the necessary meta tags to your sites <head>.
Here are some resources I managed to find after a quick search that will help you configure it yourself:
- https://developers.netlify.com/guides/blocking-ai-bots-and-controlling-crawlers/
- https://datadome.co/learning-center/block-ai-bots/
- https://www.cloudways.com/blog/block-ai-crawlers/
——–
What’s your thoughts? Will you be blocking AI crawlers from your site?
P.S
I have found Hostinger to be a really user-friendly hosting platform and are very affordable for what you get. At the time of posting this, they are running a crazy cheap Black-Friday sale for hosting services. If interested, please consider using this referral link which goes to supporting the Sentient Blogs platform. ✌️
