With the rise of AI models like ChatGPT, Google Gemini, and Perplexity, more crawlers are accessing public websites to collect content for training datasets. If your site is hosted with Hosting Australia on cPanel, this guide explains how to control which bots can crawl your site, using tools built right into your hosting panel.
AI crawlers work like traditional search engine bots, but instead of indexing pages for search results, they scan content to train language models. Examples include:
ChatGPT-User (OpenAI)Google-Extended (used by Google Gemini)CCBot (used by Common Crawl)These crawlers may pull large amounts of text from your public pages. If you prefer to control how your content is used, you can restrict access directly from your cPanel account.
robots.txt File via cPanelpublic_html).robots.txt exists. If not, right-click anywhere and choose Create New File, name it robots.txt.robots.txtEdit the file and add the following:
User-agent: ChatGPT-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This tells those specific bots not to crawl your website. Reputable bots will obey these instructions, though some third-party scrapers may not.
Once done, save the file and confirm it’s publicly accessible at:
https://yourdomain.com/robots.txt
For more granular control, you can add meta tags inside the <head> section of specific pages. For example:
<meta name="robots" content="noindex, nofollow">
This prevents indexing and link following for a given page. If you’re using WordPress, SEO plugins like Yoast or Rank Math let you control this per page or post.
If you notice strange behavior, you can check visitor logs from cPanel:
ChatGPT-User or anything suspiciousAI crawlers are now a common part of web traffic. If you want to protect your content or reduce unwanted load, you can manage access easily through cPanel by editing your robots.txt file. This gives you full control over which bots can scan your site.
If you're unsure how to implement this or need assistance with anything in cPanel, reach out to Hosting Australia's support team. We're happy to walk you through the process or make the changes for you.
