Google’s John Mueller answers a question about using robots.txt to block special files, including .css and .htacess.
This topic was discussed in some detail in the latest edition of the Ask Google Webmasters video series on YouTube.
Here is the question that was submitted:
“Regarding robots.txt, should I ‘disallow: /*.css$’, ‘disallow: /php.ini’, or even ‘disallow: /.htaccess’?”
In response, Mueller says Google can’t stop site owners from disallowing those files. Although it’s certainly not recommended.
“No. I can’t disallow you from disallowing those files. But that sounds like a bad idea. You mention a few special cases so let’s take a look.”
In some cases blocking special files is simply redundant, although in other cases it could seriously impact Googlebot’s ability to crawl a site.
Here’s an explanation of what will happen when each type of special file is blocked.
Crawling CSS is absolutely critical as it allows Googlebot to properly render pages.
Site owners may feel it’s necessary to block CSS files so the files don’t get indexed on their own, but Mueller says that usually doesn’t happen.
Google needs the file regardless, so even if a CSS file ends up getting indexed it won’t do as much harm as blocking it would.
This is Mueller’s response:
“‘*.css’ would block all CSS files. We need to be able to access CSS files so that we can properly render your pages.
This is critical so that we can recognize when a page is mobile-friendly, for example.
CSS files generally won’t get indexed on their own, but we need to be able to crawl them.”
Using robots.txt to block php.ini isn’t necessary because it’s not a file that can be readily accessed anyway.
This file should be locked down, which prevents even Googlebot from accessing it. And that’s perfectly fine.
Blocking PHP is redundant, as Mueller explains:
“You also mentioned PHP.ini – this is a configuration file for PHP. In general, this file should be locked down, or in a special location so nobody can access it.
And if nobody can access it then that includes Googlebot too. So, again, no need to disallow crawling of that.”
Like PHP, .htaccess is a locked down file. That means it can’t be accessed externally, even by Googlebot.
It does not need to be disallowed because it can’t be crawled in the first place.
“Finally, you mentioned .htaccess. This is a special control file that cannot be accessed externally by default. Like other locked down files you don’t need to explicitly disallow it from crawling since it cannot be accessed at all.”
Mueller capped off the video with a few short words on how site owners should go about creating a robots.txt file.
Site owners tend to run into problems when they copy another site’s robots.txt file and use it as their own.
Mueller advises against that. Instead, think critically about which parts of your site you do not want to be crawled and only disavow those.
“My recommendation is to not just reuse someone else’s robots.txt file and assume it’ll work. Instead, think about which parts of your site you really don’t want to have crawled and just disallow crawling of those.”
Once upon a time, Microsoft Office ruled the business world. By the late ‘90s and early 2000s, Microsoft’s office suite had brushed aside rivals such as WordPerfect Office and Lotus SmartSuite, and there was no competition on the horizon.
Then in 2006 Google came along with Google Docs & Spreadsheets, a collaborative online word processing and spreadsheet duo that was combined with other business services to form the Google Apps suite, later rebranded as G Suite, and now as Google Workspace. Although Google’s productivity suite didn’t immediately take the business world by storm, over time it has gained both in features and in popularity, boasting 6 million paying customers, according to Google’s most recent public stats in March 2020.
Microsoft, meanwhile, has shifted its emphasis away from its traditional licensed Office software to Microsoft 365 (formerly Office 365), a subscription-based version that’s treated more like a service, with frequent updates and new features. Microsoft 365 is what we’ve focused on in this story.
Nowadays, choosing an office suite isn’t as simple as it once was. We’re here to help.
Google Workspace vs. Microsoft 365
Google Workspace and Microsoft 365 have much in common. Both are subscription-based, charging businesses per-person fees every month, in varying tiers, depending on the capabilities their customers are looking for. Although Google Workspace is web-based, it has the capability to work offline as well. And while Microsoft 365 is based on installed desktop software, it also provides (less powerful) web-based versions of its applications.
Both suites work well with a range of devices. Because it’s web-based, Google Workspace works in most browsers on any operating system, and Google also offers mobile apps for Android and iOS. Microsoft provides Office client apps for Windows, macOS, iOS, and Android, and its web-based apps work across browsers.