This topic was discussed in some detail in the latest edition of the Ask Google Webmasters video series on YouTube.
Here is the question that was submitted:
“Regarding robots.txt, should I ‘disallow: /*.css$’, ‘disallow: /php.ini’, or even ‘disallow: /.htaccess’?”
In response, Mueller says Google can’t stop site owners from disallowing those files. Although it’s certainly not recommended.
“No. I can’t disallow you from disallowing those files. But that sounds like a bad idea. You mention a few special cases so let’s take a look.”
In some cases blocking special files is simply redundant, although in other cases it could seriously impact Googlebot’s ability to crawl a site.
Here’s an explanation of what will happen when each type of special file is blocked.
Crawling CSS is absolutely critical as it allows Googlebot to properly render pages.
Site owners may feel it’s necessary to block CSS files so the files don’t get indexed on their own, but Mueller says that usually doesn’t happen.
Google needs the file regardless, so even if a CSS file ends up getting indexed it won’t do as much harm as blocking it would.
This is Mueller’s response:
“‘*.css’ would block all CSS files. We need to be able to access CSS files so that we can properly render your pages.
This is critical so that we can recognize when a page is mobile-friendly, for example.
CSS files generally won’t get indexed on their own, but we need to be able to crawl them.”
Using robots.txt to block php.ini isn’t necessary because it’s not a file that can be readily accessed anyway.
This file should be locked down, which prevents even Googlebot from accessing it. And that’s perfectly fine.
Blocking PHP is redundant, as Mueller explains:
“You also mentioned PHP.ini – this is a configuration file for PHP. In general, this file should be locked down, or in a special location so nobody can access it.
And if nobody can access it then that includes Googlebot too. So, again, no need to disallow crawling of that.”
Like PHP, .htaccess is a locked down file. That means it can’t be accessed externally, even by Googlebot.
It does not need to be disallowed because it can’t be crawled in the first place.
“Finally, you mentioned .htaccess. This is a special control file that cannot be accessed externally by default. Like other locked down files you don’t need to explicitly disallow it from crawling since it cannot be accessed at all.”
Mueller capped off the video with a few short words on how site owners should go about creating a robots.txt file.
Site owners tend to run into problems when they copy another site’s robots.txt file and use it as their own.
Mueller advises against that. Instead, think critically about which parts of your site you do not want to be crawled and only disavow those.
“My recommendation is to not just reuse someone else’s robots.txt file and assume it’ll work. Instead, think about which parts of your site you really don’t want to have crawled and just disallow crawling of those.”
Google ska betala $391,5 miljoner för uppgörelse över platsspårning, säger statliga AG:er
Google has agreed to pay a $391.5 million settlement to 40 states to resolve accusations that it tracked people’s locations in violation of state laws, including snooping on consumers’ whereabouts even after they told the tech behemoth to bug off.
Louisiana Attorney General Jeff Landry said it is time for Big Tech to recognize state laws that limit data collection efforts.
“I have been ringing the alarm bell on big tech for years, and this is why,” Mr. Landry, a Republican, said in a statement Monday. “Citizens must be able to make informed decisions about what information they release to big tech.”
The attorneys general said the investigation resulted in the largest-ever multistate privacy settlement. Connecticut Attorney General William Tong, a Democrat, said Google’s penalty is a “historic win for consumers.”
“Location data is among the most sensitive and valuable personal information Google collects, and there are so many reasons why a consumer may opt out of tracking,” Mr. Tong said. “Our investigation found that Google continued to collect this personal information even after consumers told them not to. That is an unacceptable invasion of consumer privacy, and a violation of state law.”
Location tracking can help tech companies sell digital ads to marketers looking to connect with consumers within their vicinity. It’s another tool in a data-gathering toolkit that generates more than $200 billion in annual ad revenue for Google, accounting for most of the profits pouring into the coffers of its corporate parent, Alphabet, which has a market value of $1.2 trillion.
The settlement is part of a series of legal challenges to Big Tech in the U.S. and around the world, which include consumer protection and antitrust lawsuits.
Though Google, based in Mountain View, California, said it fixed the problems several years ago, the company’s critics remained skeptical. State attorneys general who also have tussled with Google have questioned whether the tech company will follow through on its commitments.
The states aren’t dialing back their scrutiny of Google’s empire.
Last month, Texas Attorney General Ken Paxton said he was filing a lawsuit over reports that Google unlawfully collected millions of Texans’ biometric data such as “voiceprints and records of face geometry.”
The states began investigating Google’s location tracking after The Associated Press reported in 2018 that Android devices and iPhones were storing location data despite the activation of privacy settings intended to prevent the company from following along.
Arizona Attorney General Mark Brnovich went after the company in May 2020. The state’s lawsuit charged that the company had defrauded its users by misleading them into believing they could keep their whereabouts private by turning off location tracking in the settings of their software.
Arizona settled its case with Google for $85 million last month. By then, attorneys general in several other states and the District of Columbia had pounced with their own lawsuits seeking to hold Google accountable.
Along with the hefty penalty, the state attorneys general said, Google must not hide key information about location tracking, must give users detailed information about the types of location tracking information Google collects, and must show additional information to people when users turn location-related account settings to “off.”
States will receive differing sums from the settlement. Mr. Landry’s office said Louisiana would receive more than $12.7 million, and Mr. Tong’s office said Connecticut would collect more than $6.5 million.
The financial penalty will not cripple Google’s business. The company raked in $69 billion in revenue for the third quarter of 2022, according to reports, yielding about $13.9 billion in profit.
Google downplayed its location-tracking tools Monday and said it changed the products at issue long ago.
“Consistent with improvements we’ve made in recent years, we have settled this investigation which was based on outdated product policies that we changed years ago,” Google spokesman Jose Castaneda said in a statement.
Google product managers Marlo McGriff and David Monsees defended their company’s Search and Maps products’ usage of location information.
“Location information lets us offer you a more helpful experience when you use our products,” the two men wrote on Google’s blog. “From Google Maps’ driving directions that show you how to avoid traffic to Google Search surfacing local restaurants and letting you know how busy they are, location information helps connect experiences across Google to what’s most relevant and useful.”
The blog post touted transparency tools and auto-delete controls that Google has developed in recent years and said the private browsing Incognito mode prevents Google Maps from saving an account’s search history.
Mr. McGriff and Mr. Monsees said Google would make changes to its products as part of the settlement. The changes include simplifying the process for deleting location data, updating the method to set up an account and revamping information hubs.
“We’ll provide a new control that allows users to easily turn off their Location History and Web & App Activity settings and delete their past data in one simple flow,” Mr. McGriff and Mr. Monsees wrote. “We’ll also continue deleting Location History data for users who have not recently contributed new Location History data to their account.”
• This article is based in part on wire service reports.
LinkedIn tillkännager utökad utrullning av nytt "Focused Inbox"-format för InMail
Googles höjdpunkter Ladda ner din Google-företagsprofildata när du tar bort dina uppgifter
Vad det betyder för SEO
YouTube tillkännager uppdaterade riktlinjer för intäktsgenerering, nya analysverktyg i mobilappen
Google Sök Ny ruta "Kartor om detta ämne" och "Från dina prenumerationer"
Länkbyggnad för SEO: Nybörjarguiden
Hur man sökmotoroptimerar ditt blogginnehåll
Den här veckan på Xbox: spännande nya releaser, uppdateringar och mer
TikTok tillkännager uppdaterad CAP University Marketing Education Course
Vad 2022 SEO-förändringar kan betyda för 2023 och därefter [Webinar]
B2C-marknadsföring: En guide för marknadsförare
Marketos oktobersläpp: En chefsguide
Veckans erbjudanden med guld- och spotlight-rea, plus Xbox Black Friday-rea
Vampire Survivors tillgängliga idag med Xbox Game Pass för Xbox Series X|S och Xbox One
Identifiera en effektiv B2B-målmarknad för annonser
Xbox delar Community Safety Approach i Transparency Report
Hjälpa affiliates att skapa tillfredsställande innehåll i långa format
För- och nackdelar med ditt varumärke genom att använda affiliate-länkar
Twitters bortgång skulle kosta marknadsförare en viktig och användbar kanal
8 marknadsföringsstrategier för e-handel för 2022 och framåt
SEO6 dagar sedan
En enkel (men komplett) SEO-handledning för nybörjare i 7 steg
MARKNADSFÖRING7 dagar sedan
Vad är försäljningsacceleration?
SPEL6 dagar sedan
Upplev hösten i Phantasy Star Online 2: New Genesis
SÖKMOTORER7 dagar sedan
Matt Cutts slogs med Sergey Brin och Larry Page över spamproblem i Google Sök