Google On Staging Sites & Preventing Accidental Indexing

The latest episode of Google’s Search Off The Record podcast discusses the challenges of launching websites and preventing staging sites from being indexed by search engines.

Staging sites used for testing and development should ideally not be indexed, and Google highlights password protection as one method of preventing accidental indexing.

Hosts John Mueller, Gary Illyes, and Lizzy Sassman delve into the significance of clicking the “launch” button on different platforms and how site launches and migrations are connected.

The discussion emphasizes the importance of balancing launching a large number of pages, maintaining content quality, and selecting an appropriate domain name for branding and visibility purposes.

Here are the highlights from Google’s recent podcast about staging and launching websites.

Accidental Leaks & Importance Of DNS Setup

Mueller and Illyes discuss keeping staging sites hidden from search engines and share anecdotes of accidental leaks from Google’s past.

They speak to the launch of Google’s search status dashboard and strategies to conceal it before its official release.

Illyes says they did “literally nothing” to keep the dashboard a secret before announcing it to everyone.

It was live for months, and no one noticed because no links were pointing to it.

Illyes states:

“If you are not linking to something on the internet then it’s very hard to notice it, or you’re not mentioning it anywhere, and that’s what we were doing. Not mentioning it anywhere, not linking to it, not having it in random JavaScript files. And by the way, this is one more reason not to do JavaScript. And it just worked, it was risky, I would say.”

Sassman adds that there was no sitemap for the new Google dashboard, which made it even more challenging to find.

The conversation also touches on the critical role of DNS setup and the distinction between using a subdomain and a new domain.

Despite the risks associated with their launch strategy, the hosts agree that potential consequences were minimal.

Takeaway: A viable, yet risky, way to keep a staging site from leaking it to not mention or link to it from anywhere.

Other Ways To Block Staging Sites

Illyes explains that robots.txt is typically sufficient for blocking search engines from crawling staging sites.

When asked about alternative methods for keeping a site hidden before launch, Illyes suggests using a noindex tag on each page.

You could also utilize password protection or IP-allow lists for staging sites.

This led to a story from Mueller about using static IP addresses for server backend access, which led to a loss of access when his IP address changed.

Launching Sites With A Large Number Of Pages & Choosing Domain Names

The discussion moves on to launching many pages simultaneously and the importance of selecting the right domain name.

Illyes believes that if a server has sufficient resources, launching 7,000 articles simultaneously should not be an issue, although quality assurance might be challenging.

Quality, rather than quantity, is emphasized when discussing the prospect of publishing a million pages at once.

The hosts also discuss using free hostnames provided by hosting providers, compared to custom domain names.

Illyes says the choice depends on the business’s needs and branding objectives, with custom domain names being more suitable for a long-term online presence.

For more insights from Google’s Search Relations team, listen to the full podcast.

Featured Image: sacitarios/Shutterstock

Source: Search Off The Record