The author’s views are entirely his or her own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.
Introduction to Googlebot spoofing
In this article, I’ll describe how and why to use Google Chrome (or Chrome Canary) to view a website as Googlebot.
We’ll set up a web browser specifically for Googlebot browsing. Using a user-agent browser extension is often close enough for SEO audits, but extra steps are needed to get as close as possible to emulating Googlebot.
Why should I view a website as Googlebot?
Originally, web servers sent complete websites (fully rendered HTML) to web browsers. These days, many websites are rendered client-side (in the web browser itself) – whether that’s Chrome, Safari, or whatever browser a search bot uses – meaning the user’s browser and device must do the work to render a webpage.
Attempting to get around potential SEO issues, some websites use dynamic rendering, so each page has two versions:
Viewing a website as Googlebot means we can see discrepancies between what a person sees and what a search bot sees. What Googlebot sees doesn’t need to be identical to what a person using a browser sees, but main navigation and the content you want the page to rank for should be the same.
That’s where this article comes in. For a proper technical SEO audit, we need to see what the most common search engine sees. In most English language-speaking countries, at least, that’s Google.
Why use Chrome (or Chrome Canary) to view websites as Googlebot?
Can we see exactly what Googlebot sees?
The aim is to emulate Googlebot’s mobile-first indexing as closely as possible.
When auditing, I use my Googlebot browser alongside Screaming Frog SEO Spider’s Googlebot spoofing and rendering, and Google’s own tools such as URL Inspection in Search Console (which can be automated using SEO Spider), and the render screenshot and code from the Mobile Friendly Test.
Even Google’s own publicly available tools aren’t 100% accurate in showing what Googlebot sees. But along with the Googlebot browser and SEO Spider, they can point towards issues and help with troubleshooting.
Why use a separate browser to view websites as Googlebot?
Having a dedicated browser saves time. Without relying on or waiting for other tools, I get an idea of how Googlebot sees a website in seconds.
While auditing a website that served different content to browsers and Googlebot, and where issues included inconsistent server responses, I needed to switch between the default browser user-agent and Googlebot more often than usual. But constant user-agent switching using a Chrome browser extension was inefficient.
Aside from having a coder who can code a headless Chrome solution, the “Googlebot browser” setup is an easy way to spoof Googlebot.
2. Improved accuracy
Browser extensions can impact how websites look and perform. This approach keeps the number of extensions in the Googlebot browser to a minimum.
It’s easy to forget to switch Googlebot spoofing off between browsing sessions, which can lead to websites not working as expected. I’ve even been blocked from websites for spoofing Googlebot, and had to email them with my IP to remove the block.
For which SEO audits are a Googlebot browser useful?
The most common use-case for SEO audits is likely websites using client-side rendering or dynamic rendering. You can easily compare what Googlebot sees to what a general website visitor sees.
Even with websites that don’t use dynamic rendering, you never know what you might find by spoofing Googlebot. After over eight years auditing e-commerce websites, I’m still surprised by issues I haven’t come across before.
Example Googlebot comparisons for technical SEO and content audits:
Is the main navigation different?
Is Googlebot seeing the content you want indexed?
Do URLs return different server responses? For example, incorrect URLs can return 200 OK for Googlebot but 404 Not Found for general website visitors.
Is the page layout different to what the general website visitor sees? For example, I often see links as blue text on a black background when spoofing Googlebot. While machines can read such text, we want to present something that looks user-friendly to Googlebot. If it can’t render your client-side website, how will it know? (Note: a website might display as expected in Google’s cache, but that isn’t the same as what Googlebot sees.)
Do websites redirect based on location? Googlebot mostly crawls from US-based IPs.
It depends how in-depth you want to go, but Chrome itself has many useful features for technical SEO audits. I sometimes compare its Console and Network tab data for a general visitor vs. a Googlebot visit (e.g. Googlebot might be blocked from files that are essential for page layout or are required to display certain content).
How to set up your Googlebot browser
Once set up (which takes about a half hour), the Googlebot browser solution makes it easy to quickly view webpages as Googlebot.
Step 1: Download and install Chrome or Canary
If Chrome isn’t your default browser, use it as your Googlebot browser.
If Chrome is your default browser, download and install Chrome Canary. Canary is a development version of Chrome where Google tests new features, and it can be installed and run separately to Chrome’s default version.
Named after the yellow canaries used to detect poisonous gases in mines, with its yellow icon, Canary is easy to spot in the Windows Taskbar:
As Canary is a development version of Chrome, Google warns that Canary “can be unstable.” But I’m yet to have issues using it as my Googlebot browser.
Step 2: Install browser extensions
I installed five browser extensions and a bookmarklet on my Googlebot browser. I’ll list the extensions, then advise on settings and why I use them.
For emulating Googlebot (the links are the same whether you use Chrome or Canary):
User-Agent Switcher extension
User-Agent Switcher does what it says on the tin: switches the browser’s user-agent. Chrome and Canary have a user-agent setting, but it only applies to the tab you’re using and resets if you close the browser.
I take the Googlebot user-agent string from Chrome’s browser settings, which at the time of writing will be the latest version of Chrome (note that below, I’m taking the user-agent from Chrome and not Canary).
To get the user-agent, access Chrome DevTools (by pressing F12 or using the hamburger menu to the top-right of the browser window, then navigating to More tools > Developer tools). See the screenshot below or follow these steps:
Go to the Network tab
From the top-right Network hamburger menu: More tools > Network conditions
Click the Network conditions tab that appears lower down the window
Untick “Use browser default”
- Select “Googlebot Smartphone” from the list, then copy and paste the user-agent from the field below the list into the User-Agent Switcher extension list (another screenshot below). Don’t forget to switch Chrome back to its default user-agent if it’s your main browser.
At this stage, if you’re using Chrome (and not Canary) as your Googlebot browser, you may as well tick “Disable cache” (more on that later).
To access User-Agent Switcher’s list, right-click its icon in the browser toolbar and click Options (see screenshot below). “Indicator Flag” is text that appears in the browser toolbar to show which user-agent has been selected — I chose GS to mean “Googlebot Smartphone:”
I added Googlebot Desktop and the bingbots to my list, too.
Why spoof Googlebot’s user agent?
Web servers detect what is browsing a website from a user-agent string. For example, the user-agent for a Windows 10 device using the Chrome browser at the time of writing is:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36
If you’re interested in why other browsers seem to be named in the Chrome user-agent string, read History of the user-agent string.
Web Developer extension
Long answer: that would be a whole other article.
Windscribe (or another VPN)
Windscribe (or your choice of VPN) is used to spoof Googlebot’s US location. I use a pro Windscribe account, but the free account allows up to 2GB data transfer a month and includes US locations.
I don’t think the specific US location matters, but I pretend Gotham is a real place (in a time when Batman and co. have eliminated all villains):
Ensure settings that may impact how webpages display are disabled — Windscribe’s extension blocks ads by default. The two icons to the top-right should show a zero.
For the Googlebot browser scenario, I prefer a VPN browser extension to an application, because the extension is specific to my Googlebot browser.
Why spoof Googlebot’s location?
Googlebot mostly crawls websites from US IPs, and there are many reasons for spoofing Googlebot’s primary location.
Some websites block or show different content based on geolocation. If a website blocks US IPs, for example, Googlebot may never see the website and therefore cannot index it.
Another example: some websites redirect to different websites or URLs based on location. If a company had a website for customers in Asia and a website for customers in America, and redirected all US IPs to the US website, Googlebot would never see the Asian version of the website.
With Link Redirect Trace, I see at a glance what server response a URL returns.
The View Rendered Source extension enables easy comparison of raw HTML (what the web server delivers to the browser) and rendered HTML (the code rendered on the client-side browser).
Step 3: Configure browser settings to emulate Googlebot
Next, we’ll configure the Googlebot browser settings in line with what Googlebot doesn’t support when crawling a website.
What doesn’t Googlebot crawling support?
Service workers (because people clicking to a page from search results may never have visited before, so it doesn’t make sense to cache data for later visits).
Permission requests (e.g. push notifications, webcam, geolocation). If content relies on any of these, Googlebot will not see that content.
Googlebot is stateless so doesn’t support cookies, session storage, local storage, or IndexedDB. Data can be stored in these mechanisms but will be cleared before Googlebot crawls the next URL on a website.
These bullet points are summarized from an interview by Eric Enge with Google’s Martin Splitt:
Step 3a: DevTools settings
To open Developer Tools in Chrome or Canary, press F12, or using the hamburger menu to the top-right, navigate to More tools > Developer tools:
The Developer Tools window is generally docked within the browser window, but I sometimes prefer it in a separate window. For that, change the “Dock side” in the second hamburger menu:
If using normal Chrome as your Googlebot browser, you may have done this already.
Otherwise, via the DevTools hamburger menu, click to More tools > Network conditions and tick the “Disable cache” option:
Block service workers
To block service workers, go to the Application tab > Service Workers > tick “Bypass for network”:
Step 3b: General browser settings
In your Googlebot browser, navigate to Settings > Privacy and security > Cookies (or visit chrome://settings/cookies directly) and choose the “Block all cookies (not recommended)” option (isn’t it fun to do something “not recommended?”):
Also in the “Privacy and security” section, choose “Site settings” (or visit chrome://settings/content) and individually block Location, Camera, Microphone, Notifications, and Background sync (and likely anything that appears there in future versions of Chrome):
Step 4: Emulate a mobile device
Finally, as our aim is to emulate Googlebot’s mobile-first crawling, emulate a mobile device within your Googlebot browser.
Towards the top-left of DevTools, click the device toolbar toggle, then choose a device to emulate in the browser (you can add other devices too):
Whatever device you choose, Googlebot doesn’t scroll on webpages, and instead renders using a window with a long vertical height.
I recommend testing websites in desktop view, too, and on actual mobile devices if you have access to them.
How about viewing a website as bingbot?
To create a bingbot browser, use a recent version of Microsoft Edge with the bingbot user agent.
Bingbot is similar to Googlebot in terms of what it does and doesn’t support.
Yahoo! Search, DuckDuckGo, Ecosia, and other search engines are either powered by or based on Bing search, so Bing is responsible for a higher percentage of search than many people realize.
Summary and closing notes
So, there you have your very own Googlebot emulator.
Using an existing browser to emulate Googlebot is the easiest method to quickly view webpages as Googlebot. It’s also free, assuming you already use a desktop device that can install Chrome and/or Canary.
Questions? Something I missed? Tweet me @AlexHarfordSEO. Thanks for reading!
Content Marketers Share Salaries, Career Paths, and More in 2023 [New Research]
You certainly know your answers to these questions. But, until now, little industry research has dived into content marketing careers.
We set out to find answers. Our goal is to help content marketers understand their opportunities and positions – and help companies develop meaningful roles and the resources and opportunities to retain them.
So, earlier this year, we asked content marketers about their work satisfaction, career development, and salary expectations.
More than 1,100 content professionals had their say. You can read the full story – including salary breakdowns by role, gender, and generation – in the Content Marketing Career & Salary 2023 Outlook (gated).
Let’s take a sneak peek at some of the intriguing findings.
You (mostly) like your content marketing jobs
More than half of the content pros (56%) tell us they’re very or extremely satisfied with their current position.
One content marketer explains: “I can be creative while being tied to business impact. Content marketing offers the fulfillment and growth of a creative career with the stability and compensation of a corporate career. It’s the best of both worlds, and it’s sometimes hard to believe it’s possible.”
Another offers this explanation: “I love seeing all the pieces come together; how great words and innovative designs can affect and influence consumers and audiences. And I love working behind the scenes, getting to turn the cogs of the content machine.”
Satisfaction rates stay roughly the same from millennials to Gen Xers to baby boomers. (We had too few Gen Z respondents to report on their segment with confidence.)
Of course, that’s not to say the job is easy. When asked about stress levels, 24% of content marketers say they are “very” or “extremely” stressed.
One survey taker explains, “The pace of work can be relentless. Just when you’ve completed one big project, another is right behind it.”
And some kudos go to employers. A significant majority (74%) said they feel their employers care about their stress levels and mental health.
HANDPICKED RELATED CONTENT:
You’re well educated – and eager to learn more
Among the surveyed group, one in three has a master’s, doctorate, or another advanced degree. As you probably know from your and your colleagues’ career pathways, people come into content marketing from many backgrounds (some come from multiple fields), including:
And content marketers are eager to expand their knowledge base:
- Over 45% want to advance their skills in SEO, data analytics, audience development/segmentation, and integrating new technologies.
- 40% show interest in honing their writing and editing skills.
- One in three wants to hone their audio and video skills (filming, editing, and production).
Content marketers clearly rank high on the “digital dexterity” scale – the ability to learn new skills and adapt to new environments. That’s a sign of an adaptable, resilient workforce ready to meet whatever the future brings.
As Jean-Marc Laouchez, president of the Korn Ferry Institute, says in a Computerworld article: “Constant learning – driven by both workers and organizations – will be central to the future of work, extending far beyond the traditional definition of learning and development.”
And yet, many content marketers are looking for new positions
Content marketers like their jobs and are ready to learn. And yet, most (57%) say they plan to find another position within the next year or are unsure about their next steps.
Looking at it from another angle: Only 43% say they won’t be looking for a new job in the next year.
What’s driving this restlessness? Is it a persistent echo of the Great Resignation? Or a wave of “quiet quitting” in content marketing?
I don’t think so. Instead, the research points to something at the heart of content marketing careers.
Content marketing lacks a clear career path
The data highlights a troubling phenomenon: Only 23% of content marketers say they have a clear path for advancement inside their current company.
Nearly all the rest (69%) say they must leave their companies to advance or simply can’t visualize the path forward. (A small share – 8% – say they’ve reached the pinnacle of their careers and aren’t looking for advancement.)
This isn’t a new phenomenon. Robert Rose, our chief strategy advisor, has written about this problem: “Content marketing is growing exponentially. But the advancement ladder for content practitioners is missing most of its rungs.”
Companies that don’t address the content marketing career ladder will struggle to keep these highly educated, adaptable employees.
Content marketers want better-defined career paths and are eager to advance their skills. So, where to begin nurturing their ambitions? With dialogue.
If you’re an individual contributor on a content team, speak up about your needs and wants.
If you’re a team leader, involve your creative, results-driven professionals in open, honest conversations. Invite them to help shape their career paths based on their aspirations. Then partner with HR and executive leadership to provide what they need to achieve their goals.
After all, investing in their future also pays off for the brand.
Content Marketing Career & Salary 2023 Outlook offers more insights into:
- Content marketers’ income
- Unique career priorities by age and gender
- Advice on how companies can recruit and retain the best content marketing talent
I hope you’ll download the e-book to learn more. In the meantime, I’d love to hear from you. How do these findings align with your experience? What would you tell the next generation about content marketing as a career? Let me know in the comments.
Cover image by Joseph Kalinowski/Content Marketing Institute
Meta Will Shut Down its Newsletter Platform Early Next Year
Instagram Is Now Putting Ads In Your Profile Feed
Wild Google Organic Ranking Swings With Google Algorithm Updates
How to Expand Your eCommerce Business Globally
Content Marketers Share Salaries, Career Paths, and More in 2023 [New Research]
Meta Shares Insights into How Consumers View the Next Big Tech Shifts, Including the Metaverse
Is RankBrain A Ranking Factor In Google Search?
How Computer Vision Paired with AR Can Be Used for Navigation Aide
Google Home App Gets an Overhaul, Rolling Out Soon
Unusual Spike In Google Business Profile Insights Traffic For Some
Explore the Path to Digital Future: Interconnect, Integrate and Innovate
Google Updates Documentation On Meta Descriptions
Daily Search Forum Recap: September 5, 2022
7 Tips For Creating Instagram Story Ads That Convert
How To Launch Your First Google Ads Remarketing Campaign
Daily Search Forum Recap: September 30, 2022
Microsoft Advertising Gains Pinterest Import, More Google Imports, & More
Google Again Says Spikes In Crawling Activity Not A Sign Of The Helpful Content Update Rollout
The Ultimate Timeline of Google Algorithm Updates (+ Recommendations)
Confusion Over Google Search Console’s HTTPS Is Invalid And Might Prevent It From Being Indexed