Ta kontakt med oss

MARKNADSFÖRING

Hur man använder Chrome för att visa en webbplats som Googlebot

Publicerad

How to Use Chrome to View a Website as Googlebot

Författarens åsikter är helt hans eller hennes egna (exklusive den osannolika händelsen av hypnos) och återspeglar kanske inte alltid Moz åsikter.

Introduction to Googlebot spoofing

In this article, I’ll describe how and why to use Google Chrome (or Chrome Canary) to view a website as Googlebot.

We’ll set up a web browser specifically for Googlebot browsing. Using a user-agent browser extension is often close enough for SEO audits, but extra steps are needed to get as close as possible to emulating Googlebot.

Skip to “How to set up your Googlebot browser”.

Why should I view a website as Googlebot?

For many years, us technical SEOs had it easy when auditing websites, with HTML and CSS being webbdesign’s cornerstone languages. JavaScript was generally used for embellishments (such as small animations on a webpage).

Increasingly, though, whole websites are being built with JavaScript.

Originally, web servers sent complete websites (fully rendered HTML) to web browsers. These days, many websites are rendered client-side (in the web browser itself) – whether that’s Chrome, Safari, or whatever browser a search bot uses – meaning the user’s browser and device must do the work to render a webpage.

Annons

SEO-wise, some search bots don’t render JavaScript, so won’t see webpages built using it. Especially when compared to HTML and CSS, JavaScript is very expensive to render. It uses much more of a device’s processing power — wasting the device’s battery life— and much more of Google’s, Bing’s, or any search engine’s server resource.

Even Googlebot has difficulties rendering JavaScript and delays rendering of JavaScript beyond its initial URL discovery – sometimes for days or weeks, depending on the website. When I see “Discovered – currently not indexed” for several URLs in Google Search Console’s Coverage (or Pages) section, the website is more often than not JavaScript-rendered.

Attempting to get around potential SEO issues, some websites use dynamic rendering, so each page has two versions:

Generally, I find that this setup overcomplicates websites and creates more technical SEO issues than a server-side rendered or traditional HTML website. A mini rant here: there are exceptions, but generally, I think client-side rendered websites are a bad idea. Websites should be designed to work on the lowest common denominator of a device, with progressive enhancement (through JavaScript) used to improve the experience for people, using devices that can handle extras. This is something I will investigate further, but my anecdotal evidence suggests client-side rendered websites are generally more difficult to use for people who rely on accessibility devices such as a screen reader. There are instances where technical SEO and usability crossover.

Technical SEO is about making websites as easy as possible for search engines to crawl, render, and index (for the most relevant keywords and topics). Like it or lump it, the future of technical SEO, at least for now, includes lots of JavaScript and different webpage renders for bots and users.

Viewing a website as Googlebot means we can see discrepancies between what a person sees and what a search bot sees. What Googlebot sees doesn’t need to be identical to what a person using a browser sees, but main navigation and the content you want the page to rank for should be the same.

That’s where this article comes in. For a proper technical SEO audit, we need to see what the most common search engine sees. In most English language-speaking countries, at least, that’s Google.

Why use Chrome (or Chrome Canary) to view websites as Googlebot?

Can we see exactly what Googlebot sees?

Annons

No.

Googlebot itself uses a (headless) version of the Chrome browser to render webpages. Even with the settings suggested in this article, we can never be exakt sure of what Googlebot sees. For example, no settings allow for how Googlebot processes JavaScript websites. Sometimes JavaScript breaks, so Googlebot might see something different than what was intended.

The aim is to emulate Googlebot’s mobile-first indexing as closely as possible.

When auditing, I use my Googlebot browser alongside Screaming Frog SEO Spider’s Googlebot spoofing and rendering, and Google’s own tools such as URL Inspection in Search Console (which can be automated using SEO Spider), and the render screenshot and code from the Mobile Friendly Test.

Even Google’s own publicly available tools aren’t 100% accurate in showing what Googlebot sees. But along with the Googlebot browser and SEO Spider, they can point towards issues and help with troubleshooting.

Why use a separate browser to view websites as Googlebot?

1. Convenience

Having a dedicated browser saves time. Without relying on or waiting for other tools, I get an idea of how Googlebot sees a website in seconds.

While auditing a website that served different content to browsers and Googlebot, and where issues included inconsistent server responses, I needed to switch between the default browser user-agent and Googlebot more often than usual. But constant user-agent switching using a Chrome browser extension was inefficient.

Some Googlebot-specific Chrome settings don’t save or transport between browser tabs or sessions. Some settings affect all open browser tabs. E.g., disabling JavaScript may stop websites in background tabs that rely on JavaScript from working (such as task management, social media, or email applications).

Annons

Aside from having a coder who can code a headless Chrome solution, the “Googlebot browser” setup is an easy way to spoof Googlebot.

2. Improved accuracy

Browser extensions can impact how websites look and perform. This approach keeps the number of extensions in the Googlebot browser to a minimum.

3. Forgetfulness

It’s easy to forget to switch Googlebot spoofing off between browsing sessions, which can lead to websites not working as expected. I’ve even been blocked from websites for spoofing Googlebot, and had to email them with my IP to remove the block.

For which SEO audits are a Googlebot browser useful?

The most common use-case for SEO audits is likely websites using client-side rendering or dynamic rendering. You can easily compare what Googlebot sees to what a general website visitor sees.

Even with websites that don’t use dynamic rendering, you never know what you might find by spoofing Googlebot. After over eight years auditing e-commerce websites, I’m still surprised by issues I haven’t come across before.

Example Googlebot comparisons for technical SEO and content audits:

  • Is the main navigation different?

  • Is Googlebot seeing the content you want indexed?

  • If a website relies on JavaScript rendering, will new content be indexed promptly, or so late that its impact is reduced (e.g. for forthcoming events or new product listings)?

  • Do URLs return different server responses? For example, incorrect URLs can return 200 OK for Googlebot but 404 Not Found for general website visitors.

  • Is the page layout different to what the general website visitor sees? For example, I often see länkar as blue text on a black background when spoofing Googlebot. While machines can read such text, we want to present something that looks user-friendly to Googlebot. If it can’t render your client-side website, how will it know? (Note: a website might display as expected in Google’s cache, but that isn’t the same as what Googlebot sees.)

  • Do websites redirect based on location? Googlebot mostly crawls from US-based IPs.

It depends how in-depth you want to go, but Chrome itself has many useful features for technical SEO audits. I sometimes compare its Console and Network tab data for a general visitor vs. a Googlebot visit (e.g. Googlebot might be blocked from files that are essential for page layout or are required to display certain content).

How to set up your Googlebot browser

Once set up (which takes about a half hour), the Googlebot browser solution makes it easy to quickly view webpages as Googlebot.

Step 1: Download and install Chrome or Canary

If Chrome isn’t your default browser, use it as your Googlebot browser.

Annons

If Chrome är your default browser, download and install Chrome Canary. Canary is a development version of Chrome where Google tests new features, and it can be installed and run separately to Chrome’s default version.

Named after the yellow canaries used to detect poisonous gases in mines, with its yellow icon, Canary is easy to spot in the Windows Taskbar:

Screenshot of the yellow Chrome Canary icon in a Windows 10 taskbar

As Canary is a development version of Chrome, Google warns that Canary “can be unstable.” But I’m yet to have issues using it as my Googlebot browser.

Step 2: Install browser extensions

I installed five browser extensions and a bookmarklet on my Googlebot browser. I’ll list the extensions, then advise on settings and why I use them.

For emulating Googlebot (the länkar are the same whether you use Chrome or Canary):

Not required to emulate Googlebot, but my other favorites for technical SEO auditing of JavaScript websites:

User-Agent Switcher extension

User-Agent Switcher does what it says on the tin: switches the browser’s user-agent. Chrome and Canary have a user-agent setting, but it only applies to the tab you’re using and resets if you close the browser.

I take the Googlebot user-agent string from Chrome’s browser settings, which at the time of writing will be the latest version of Chrome (note that below, I’m taking the user-agent from Chrome and not Canary).

To get the user-agent, access Chrome DevTools (by pressing F12 or using the hamburger menu to the top-right of the browser window, then navigating to More tools > Developer tools). See the screenshot below or follow these steps:

Annons
  1. Go to the Network tab

  2. From the top-right Network hamburger menu: More tools > Network conditions

  3. Click the Network conditions tab that appears lower down the window

  4. Untick “Use browser default”

  5. Select “Googlebot Smartphone” from the list, then copy and paste the user-agent from the field below the list into the User-Agent Switcher extension list (another screenshot below). Don’t forget to switch Chrome back to its default user-agent if it’s your main browser.
    • At this stage, if you’re using Chrome (and not Canary) as your Googlebot browser, you may as well tick “Disable cache” (more on that later).

Screenshot of DevTools showing the steps described above

To access User-Agent Switcher’s list, right-click its icon in the browser toolbar and click Options (see screenshot below). “Indicator Flag” is text that appears in the browser toolbar to show which user-agent has been selected — I chose GS to mean “Googlebot Smartphone:”

Screenshot showing User-Agent Switcher options described in the paragraph above

I added Googlebot Desktop and the bingbots to my list, too.

Why spoof Googlebot’s user agent?

Web servers detect what is browsing a website from a user-agent string. For example, the user-agent for a Windows 10 device using the Chrome browser at the time of writing is:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36

If you’re interested in why other browsers seem to be named in the Chrome user-agent string, read History of the user-agent string.

Web Developer extension

Web Developer is a must-have browser extension for technical SEOs. In my Googlebot browser, I switch between disabling and enabling JavaScript to see what Googlebot might see with and without JavaScript.

Why disable JavaScript?

Short answer: Googlebot doesn’t execute any/all JavaScript when it first crawls a URL. We want to see a webpage before any JavaScript is executed.

Long answer: that would be a whole other article.

Windscribe (or another VPN)

Windscribe (or your choice of VPN) is used to spoof Googlebot’s US location. I use a pro Windscribe account, but the free account allows up to 2GB data transfer a month and includes US locations.

I don’t think the specific US location matters, but I pretend Gotham is a real place (in a time when Batman and co. have eliminated all villains):

Annons
Windscribe browser extension showing location set to New York: Gotham, with a background of the United States of America flag behind a blue overlay

Ensure settings that may impact how webpages display are disabled — Windscribe’s extension blocks ads by default. The two icons to the top-right should show a zero.

For the Googlebot browser scenario, I prefer a VPN browser extension to an application, because the extension is specific to my Googlebot browser.

Why spoof Googlebot’s location?

Googlebot mostly crawls websites from US IPs, and there are many reasons for spoofing Googlebot’s primary location.

Some websites block or show different content based on geolocation. If a website blocks US IPs, for example, Googlebot may never see the website and therefore cannot index it.

Ett annat exempel: vissa webbplatser omdirigerar till olika webbplatser eller webbadresser baserat på plats. Om ett företag hade en webbplats för kunder i Asien och en webbplats för kunder i Amerika och omdirigerade alla amerikanska IP-adresser till den amerikanska webbplatsen, skulle Googlebot aldrig se den asiatiska versionen av webbplatsen.

Andra Chrome-tillägg som är användbara för granskning av JavaScript-webbplatser

Med Länkomdirigeringsspårning, Jag ser med ett ögonkast vilket serversvar en URL returnerar.

De Visa renderad källa tillägg möjliggör enkel jämförelse av rå HTML (vad webbservern levererar till webbläsaren) och renderad HTML (koden som återges i webbläsaren på klientsidan).

Jag har också lagt till NoJS Side-by-Side bokmärke till min Googlebot-webbläsare. Den jämför en webbsida med och utan JavaScript aktiverat, inom samma webbläsarfönster.

Steg 3: Konfigurera webbläsarinställningar för att emulera Googlebot

Därefter kommer vi att konfigurera Googlebot webbläsarinställningar i linje med vad Googlebot inte stöd vid genomsökning av en webbplats.

Annons

Vad stöder inte Googlebot genomsökning?

  • Servicearbetare (eftersom personer som klickar till en sida från sökresultat kanske aldrig har besökt tidigare, så det är inte meningsfullt att cachelagra data för senare besök).

  • Tillståndsbegäranden (t.ex. push-meddelanden, webbkamera, geolokalisering). Om innehållet är beroende av något av dessa kommer inte Googlebot att se det innehållet.

  • Googlebot är statslös och stöder inte cookies, sessionslagring, lokal lagring eller IndexedDB. Data kan lagras i dessa mekanismer men kommer att rensas innan Googlebot genomsöker nästa webbadress på en webbplats.

Dessa punktpunkter sammanfattas från en intervju av Eric Enge med Googles Martin Splitt:

Steg 3a: DevTools-inställningar

För att öppna utvecklarverktyg i Chrome eller Canary, tryck på F12, eller använd hamburgermenyn uppe till höger och navigera till Fler verktyg > Utvecklarverktyg:

Screenshot showing the steps described above to access DevTools

Utvecklarverktygsfönstret är vanligtvis dockat i webbläsarfönstret, men ibland föredrar jag det i ett separat fönster. För det, ändra "Dock sida" i den andra hamburgermenyn:

Screenshot showing the 'Dock side' of DevTools
Inaktivera cache

Om du använder normal Chrome som din Googlebot-webbläsare kan du ha gjort detta redan.

Annars, via DevTools hamburgermeny, klicka på Fler verktyg > Nätverksvillkor och markera alternativet "Inaktivera cache":

DevTools screenshot showing the actions described above to disable cache
Blockera servicearbetare

För att blockera tjänstearbetare, gå till fliken Applikationer > Servicearbetare > kryssa för "Kringgå för nätverk":

Screenshot showing the steps described above to disable service workers

Steg 3b: Allmänna webbläsarinställningar

I din Googlebot-webbläsare, navigera till Inställningar > Sekretess och säkerhet > Cookies (eller besök chrome://settings/cookies direkt) och välj alternativet "Blockera alla cookies (rekommenderas inte)" (är det inte kul att göra något "rekommenderas inte?"):

Screenshot showing how to block cookies in Chrome settings

I avsnittet "Sekretess och säkerhet" väljer du också "Webbplatsinställningar" (eller besök chrome://settings/content) och individuellt blockera plats, kamera, mikrofon, aviseringar och bakgrundssynkronisering (och troligen allt som visas där i framtida versioner av Chrome):

Screenshot of Chrome's privacy settings

Steg 4: Emulera en mobil enhet

Slutligen, eftersom vårt mål är att emulera Googlebots genomsökning på mobilen, emulera en mobil enhet i din Googlebot-webbläsare.

Uppe till vänster i DevTools, klicka på enhetens verktygsfältsväxling och välj sedan en enhet att emulera i webbläsaren (du kan lägga till andra enheter också):

Screenshot showing mobile device emulation in Chrome

Vilken enhet du än väljer rullar inte Googlebot på webbsidor utan renderar istället med ett fönster med en lång vertikal höjd.

Jag rekommenderar att du testar webbplatser i skrivbordsvy också, och på faktiska mobila enheter om du har tillgång till dem.

Vad sägs om att se en webbplats som bingbot?

För att skapa en bingbot-webbläsare, använd en senaste version av Microsoft Edge med bingbot användaragent.

Bingbot liknar Googlebot när det gäller vad den gör och inte stöder.

Yahoo! Sök, DuckDuckGo, Ecosia och andra sökmotorer drivs antingen av eller baseras på Bing-sökning, så Bing är ansvarig för en högre procentandel av sökningen än vad många inser.

Annons

Sammanfattning och avslutande anteckningar

Så där har du din alldeles egna Googlebot-emulator.

Att använda en befintlig webbläsare för att emulera Googlebot är den enklaste metoden för att snabbt visa webbsidor som Googlebot. Det är också gratis, förutsatt att du redan använder en stationär enhet som kan installera Chrome och/eller Canary.

Det finns andra verktyg som hjälper dig att "se" vad Google ser. jag uppskattar testar Googles Vision API (för bilder) och deras Natural Language API.

Granskning av JavaScript-webbplatser – särskilt när de renderas dynamiskt – kan vara komplicerat, och en Googlebot-webbläsare är ett sätt att göra processen enklare. Om du vill lära dig mer om granskning av JavaScript-webbplatser och skillnaderna mellan vanliga HTML- och JavaScript-renderade webbplatser rekommenderar jag att du letar upp artiklar och presentationer från Jamie Indigo, Joe Hall och Jess Peck. Två av dem bidrar i videon nedan. Det är en bra introduktion till JavaScript SEO och berör punkter jag nämnde ovan:

Frågor? Något jag missat? Tweet mig @AlexHarfordSEO. Tack för att du läser!



Källlänk

Annons

MARKNADSFÖRING

De 21 bästa verktygen för generering av potentiella kunder 2022

Publicerad

The 21 Best Lead Generation Tools in 2022

Uppkopplad generering av leads är svårt. Det finns ingen väg runt det. Du kan dock ge ditt varumärke ett försprång genom att optimera leadsgenereringsvägar på din webbplats.

(mer …)

Fortsätt läsa

MISSA INTE NÅGRA VIKTIGA NYHETER!
Prenumerera på vårt nyhetsbrev
Vi lovar att inte spamma dig. Avsluta prenumerationen när som helst.
Ogiltig e-postadress

Trendigt

sv_SESvenska