{"id":137785,"date":"2022-01-12T10:01:11","date_gmt":"2022-01-12T15:01:11","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=137785"},"modified":"2025-01-18T14:43:23","modified_gmt":"2025-01-18T19:43:23","slug":"googlebot","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/googlebot\/","title":{"rendered":"What Is Googlebot & How Does It Work?"},"content":{"rendered":"\n
Googlebot is the web crawler used by Google to gather the information needed and build a searchable index of the web. Googlebot has mobile and desktop crawlers, as well as specialized crawlers for news, images, videos, and more.<\/div>\n\n\n\n

There are more crawlers Google uses for specific tasks<\/a>, and each crawler will identify itself with a different string of text called a \u201cuser agent.\u201d Googlebot is evergreen, meaning it sees websites as users would in the latest Chrome browser.<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
Googlebot name<\/th>User-agent string<\/th>\n<\/tr>\n<\/thead>\n
Googlebot Smartphone<\/td>Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/W.X.Y.Z Mobile Safari\/537.36 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)<\/td>\n<\/tr>\n
Googlebot Desktop<\/td>Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html) Chrome\/W.X.Y.Z Safari\/537.36
\n

\nSometimes:
\n

    \n
  • Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)<\/li>
    \n
  • Googlebot\/2.1 (+http:\/\/www.google.com\/bot.html)<\/li>
    \n<\/ul><\/td>\n<\/tr>\n
Googlebot Image<\/td>Googlebot-Image\/1.0<\/td>\n<\/tr>\n
Googlebot Video<\/td>Googlebot-Video\/1.0<\/td>\n<\/tr>\n
Googlebot News<\/td>uses standard Googlebot strings<\/td>\n<\/tr>\n
Google StoreBot Mobile<\/td>Mozilla\/5.0 (Linux; Android 8.0; Pixel 2 Build\/OPD3.170816.012; Storebot-Google\/1.0) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/W.X.Y.Z Mobile Safari\/537.36<\/td>\n<\/tr>\n
Google StoreBot Desktop<\/td>Mozilla\/5.0 (X11; Linux x86_64; Storebot-Google\/1.0) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/W.X.Y.Z Safari\/537.36<\/td>\n<\/tr>\n
Google-InspectionTool Mobile<\/td>Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/W.X.Y.Z Mobile Safari\/537.36 (compatible; Google-InspectionTool\/1.0;)<\/td>\n<\/tr>\n
Google-InspectionTool Desktop<\/td>Mozilla\/5.0 (compatible; Google-InspectionTool\/1.0;)<\/td>\n<\/tr>\n
GoogleOther<\/td>

    \n
  • Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/W.X.Y.Z Mobile Safari\/537.36 (compatible; GoogleOther)<\/li>
    \n
  • Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome\/W.X.Y.Z Safari\/537.36<\/li>
    \n<\/ul><\/td>\n<\/tr>\n
GoogleOther-Image<\/td>GoogleOther-Image\/1.0<\/td>\n<\/tr>\n
GoogleOther-Video<\/td>GoogleOther-Video\/1.0<\/td>\n<\/tr>\n
Google-CloudVertexBot<\/td>Google-CloudVertexBot<\/td>\n<\/tr>\n
Google-Extended<\/td>uses standard Googlebot strings<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

Chrome\/W.X.Y.Z in the user-agent strings is a placeholder and will be the latest version of Chrome used by Googlebot.<\/p>\n\n\n\n

Googlebot runs on thousands of machines. They determine how fast and what to crawl on websites. But they will slow down their crawling so as to not overwhelm websites. 

Googlebot is the fastest crawler on the web according to Cloudflare Radar, with Ahrefsbot being the 2nd fastest.

\"Googlebot<\/p>\n

If we look at that by the percentage of HTTP requests, Googlebot is 23.7% of the overall requests from good bots. Ahrefsbot is 14.27% and just for comparison, Bingbot is 4.57% and Semrushbot is 0.6%.<\/p>\n

\"percent<\/p>\n\n\n\n

Let\u2019s look at their process for building an index of the web.<\/p>\n

Contents<\/div><\/span><\/span><\/span><\/a><\/div>\n

\n\n<\/p>\n