How Search Engines Work

Joshua Hardwick
Head of Content @ Ahrefs (or, in plain English, I'm the guy responsible for ensuring that every blog post we publish is EPIC).
Article stats
  • Monthly traffic 1,753
  • Linking websites 245
  • Tweets 74
Data from Content Explorer

Shows how many different websites are linking to this piece of content. As a general rule, the more websites link to you, the higher you rank in Google.

Shows estimated monthly search traffic to this article according to Ahrefs data. The actual search traffic (as reported in Google Analytics) is usually 3-5 times bigger.

The number of times this article was shared on Twitter.

Search engines work by crawling billions of pages using web crawlers. Also known as spiders or bots, crawlers navigate the web and follow links to find new pages. These pages are then added to an index that search engines pull results from.

Understanding how search engines function is crucial if you’re doing SEO. After all, it’s hard to optimize for something unless you know how it works.

That’s what you’ll learn in this guide.

Contents


    Let’s begin by exploring what search engines are, why they exist, and how they make money. 

    What are search engines?

    Search engines are searchable databases of web content. They’re made up of two main parts: 

    1. Search index. A digital library of information about web pages. 
    2. Search algorithm(s). Computer program(s) that rank matching results from the search index.

    What is the aim of search engines?

    Every search engine aims to provide the best, most relevant results for users. That’s partly how they gain market share.

    How do search engines make money?

    Search engines have two types of search results: 

    1. Organic results from the search index. You can’t pay to be here.
    2. Paid results from advertisers. You can pay to be here.

    Each time someone clicks a paid search result, the advertiser pays the search engine. This is known as pay-per-click (PPC) advertising and it’s why market share matters. More users means more ad clicks and more revenue.

    How search engines make money

    Each search engine has its own process for building a search index. Below is a simplified version of the process Google uses.[1]

    How Google builds its search index

    Let’s break it down.

    URLs

    Everything begins with a known list of URLs. Google discovers these in many ways, but the three most common are: 

    • From backlinks. Google has an index of hundreds of billions of web pages.[2] If someone links to a new page from a known page, Google can find it from there.
    • From sitemaps. Sitemaps tell Google which pages and files you think are important on your site.[3]
    • From URL submissions. Google lets site owners request crawling of individual URLs in Google Search Console.

    Crawling

    Crawling is where a computer bot called a spider visits and downloads known URLs. Google’s crawler is Googlebot.[4]

    Processing & rendering

    Processing is where Google works to understand and extract key information from crawled pages. To do this, it has to render the page, which is where it runs the page’s code to understand how it looks for users. 

    Nobody outside of Google knows every detail about this process. But it doesn’t matter. All we really need to know is that it involves extracting links and storing content for indexing. 

    Indexing

    Indexing is where processed information from crawled pages gets added to the search index.

    The search index is what you search when you use a search engine. That’s why getting indexed in major search engines like Google and Bing is so important. Users can’t find you unless you’re in the index. 

    Did you know?

    Google owns 91.43% of the search engine market. It can send you more traffic than other search engines as it’s the one most people use.[5]


    Discovering, crawling, and indexing content is only the first part of the puzzle. Search engines also need a way to rank matching results when a user performs a search. This is the job of search algorithms.

    What are search algorithms?

    Search algorithms are formulas that match and rank relevant results from the index. Google uses many factors in their algorithms.

    Key Google ranking factors

    Nobody knows every Google ranking factor because Google hasn’t disclosed them. But we do know some key ones. Let’s look at a few of them.

    Backlinks

    Backlinks are links from a page on one website to another. They’re one of Google’s strongest ranking factors.[6] This is probably why we saw a strong correlation between linking domains and organic traffic in our study of over a billion pages.[7]

    The correlation between referring domains and search traffic

    It’s not all about quantity, though. Quality matters too. Pages with a few high-quality backlinks often outrank those with many low-quality backlinks.

    Did you know?

    You can check backlinks to your website for free in Ahrefs. 

    Sign up for a free Ahrefs Webmaster Tools account, enter your domain in Site Explorer, and go to the Backlinks report.

    Backlinks report in Ahrefs' Site Explorer

    Our crawler is the fifth most active on the web,[8] so you’ll see a pretty complete view of your backlinks here. 

    Relevance

    Relevance is the usefulness of a given result for the searcher. Google has many ways of determining this. At the most basic level, it looks for pages containing the same keywords as the search query. It also looks at interaction data to see if others found the result useful.[9]

    Freshness

    Freshness is a query-dependent ranking factor. It’s stronger for searches that call for fresh results.[9] That’s why you see a recently published top result for “new netflix series” but not “how to solve a rubik’s cube.” 

    Freshness is a query-dependant Google ranking factor

    Page speed

    Page speed is a ranking factor on desktop and mobile.[10][11] But it’s more of a negative ranking factor than positive. This is because it negatively affects the slowest pages rather than positively affecting lightning-fast pages.

    Did you know?

    You can check your page speed for free in Ahrefs.

    Sign up for a free Ahrefs Webmaster Tools account, crawl your website with Ahrefs’ Site Audit, then go to the Performance report. In general, the less red you see, the better. 

    Performance report in Ahrefs' Site Audit

    Mobile-friendliness

    Mobile-friendliness has been a ranking factor on mobile and desktop since Google’s switch to mobile-first indexing in 2019.[12]


    Google tailors search results for each user. They use information such as your location, language, and search history to do this.[9] Let’s take a closer look at these things.

    Location

    Google uses your location to personalize the results for searches with local intent. That’s why all results for “Italian restaurant” are from or about local restaurants. Google knows you’re unlikely to fly halfway around the world for lunch.

    Language

    Google knows there’s no point in showing English results to Spanish users. That’s why it ranks localized versions of content (if available) to users who speak different languages.

    Google ranks different versions of pages for different languages

    Search history

    Google saves the things you do and places you go to give you a more personalized search experience.[13] You can opt out of this, but most people probably don’t. 

    Key takeaways

    • Search engines consist of two main parts: index and algorithms.
    • To build their index, they crawl known pages and follow links to find new ones. 
    • The aim of search algorithms is to return the best, most relevant results.
    • Search result quality is important for building market share.
    • Nobody knows all Google’s ranking factors for organic results.
    • Key ranking factors include backlinks, relevance, and freshness. 
    • Google personalizes its results based on your location, language and search history. 

    References

    1. Understand JavaScript SEO Basics”. Google. Retrieved August 16th 2022. 
    2. Organizing Information — How Google Search Works”. Google. Retrieved 16th August 2022
    3. Learn about sitemaps”. Google. Retrieved 16th August 2022
    4. Googlebot”. Google. Retrieved 16th August 2022
    5. Search Engine Market Share Worldwide”. Statcounter. Retrieved 16th August 2022
    6. Google Q&A+ #March”. YouTube. Retrieved 16th August 2022
    7. 90.63% of Content Gets No Traffic From Google. And How to Be in the Other 9.37%”. Ahrefs. January 31, 2020. Retrieved 16th August 2022
    8. CloudFlare Radar”. CloudFlare. Retrieved 16th August 2022
    9. Ranking Search Results — How Google Search Works”. Google. Retrieved 16th August 2022
    10. Using site speed in web search ranking”. Google. Retrieved 16th August 2022
    11. Using page speed in mobile search ranking”. Google. Retrieved 16th August 2022
    12. Mobile-first indexing best practices”. Google. Retrieved 16th August 2022
    13. Find & control your Web & App Activity”. Google. Retrieved 16th August 2022
    • Monthly traffic 1,753
    • Linking websites 245
    • Tweets 74
    Data from Content Explorer