Searching a copy, not the live web
When you type a query, the search engine does not scan the whole web in that moment. That would take far too long. Instead, it searches its own prepared copy of the web. The work is done ahead of time in three stages: crawling, indexing and ranking. The first two happen continuously in the background, and the third happens when you search.
Crawling: finding pages
Search engines use programs known as crawlers or spiders. A crawler starts from a set of known pages, reads each one, and notes the links on it. Each link points to another page, which can be added to the list of places to visit. By following links over and over, a crawler can discover a large part of the web. Site owners can also submit a list of their pages, called a sitemap, to help the engine find them.
Crawlers are polite by design. A site can include a file called robots.txt that tells crawlers which areas to avoid, and crawlers usually limit how fast they request pages so as not to overload a server. They also revisit pages from time to time, since pages change. Pages that change often or are considered important may be revisited more frequently, while others are revisited rarely.
Not everything is reachable. Pages behind a login, pages that no one links to, and pages that site owners have blocked will typically not be found. The part of the internet that search engines cannot see is sometimes called the deep web, which is mostly mundane material such as private accounts and databases rather than anything sinister.
Indexing: organizing what was found
After fetching a page, the engine analyzes it. It extracts the text, the title, headings, links, images and other details, and works out what language the page is in and what it seems to be about. It may detect pages that are near-duplicates of others and choose one to represent them.
The results are stored in an index. A helpful comparison is the index at the back of a book, which lists each term and the pages where it appears. A search index works the same way at huge scale: for each word or phrase, it records which pages contain it. This structure is often called an inverted index, and it is what allows a query to be answered in a fraction of a second. The engine looks up the words in your query and quickly finds the pages that match, without reading any pages again.
Not every crawled page makes it into the index. Pages that are low quality, duplicated, blocked or broken may be left out.
Understanding the query
Before ranking, the engine tries to work out what you want. It may correct spelling, treat related words as equivalent, recognize that “jaguar” could be an animal or a car and use context to guess, and notice whether a query suggests a place, a product, a question or a news event. Modern engines rely heavily on machine learning for this, and some use language models to better capture meaning rather than just matching words. The description here stays at a conceptual level, as the exact methods are proprietary and change often.
Ranking: putting results in order
Many pages may match a query, so the engine has to decide which come first. It does this by combining many signals, which are pieces of evidence about how relevant and useful a page is. The details are not public, but commonly discussed categories include:
- Relevance. How well the page’s content matches the words and meaning of the query, including whether those words appear in prominent places like the title.
- Links. A page that many other respected pages link to is often treated as more trustworthy. An early and well-known idea, PageRank, treated links as a kind of vote, with votes from important pages counting for more.
- Quality and trust signals. Attempts to judge whether a page is original, accurate and from a reliable source, especially on health or money topics.
- Freshness. For topics such as news, recent pages matter more. For stable topics, older pages may do just as well.
- Usability. Whether the page loads quickly and works on mobile devices.
- Context. Your location, language and sometimes your past searches can change the results, which is a reason two people searching the same words may see different pages.
No single signal decides the order. Engines also adjust their methods regularly, and they test changes to see whether users find results more helpful.
What you see on the results page
Many results pages go beyond a list of links. They may show direct answers, maps, images, videos, shopping results, or summaries drawn from several sources. Some of these come from separate databases and are not ranked in the same way as ordinary web pages. Some results are advertisements, which are normally labeled and which companies pay for. Ads are placed by an auction-style system rather than by the ranking process described above.
Because the page you get depends partly on tracking of your past activity and device, the topic overlaps with web tracking, which is worth understanding if you care about how personalized your results are.
Limits and cautions
A search engine finds pages that match and seem relevant. It does not verify that each page is correct. A highly ranked page can still be wrong, and a helpful page can rank poorly. Search results can also be manipulated by people trying to rank higher with low-quality content, and engines work to fight this, with mixed success. Knowing that, it is wise to apply some source evaluation, such as checking who published a page and whether other reliable sources agree. This matters even more with online misinformation, which can spread widely.
Key points
A search engine crawls the web by following links, stores what it finds in an index that works like a giant book index, and ranks matching pages using many signals when you search. The result is a prepared, best-guess ordering of pages, not a guarantee of truth.