You type “best coffee Melbourne” into Google. 0.43 seconds later, you have millions of results ranked in order of relevance.

Most people have no idea how that happens. The technology is more interesting than you’d think.

The Three Core Processes

Search engines do three main things: crawling, indexing, and ranking.

Crawling is discovering web pages. Google’s bots (they call them spiders or crawlers) constantly browse the web, following links from page to page, cataloguing what they find.

Indexing is storing and organising that information. The crawler doesn’t just save the full page—it analyses and categorises the content, extracting keywords, understanding structure, and storing it in massive databases.

Ranking is deciding which results to show and in what order when someone searches. This is the complex part.

How Crawling Works

Google’s crawlers start with known pages—usually popular sites or pages that have been submitted directly. They load each page, note the content, and follow every link they find.

Those links lead to new pages, which get crawled, which contain more links, and so on. It’s a continuous process. Google recrawls popular sites multiple times per day. Lesser-known sites might get crawled weekly or monthly.

Not everything gets crawled. Pages blocked by robots.txt files (instructions that tell crawlers not to visit) get skipped. Pages that require login aren’t usually crawled. Pages with no inbound links might never be discovered.

This is why SEO (search engine optimisation) exists—making sure crawlers can find, access, and understand your content.

The Indexing Process

Once a page is crawled, it gets indexed. Think of the index as a massive library catalogue.

Google doesn’t store the full visual page as you see it. It extracts:

• Text content and keywords
• Images and alt text
• Links and their anchor text
• Page structure (headings, paragraphs, lists)
• Metadata (title tags, descriptions, structured data)
• Page speed and performance metrics
• Mobile-friendliness

All of this gets processed and stored in a searchable format. When you search, Google isn’t scanning the entire internet in real-time—it’s searching its index, which is much faster.

Google’s index is estimated to contain hundreds of billions of pages. The infrastructure required to store and search that is staggering.

The Ranking Algorithm

This is where it gets complicated. Google uses over 200 factors to determine which results to show and how to rank them.

Some confirmed factors:

• Relevance: Does the content actually match the search query?
• Authority: Is the site credible? This largely comes from backlinks—other sites linking to you
• User experience: Page speed, mobile-friendliness, lack of intrusive popups
• Content quality: Depth, accuracy, uniqueness
• Freshness: Newer content often ranks higher for time-sensitive topics

Google weighs these factors differently depending on the query. For “breaking news,” freshness matters more. For “how to tie a tie,” content quality matters more.

PageRank and Backlinks

Google’s original breakthrough was PageRank, an algorithm that treats links like votes. If important sites link to your page, your page must be important too.

This still matters, though it’s no longer the dominant factor. A link from a major news site carries more weight than a link from a random blog. Links from relevant sites in your industry matter more than random links.

People gamed this system for years—buying links, creating link farms, spamming comments with links. Google’s gotten much better at identifying and discounting these manipulative tactics.

The Personalisation Layer

Search results aren’t the same for everyone. Google personalises based on:

• Your location (search “pizza” and you get local results)
• Your search history (previous searches influence future results)
• Your language and region settings
• Device type (mobile vs desktop)

This is why clearing cookies and searching in incognito mode can show different results—you’re removing some personalisation factors.

How Google Handles Different Query Types

Navigational queries (searching for a specific site): Google tries to show that exact site first. Search “Facebook” and facebook.com is result #1.

Informational queries (seeking knowledge): Google tries to answer directly or show the most authoritative sources. Search “how tall is Mt Everest” and you might get the answer right at the top without clicking anything.

Transactional queries (looking to buy or do something): Google shows commercial results—product pages, services, local businesses.

The same algorithm handles all of these differently based on understanding what you’re actually trying to accomplish.

The Featured Snippet

Google increasingly tries to answer questions directly rather than just linking to answers. These “featured snippets” or “position zero” results appear above traditional results.

To get your content into snippets, you need:

• Clear, concise answers to common questions
• Proper formatting (lists, tables, definitions)
• Content that matches how people actually search

This has created issues for publishers—why would someone click your link if Google already displayed your answer?

Machine Learning and AI

Modern Google relies heavily on AI, particularly systems like RankBrain and BERT.

These systems help Google understand:

• Natural language and context (“What time is it in Tokyo?” vs “Tokyo time”)
• Synonyms and related concepts
• User intent beyond literal keywords
• Content quality at scale

Google’s AI can now understand that “best phone 2026” and “top smartphones this year” are seeking the same information, even though the keywords differ.

Why Some Results Seem Wrong

Google optimises for what most people click on, not necessarily what’s most accurate.

If a misleading page gets more clicks and engagement than a factually correct but boring page, the misleading page might rank higher. This has created problems with misinformation.

Google’s working to address this with “E-E-A-T” (Experience, Expertise, Authoritativeness, Trustworthiness) as ranking factors, but it’s an ongoing challenge.

The Commercial Pressure

Google makes money from ads. Paid results appear above organic results.

The algorithm for ranking ads is different—it’s based on bid amount, ad quality, and expected click-through rate. But the visual similarity between ads and organic results can be misleading.

Google also favours its own properties. Search for something and you might see Google Maps results, YouTube videos, or Google Shopping before traditional website links.

The Competition

Google dominates search in Australia (90%+ market share), but Bing and DuckDuckGo exist.

Bing uses similar principles but different weighting. DuckDuckGo focuses on privacy—no personalisation or tracking, which means more generic results but more privacy.

The basic technology is similar across search engines, but the implementation details and priorities differ.

Why This Matters

Understanding search engines helps you:

• Find better information (knowing how to phrase queries effectively)
• Evaluate result quality (recognising the difference between authoritative sources and SEO spam)
• Build websites that people can actually find
• Understand the power and limitations of search technology

Search engines aren’t neutral arbiters of information. They’re algorithmic systems optimised for specific goals (usually engagement and ad revenue), with all the biases and limitations that implies.

The next time you search for something, you’ll know there’s a sophisticated, imperfect, constantly evolving system working behind that simple search box to guess what you actually want and serve it up in under a second.

That’s genuinely impressive, even if the results aren’t always perfect.