How to Build a Search Engine: Crawling, Indexing, Ranking and Vectors

A catalog team adds vector search because the demo looked good, and a week later a buyer searches for a part number and gets lookalike parts instead of the part.

How to build a search engine in 2026 is mostly a data and evaluation problem: get clean documents into a Lucene-based index, rank with BM25, and add embeddings and learned ranking where a judgment list shows they win.

What is a search engine?

A search engine is software that takes a query and returns the documents from a collection that best answer it, in ranked order. The collection, called the corpus, can be the public web, one website, a product catalog, a ticketing system or a company's document store. The mechanics are the same in each case; the scale and the ranking signals change.

A database query with LIKE '%term%' finds rows that contain a string, by scanning them. A search engine reads a precomputed index instead, so it answers in milliseconds over millions of documents, matches running to run, tolerates typos and scores every match by how well it fits the query.

A vertical search engine is a general engine narrowed to one domain: jobs, case law, products, hotels. Most search engines a company builds are of this kind. A metasearch engine is a different system. It owns no index of its own; it sends each query to other engines or supplier APIs and merges what comes back, which is how flight and hotel comparison sites work, and our technical deep dive into metasearch engines covers that architecture. Everything below is about engines that build and own their index.

How do search engines work?

An engine that owns its index runs two paths. The indexing path runs offline: it collects documents, turns them into terms and vectors, and writes the index. The query path runs on every request, within tens of milliseconds: it parses the query, retrieves candidates from the index, ranks them and returns a page of results. Google describes its own engine in three stages, crawling, indexing and serving, and a site search over 50,000 pages has the same parts at a smaller size.

Crawling or ingestion

A web search engine finds pages by crawling. It starts from seed URLs and sitemaps, fetches each page, extracts its links and queues the new ones. The hard part is the scheduler: Googlebot decides algorithmically which sites to crawl, how often and how many pages to fetch from each, so that it covers billions of pages without overloading any single server.

Most search engines a company builds do not crawl at all. A catalog, a help center or a document store already holds its content in a database or a content management system, so the engine ingests it: a bulk load for the first index, then a stream of changes (change data capture from the database log, or events on a queue) that reaches the index within seconds.

Parsing and text analysis

Raw documents arrive as HTML, PDF, Office files or JSON records. Parsing extracts the text and the fields that matter (title, headings, body, price, date, language) and drops navigation, footers and ads.

An analyzer then turns each text field into terms. A tokenizer splits the text; filters lowercase it, fold accents, remove stop words and reduce words to a stem, so Running Shoes is stored as run and shoe. The same analyzer runs on the query, which is why a search for running shoe matches. A choice made here cannot be repaired by ranking later: a term that was never indexed cannot be found.

The inverted index

The inverted index maps every term to a posting list: the documents that contain it, with the term frequencies and positions the ranker needs.

run    -> doc 3 (tf 2), doc 8 (tf 1), doc 41 (tf 4)
shoe   -> doc 3 (tf 1), doc 41 (tf 2), doc 77 (tf 1)
trail  -> doc 41 (tf 1), doc 77 (tf 3)

A query for trail running shoe reads three short lists and merges them instead of scanning every document, so lookup cost grows with the number of matches rather than with the size of the corpus. Lucene, the Java library under Elasticsearch, OpenSearch and Solr, writes the index as immutable segments and merges them in the background. A new document becomes searchable when the segment holding it is opened, which Elasticsearch does every second by default, on indices that received a search in the last 30 seconds. Search is near real-time, not real-time.

Next to the postings, Lucene keeps columnar doc values for sorting, filters and facets, and a separate structure for vectors.

Ranking: BM25, link signals and learned ranking

Matching produces candidates; ranking puts them in order. The default text scorer in Lucene-based engines is BM25. It scores a document higher when the query terms appear in it more often, with diminishing returns (parameter k1, default 1.2), when those terms are rare across the corpus (inverse document frequency), and when the document is short relative to the average (length normalization b, default 0.75), as the Elasticsearch similarity reference sets out. BM25 needs no training data and no GPU, and it gets exact matches right: part numbers, names, error codes.

Text match is one signal among several. Web search adds link signals: a page linked from many well-linked pages is more likely to be an authority, the idea behind PageRank, and the anchor text of those links describes the page in other people's words. A catalog has its own equivalents in sales, margin, stock, ratings and recency.

Learning to rank (LTR) combines the signals with a model trained on graded examples of good and bad results for real queries. The usual model is a gradient-boosted tree ensemble trained with LambdaMART, such as XGBoost's ranker; Elastic's LTR guide picks this family for strong ranking at low inference latency. It reranks the top 100 or so BM25 candidates, because scoring every match with a model would be too slow.

Vector search, embeddings and hybrid retrieval

An embedding model turns a text into a vector of a few hundred to a few thousand numbers, placed so that texts with similar meaning sit close together. Embed every document (or every passage of a long one), embed the query with the same model, and a nearest-neighbor search finds documents that match the meaning without sharing a word: laptop that stays cool finds a product page about thermal design. Comparing the query with every vector is too slow beyond a few million, so engines use approximate indexes, most often the HNSW graph, which Lucene implements natively.

Vectors fail where BM25 is strong. The embedding of SKU 4471-B lands near other SKUs, not on that one, and a model trained on general web text has never seen your product names. The BEIR benchmark, which tested ten retrieval systems on 18 datasets, found BM25 a strong baseline across domains, dense retrievers often behind the other approaches outside the data they were trained on, and rerankers best on average at a high computational cost.

Production engines in 2026 therefore run hybrid retrieval: a BM25 query and a vector query in parallel, merged into one list. The simplest merge is reciprocal rank fusion (RRF), which ignores the raw scores and sums 1/(k + rank) for each document across the lists, with k set to 60 by default in Elasticsearch. OpenSearch offers RRF and score normalization in its hybrid search pipelines. A reranker on the fused top 50 to 100 is the usual last stage.

Retrieval-augmented generation (RAG) uses the same stack and passes the top results to a language model. The search engine underneath decides what the model gets to read, so a RAG system is only as good as its retrieval.

How to build a search engine, step by step

The steps follow the order in which decisions depend on each other: the corpus decides how documents are collected, the text decides the analyzers, and no ranking change can be judged until step 7 exists. Build a thin version of all seven first, then iterate on steps 5 to 7.

1. Define the corpus, the queries and the latency budget

Write down three things before choosing a tool.

• The corpus: which documents, how many, how often they change and in which languages. Five million products whose prices change every few minutes and two billion web pages recrawled monthly need different systems.
• The queries: collect 200 to 500 real ones from logs, support tickets or an existing search box, and sort them by type (exact identifiers, navigational, descriptive, questions). The mix predicts how much vectors will help. A log full of part numbers needs BM25 and careful tokenization; a log full of something for a rainy hike needs semantics.
• The budget: latency at the 99th percentile (p99), queries per second at peak, freshness (how long until an edit is searchable) and the monthly infrastructure spend.

If you need results from the open web, decide now how to get them. Microsoft retired the Bing Search APIs on 11 August 2025 and pointed customers to Grounding with Bing Search in its AI agent service, so a web index you rank yourself means crawling, licensing a commercial index API or starting from the Common Crawl archive.

2. Crawl or ingest the documents

For the open web, start from an existing crawler. Apache Nutch crawls in batches on Hadoop; Apache StormCrawler (version 3.7.0 at the time of writing) crawls as a stream on Apache Storm. Both are Java under the Apache license and handle the URL frontier, politeness, robots.txt and retries. If you write your own fetcher, the crawler-commons library parses robots.txt and sitemaps.

The cheapest start is the Common Crawl archive. Its September 2026 crawl holds 2.17 billion pages, 361.4 TiB uncompressed, free to download as WARC files, with the extracted plain text published alongside.

For site, catalog or document search, ingestion is a data integration job. Load a full snapshot for the first build, then stream changes: change data capture from the database log with a tool such as Debezium, or events the application publishes to Kafka. Keep a raw copy of every document outside the search engine. When an analyzer or the mapping changes, reindexing from that copy takes hours; without it, it takes a re-crawl.

3. Parse and normalize the text

Jsoup parses HTML in Java, and Apache Tika extracts text and metadata from over a thousand file types, PDF and Office among them. Remove boilerplate before indexing: a footer repeated on 40,000 pages adds the same terms to every document and skews the corpus statistics BM25 depends on. Detect the language of each document and send each language to its own analyzer.

Put the structured fields into a typed schema: prices as numbers, dates as dates, categories as keywords. Filters and facets run on these exact values, never on analyzed text. Compute near-duplicate fingerprints at this stage, and an embedding per passage if you plan to use vectors. Split long documents into passages of a few hundred tokens, because one vector for a 30-page PDF says little about any page in it.

4. Build the index with Lucene, Elasticsearch or OpenSearch

Writing your own inverted index teaches how search works, but it is a poor way to ship one. Unless search is your product, use a Lucene-based engine and spend the effort on data and ranking. Apache Lucene is at version 10.5 in October 2026. Elastic announced the AGPL as a license option for Elasticsearch in August 2024, and OpenSearch is a Linux Foundation project on its 3.x release line.

Design the mapping before loading data. A minimal Elasticsearch index for text with vectors, created under a versioned name and exposed through an alias:

PUT /docs_v1
{
  "mappings": {
    "properties": {
      "url":     { "type": "keyword" },
      "lang":    { "type": "keyword" },
      "title":   { "type": "text", "analyzer": "english" },
      "body":    { "type": "text", "analyzer": "english" },
      "updated": { "type": "date" },
      "body_vector": {
        "type": "dense_vector",
        "dims": 1024,
        "similarity": "cosine",
        "index_options": { "type": "bbq_hnsw" }
      }
    }
  }
}

POST /_aliases
{ "actions": [ { "add": { "index": "docs_v1", "alias": "docs" } } ] }

Three decisions are expensive to change because each one forces a full reindex: the analyzer of each field, the embedding model and its dimensions, and the number of primary shards. The alias makes a reindex routine. Build docs_v2 beside the live index, compare the two on the judgment list, then move the alias in one atomic call.

5. Rank with BM25 first, then add learning to rank or hybrid retrieval

Start with BM25 across several fields with boosts (a title match worth more than a body match), phrase proximity, and filters for hard constraints such as availability or language. Add business signals as function scores: popularity, freshness, stock. Keep each change only if it improves the judgment list score from step 7.

When BM25 stops improving, there are two upgrades, and they fix different problems.

Hybrid retrieval fixes recall: relevant documents that share no words with the query never reach the ranker under BM25 alone. In Elasticsearch it is one retriever:

GET /docs/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": {
            "query": {
              "multi_match": {
                "query": "waterproof trail running shoe",
                "fields": ["title^3", "body"]
              }
            }
        } },
        { "knn": {
            "field": "body_vector",
            "query_vector": [0.012, -0.044, ...],
            "k": 50,
            "num_candidates": 200
        } }
      ],
      "rank_window_size": 100,
      "rank_constant": 60
    }
  },
  "size": 10
}

Learning to rank fixes ordering: the right documents are in the top 100 in the wrong order. It needs graded judgments, from human raters or from click logs corrected for position bias, and features for each query and document pair. Elasticsearch runs XGBoost models uploaded with its eland client as a rescorer, on paid subscription levels; OpenSearch ships a learning to rank plugin under the Apache license.

Cross-encoder rerankers, which read the query and each candidate together, sit between the two: no training data of your own, better ordering than BM25, and a cost in latency and compute on every query, so they run on the top 50 to 100 candidates only. The two-stage pattern is the one our guide to building a recommendation system describes for recommendations: cheap retrieval over everything, expensive ranking over a few hundred.

6. Serve the query path within a latency budget

A query passes through several stages, and each one takes part of the budget. Here is the split we would start from for a catalog or site search answering in 200 ms at p99; the numbers are a starting point to measure against, not a benchmark.

Caching carries the head of the query distribution. A small set of queries repeats all day, so a result cache keyed on the normalized query and its filters, invalidated on index refresh, answers them without touching the cluster. Below that, the engine's filter cache and the operating system's page cache keep the hot parts of the index in memory. Give search nodes enough RAM for the index files, and keep the JVM heap small enough that the page cache gets the rest; garbage collection tuning matters as much here as in any low-latency Java application.

The result page belongs to the query path too. Highlighted snippets show why a result matched, facets are computed over the matching set, and autocomplete runs against a separate prefix index so that it answers in a few milliseconds as the user types.

7. Evaluate relevance offline and online

Build the evaluation set before tuning anything. Take the real queries from step 1, pool the top 10 to 20 results for each from every configuration you are testing, and grade every query and document pair from 0 (irrelevant) to 3 (exactly right). That is the judgment list.

Score each configuration with nDCG@10 for ranked quality, recall@100 for the retrieval stage (a reranker cannot fix what retrieval missed), and mean reciprocal rank for navigational queries with one right answer. Elasticsearch's ranking evaluation API computes these metrics against a judgment list. A language model can grade at scale, once its grades have been checked against a few hundred human ones.

Offline scores predict how a change will do, and an online test confirms it. Ship each ranking change behind an A/B test or interleaving and watch click-through on the first results, the zero-result rate, reformulations (a user who rephrases has failed to find something) and, for a catalog, conversion per search. Log every query with the result IDs shown and the clicks, and the logs become the next judgment list.

Things to watch out for when building a search engine

robots.txt and crawl politeness

robots.txt became an internet standard with RFC 9309 in September 2022. Crawlers must parse at least 500 KiB of the file and should not use a cached copy for more than 24 hours. Google applies the same 500 KiB limit and does not support crawl-delay, so a crawler that wants to be polite cannot rely on site owners to state a rate.

Set the rate yourself: a cap on concurrent connections per host, backoff on 429 and 5xx responses, a user agent that names the crawler and links to a contact page, and respect for noindex and nofollow. Crawlers that skip these get blocked by site owners and by CDN bot protection, and a blocked host is a gap in the index.

Duplicate and near-duplicate content

Many crawled URLs are copies: the same page with tracking parameters, print versions, mirrors, syndicated articles. Left in, they waste crawl capacity and fill a result page with the same content ten times. Normalize URLs before queuing them (strip session IDs and tracking parameters, follow rel=canonical), hash the extracted text to catch exact copies, and use SimHash or MinHash fingerprints for near-duplicates, as Google described for its crawler in Detecting near-duplicates for web crawling. Google's indexer still groups similar pages and picks one canonical to show. Catalogs meet the same problem as one product listed by six sellers.

Index freshness against cost

Every refresh opens a new segment, and every segment adds merge work later. On an index taking heavy writes, a one-second refresh spends CPU and disk I/O that queries need; raising refresh_interval to 30 seconds, or switching it off during bulk loads, gives that capacity back. Set freshness per kind of change. A price or availability update may need seconds; a new article can wait a minute.

When freshness and query volume are both extreme, a general-purpose search engine can be the wrong store for the hot data. A UK hotel metasearch engine we worked on answered about 200 million availability queries a day from a cluster of dozens of SQL Server machines, where one more machine added about 2% to the cost and 0.1% to the throughput. We moved availability search into an in-memory data grid that holds everything a query needs, filled by a fast loader and kept current by a message queue of intraday updates, with SQL Server kept as the system of record. The engine went on to serve 300 million queries a day, with response time down 60% and infrastructure cost down 80% (case study). Full-text ranking rarely needs this; structured lookups at high rates over constantly changing data often do, and our post on what Hazelcast is used for covers the pattern.

Relevance without evaluation data

The failure we see most often is tuning by anecdote. Someone senior searches for one thing, dislikes the result, and a boost is added. After fifty boosts nobody knows which ones still help.

Without a judgment list, a switch to vectors or a new embedding model is a guess, and a regression on query types nobody tried stays hidden until users complain. The 200-query set from step 1 takes days to build and makes every later decision cheaper.

Multilingual text and tokenization

Analyzers are language specific. An English analyzer applied to German keeps compounds such as Hotelzimmerreservierung as one token, so a search for Hotelzimmer misses it; German needs a decompounding filter. Chinese and Japanese text has no spaces between words, so whitespace tokenization fails outright. Use a dictionary-based tokenizer such as Kuromoji for Japanese or SmartCN for Chinese, or ICU segmentation.

Index each language in its own field with its own analyzer, and detect the language of the query. Multilingual embedding models match meaning across languages, but like any embedding they miss rare names and identifiers, in every language.

Query latency at the tail

Averages hide the problem. A search cluster fans each query out to every shard and waits for the slowest one. In The Tail at Scale, Jeffrey Dean and Luiz André Barroso show that if one request in a hundred takes over a second on a single server, a request that needs 100 servers in parallel takes over a second 63% of the time. Garbage collection pauses, segment merges and noisy neighbors produce exactly those outliers.

Measure p99 and p99.9, never only the mean. Keep the shard count no higher than the data needs, since every extra shard is one more server to wait for. Hedged requests, sending a second copy to another replica when the first is slow, cut 99.9th-percentile latency for a 1,000-key fetch from 1,800 ms to 74 ms in the same paper, at the cost of 2% more requests.

The cost of vector indexes

HNSW is fast only while the vectors and the graph stay in memory, in the operating system's page cache rather than the JVM heap. By Elastic's sizing formulas, 1 million float vectors of 1,024 dimensions with the default graph settings need 3.87 GB of off-heap RAM per copy of the index. Ten million vectors with one replica need over 77 GB for the vectors alone, before any text index.

Quantization cuts the bill: int8 stores one byte per dimension and Better Binary Quantization (BBQ) about one bit, while the full-precision vectors stay on disk to rescore the top candidates. Embeddings also cost at ingestion. Every change of embedding model means re-embedding the whole corpus, so measure what vectors add over BM25 on your judgment list before paying for them at full scale.

Abuse and spam

A public search box is an API anyone can call. Scrapers page through results to copy a catalog, and bots send queries that cost far more than normal ones: leading wildcards, deep pagination, huge facet requests. Rate limit per client, cap result depth (Elasticsearch refuses results beyond from + size of 10,000 by default, through index.max_result_window), reject leading wildcards and cache what repeats.

On the index side, content is written to game the ranker: keyword-stuffed pages and link farms on the web, sellers repeating every brand name in a product title on a marketplace. Penalize repeated terms, weight links by the trust of the page they come from, and expect any ranking signal a third party controls to be inflated.

Most of these problems surface only at production volume, when the query log is real and the index is ten times the size of the test set. That is where we usually come in: the hotel engine above came to us when adding hardware had stopped adding throughput. If your search is getting slower or more expensive as it grows, book a scoping call with one of our architects and we will start by measuring where the time and the money go.