Meta search engine development: where the difficulty actually is
A meta search engine queries many suppliers at once and returns one ranked answer in real time. It looks like a product problem and is a latency problem. We have run one at 300 million queries a day; this page explains what that took.
The engine behind this page
- 300,000,000
- queries a dayReal-time price and availabilityMetasearch case study
- 60%
- lower response timeAgainst the engine it replacedMetasearch case study
- 80%
- lower infrastructure costHotel metasearch platformMetasearch case study
The four problems a metasearch engine has to solve
Supplier aggregation
Querying many suppliers in parallel and returning before the slowest one answers, which means deciding what a partial answer looks like.
Caching strategy
Deciding per data type how stale an answer may be. Some prices tolerate minutes, some seconds, some nothing. This decision, not hardware, is what decides perceived speed.
Ranking and filtering
Sorting and filtering results without a second round trip per interaction, which pushes the data into memory close to the request.
Failure behaviour
What a customer sees when a supplier times out or answers nonsense, designed rather than discovered.
Almost all of the engineering is caching boundaries, timeouts and partial results
Prices live in systems you do not control, they change constantly, and any design that waits for every supplier will be as slow as the worst one. The engine we rebuilt for a UK hotel metasearch operator serves 300 million queries a day at 60 percent lower response time and 80 percent lower infrastructure cost than the one it replaced, and most of that came from removing repeated work and defining staleness per data type, not from faster machines.
If your supplier count and traffic are modest, this is not hard and you do not need us. Ask when the numbers are large enough that the architecture has started to matter.
Metasearch engineering notes
Book a scoping call
Thirty minutes with an architect, not a salesperson. You leave with a written view of scope, price model and whether we are the right team for it.