Meta search engine development: where the difficulty actually is
A meta search engine queries many suppliers at once and returns one ranked answer in real time. It looks like a product problem and is a latency problem. We have run one at 300 million queries a day; this page explains what that took.
The engine behind this page
- 300,000,000
- queries a dayReal-time price and availabilityMetasearch case study
- 60%
- lower response timeAgainst the engine it replacedMetasearch case study
- 80%
- lower infrastructure costHotel metasearch platformMetasearch case study
- For
- Travel and comparison platforms whose search slows at peak or costs too much to run.
- Starting point
- A scoping call and a written ballpark; an existing engine starts with measurement.
- Pricing
- Time and materials against the written ballpark you receive after the scoping call.
- Cost drivers
- The number of suppliers, the traffic you expect and how fresh prices have to be.
The four problems a metasearch engine has to solve
Supplier aggregation
Querying many suppliers in parallel and returning before the slowest one answers, which means deciding what a partial answer looks like.
Caching strategy
Deciding per data type how stale an answer may be. Some prices tolerate minutes, some seconds, some nothing. This decision, not hardware, is what decides perceived speed.
Ranking and filtering
Sorting and filtering results without a second round trip per interaction, which pushes the data into memory close to the request.
Failure behavior
What a customer sees when a supplier times out or answers nonsense, designed rather than discovered.
They're very skilled technically and are also able to see the bigger picture. Stratoflow can actually think about solutions, not just the technical task at hand, which they've been assigned.
Arnd Jan PrauseChief Operating Officer, musQueteer
Most of the engineering is caching, timeouts and partial results
Prices live in systems you do not control and change constantly, so a design that waits for every supplier is only as fast as the slowest one. The work goes into deciding how stale each kind of data may be, how long to wait for each supplier and what to show when one does not answer.
That is how we rebuilt a UK hotel metasearch engine. It now serves 300 million queries a day at 60 percent lower response time and 80 percent lower infrastructure cost, mostly from removing repeated work rather than buying faster machines.
If your supplier count and traffic are modest, a standard platform will do. Talk to us when the numbers are large enough that the architecture decides your speed and your costs.
How metasearch engine development with Stratoflow works
Each phase is agreed with you first, and results are measured against your current engine.
Scope the search
We begin with a scoping call on your suppliers, your traffic and how fresh prices must be, and you receive a written ballpark.
Measure or design
In the second phase, we measure the current engine or design a new one: supplier calls, caching, timeouts and failure behavior.
Build in increments
Then we build or rework the engine in increments, testing each against real supplier responses and load tests at the traffic you expect.
Release and compare
Finally, we release it, compare speed and running cost against the baseline, and hand over the code and the operating guidance.
Our projects
- Flagship · Travel · 2019
UK hotel metasearch: 300 million queries a day at 80 percent lower cost
A SQL Server cluster that stopped scaling replaced by an in-memory data grid: 50 percent more traffic, 60 percent lower response time, 80 percent lower infrastructure cost.
300,000,000 queries / day
- Travel · Performance
Flight schedule calculation 65 percent faster on open-source Java
A DB2 and proprietary Java calculation engine at its scaling limit replaced in a proof of concept, tested against the same production input.
-65% calculation time
- Ecommerce · Banking and fintech
Mennica Skarbowa: real-time pricing engine for an omnichannel precious-metals platform
Intraday base prices from London Metal Exchange and distributor feeds, a rules-driven margin engine, and price distribution to web stores and physical shops.
Intraday repricing, every channel
Metasearch guides from our engineers
- Travel
An In-Depth Guide to Meta Search Engines: Architecture, Ranking, Comparison
How meta search engines work under the hood, how they differ from search engines, and how eight of them compare.
- Travel
Introduction to Meta Search Engines: What They Are, How They Work, Examples
What meta search engines are, how they turn one search into one ranked list, and what it takes to build one.
- Travel
How to Build a Travel Search Engine: Architecture, Caching and Steps
How travel search engines work, why travel search is harder than web search, and how to build one in eight steps.
Discuss your metasearch engine
Bring your supplier count, peak traffic and how fresh prices must be. We will show you where the time and money go.