Perplexity’s AI-native search stack retrieves and ranks candidate pages, surfacing the most relevant content for the model to ground its answers. Originally, this stage ran on an open-source engine that we adapted to our system. However, as our index and workloads grew, we hit hard limits on costs, tail latency, and node recovery time that we couldn’t resolve within the existing third-party platform. (View Highlight)
To overcome these challenges, we built Photon, a new in-house retrieval and ranking engine. This continues our approach of building AI-first search infrastructure from scratch, as we did with CobbleDB, our distributed key-value store for serving page records, and our Search API. (View Highlight)
Photon uses compact data formats to read and decode only what each query needs, with batch-aware caching and asynchronous reads to reduce disk access and overlap waits. Index building runs separately from query serving, updating one serving group at a time and warming its caches before routing live queries to it. (View Highlight)
Photon now powers our search pipeline and is also the foundation of a new fast preset for our Search API, with single-search-call latency of 160 ms at p50 and 230 ms at p95. Across six public benchmarks, the fast preset reduced estimated model-plus-search cost per task by 68% while maintaining aggregate task quality comparable to the default preset. We are releasing it today so developers can access these improvements. (View Highlight)
For each query, the retrieval and ranking engine searches a large, distributed index to select and order a small set of relevant pages within the search stack’s latency budget. The system must also keep the index up to date as new pages arrive and existing pages change, without letting that work slow down live queries. (View Highlight)
We evaluated several open-source options and chose a mature, general-purpose retrieval and ranking engine that met our initial needs. We then adapted it to our architecture and workloads, maintaining stable latency and throughput at production scale. (View Highlight)
We maintained our own fork to adapt the engine to our growing index, but addressing key limitations increasingly required redesigning large parts of the platform. Several constraints proved difficult to resolve without fundamental architectural changes: (View Highlight)
1. Scaling costs. Both the number of documents we store and the amount of per-document data we need to retrieve and rank have been growing over time. This scaling put pressure on our hardware, especially the amount of RAM available. To control costs, we needed more compact data representations and finer control over what stays in memory and what is read from disk on demand. (View Highlight)
2. Poor tail latency. On production traffic, the old system’s p99 response time was about 800 ms. The dataset exceeded available RAM, so we could not use mlock to keep it entirely in memory. Accessing nonresident data could trigger major page faults, stalling query execution while the operating system read from disk. (View Highlight)
The engine also periodically merged its main disk index with smaller indexes flushed from memory. During this disk index fusion process, the number of page faults rose even higher, with p99 latency reaching approximately 1.2 s for 10–15 minutes. As the caches warmed up, the frequency of page faults and latency both decreased, suggesting that cold reads were a major contributor to the spikes. Preloading data and warming the index reduced the spikes but did not adequately address high tail latency. (View Highlight)
3. Lengthy recovery and synchronization. In the previous system, deploying an additional cluster and synchronizing its data could take more than a week. Synchronization was also required when adding groups of nodes to hold additional data copies or recovering from node failures. During recovery, a larger share of search responses was partial, which lowered final-answer quality scores. (View Highlight)
Retrieval and ranking access document data differently. Lexical retrieval starts with query terms and finds documents that contain them. Ranking then needs detailed information about each selected document, including term frequencies and positions. (View Highlight)
Photon uses two complementary representations to support these access patterns. An inverted index stores a list of matching documents for each term, called a posting list, to support candidate selection. Separate per-document records provide the data needed for ranking. We also optimize how posting lists are stored and traversed to improve selection within a fixed work budget. (View Highlight)
Specialized data representations enable us to store key data more efficiently. A more compact inverted index reduces the amount of data read or moved per query, improving latency by eliminating unnecessary I/O operations. These formats also make it possible to check whether a document appears in a list without fully decoding it, a property our traversal logic relies on to control how much work a query does under a fixed budget. Optimizing per-document records lets us store more data per document for ranking, while also reducing the scaling costs associated with our previous system. (View Highlight)
For each term, the inverted index stores a posting list. These lists can range from a handful of documents to millions, so no single format works well for every list. Photon chooses the format based on the number of documents containing the term. (View Highlight)
Short lists use an inline representation that fits within a single page and requires no block table. Longer lists that don’t fit the inline format are divided into blocks covering fixed ranges of document identifiers. Each block’s representation depends on its density: sparse blocks store sorted arrays of document-ID offsets, while dense blocks use bitmaps. Sparse blocks use galloping search, which advances in increasingly large steps to skip regions of the list. Dense blocks allow membership checks by looking up the bit corresponding to a document. (View Highlight)
A term’s query frequency determines where its posting list is placed on disk. Frequently queried lists are stored near one another, concentrating hot data in a smaller storage region. Within each list, documents are ordered by a separate usefulness signal so that more promising candidates are encountered earlier. (View Highlight)
Candidate selection requires traversing posting lists to find potentially relevant documents. Exhaustively scanning every list is expensive, so Photon uses a WAND-like algorithm that traverses only some lists while still accounting for matches in the others. (View Highlight)
The algorithm divides lists into two groups. Driving lists advance the traversal and propose candidate documents. Probe lists contribute to the scores of those candidates but do not generate candidates themselves. (View Highlight)
Candidate scoring combines weighted query-term coverage and a frequency contribution, with document-length normalization and a cap on gains from repeated terms. Photon first checks term presence in the arrays and bitmaps to bound each candidate’s maximum possible score. It reads exact term frequencies only for candidates that could still meet the selection threshold. (View Highlight)
Selection then proceeds in three steps:
• Traversal: Start with a strict query-coverage requirement. If too few candidates are found, repeat with a relaxed requirement.
• Retention: Candidates are held in a bounded set using a score that combines coverage and frequency. As the collected top candidates improve, the admission threshold rises.
• Final selection: Final results for the pass are chosen from the retained set based on the frequency scores. Each field runs its own pass, and the results are merged using field weights. (View Highlight)
The algorithm produces a candidate set of relevant documents to be used in further ranking. (View Highlight)
After candidate pages are selected, the ranking phase needs term frequencies, field coverage, and the locations and proximity of occurrences. Obtaining this data through the inverted index would require scattered accesses across many term-document pairs. In Photon, we store a compact per-document record called docblob that contains term frequencies, which fields a term appeared in, and where in the document it occurred. As a result, ranking a candidate only requires one lookup per document. (View Highlight)
The format is built to avoid decoding more than necessary. Within a docblob, fields are grouped by text analyzer. Each group stores a sorted term list encoded with Elias-Fano, so a query term can be looked up directly without expanding the entire list first. A field mask marks where a term occurred, a frequency count tracks how often, and actual positions live in separate blocks per term-field pair encoded as position differences. Only the blocks for terms that actually matched get decoded, while everything else in the record stays in the compressed format. (View Highlight)
Even with the more efficient data representations, RAM still runs out before other resources, so we frequently need to access data stored on disk. Photon uses asynchronous disk reads and a cache optimized for batch reads to speed up these accesses, cutting the per-request wait time. This contributes to a significant improvement in tail latency compared to the previous system. (View Highlight)
FIG. 07Photon checks a batch of known record offsets in its cache before issuing disk reads. Cache hits are copied into a prepared buffer; misses are submitted through io_uring so disk waits can overlap, and returned data populates the cache. The records and timelines are schematic, not measured latencies. (View Highlight)
Photon takes advantage of the fact that record offsets are known in advance, so we can submit several reads at once. This is managed by the reactor: dedicated worker threads submit batches through io_uring and pass the data they read to the calling code. Each thread maintains its own io_uring; batches are sent to the least-loaded reactor. Preregistered file descriptors and buffer pools reduce the overhead of repeated reads. This asynchronous execution allows the waits for several operations to overlap, reducing tail latency. (View Highlight)
The cache is designed for batched reads. Specifically:
• There’s no separate allocation on a hit. Instead, data is copied into a buffer prepared in advance by the calling code.
• There’s no lock acquisition by the reader. Instead, it checks the data version and retries if it conflicts with a write.
• It uses CLOCK eviction, rather than requiring updates to a shared LRU list on every read.
• It uses CPU cache prefetching, where metadata and data for upcoming records are requested in advance. (View Highlight)
In Photon, index building and query serving run on different nodes. Indexers transform data into ready storage structures, while serving nodes read them and perform retrieval and ranking. As a result, index building doesn’t compete with search queries for CPU, and resources for these tasks can be scaled independently. (View Highlight)
This separation addresses the recovery and synchronization bottlenecks present in the previous implementation. Because index building ran inside the same cluster that served queries, recovering a failed node or adding a new copy of the data could mean reindexing documents in place, a process that took days and increased partial responses during recovery. Photon removes that coupling; indexers run as distributed jobs that produce versioned, ready-to-serve index structures independently of the serving nodes, so building the next version of the index no longer competes with live traffic. (View Highlight)
