Production Ranking Systems: A Review

Murium Iqbal Nishan Subedi Kamelia Aryafar

Overstock.com Overstock.com Overstock.com
ABSTRACT critical to the business of any such organization, and thus even a
The problem of ranking is a multi-billion dollar problem. In this small improvement to these systems can yield significant growth
paper we present an overview of several production quality ranking for the business.
systems. We show that due to conflicting goals of employing the The need to increase user engagement has spurred an iterative ex-
most effective machine learning models and responding to users in periment driven approach to improving ranking systems. The field
real time, ranking systems have evolved into a system of systems, has thus evolved into an intersection of research and application,
where each subsystem can be viewed as a component layer. We with each system being built to simultaneously leverage complex
view these layers as being data processing, representation learning, machine learning (ML) methodologies and adhere to the constraints
candidate selection and online inference. Each layer employs dif- and tools required to support millions of users in real time. These
ferent algorithms and tools, with every end-to-end ranking system methodologies select the most relevant items from catalogs of mil-
spanning multiple architectures. Our goal is to familiarize the gen- lions and present them in decreasing relevance to users. The need
eral audience with a working knowledge of ranking at scale, the to accommodate experimentation with complex nonlinear models
tools and algorithms employed and the challenges introduced by such as those based on deep learning, while still working within
adopting a layered approach. constraints, such as low latency, limited compute power and high
parallelization, has driven ranking systems to evolve from a single
CCS CONCEPTS system to a system of systems. These conflicting concerns are sepa-
rated by isolating functionality within the systems. As such ranking
• Information systems → Learning to rank; • Computer sys-
systems are built with several layers of subsystems, including off
tems organization → Real-time system architecture; • Comput-
line models which allow for flexibility of complex experimentation
ing methodologies → Learning settings.
and online models which cater to live system constraints.
ACM Reference Format: We view production ranking systems as having the following
Murium Iqbal, Nishan Subedi, and Kamelia Aryafar. 2019. Production Rank- component layers:
ing Systems: A Review. In Proceedings of ACM SIGIR Workshop on eCommerce
(SIGIR 2019 eCom). ACM, New York, NY, USA, 10 pages.
data processing responsible for aggregating and featurizing
1 INTRODUCTION raw data from various sources into training data for models
offline representation learning responsible for transform-
The domain of ranking has its roots in the field of Information
ing raw data into embeddings or graph representations
Retrieval (IR). Early automated IR systems, used in the 1950s were
candidate selection responsible for leveraging the learned
first applied to library indexing and employed statistics to retrieve
representations to populate distributed databases with a se-
documents from catalogs of thousands [18, 44]. As the Internet has
lection of relevant candidates given a query
grown, an increasing number of industries rely on web and mobile
ranking model responsible for loading candidates from the
platforms to reach end users. This has resulted in vastly larger cata-
distributed database or inverted index and ranking them in
logs of both public and private data. Ranking systems have emerged
decreasing relevance given some context
over time to extend the original IR systems to balance the goals of
understanding user intent, scoring the relevance of an increasing
number of items, and presenting users with results within frac- Ranking systems are deployed to support recommendations,
tions of a second. Organizations which use the Internet to interface search, and sponsored advertising. In this work we examine pro-
with their users rely on ranking technologies to parse catalogs of duction ranking systems as a general framework irrespective of
millions or billions of items and surface the most relevant ones. their application. As such, in the context of this paper a query is
These items range from music and movies available on streaming generally the prompt to which the ranking system responds. This
content services, to products for sale on e-commerce platforms, to can be a text string used in search, a user or an item in recom-
web pages on the Internet cataloged by search engines, to adver- mendations, or a keyword in sponsored advertising. We use the
tisements for sponsored advertising and more. As such, ranking term item to refer generally to any listing within a catalog, such
systems have become a core technology powering sales and user as products for sale, advertisements, web pages, and more. We ex-
engagement. Users’ interactions with the surfaced information is amine different approaches used in each of the layers of a ranking
system across industries, but a convergence of methodology on any
Permission to make digital or hard copies of part or all of this work for personal or layer is still not apparent. Each individual application of ranking
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation systems technologies requires it’s own problem and data specific
on the first page. Copyrights for third-party components of this work must be honored. approaches to be developed. Instead of tabulating all approaches
For all other uses, contact the owner/author(s).
and caveats, which would be outside the scope of this work, we
SIGIR 2019 eCom, July 2019, Paris, France
© 2019 Copyright held by the owner/author(s). will examine the most popular general methodologies adopted for
each layer of a ranking system.
SIGIR 2019 eCom, July 2019, Paris, France Iqbal, Subedi, Aryafar

The rest of the paper is organized as follows: Section 2 reviews systems. As such, one-box architectures are attractive. These archi-
how the conflicting goals of training models to rank items and serv- tectures pull processed training data from distributed data stores
ing models to support millions of users in real time has given rise and train complex models in memory on a single machine [13, 23].
to a system of systems. Section 3 reviews ways to aggregate and An end to end ranking system employs both of these archi-
normalize raw data into training instances for future layers. Section tectures, with distributed architectures providing a sink for raw
4 reviews various representations which are built to simplify the data, embeddings, candidates and model predictions, and a source
task of retrieval. Section 5 reviews how learned representations of training data and features for one-box models. This system of
can be leveraged by online ranking systems to select initial can- systems requires orchestration across frameworks, which can be
didates. Section 6 reviews the live models used to infer item to handled via schedulers that are responsible for coordinating work-
query relevance and how they are served. Section 7 reviews state flows for training and deployment of online models [3]. Each set of
of the architectures necessary to deploy ranking systems. Section 8 tools employed by the layers of the ranking model must be selected
reviews several ways to validate new components within a ranking with care, as increasing the number of employed tools increases
system, and possible faults that can arise. the complexity of the overall ranking system. This can cause some
layers to be poorly configured, as practitioners are required to mas-
ter many technologies. In some cases, separation of concerns can
lead to isolation of practitioners who specialize on specific layers
2 SEPARATION OF CONCERNS within the ranking system. This can cause further poor configura-
The separation of concerns across the ranking system layers has al- tions, as practitioners may treat other layers of the system as black
lowed organizations to create models which can approximate ideal boxes, leading to layer specific optimizations which may give rise
ranking, learn query intent, perform query re-writing, and diversify to suboptimal behavior across the entire system [42].
retrieved results. All of these applications are addressed by train- We feel the separation of concerns is a useful tool, but only when
ing offline models to capture relevant relationships within their employed with care. Research on productionized ranking systems
representations. The necessary, often massive computations can tends to focus on just a single layer [37, 48, 49, 51]. Practitioners
be performed in batch, allowing for feature spaces which encode must be careful to examine the individual component layers as well
desired relationships within the geometry of the representation as behavior across the entire system. Only the ability to interpret
space. These relationships can then be directly leveraged by candi- the system at both scales can yield an understanding of behavior
date selection or the online model. As the computationally heavy and enable proper iteration to improve results. A diagram of the ar-
processes are captured in offline representation learning and candi- chitectural components necessary to support a full ranking system
date selection, online models can be built as less computationally is provided by in Figure 1
complex models, e.g. linear or shallow models, allowing for low
latency response time. 3 DATA PROCESSING
This separation allows for highly complex systems to be built
and leveraged, but introduces failure points in the form of decreased 3.1 Datasets
interpretability of the ranking system behavior, difficulty in tuning, Increasing user interactions is the primary goal of ranking systems
and increased difficulty of validation. This makes it hard to interpret [10, 30]. As such, user interaction logs captured by the platform
experimental test results and improve upon previous iterations. Ar- often serves to be the richest source of training data for the system.
chitectural decisions within the various layers of a ranking system Some user interactions are ubiquitous, such as user clicks or user
also introduce corresponding assumptions into the overall system. item ratings. Others are platform specific, such as purchases on
Often these assumptions although necessary, can obfuscate biases e-commerce platforms, duration of viewing time on streaming con-
and weaknesses. tent platforms, or likes on social media platforms. Data about the
Ranking systems have developed two architectures which have items themselves, such as title, category, associated text and cost,
facilitated the separation of concerns. These are distributed tech- is referred to as side information. Side information is gathered by
nologies and one box models. As item catalogs have grown, tech- platforms either by user feedback, such as tagging on social media
nologies which require the housing of indices in memory are no platforms, or by content providers, such as attribute labels provided
longer feasible. As such, distributed databases which can house an by vendors on e-commerce platforms, or by the platform itself such
index across a cluster of machines and coordinate retrieval from as item categories.
this index have become prevalent [16, 28, 43]. These distributed Most data used by ranking systems are sparse high dimension-
database technologies are able to house petabytes of data and run ality vocabularies. This is especially true for user interaction data,
computations over their entirety. In parallel to the development where each item in a catalog can be seen as a word in the vocabulary
of distributed computing technologies, ML models have grown with few user interactions [53]. Side information contains a mix of
in complexity, especially with the advent of deep learning. These sparse and dense data, such as sparse multi-hot encodings of text, or
models require specialty hardware to support their high computa- dense data of item price and size or item images. Prior to represen-
tional complexity, such as CUDA enabled graphics processing units tation learning and model training, raw data must be normalized,
(GPUs) [38]. Often the computations necessary for these models are featurized and formatted. To reduce noise and reduce computation
infeasible to translate to distributed system frameworks, and the time, large cardinality spaces can be trimmed via thresholding to
volume of model parameter updates required to coordinate across drop highly sparse dimensions from the vocabulary. Out of vocabu-
the clusters is prohibitive to deploying these models on distributed lary components can either be dropped or mapped to a default null
Production Ranking Systems: A Review SIGIR 2019 eCom, July 2019, Paris, France

Figure 1: General system architecture that covers the life cycle for a ranking request. The query can be items, user provided query strings
or users in generalized ranking approach, and thus query processor can either act as a query re-writer, user personalization, or an item
diversification layer.

representation [49]. Dense features which contain high variability hour be considered related? Adjusting these data aggregation layers
or follow an exponential distribution can be normalized and/or and their underlying assumptions dictates which correlations are
smoothed prior to ingestion by models. and are not captured within the dataset. Class labels on training
instances per user click based relevancy can also change depending
on the assumptions made [49]. These decisions can be seen as a
3.2 Data Aggregation And Normalization
form of data tuning which build the assumptions into the datasets
User interaction data can be aggregated by various methods, each used for both training, tuning and evaluating ranking systems.
building its own assumptions into the ranking system. Excepting
reviews and ratings, user interaction data is often referred to as im-
plicit feedback data. This is due to the fact that although platforms 3.3 Creating A Balanced Dataset
can present items to users and track specific user interactions, the The Learning To Rank (LTR) framework poses ranking as a super-
relationship between those interactions is only implied. Negative vised learning problem [30]. As such, both positive and negative
signals, indicated by lack of user interaction, can also only be im- samples need to be inferred from user interactions. Just as with
plied as it is impossible to know exactly what items were viewed positive samples, there are numerous ways to attribute negative
by the user. Thus, a complete set of ranking labels for all items is samples, for example, if a user exited a video stream before finish-
impossible to capture [24]. To reduce noise within this dataset, out- ing the entire video. There are far more possibilities for negative
liers, such as those associated with bot-like behavior or accidental samples, represented in the lack of user interactions, than positive
clicks, are removed. This is generally done by thresholding and samples. Various methods for balancing the dataset are employed.
weighting user interactions by dwell time [21, 48]. Selection of the One popular method is for each positive sample defined in the
date range over which to populate data can also affect models, as dataset only sampling one corresponding negative sample. Differ-
the volume and sparsity of data changes over the training window. ent mechanisms of negative item sampling are employed to select
Seasonal trends must also be accounted for when aggregating user negative samples such that specific relationships are reflected in
interaction data. These aggregations, although necessary, and often balanced pairs of negative samples and positive samples. This could
specific to dataset and application, propagate assumptions through include choosing negative items from categories that are far away
all layers of the ranking system and should be made with care [42]. from the item, excluding highly co-viewed or co-engaged prod-
Attributing user clicks to searches or recommendation carousels ucts from the negatives list [25]. Selecting negative interactions
affects both model training and evaluation [15, 27]. This is not from only a window around a positive interaction can allow an
limited to sponsored advertising and affects all ranking problems. assumption of user impression.
For example, in gathering training data for search, do products Ranking systems have an intrinsic positional bias associated
associated with the search only include those clicked immediately with them [40]. Users click on higher presented results irrespective
after the search? Or should they include items clicked with subse- of query relevance, leading to false positive labels. Ignoring this
quent searches assuming that these searches are refinements on the bias and training on naively collected data can lead to models that
original search? Should two items clicked by the same user across simply fit the behavior of the existing global ranking function. The
days be considered related or only those clicked within the same FairPairs method modifies search results in a non-invasive manner
SIGIR 2019 eCom, July 2019, Paris, France Iqbal, Subedi, Aryafar

that allows us to collect pairs of results that are unaffected by this vocabulary can be categorical user actions, a discretized continuous
presentation bias by randomizing the order of results between a feature, or English words [22]. User action data can be taken strictly
small window of items during presentation. [31]. as words, or as sequences of actions, in which case embeddings can
be built from skip-grams of user action sequences [22, 49]. Multi-hot
3.4 Discussion user action vectors can also be weighted by dwell time [22]. This al-
Although user interaction data is rich, incorporation of side infor- lows us to learn query and item representations unsupervised from
mation is necessary in any ranking system. This data is employed user engagement data. To handle cold start, out of vocabulary items
to combat popular item bias, and to address the cold-start problem can be taken as linear combinations of embeddings of associated in
[29]. This side information allows correlations learned from user vocabulary items, or corresponding content data [22, 25, 49].
interactions to be extended to items with few impressions based
on relationships which exist in item descriptions, item category
4.2 Multi Modal Representations
information, etc. Although proper processing of user interaction
data is required for training high performance ranking models, we User interaction, text based side information and image based side
find effective ways to leverage side information affords the highest information are each distinct datasets from various sources that
coverage and most diversity. Models which rely too heavily on user follow distinct distributions. They are, however, related in that they
interaction data overfit to head queries and popular items. express information about the same items. To simultaneously lever-
Processing this data, which is often multiple gigabytes, must age all datasets and learn a single representation ranking systems
be done over a distributed computing platform, as it would be in- employ multimodal learning [36]. The simplest approach to this
feasible to fit the dataset in memory on a single machine. These end is concatenation of raw vectors to generate a single input to
processes are often done in batch, with user interaction data being a model which learns a representation [37]. Although simple, this
processed into new training data at regular intervals. As the need method fails to take advantage of the separate structures within
for real time personalization and sponsored advertising increases, each modality. Another approach is to train separate embedding
stream processing methodologies are emerging which allow data layers for each modality and use these as input to another model
to be processed and featurized in near real time. This trend allows which learns a combined embedding. This methodology allows sep-
ranking systems to become increasingly responsive to user interac- arate architectures to be employed for each modality [48], but adds
tions and increase user engagement by modifying representations complexity in that the individual architectures have no obvious
of users and query intent as users progress through a platform. method of validation. Furthermore if one modality of data is not
present for an item, this could cause unexpected results in the final
representation.
4 REPRESENTATION LEARNING
Extensions to this method train separate embeddings for each
Vector representations, learned over processed data, facilitate com- modality but with a cost function used across the separate models
munication across the different layers of ranking systems. Repre- to force them to map to the same space [48]. Items can then be
sentation learning leverages complex state of the art models, often taken as a weighted sum of the embeddings of their associated
relying on deep learning architectures. These models are highly data points. This method allows for items with incomplete side
platform specific, with architectures and solutions which yield information. Mapping modalities to the same embedding space
large lift in one domain not necessarily providing effective repre- can also be performed in a the methodology of search2vec [22],
sentations on another platform. As such, much recent research on in which vectors built from side information are first initialized
production ranking systems revolves around learned representa- to the corresponding user action vectors. Side information is then
tions and much of a practitioners time is spent on this layer. These sampled to form n-grams which are subsequently embedded into
models are tuned to transform raw data into succinct expressive the same space as the user action vectors, and only those within
representations which can be used either for candidate set retrieval a minimum cosine similarity to the user action vector are kept.
or reranking. These representations can be seen as mappings which Embeddings can be trained jointly over modalities, by employing
project input data into low dimensional embedding spaces where siamese networks [26], or by allowing certain subgraphs within a
distance is inversely related to relevance. Representation learning network share weights [49]. In these architectures side information
is performed offline, which allows this layer to leverage complex is used as features to learn supervised embeddings with the user in-
nonlinear ML architectures without the constraints of real-time teractions providing the relevance to encode within the space. This
systems. This allows the computational burden inherent in rep- methodology introduces training and architectural complexities
resentation learning to be placed ahead of the live ranking layer. but provides powerful representations.
These representations are often learned from multiple modalities,
incorporating both user interaction data and side information. We
examine several methods of learning representations from these 4.3 Multi Task Learning
modalities both jointly and separately. Multi-task learning aims to train robust representations by jointly
training representations for multiple applications [37, 49]. Each
4.1 Shallow Embeddings task can be viewed as a regularization to the other tasks. This can
The simplest architectures to form representations employ shallow yield powerful generalizable representations, but requires tasks
architectures such as word2vec over a combination of vectorized to be related. Multi-task learning can yield unstable results when
interaction data and side information. Here a "word" within the poorly configured.
Production Ranking Systems: A Review SIGIR 2019 eCom, July 2019, Paris, France

4.4 Graph Representations 5.1 Approximate Nearest Neighbors

User action data can also be represented as graphs instead of vector Hash based. Hashing based approaches, such as locality sensitive
embeddings. Here nodes are individual items and edges are user in- hashing or FALCONN [41], are simple models which can be scaled
teractions. Graphs are initialized with raw user interaction data and using distributed frameworks as each item can be hashed indepen-
side information. The final structure of the graph is learned by train- dent of others within the catalog. These methods compress high
ing models to prune edges within the graph via logistic regression, dimensional data, via hashes, and assume similar items’ hashes will
gradient boosted decision trees or multi-layer perceptrons [13, 51]. result in collisions [1]. This form of ANN has drawbacks in that
Hierarchical graph structures can also be built, where each level in high dimensions false positives and false negatives can appear,
of the graph represents a type of node. For example, in sponsored as the data is highly sparse and the randomness inherent in the
search setting, the first tier can represent a query signal, the second selected hashing function can incorrectly cluster items due to the
tier can represent keywords, and the third tier can represent ads. curse of dimensionality [47].
These representations can be used directly with graph based candi-
Tree based. Tree based methods are frequently built as in-memory
date selection methods without the use of embeddings as described
models with many open source implementations available [4, 34,
in Section 5. Once in graphical form, random walks can be em-
35]. Trees are built to be balanced by applying splits along differ-
ployed to transform graph representations into sample data points.
ent dimensions of input data. Nearest neighbors search is then
These samples can then be used as input to an embedding model
performed by traversing the tree starting at the query node and
similar to the raw sequences of user clicks. Embeddings built off of
finding nodes within minimum traversals. This method works well
these samples purportedly capture higher-order item similarities
on low dimensional data, but at higher dimensions, performance
than direct sampling of user interaction sequences [13, 48].
degrades as tree based approaches are complex, often relying on
several trees to obtain high performance and traversal of several
4.5 Discussion trees is time consuming [33].
As state of the art work in deep learning continues to produce Graph based. Unlike hashing and tree based approaches, graph
more effective representation learning techniques, we find the best based approaches function over graph representations of raw data,
approach is to employ embeddings which can surface relationships instead of embeddings. Graphs can be initialized with all incidences
within the underlying data. Validation of this layer is difficult, and of user interaction indicating an edge, which populates a highly
not often discussed in the literature. Instead this layer is often dense graph. The graph is then pruned, with models trained to
validated in conjunction with the subsequent layers. This can cause learn relevance [13, 51]. Neighbors are discovered by employing
improvements within this layer to be hidden by poor tuning in navigable small world (NSW) or walk based proximity algorithms
these subsequent layers. As such effective use of representation [32].
learning requires development of clear validation of this layer in
isolation to the ranking system. 5.2 WAND
Weighted And (WAND) method is a candidate selection algorithm
5 CANDIDATE SELECTION adopted by some production ranking systems [7]. This method
matches queries directly to items using raw features. Items are
In ranking systems, candidate selection functions over the learned
scored to relevance with queries by a weighted average of all fea-
representations output from the offline models and populates databases
tures which they have in common with the query. The top scored
which can be read from rapidly by online models. It should be noted
items are taken as candidates. This method requires a weight matrix
that representations used for candidate selection can be distinct
to be learned via constrained feature selection, but provides low
and separate from representations used as features to the online
latency response with minimal model complexity.
model. Representations used by candidate selection support pro-
jecting queries into a shared representation space with items that
encapsulates similarity, query intent, and support personalization.
5.3 Serving Candidates
The goal of candidate selection algorithms is to use the representa- The databases used to serve the results from candidate selection
tions to populate a distributed database with a relatively small set are distributed frameworks to allow for rapid responses. These are
of candidates for each query. [22]. updated in batch, with new candidates being populated on regular
After offline systems build embedding spaces in which spatial schedules. Search and advertising systems leverage distributed in-
relationships encode relevance, candidates are selected by near- verted index technologies such as Solr and Elasticsearch [19, 20].
est neighbor searches. A query is represented in the embedding Other similar architectures leverage distributed key value stores
space, and all items within the catalog closest to it, given some which prioritize high availability, such as Cassandra or Redis [9, 28].
distance metric, are selected as candidates. As directly computing These systems are built to be highly fault tolerant and scalable to
exact k −N N from catalogs of millions is prohibitively computation- support growing numbers of users and items.
ally expensive, various methods for approximate nearest neighbor
(ANN) searches are employed. All of these methods populate an 5.4 Discussion
optimized lookup index. Approximate nearest neighbor methods Concerns with this layer of ranking and form of separation is prop-
can be broadly broken into three categories, hash based, tree based agation of error from new embedding experiments leaking into
and graph based. later layers due to proper lack of tuning of this layer. We find that
SIGIR 2019 eCom, July 2019, Paris, France Iqbal, Subedi, Aryafar

Figure 2: This figure highlights how ranking systems enable real time interaction with users. Users interact with the platform and the system
processes their interactions to update their query results. The first row illustrates a global result. The second row shows results for the same
query when the user signs in, and provides context based on their past history. The third row presents results for the user for the same query
once a product has been added to the cart. This updates the context to the ranking model by providing an additional item, thus narrowing
down to the user’s information need. At each stage the system is dynamically ranking candidates after interpreting new information about
the user to better infer their intent and each items relavance to the intent.
Production Ranking Systems: A Review SIGIR 2019 eCom, July 2019, Paris, France

hashing based methodologies are likely the most relevant to rank- is the most popular approach as it allows for the model to con-
ing, as much work is spent on representation learning. Graph based sider relationships between candidates, rather than scoring them
approaches are weak in that they require massive datasets, and only independently. This approach requires an additional sorting to be
the largest organizations can afford to employ them. Graph based performed based on the scoring which causes further computation
approaches have the further complication of lacking an obvious overhead. Pairwise approaches allow for regression models to be
approach to addressing the cold start problem, as graph based rep- used, such as logistic regression and gradient boosted decision trees.
resentations cannot directly represent items which do not already Although gradient boosted trees provide better results, the logistic
exist within the graph. regression still provides the lowest latency response. Popular loss
functions used to train these models are binary cross entropy and λ
6 RANKING MODELS loss function utilized by LambdaRank [8]. This approach is detailed
Online ranking models return candidates in descending order of in Algorithm 2.
relevance to a query. Shallow models are used for real time services,
making use of offline learned representations as input features [37]. 6.3 Listwise Approach
This allows online models to optimize for latency, limited storage Listwise approaches require models to be trained over an entire list
and limited compute available to real time services. Figure 2 depicts of items simultaneously. Formulation of a loss function for such
how the online ranking inference models allow platforms to better a model is difficult, as the true ranking of an entire list is not pos-
interact with and engage users. There are three distinct ways to sible to populate. One method extends pointwise approaches by
formulate the ranking problem to train online models; these are assuming a multinomial instead of a multivariate bernoulli. This
pointwise, pairwise and listwise approaches [30]. methodology forces candidates to compete with one another for
limited probability mass. Other methods try to directly maximize
6.1 Pointwise Approach NDCG as their objective [46, 50]. This approach is detailed in Algo-
In pointwise approaches a model is trained to individually score the rithm 3.
relevance of each candidate to the query. The problem is formulated
as a binary classification, with a positive class indicating a user 6.4 Discussion
interaction such as a click or purchase, and the negative class indi- The most common approaches for real time models are shallow bi-
cating a lack of interaction. Training instances are taken from logs nary classifications, especially those relying on pairwise approaches,
of user interactions. The model provides a score for each candidate which can rank an entire set of candidates against one another while
reflecting its probability of eliciting an interaction given the query still posing the problem as a binary classification[37]. These ap-
and any context of the user and items captured in the provided proaches are best suited for machine learning, as binary classifiers
features. Candidates can then be ranked in descending order of like- are a well studied set of models. We view both pointwise and pair-
lihood for an interaction. This approach affords the use of shallow wise approaches as being the most optimal solutions. The challenge
models, such as logistic regression. This methodology suffers from with these models is addressed by careful feature engineering in
a lack of context about other candidates, as each candidate is scored data processing and representation learning layers. Listwise ap-
individually. This causes an assumption that the output space of proaches seem ill-posed in comparison, as true ranking data is
the candidates is a multi-variate Bernoulli, where each candidate’s impossible to obtain with implicit feedback and the models suffer
score is independent of each other. This is a poor assumption, as
users view several items at once, and choose from among them.
As such an item’s probability of being clicked is affected by its Algorithm 2 Pairwise Approach (Users, Queries, Clicks)
neighbors. This approach is detailed in Algorithm 1.
1: Nonclicks ← sample_nonclick(c) ∀ c ∈ Clicks
6.2 Pairwise Approach 2: S ← s ∼ Bernoulli ∀ c ∈ Clicks
3: Labels, Interactions ← stack(swap_or_not(s, c, n)) ∀ tuple(s, c,
In pairwise approaches a binary classifier is trained to score a pair n) ∈ (S, Clicks, Nonclicks)
of candidates simultaneously. The positive class indicates that the 4: Features = featurize(u, q, i) ∀ tuple(u,q,i) ∈ (Users, Queries,
first candidate is more likely to be interacted with than the second, Interactions)
and the negative class indicates the opposite. Training data is initial- ®
5: P ← sigmoid(W T (Features) + b)
ized with all positive classes and balanced by randomly swapping
6: C ← Labels × log(P)
the order of items within a pair with a fifty percent chance. This

Algorithm 1 Pointwise Approach (Users, Queries, Clicks) Algorithm 3 Listwise Approach (Users, Queries, Clicks)
1: Nonclicks ← sample_nonclick(c) ∀ c ∈ Clicks 1: Nonclicks ← sample_nonclick(c) ∀ c ∈ Clicks
2: Labels, Interactions ← stack([0, Nonclicks], [1, Clicks]) 2: Labels, Interactions ← stack([0, Nonclicks], [1, Clicks])
3: Features = featurize(u, q, i) ∀ tuple(u,q,i) ∈ (Users, Queries, 3: Features = featurize(u, q, i) ∀ tuple(u,q,i) ∈ (Users, Queries,
Interactions) Interactions)
4: P ← sigmoid(W T (Features) + b)® 4: P ← softmax(sigmoid(W T (Features) + b))®
5: C ← Labels × log(P) 5: C ← Labels × log(P)
SIGIR 2019 eCom, July 2019, Paris, France Iqbal, Subedi, Aryafar

from complexities of trying to learn relevancy of multiple classes development of containerized solutions allows one-box architec-
at once. tures to simultaneously support a number of tools and solutions
by isolating system dependencies [6]. Practitioners are thus free to
7 SERVING COMPLEX MODELS employ complex models without the need to work within system
Different architectures are used to deploy various types of rank- specific paradigms. One-box architectures are limited to in-memory
ing systems. Each architecture has specific purposes and supports computations and cannot fully leverage the entire dataset and thus
various layers of the overall ranking system. are still reliant on distributed datastores to pre-compute data.

7.1 Distributed Architecture

Distributed databases employ a cluster of machines and coordinate
data storage and computations across the cluster. This architecture 7.3 Serving Deep Learning Models
can be leveraged for data processing layers of ranking systems As research on deep learning has progressed, a push to use these
to aggregate, normalize, and featurize training data [14, 39]. Re- complex models in real time ranking applications has been made.
cent work has also employed this architecture for representation This is a divergence from prior architectures which employ shallow
learning via proprietary distributed computing technologies, such models for real time inference. Several methods have been employed
as those used for distributed computation of embeddings [14, 22] to enable deep learning models to be served and respond in real
or via open source libraries such as those available in Spark [52]. time. Each has with its own assumptions and trade-offs.
Several forms of candidate selection, those based on hashing, can Deep learning training can be scaled by distributing training
also be performed on distributed databases. These architectures are over multiple GPUs by leveraging recent open source tools, such as
used to serve learned representations and results from candidate Horizon [14]. These tools provide support for training on CPU, GPU
selection to online models. Distributed databases further provide and multi-GPU in one box architectures and can provide the ability
the ability to simultaneously write billions of records, affording to conduct training on many GPUs distributed over numerous
the ability to capture user interaction data from millions or bil- machines. This framework requires GPU servers, which can be cost
lions of users simultaneously. Distributed databases also afford prohibitive to purchase and maintain.
high scalability, as new machines can be added to a cluster ad hoc Other platforms allow for distributed embeddings by adopting a
to support increases in volume of data. These systems have their parameter server (PS) paradigm which employs a cluster of servers.
own limitations though. One such limitation is described by the Model updates are communicated across the cluster through a
CAP theorem which maintains that a distributed data store can central parameter server. These systems are designed with the
not simultaneously provide consistency, availability and partition following constraints to allow for fast training: Column-wise par-
tolerance [17]. Thus each distributed framework trades off one of titioning of feature vectors among PS shards; No transmission of
these goals for the others, some prioritizing rapid capturing of user word vectors across the network; PS shard-side negative sampling
interaction with others prioritizing high availability to support real and computation of partial vector dot products [11]. Here a shard is
time response. This may require a single ranking system to employ a subset of the entire dataset which is acted upon independently by
several different distributed datastores, one to capture and process one server in the cluster. These systems are complex to maintain
data, another to make data highly available to online models. Dis- and support as updates are made asynchronously to the model over
tributed data stores also require computations to be written within each shard.
specific programming paradigms, such as MapReduce, which do Embedding layers within deep learning models are fully con-
not easily represent deep learning computations or graph based nected, requiring many parameters. These computations are incred-
computations [23]. Furthermore, model training and prediction ibly expensive. To allow for real time embedding of raw signals,
on distributed data stores require transference of model weights model compression via quantization of model parameters is em-
across entire clusters, which is infeasible for complex deep learning ployed [39]. For example, in some layers floating point precision of
models with many parameters. weights is reduced to 8 and 16 bits. This comes with a reduction
in precision, but allows for a smaller memory foot-print. In this
7.2 One-box Architecture approach representation learning must be done such that models
One-box architectures allow for the use of complex modeling tech- and their hardware are co-designed. The complexity of training
niques to be employed in ranking systems. Here a single machine such a model is much higher. Different types of quantization are
loads training data from a distributed database and trains a model performed given the acceptable drop in precision at each layer.
in memory [13, 23]. These architectures are used for representa- Each layer is individually optimized, as well as the entire graph as a
tion learning via deep learning models or graph based approaches. whole. Reduction of model parameters can be performed by model
Candidate selection can also performed on one-box architectures architecture decisions as well. For example, in the case of sequential
via tree or graph based methods, which are described further in models, GRUs are chosen to learn representations instead of LSTMs
Section 5. Online models are often trained via one-box solutions, as they require fewer parameters [49].
and can be served in parallel to scale to support requests from high To allow model training to be completed in a timely manner, mod-
volumes of users, such as millions or billions [5]. This architecture els with many parameters trained over large datasets can be trained
affords ML practitioners the freedom to use different tools, with- incrementally, with a one-time training occurring infrequently and
out the constraints imposed by distributed architectures. Recent weight incremental updates being calculated daily [37].
Production Ranking Systems: A Review SIGIR 2019 eCom, July 2019, Paris, France

7.4 Discussion tests to be run simultaneously [45]. Experiment duration can also
Current design paradigms rely heavily on the coordination of tasks be decreased by employing variance reduction techniques, which
across both distributed and one-box architectures. Data process- separate users within the test group into two strata: those with
ing and candidate selection are performed distributed, while rep- prior purchase behavior and those without. For those with prior
resentation learning and online models are handled via one-box purchase behavior, this past data is used as a control covariate for
architectures. Tasks are coordinated via schedulers [3] and one box additional variance reduction. This has been shown to reduce the
architectures are containerized and deployed in parallel to support duration of experimentation by half while maintaining equivalent
requests from many users [5]. confidence [12].
We find that recent trends aim to allow service of complex mod-
els directly by making complex models more efficient [11, 39, 49, 51]. 8.2 Offline evaluation
This methodology has the potential to reduce the layers and separa- Even with such methods, bandwidth for tests is limited. As such new
tion of concerns within ranking systems. This would be a powerful model experiments must prove a significant improvement on offline
improvement as it would reduce the complexity of the overall model, validation to be selected for a live A/B test. Common offline metrics
allowing for a unified approach. Improvements to streaming data used are area under the curve (AUC) for the receiver operating
processing technologies could further support these developments characteristic (ROC) and normalized discounted cumulative gain
as complex computations can be run over distributed data caches (NDCG) as these correlate well with expected click through rate [22,
in near real time to transform data into input features for deep 37, 49, 51]. Simulated experiments can also be employed to estimate
learning models which are served live. Although promising, this performance of models prior to A/B tests. These simulations must
allows for deep learning models to be deployed in real time, but the be calibrated to avoid incorporating bias leading to poor estimates
work lacks a generalized approach, lacks open source support and of expected click through rate [2].
thus lacks wide spread adoption.
8.3 Discussion
8 VALIDATING A SYSTEM OF SYSTEMS Focus on ranking metrics for the overall system is necessary, but
We have thus far shown that a single ranking system is composed we propose that each layer requires its own independent metrics as
of several layers, each with its own complexity. The need for exper- well to avoid obfuscating errors and biases. Data processing layers
imentation to increase user engagement requires both offline vali- should document assumptions with metrics dashboards, and gauge
dation and online test results. Validating a ranking system, which distributions of data as well as any underlying shifts within these
spans several frameworks is not straightforward. Each individual distributions over time. Validation on learned representations is
layer should be validated in isolation in addition to the whole, but not documented in most production ranking system architectures,
often in ranking systems, the reported results are only those of instead they are only measured in their improvement of applications
the candidate selection or the final ranking model. Such complex for modeling and candidate selection. Requiring each component
systems thus lend themselves to a change one thing change ev- to have independent functionality tests as well as tests of the entire
erything (CACE) data dependency and system entanglement [42]. system can more clearly surface errors [14, 42].
This makes metrics unreliable, obfuscating errors and making it
difficult to prove improvements to the overall system. Despite these 9 CONCLUSION
reservations, both offline and online testing is performed primarily We examine production ranking systems and find that a layered ap-
after candidate selection and after ranking. Depending on applica- proach is adopted in every case. This is necessary to offset the com-
tion the system and its component layers are generally tested for putational cost of leveraging the most effective machine learning
improvement on click-through-rates, purchases, dwell times and models, which are unable to produce real time inference for users.
advertisement engagement to name a few. Selection of the desired This layered approach causes ranking systems to be composed of a
metric to optimize for must be done with user behavior in mind, system of systems, each layer employing different algorithms over
but often suffers from biases and is also affected by functionality different architectures. This approach allows for rapid experimen-
outside of the ranking system itself, such as user interfaces. tation both within and across layers and allows practitioners to
employ state of the art modeling techniques while still adhering to
8.1 Increasing experiment bandwidth real time service constraints such as low latency and limited avail-
Speed of experimentation is hindered by the bandwidth for online able memory. However, this same layered approach causes ranking
testing, as there is a finite amount of traffic that lends itself to each systems to be incredibly complex, with each layer introducing its
particular test. One approach to improve throughput of testing own assumptions and requiring its own tuning. This can obfuscate
involves early detection of poor or invalid experiments. This afford errors and makes it difficult to measure iterative successes. As rank-
greater throughput of experiments as poorly performing tests are ing systems develop new methods to facilitate the direct serving of
detected and terminated early. To allow for this standard metrics more complex systems, reliance on this layered approach could be
are populated frequently and made consistent across all related reduced.
experiments. A multi-layer experiment architecture can also be
employed where experiments are grouped into statistically inde-
REFERENCES
[1] Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. 2019. ANN-
pendent layers. Each user is then simultaneously used as a data benchmarks: A benchmarking tool for approximate nearest neighbor algorithms.
point for multiple tests, one from each layer, allowing multiple Information Systems (2019).
SIGIR 2019 eCom, July 2019, Paris, France Iqbal, Subedi, Aryafar

[2] Gang Bai, Zhihui Xie, and Liang Wang. 2018. Practical Constrained Optimization [31] Corey Lynch, Kamelia Aryafar, and Josh Attenberg. 2016. Images don’t lie:
of Auction Mechanisms in E-Commerce Sponsored Search Advertising. arXiv Transferring deep visual semantic features to large-scale multimodal learning
preprint arXiv:1807.11790 (2018). to rank. In Proceedings of the 22nd ACM SIGKDD International Conference on
[3] Maxime Beauchemin et al. 2016. Airflow. Knowledge Discovery and Data Mining. ACM, 541–548.
[4] Erik Bernhardsson et al. 2018. Annoy (Approximate Nearest Neighbors Oh Yeah). [32] Yury A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate
[5] David Bernstein. 2014. Containers and cloud: From lxc to docker to kubernetes. nearest neighbor search using hierarchical navigable small world graphs. IEEE
IEEE Cloud Computing 1, 3 (2014), 81–84. transactions on pattern analysis and machine intelligence (2018).
[6] Carl Boettiger. 2015. An introduction to Docker for reproducible research. ACM [33] Marius Muja and David G Lowe. 2014. Scalable nearest neighbor algorithms
SIGOPS Operating Systems Review 49, 1 (2015), 71–79. for high dimensional data. IEEE transactions on pattern analysis and machine
[7] Fedor Borisyuk, Krishnaram Kenthapadi, David Stein, and Bo Zhao. 2016. CaS- intelligence 36, 11 (2014), 2227–2240.
MoS: A framework for learning candidate selection models over structured [34] Marius Muja and David G Lowe. 2015. Fast library for approximate nearest
queries and documents. In Proceedings of the 22nd ACM SIGKDD International neighbors.
Conference on Knowledge Discovery and Data Mining. ACM, 441–450. [35] Bilegsaikhan Naidan and Leonid Boytsov. 2015. Non-metric space library manual.
[8] Christopher JC Burges. [n. d.]. From ranknet to lambdarank to lambdamart: An arXiv preprint arXiv:1508.05470 (2015).
overview. ([n. d.]). [36] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and An-
[9] Josiah L Carlson. 2013. Redis in action. Manning Publications Co. drew Y Ng. 2011. Multimodal deep learning. In Proceedings of the 28th international
[10] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks conference on machine learning (ICML-11). 689–696.
for youtube recommendations. In Proceedings of the 10th ACM conference on [37] Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, Anxiang Zeng, and Luo
recommender systems. ACM, 191–198. Si. 2018. Perceive Your Users in Depth: Learning Universal User Representa-
[11] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark tions from Multiple E-commerce Tasks. In Proceedings of the 24th ACM SIGKDD
Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012. Large scale International Conference on Knowledge Discovery & Data Mining. ACM, 596–605.
distributed deep networks. In Advances in neural information processing systems. [38] CUDA Nvidia. 2010. Programming guide.
1223–1231. [39] Jongsoo Park, Maxim Naumov, Protonu Basu, Summer Deng, Aravind Kala-
[12] Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the sensitivity iah, Daya Khudia, James Law, Parth Malani, Andrey Malevich, Satish Nadathur,
of online controlled experiments by utilizing pre-experiment data. In Proceedings et al. 2018. Deep Learning Inference in Facebook Data Centers: Characteriza-
of the sixth ACM international conference on Web search and data mining. ACM, tion, Performance Optimizations and Hardware Implications. arXiv preprint
123–132. arXiv:1811.09886 (2018).
[13] Chantat Eksombatchai, Pranav Jindal, Jerry Zitao Liu, Yuchen Liu, Rahul Sharma, [40] Filip Radlinski and Thorsten Joachims. 2006. Minimally invasive randomization
Charles Sugnet, Mark Ulrich, and Jure Leskovec. 2018. Pixie: A system for for collecting unbiased preferences from clickthrough logs. In Proceedings of the
recommending 3+ billion items to 200+ million users in real-time. In Proceedings National Conference on Artificial Intelligence, Vol. 21. Menlo Park, CA; Cambridge,
of the 2018 World Wide Web Conference on World Wide Web. International World MA; London; AAAI Press; MIT Press; 1999, 1406.
Wide Web Conferences Steering Committee, 1775–1784. [41] Ilya Razenshteyn and Ludwig Schmidt. 2018. FALCONN-FAst Lookups of Cosine
[14] Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary and Other Nearest Neighbors.
Kaden, Vivek Narayanan, and Xiaohui Ye. 2018. Horizon: Facebook’s Open Source [42] David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Diet-
Applied Reinforcement Learning Platform. arXiv preprint arXiv:1811.00260 (2018). mar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan
[15] Sahin Cem Geyik, Abhishek Saxena, and Ali Dasdan. 2014. Multi-touch attribu- Dennison. 2015. Hidden technical debt in machine learning systems. In Advances
tion based budget allocation in online advertising. In Proceedings of the Eighth in neural information processing systems. 2503–2511.
International Workshop on Data Mining for Online Advertising. ACM, 1–9. [43] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler, et al. 2010.
[16] Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 2003. The Google file The hadoop distributed file system.. In MSST, Vol. 10. 1–10.
system. (2003). [44] Amit Singhal et al. 2001. Modern information retrieval: A brief overview. (2001).
[17] Seth Gilbert and Nancy Lynch. 2002. Brewer’s conjecture and the feasibility [45] Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Overlap-
of consistent, available, partition-tolerant web services. Acm Sigact News 33, 2 ping experiment infrastructure: More, better, faster experimentation. In Proceed-
(2002), 51–59. ings of the 16th ACM SIGKDD international conference on Knowledge discovery
[18] Cyril W Gleverdon and Cyril W Cleverdon. 1962. Report on the testing and and data mining. ACM, 17–26.
analysis of an investigation into the comparative efficiency of indexing systems. [46] Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. 2008. Softrank:
(1962). optimizing non-smooth rank metrics. In Proceedings of the 2008 International
[19] Clinton Gormley and Zachary Tong. 2015. Elasticsearch: The definitive guide: A Conference on Web Search and Data Mining. ACM, 77–86.
distributed real-time search and analytics engine. " O’Reilly Media, Inc.". [47] Michel Verleysen and Damien François. 2005. The curse of dimensionality in data
[20] Trey Grainger and Timothy Potter. 2014. Solr in action. Manning Publications mining and time series prediction. In International Work-Conference on Artificial
Co. Neural Networks. Springer, 758–770.
[21] Mihajlo Grbovic. 2017. Search ranking and personalization at Airbnb. In Proceed- [48] Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun
ings of the Eleventh ACM Conference on Recommender Systems. ACM, 339–340. Lee. 2018. Billion-scale commodity embedding for e-commerce recommendation
[22] Mihajlo Grbovic, Nemanja Djuric, Vladan Radosavljevic, Fabrizio Silvestri, Ri- in alibaba. In Proceedings of the 24th ACM SIGKDD International Conference on
cardo Baeza-Yates, Andrew Feng, Erik Ordentlich, Lee Yang, and Gavin Owens. Knowledge Discovery & Data Mining. ACM, 839–848.
2016. Scalable semantic matching of queries to ads in sponsored search advertis- [49] Wenjin Wu, Guojun Liu, Hui Ye, Chenshuang Zhang, Tianshu Wu, Daorui Xiao,
ing. In Proceedings of the 39th International ACM SIGIR conference on Research Wei Lin, Kaipeng Liu, and Xiaoyu Zhu. 2018. EENMF: An End-to-End Neu-
and Development in Information Retrieval. ACM, 375–384. ral Matching Framework for E-Commerce Sponsored Search. arXiv preprint
[23] Pankaj Gupta, Ashish Goel, Jimmy Lin, Aneesh Sharma, Dong Wang, and Reza arXiv:1812.01190 (2018).
Zadeh. 2013. Wtf: The who to follow service at twitter. In Proceedings of the 22nd [50] Jun Xu and Hang Li. 2007. Adarank: a boosting algorithm for information
international conference on World Wide Web. ACM, 505–514. retrieval. In Proceedings of the 30th annual international ACM SIGIR conference on
[24] Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. Research and development in information retrieval. ACM, 391–398.
In Proceedings of the eighth ACM SIGKDD international conference on Knowledge [51] Su Yan, Wei Lin, Tianshu Wu, Daorui Xiao, Xu Zheng, Bo Wu, and Kaipeng Liu.
discovery and data mining. ACM, 133–142. 2018. Beyond keywords and relevance: a personalized ad retrieval framework
[25] Bhargav Kanagal and Sandeep Tata. 2018. Recommendations for all : solving in e-commerce sponsored search. In Proceedings of the 2018 World Wide Web
thousands of recommendation problems a day. In Proceedings of the 34th IEEE Conference. International World Wide Web Conferences Steering Committee,
International Conference on Data Engineering (ICDE). 1919–1928.
[26] Wang-Cheng Kang, Chen Fang, Zhaowen Wang, and Julian McAuley. 2017. [52] Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion
Visually-aware fashion recommendation and design with generative image mod- Stoica. 2010. Spark: Cluster computing with working sets. HotCloud 10, 10-10
els. In 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 207–216. (2010), 95.
[27] PK Kannan, Werner Reinartz, and Peter C Verhoef. 2016. The path to purchase [53] Meizi Zhou, Zhuoye Ding, Jiliang Tang, and Dawei Yin. 2018. Micro behaviors:
and attribution modeling: Introduction to special section. A new perspective in e-commerce recommender systems. In Proceedings of the
[28] Avinash Lakshman and Prashant Malik. 2010. Cassandra: a decentralized struc- eleventh ACM international conference on web search and data mining. ACM,
tured storage system. ACM SIGOPS Operating Systems Review 44, 2 (2010), 35–40. 727–735.
[29] Blerina Lika, Kostas Kolomvatsos, and Stathes Hadjiefthymiades. 2014. Facing
the cold start problem in recommender systems. Expert Systems with Applications
41, 4 (2014), 2065–2073.
[30] Tie-Yan Liu et al. 2009. Learning to rank for information retrieval. Foundations
and Trends® in Information Retrieval 3, 3 (2009), 225–331.