How we built a destination recommendation model that helps users spark inspiration and narrow down choices to make journeys smoother.

By: Weiwei Guo, Bin Xu, Sundara Rajan Srinivasavaradhan, Jie Tang, Xiaowei Liu, Bharathi Thangamani, Liwei He, Huiji Gao, Tracy Yu, Hui Gao, Stephanie Moyerman, Sanjeev Katariya

Airbnb users in the trip planning stage may not have a clear idea of travel destinations, travel dates, or other preferences. They exhibit different behaviors compared to users who have a clear itinerary in mind. More exploratory users visit the Airbnb platform less often and are less likely to book listings in the near future; they’re more likely to search for a broad area such as “France” looking for inspiration. We believe that by helping users in the exploration stage, we can spark inspiration, reduce decision friction, and drive improvements in engagement and conversions.

In this blog post, we describe how we help users in the exploration stage by recommending travel destinations. There are multiple unique challenges in modeling destination intent: for example, how to effectively integrate diverse signals (users’ long term interests vs. short term interests), how to balance dormant user behavior vs. active user behavior, and how to encode rich geolocation knowledge.

To address these challenges, we developed a framework that predicts users’ destination intent based on their actions on the Airbnb platform. While the framework is inspired by language modeling, we introduce several key adaptations in training data creation, model architecture, and loss function to tailor it to the destination recommendation problem in the travel domain. Lastly, we present two applications, autosuggest and abandoned search email notifications, that help users explore destination possibilities and facilitate booking decisions.

Model architecture

Travel destination is one of the primary aspects users explore during trip planning, as it largely determines subsequent decisions such as travel timing, budget, and accommodation preferences. User travel destination preferences are driven by a combination of historical behavior, contextual signals, and temporal factors, etc. For instance, users who previously booked listings in Hawaii may exhibit a preference for beach or tropical destinations, while seasonal context (e.g., summer) may shift their intent toward cooler locations.

In our model, we generalize the destination prediction based on historical user preference data. (Users are able to opt out of this personalization.) As shown in Figure 1, we treat each user action as a token, inspired by language modeling. We use transformers to model sequences of user actions as recorded in various sources: booking history, view history, and search history. Each action is represented by the sum of embeddings of city / region / days to today. We also use contextual information, such as the current time, to capture seasonality. This setup enables the model to summarize user’s short-term interests (views, searches), and long-term interests (bookings), and make a holistic prediction of destination intent.

Figure 1: model architecture.

Balancing active users and dormant users

At Airbnb, we need to make predictions not only for “active users,” but also for “dormant users”. They exhibit different behaviors, for example:

• Active users: User A recently issued a search in the California Bay Area last week. She is currently looking for more affordable listings in the Bay Area.
• Dormant users: User B made several bookings in 2025, and hasn’t returned to Airbnb since then. He is currently exploring ideas for a summer vacation in 2026.

Motivated by these two different types of goals, we design the training data shown in Figure 2. For each booking, we create 14 training examples in total. There are two parts:

1. Seven training examples for active users, from 1, 2, 3…7 days before the booking date. For these 7 examples, we use the up-to-date booking/view/search data. This is to mimic the late booking stage when users have a rough idea where to go.
2. Seven training examples for dormant users, randomly sampled from 8 to 365 days before the booking. For these 7 examples, we only use booking data, to mimic the early planning stage when users don’t have a concrete idea and haven’t come to Airbnb.

Figure 2: T is the date for the latest booking. The arrows at the bottom show the training examples used for the planning stage; the arrows in the upper-right corner illustrate the training examples used for the booking stage.

Improving location understanding

At Airbnb, we have rich geolocation information about cities and their relationships. For example, the California Bay Area contains many closely related cities; a user interested in staying in San Francisco may also consider nearby cities such as San Jose. For the purposes of destination recommendation, the Bay Area can be viewed as a broader “region” that encompasses multiple cities.

To incorporate this information into our framework, we use multi-task learning. Specifically, we add multiple prediction heads at the final layer of the model, each corresponding to a different prediction task. As shown in Figure 1, the model is trained to predict both the region-level and the city-level destination. By jointly learning these tasks and encouraging consistency between region and city predictions, the model learns richer geolocation representations of cities.

Applications

We deployed the resulting model in two features of the Airbnb platform. The first is autosuggest. When users click on the search bar, multiple city recommendations are presented. Online A/B testing shows significant booking gains in regions where English is not the primary language; further analysis indicates that these recommendations benefit not only users who have not yet decided on a destination, but also users who are open to booking more affordable listings in neighboring cities.

The second application is abandoned search email notifications. When a user abandons a search on Airbnb, we send follow-up emails featuring listings from areas predicted by the destination recommendation model. This helps drive bookings by encouraging users to explore alternative listings within the recommended destinations and re-engaging them to complete a booking on Airbnb.

Conclusion

In this post, we described a destination recommendation framework designed to support users in the exploration stage of trip planning, when intent is often ambiguous and preferences are still forming. Our framework includes several key innovations: modeling multiple sequences of user actions to balance short-term and long-term interests, designing training data to accommodate both active and dormant user behaviors, and using multi-task learning to incorporate rich geolocation information. Deployed in autosuggest and abandoned search email notifications, the model helps users discover relevant destination alternatives and drives measurable booking gains. Looking ahead, this framework provides a solid foundation for modeling other preferences, such as travel times and price preferences, enabling broader and deeper personalization across the travel planning journey.

If this type of work interests you, check out some of our open roles.

Acknowledgments

We would like to especially thank the following people for their great collaboration throughout this project: Kidai Kwon, Phanindra Ganti, Kedar Bellare, Malay Haldar, Soumyadip Banerjee, Michael Kinoti, Yi Li, Amisha Patel, Rachel Zhao, Zhentao Sun, Wei Jiang, Jackie Liu, Ying Xiao, Hongzhao Huang, Chen Qian, Haiyang Han, Pengyu Hou, Haichun Chen, Sherry Chen, Pavan Tapadia, Stephen Simburg, Clarence Quah, Chris Tarello, Eric Kostenbauder, Linda Yu, Gary Chang.

Recommending Travel Destinations to Help Users Explore was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we changed our Observability as Code alert review process and cut development cycles from weeks to minutes.

Observability as Code (OaC) — defining alerts, dashboards, and SLOs via code rather than UI — is table stakes for large engineering organizations. With OaC, observability adopts software development’s version control, code review, and testing processes, achieving the same level of discipline as a result. At Airbnb’s scale (thousands of engineers and services), this is the foundation that lets teams ship confidently while maintaining the reliability our guests and hosts depend on.

Yet there’s a critical gap in most OaC workflows. While we bring rigor to alert definitions through code review and version control, the actual behavior of those alerts often can’t be validated until they’re live. Production becomes the proving ground. Problems surface either as noise that erodes trust or silence that hides real incidents.

This tolerance of high alert noise might appear to be a culture problem, but we realized it was actually a gap in the developer workflow. We solved it by building accessible, fast feedback loops to preview, validate, and surface actionable insights on alert behavior before PR submission. With these changes, development cycles collapsed from weeks to minutes, and we successfully migrated 300,000 alerts from a vendor to Prometheus, a feat that wouldn’t have been possible otherwise.

Airbnb’s OaC North Star

Our Observability as Code North Star is for product teams to receive out-of-the-box, best-practice monitoring from platform teams. When a product engineer adopts Kubernetes, a service framework, or a database, they should inherit battle-tested alerts, dashboards, and SLOs. The best monitoring gives product engineers the benefit of all of Airbnb’s infra and platform domain expertise immediately. We call this “zero touch.”

We began our OaC journey more than 10 years ago when we built Interferon, starting with 1,000 alerts. Today, we manage 300,000. By any measure, Interferon was a success, scaling our monitoring practices towards our North Star.

However, this success introduced new operational challenges. With so many alerts in production, validating any OaC changes became costly, and it became harder to iterate. Engineers faced a difficult tradeoff: tuning an alert template might reduce noise, but it also risked losing an important signal. Without a way to preview alert behavior, the safer choice was often to leave things as-is.

The problem: Traditional code review can’t validate alert behavior

The problem wasn’t Interferon itself, but rather a development workflow gap. Our North Star requires platform teams to define and maintain monitoring patterns at scale, but we lacked the necessary tooling to effectively validate those patterns against reality.

Traditional code review can validate syntax and logic, and unit tests can verify outputs. But neither can answer the questions that matter most: “How will these alerts behave in production? What noise might they generate? Will they needlessly wake up on-call engineers at 3 AM?”

That’s why you have to validate alerts against real-world data. If your assumptions about the data are wrong, your alert is wrong.

However, off-the-shelf query visualization tools are insufficient for the job. They don’t account for alert-specific parameters, and most notably, they show downsampled data that masks actual alert behavior — step sizes don’t match evaluation intervals. Also, the further you move from static configs, the harder validation becomes. With templating, reviewers must manually copy-paste queries and fill in variables. For a single alert, this is tedious and error-prone; for changes affecting hundreds of services, it’s impossible.

So in practice, developers often had to resort to a weeks-long process of deploying a new alert side-by-side with the existing one, waiting for real-world data to come in, validating, and then iterating.

The solution: Making alert behavior visible

What if an infrastructure engineer could validate a Kubernetes alert template, one that fans out to thousands of services, in 30 seconds instead of 30 days?

That question among others prompted us to rethink and rebuild our OaC platform from the ground up. Building on top of Prometheus’ open source foundations, we could develop the exact UX our engineers needed, particularly local diffs and pre-deployment validation.

The core of this workflow is local-first development: The same code and inputs run in production must run identically on a developer’s laptop and in CI. In addition, we built Change Reports that show how alerts will be modified and bulk backtesting that simulates alerts against historical data.

We rolled out this platform incrementally, with each milestone providing compounding value.

Phase 1: Text-based diffs everywhere

We first generated markdown alert diffs with field-level granularity and query links — the “terraform plan” of OaC. We met developers where they work, in terminal via CLI and in PRs via CI. This solved the basic visibility problem: engineers could finally review the OaC generated alerts without error-prone copy-pasting.

Phase 2: Dedicated Change Report UI

We then built a Change Report UI showing side-by-side alert diffs exactly as they will appear in production, removing the guesswork and mental mapping between config and UI. However, the user was still responsible for mentally simulating alert behavior, which is challenging even for Prometheus experts to get right.

Phase 3: Historical simulation via bulk backtesting

Finally, we built a backtesting system that runs proposed alerts against historical data, hooking directly into Prometheus’s rule manager. Backtesting allows users to understand which alerts would have fired, when, and why, as if they had existed the entire time. Displaying this simulated state inline in the Change Report UI answered the question that matters most: “How would this alert behave in production?”

We backtest in bulk for the entire diff — hundreds or thousands of related alerts — and surface quality signals to help reviewers focus their attention. We compute a “noisiness” metric and show alert firing timelines in the table view, letting users sort by potential problems and focus their effort.

Putting it all together

On the new platform, when a user makes an OaC change, they will now generate a Change Report via CLI or CI. We post a Change Report on all PRs.

The user reviews their changes via the UI. In this example, a one week backtest was conducted, and the changes are sorted by noisiness, seen in the “Tuning” column, to help direct users’ attention.

The user can dive into individual alerts to learn more. This one looks to be problematic, firing once per day.

The user can set overrides. Given the graph, 1.14 looks like a better trigger threshold.

They can then see the impact of their changes. No more alert firings. This looks good and is ready to ship.

Our learnings

Compatibility over novelty

A key architectural decision was compatibility over novelty. Rule groups are Prometheus’ standardized format. By taking that as input, we hooked directly into Prometheus’s rule evaluation engine rather than reimplementing it. We wrote results as Prometheus time series blocks, exposing the data via the standard query API. This meant building analysis tools once. This standardization made our system portable, allowing us to reach all developers in their existing workflows.

Guardrails aren’t optional

Simulating thousands of alerts over 30 days quickly and without service degradations required careful design. Each backtest runs in its own Kubernetes pod with autoscaling to prevent resource contention. Concurrency limits, error thresholds, and multiple circuit breakers prevent cascading failures. A backtesting system that can destabilize production is worse than no system at all.

Perfect is the enemy of shipped

Our simulator doesn’t account for recording rule dependencies. We could have built a more sophisticated dependency resolver, but users can separate this into two distinct tasks — modify the recording rule first, then backtest alerts that depend on it — assuming they know they should. The Change Report UI helps, because when modified dependencies are detected, it highlights them and prompts resolution. This turns a technical limitation into a guided workflow. We shipped the 80% solution that immediately delivers value, leveraging the UI to close gaps.

Own the full surface area

Monitoring is often an afterthought — engineers have limited time under tight deadlines. Our job is to make that time as effective as possible. Prometheus is powerful but exposes a low-level API that requires expertise most engineers don’t have. To achieve our North Star, we introduced abstractions like anomaly detection, burn rate alerts, and change detection. But abstractions only simplify things when you own all the touchpoints: the input language engineers write, the generation process, the UI that displays results, and the validation tools that provide feedback. Partial ownership creates leaky abstractions. Full ownership lets us ruthlessly optimize for developer experience.

The impact

Successful migration from a vendor to Prometheus

We migrated 300,000 alerts from a vendor to Prometheus. Rewriting every alert would have been impossible with our old workflow, but we achieved it thanks to our Change Report UI, bulk backtesting, and an additional vendor-specific integration. By codifying our domain knowledge in the UX, what originally promised to be a multi-year slog of manual effort became a structured, confident migration.

Collapsed development cycles

The typical developer workaround for making alert changes — deploy side-by-side, wait, then iterate — became obsolete. Engineers now make and validate alert changes all within a single PR. Platform teams confidently deploy template changes affecting thousands of services. What once took a month of iteration now takes an afternoon.

Culture transformation

Even though we realized we had a workflow problem, not a culture problem, solving this problem still ended up transforming our culture. We reduced companywide alert noise by 90%, and engineers stopped tolerating noisy alerts and started competing to improve them. Platform teams resumed iterating on shared patterns. Alert hygiene became a point of pride, not a chore to avoid.

Turnkey alert testing is the biggest positive improvement to alerts management in Airbnb’s history.

I’ve seen people spend hours debating the merits of changing an alert only to do nothing because of fear, uncertainty, and doubt. This new alert testing capability completely evaporates the stop energy and allows us to monitor with confidence.

- Gregory Szorc, Senior Staff Software Engineer

Conclusion

Local-first development, Change Reports, and bulk backtesting give us the necessary tools to incrementally reach our North Star. Platform teams can now confidently iterate on monitoring for their domains. Zero touch is becoming how we operate, one cycle at a time.

Now that we’ve introduced pre-deployment visibility and validation to the alert lifecycle, our next step is to introduce that same rigor to on-call analysis.

If this type of work interests you, check out some of our open roles.

Acknowledgements

Thank you to the Reliability Experience team — Kevin Goodier, Harry Shoff, Rich Unger, and Vlad Vassiliouk — and our partners across the company who helped make this a reality.

It Wasn’t a Culture Problem: Upleveling Alert Development at Airbnb was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

2025 was a big year for research at Airbnb, as we made significant progress toward our mission to use AI, data science, and machine learning to become the best travel and living platform.

Specifically, we doubled down on our presence at long-standing venues like KDD and CIKM — two of the most selective conferences in machine learning. At the same time, we expanded our research footprint by sharing our work in NLP, optimization, and measurement science at conferences such as COLING, LION, and VLDB.

Across these conferences, Airbnb researchers engaged directly with academic and industry peers by publishing and presenting papers, learning about the latest innovations, launching new collaborations, and mentoring emerging researchers. In this blog post, we’ll recap the conferences and key papers we presented in 2025, organized by research themes.

Applied machine learning for search, ranking, and personalization

KDD (Knowledge and Data Mining)

KDD is a flagship conference in data science research. Hosted annually by a special interest group of the Association for Computing Machinery (ACM), it’s where researchers learn about some of the most groundbreaking developments in data mining, knowledge discovery, and large-scale data analytics, which are critical to Airbnb’s efforts to improve core products like search and recommendations.

Our participation

We’ve been presenting at KDD since 2018, and 2025 was another strong year for us. We received multiple contributions across the applied data science track and workshops, which were well-received by the broader community and even inspired us to consider open-sourcing some of our technology. We were also inspired by the related research in this area and are eager to explore these methods through new collaborations.

Research highlights

1. Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking: While A/B tests are crucial for developing ranking algorithms and recommender systems, they’re difficult to set up and can take extensive time to reach statistical significance (especially for products with long conversion cycles, like accommodation booking). In this paper, we shared techniques for rapid pre-A/B online assessments that help teams identify the most promising experiments, streamlining the overall process without sacrificing accuracy.
2. High Precision Audience Expansion via Extreme Classification in a Two-Sided Marketplace: Airbnb search balances diverse global inventory with varied guest preferences for location, amenities, style, and price. This process requires efficient location retrieval to find the listings guests might realistically book by determining which geographic areas to query. We introduce a new approach to location retrieval by using a set of relevant, high-precision categorical location cells.

Link to all papers

• Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking (Qing Zhang, Alex Deng, Michelle Du, Huiji Gao, Liwei He, Sanjeev Katariya)
• High Precision Audience Expansion via Extreme Classification in a Two-Sided Marketplace (Dillon Davis, Huiji Gao, Thomas Legrand, Juan Manuel Caicedo Carvajal, Malay Haldar, Kedar Bellare, Moutupsi Paul, Soumyadip Banerjee, Liwei He, Stephanie Moyerman, and Sanjeev Katariya)
• TSMO: Two-sided Marketplace Optimization

CIKM (Conference on Information and Knowledge Management)

CIKM is a premier forum for discussing and presenting research at the intersection of information and knowledge management, including topics like AI, data mining, database systems, and information retrieval. Many of these topics directly intersect with our core product challenges, such as search, ranking, and recommendations.

Our participation

At CIKM 2025, Airbnb’s Relevance and Personalization team had five peer-reviewed papers accepted for publication, building on our participation in 2023 and 2024. These papers focused on advanced AI/ML techniques for search and recommendations, and sharing real-world insights from using these technologies at Airbnb’s scale. Industry and academic researchers, especially those working on two-sided marketplaces, engaged with our work and provided valuable feedback.

Research highlights

1. Augmenting Guest Search Results with Recommendations at Airbnb: When guests use overly narrow criteria to search for accommodations, they often receive insufficient results, leading to a frustrating experience. This paper introduces a recommendation system that dynamically suggests alternatives — different dates, relaxed amenities, or adjusted price ranges — to help guests find suitable accommodations and improve the platform’s booking rate. Authors: Haowei Zhang, Philbert Lin, Dishant Ailawadi, Soumyadip Banerjee, Shashank Dabriwal, Hao Li, Kedar Bellare, Liwei He, Sanjeev Katariya
2. Maps Ranking Optimization in Airbnb: Maps play a crucial role in Airbnb search and bookings, accounting for roughly 80% of search interactions. Yet map ranking has traditionally reused feed-ranking assumptions, which break down when we examine the NDCG (Normalized Discounted Cumulative Gain) metric. This paper explains why list-based NDCG fails to model user attention on maps, introduces a map-specific NDCG, and reports experiments showing that optimizing it yields booking gains. Authors: Hongwei Zhang, Malay Haldar, Kedar Bellare, Sherry Chen, Soumyadip Banerjee, Xiaotang Wang, Mustafa Abdool, Huiji Gao, Pavan Tapadia, Liwei He, Sanjeev Katariya, Stephanie Moyerman
3. BListing: Modality Alignment for Listings: To improve search ranking, we introduce BiListing (Bimodal Listing) embeddings to use unstructured text and photo listing data as ranking signals. BiListing leverages large-language models and pretrained language-image models to create unified representations of diverse unstructured data into a single embedding vector per list and modality. Our experiment results show a 0.425% increase in NDCB (Normalized Discounted Cumulative Booking) gain and drove tens of millions in incremental revenue. Authors: Guillaume Guy, Mihajlo Grbovic, Chun How Tan, Han Zhao
4. Beyond Pairwise Learning-To-Rank At Airbnb: In this paper, we introduce a method to improve the accuracy of pairwise learning-to-rank algorithms, the bedrock of modern search stacks. This approach captures interactions between items during pairwise comparisons, thereby giving us a better sense of what searchers truly want. We also share ways to implement this algorithm performantly, and results from online and offline experiments. Authors: Malay Haldar, Daochen Zha, Huiji Gao, Liwei He, Sanjeev Katariya
5. Learning to Comparison-Shop: Traditional ranking models often evaluate items in isolation, disregarding the context in which users compare multiple items on a search results page. In this paper, we propose a novel ranking architecture, the Learning-to-Comparison-Shop (LTCS) System, that explicitly models and learns users’ comparison-shopping behaviors. Our experiments show statistically significant improvements of 1.7% in Normalized Discounted Cumulative Gain (NDCG) and 0.6% in booking conversion rate. Authors: Jie Tang, Daochen Zha, Xin Liu, Huiji Gao, Liwei He, Stephanie Moyerman, Sanjeev Katariya

NLP & building LLM systems in production

EMNLP (Empirical Methods in Natural Language Processing)

EMNLP is a top-tier NLP conference that brings together practitioners and researchers to discuss new architectures and training strategies for language models, safety and evaluation strategies for LLMs, and real-world NLP applications. These research areas directly intersect with many of Airbnb’s product surfaces, such as customer support, search & discovery, and trust & safety. Additionally, each EMNLP cycle includes the release of new datasets, evaluation suites, and open-source libraries to help teams benchmark their progress against community standards.

Our participation

In 2025, we sponsored EMNLP and presented two papers on humans-in-the-loop in AI systems and advanced summarization techniques. We also used EMNLP’s community datasets to benchmark our system, which showcased where we excel and where we can build upon our success with additional best practices. The conference deepened academic collaborations through discussions on LLM evaluation, safety, and agentic AI design, including mentoring students and early-career researchers.

Research highlights

1. Agent-in-the-Loop, A Data Flywheel for Continuous Improvement in LLM-based Customer Support: To improve our LLM-based customer support system, this paper introduces an Agent-in-the-Loop (AITL) framework that leverages new interaction data to continuously enhance model performance. This flywheel can help the system stay up to date with new product features, shifting user preferences, and updated support policies and procedures. We launched a pilot in the US, and the results demonstrate significant improvement in accuracy and helpfulness. Authors: Cen Mia Zhao, Tiantian Zhang, Hanchen Su, Yufeng Wayne Zhang, Shaowei Su, Mingzhi Xu, Yu Elaine Liu, Wei Han, Jeremy Werner, Claire Na Cheng, Yashar Mehdad
2. Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback: Customer service agents multitask during support interactions, identifying core issues, tracking prior actions, and producing accurate notes. To streamline this workflow, we introduced an incremental summarization system that intelligently determines when to generate concise bullet notes during conversations, reducing agents’ context-switching effort without sacrificing quality. To improve the system over time, we also introduced a learning framework that enables agents to make real-time edits, immediately refining online note generation. Authors: Yisha Wu, Cen Mia Zhao, Yuanpei Cao, Xiaoqing Su, Yashar Mehdad, Mindy Ji, Claire Na Cheng

COLING (International Conference on Computational Linguistics)

COLING is a top-tier NLP conference that covers both foundational research and industry applications of language models, including reasoning, evaluation, multilingual NLP, and real-world LLM systems. The work presented at this conference helps validate Airbnb’s technical direction and directly informs future investments.

Our participation

In 2025, Airbnb presented at COLING for the first time, sharing a paper titled “LLM-Friendly Knowledge Representation for Customer Support” by Hanchen Su, Wei Luo, Wei Han, Yu Elaine Liu, Yufeng Wayne Zhang, Cen Mia Zhao, Ying Joy Zhang, and Yashar Mehdad. The paper presents a new format, Intent, Context, and Action (ICA), for structuring business knowledge in LLM-based QA and customer support workflows. Initial experiments in production show promising results. We also discovered relevant research in knowledge retrieval, LLM evaluation, and hallucination detection that will inspire future projects.

Optimization, causal inference, and measurement science

MIT CODE (Conference on Digital Experimentation)

MIT CODE is one of the premier venues for researchers and practitioners to discuss topics in online digital experimentation, causal inference, and data-driven product innovation. The conference supports our commitment to data-driven decision-making and using experimentation to understand the long-term impacts on guests, hosts, and marketplace health.

Our participation

In 2025, we had another strong showing at CODE, with a cohort of 6 data scientists and 3 academic collaborators. We gave talks in two sessions and presented a poster, which led to meaningful discussions with peer companies and interest in collaborating with academic research groups.

Research highlights

1. Beyond the Experiment Window: Prospective Impacts Under Long-Term Ranking Dynamics: Product teams frequently leverage A/B tests to assess different rankers. While these experiments are typically conducted over shorter periods, we also recognize the value of understanding longer-term dynamics (such as seasonality and user evolution) to further support sustained business objectives, like marketplace health. To solve this problem, we developed a causal framework that allows us to estimate the long-term impacts of ranking changes with strategic goals (like marketplace health) using A/B testing data.
2. Trustworthy Bayesian Inference in Batch-Adaptive Experimentation: Adaptive experimentation, like multi-arm bandit methods, can improve experiment efficiency by reallocating traffic toward promising treatments. Continued advancements in these approaches are expanding our ability to maintain high standards of statistical validity. This paper introduces a practical Bayesian framework for inference in batch-adaptive experiments, specifically tailored to the operational realities of online platforms.

Link to all papers

• Beyond the Experiment Window: Prospective Impacts Under Long-Term Ranking Dynamics (Lo-Hua Yuan)
• Trustworthy Bayesian Inference in Batch-Adaptive Experimentation (Yicheng Li)
• Experimental Design for Product Launches with Collaborative User Networks (Monu Kala)

INFORMS (Institute for Operations Research and the Management Sciences)

INFORMS brings together academics and industry professionals to discuss and share research across data science, machine learning, economics, behavioral science, and analytics.

Our participation

In 2025, our data science team was invited to INFORMS to present two talks in a session about bridging the gap between statistical methods and industry applications.

Research highlights

1. Beyond Multi-Arm Bandits: Tackling Challenges in Adaptive Experiments at Airbnb. In this talk, we walked through the metrics and infrastructure challenges when using classic bandit algorithms, which make it difficult to operationalize adaptive experiments. We propose a hybrid approach that incorporates bandit algorithms into A/B experiments to enable adaptive testing. We also discussed how we onboard and validate adaptive experiments across individual product domains at Airbnb.

Link to all papers

• Beyond the Experiment Window: Prospective Impacts Under Long-Term Ranking Dynamics (Lo-Hua Yuan)
• Beyond Multi-Arm Bandits: Tackling Challenges in Adaptive Experiments at Airbnb (Yicheng Li)

LION (Learning and Intelligent Optimization)

The LION conference is a premier gathering of researchers exploring the intersection of machine learning, artificial intelligence, and mathematical optimization.

Our participation

While Airbnb has attended LION in the past, 2025 was the first time we presented at the conference. Nathan Brixius presented “Optimal Matched Block Design For Multi-Arm

Experiments,” which introduces a new optimization formula using mixed-integer programming (MIP) to group subjects in multi-armed experiments, leading to more balanced groups and, in turn, more accurate experimental results. We also connected with leading experts in metaheuristics and AI fairness to help shape our future roadmap and sponsored the awards for the best papers presented at the conference.

Data systems

VLDB (Very Large Data Bases)

The VLDB Conference is one of the top 2 flagship conferences in data management and large-scale data systems, with over 1,500 researchers and practitioners attending.

Our participation

“In 2025, we published our first paper at VLDB: ‘SQL:Trek Automated Index Design at Airbnb’ by Sam Lightstone and Ping Wang. The paper presents a novel approach for automated index design (code-named SQL:Trek). It uses query compiler cost models to identify effective indexes across many relational databases, including most MySQL and PostgreSQL derivatives. Additionally, the Airbnb team attended sessions on system efficiency, graph computing, and AI databases, and had the opportunity to meet other researchers.

Conclusion

Conferences remain a big part of our research program at Airbnb, helping us validate and refine our ideas through community feedback and providing a forum to share real-world insights that advance the field. In 2025, we doubled down on this vision by publishing papers for the first time at conferences in domains such as NLP, optimization, causal inference, and data systems, reflecting our ongoing commitment to using these technologies to create the best possible travel experiences.

As we look to 2026, we’re eager to expand our presence at these conferences and discover new ways to use AI, machine learning, and data science to build a best-in-class travel and living platform. If you’re interested in doing this type of work with us, consider joining us. Apply for one of our open positions.

Academic Publications & Airbnb Tech: 2025 Year in Review was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Cosmo Qiu, Bo Teng, Siyuan Zhou, Ankur Soni, Willis Harvey

Dynamic configuration is a core infrastructure capability in modern systems. It allows developers to change runtime behavior without restarting or redeploying services, even as the number of services and requests grows. In practice, that might mean rolling out a new address form for a region launch, tightening an authorization rule, or adjusting timeouts when a dependency is slow.

Like any powerful tool, dynamic configuration is a double-edged sword. While it enables fast iteration and rapid incident response, a bad change can cause regressions or even outages. This is a common challenge across the industry: balancing developer flexibility with system reliability.

In this post, we will outline the expectations of a modern dynamic configuration platform, then walk through the high-level architecture of Airbnb’s dynamic config platform and how its core components work together to enable safe, flexible config changes.

Modern config platform essentials

As Airbnb’s business grows, our expectations for the dynamic config platform have evolved over time through our own learnings as well as industry best practices. These shape our view of what a good dynamic config platform should provide, including:

• A coherent experience for config management: The platform provides a streamlined, end-to-end experience for defining, reviewing, testing, and rolling out config changes. It covers the most common needs out of the box with rich built-in features, while still offering escape hatches for edge cases.
• Strong reliability, availability and safety guarantees: All config changes are validated, reviewed, and rolled out progressively, with clear ownership and well-defined access control. Treating config as code is a key focus: config changes are versioned, reviewed, and auditable like service code, but remain dynamic at runtime. The platform itself must be highly available so that services can reliably fetch and apply configs. Changes should be observable, with support for fast rollbacks when needed.
• Safe testing in isolated environments: Developers can validate config changes in isolated local or canary environments before they reach production.
• Flexible multi-tenant support: In a multi-tenant platform, different tenants have different risk profiles. The platform should allow config owners to customize how their configs behave per tenant, including deployment triggers, guardrails, and rollout strategy (for example, AWS zone or Kubernetes pod percentage-based rollouts).
• Fast and controlled incident response: During an incident, responders can ship emergency configs as needed with clear auditability. The platform also provides observability for config changes, so incident responders can tell what changed, who was affected, when the change was made, and who made the change. This enables them to effectively identify the culprit and take action.

High-level architecture

At Airbnb, Sitar is the internal name for our dynamic config platform. It provides a common way for teams to manage runtime behavior safely. At a high level, Sitar has four main parts: a developer-facing layer, a control plane, a data plane, and the clients and agents that run alongside application code.

The developer-facing layer is where config changes are created and reviewed. By default, configs are managed through a Git-based workflow, while a few exceptions are managed in the web interface (sitar-portal), which is also used for admin operations such as emergency deployments.

The control plane is responsible for orchestrating config changes. It enforces schema validation, ownership, and access control, and decides how each change should be rolled out: for example, which environments or AWS zone to target, what percentage of Kubernetes pods to start with, and how to progress the rollout over time. The control plane also specifies how to roll back the changes when needed, and supports routing in-flight configs to specific environments or slices of subscribers for fast testing.

The data plane provides scalable storage and efficient distribution of configs. It acts as the source of truth for config values and versions, and propagates updates to services reliably, consistently, and quickly.

On the product services side, an agent sidecar running alongside each service fetches the subscribed configs from the data plane and maintains a local cache. Client libraries inside the service then read from this cache and expose configs to application logic with fast, in-process access and optional fallbacks.

Putting these together, a typical change starts from a Git flow, proceeds through control-plane validation and rollout decisions, into the data plane for distribution, and finally to agents and client libraries that apply the config updates to application logic.

Key design choices

In this section, we highlight a few key design choices that shape how the platform looks and is operated.

Configs as code with a Git-based workflow

Config changes are by default managed by a Git-centric workflow. We use GitHub as the primary interface for managing configs, because we have an established and responsive internal team to manage GitHub Enterprise. GitHub integrates naturally with our existing CI/CD tooling, so we can reuse rich validation and deployment pipelines without re-inventing the wheel. This approach gives developers a consistent experience to make code changes: open a pull request, get reviews, merge, and deploy. GitHub also brings additional benefits such as mandatory reviewers, review and approval flows, and a change history. Configs under the same theme are grouped into tenants, with clear owners, customizable tests, and a dedicated CD pipeline.

While the Git-based flow is the default, we keep a UI portal for teams that prefer a portal-based experience and as a shortcut for specific operational needs, such as fast emergency config updates that can bypass the normal CI/CD pipeline.

Staged rollouts and fast rollbacks

When a change is proposed, schema validation (checking that the config matches the expected structure and types) and other automated checks run in CI. The change is always reviewed and approved before rollout.

Once merged in the main branch, the control plane performs a staged rollout where the change is first deployed to a limited scope, then gradually expanded to a larger scope if things look good. At each stage of this rollout, the change is evaluated, the author and the stakeholders are notified if regressions are detected, and a fast rollback can be triggered if needed. Staged rollouts can greatly reduce the blast radius of bad changes and improve the overall reliability of the platform.

Separated control and data planes

We separate the “decide” and “deliver” responsibilities. The control plane focuses on validation, authorization, and rollout decisions, while the data plane focuses on storing configs and distributing them reliably at scale. This separation allows us to evolve rollout strategies and policies without disrupting the underlying storage and delivery mechanisms, and vice versa.

Local caching and resilient clients

On the product services side, we introduced a local caching layer between the agent sidecar and the client library to improve resilience and availability. The agent sidecar runs alongside the main service container, regardless of which language the service is written in, and periodically fetches subscribed configs from the backend and persists them locally. The client libraries then read from this local cache. Even if the backend is temporarily unavailable or degraded, services can continue operating on the last known good configs from the local cache.

Impact on product teams

It is essential for the Sitar system to make life easier for product teams. In practice, its architecture changes how teams ship and operate in a few ways:

• Rollouts become safer and more predictable. New behaviors, such as refined authorization rules, can be introduced gradually, verified on a small slice of traffic or in a specific environment, and rolled back quickly if needed. Teams spend less time worrying about “big bang” releases and more time iterating on behavior.
• Teams get more flexibility in how their configs are managed and rolled out. Each team can tailor a config flow to its own risk profile and release schedules. For example, teams can choose between automatic, manual, or cron rollouts, select the rollout strategy, and add extra checks. This lets teams keep their existing ways of working while still benefiting from a common platform and shared guardrails.
• Incident mitigation becomes faster and more controlled. When something goes wrong in production, incident responders can use observability tools that integrate config events to quickly locate the culprit change, then take quick action using the portal’s emergency flow. These emergency updates are fully auditable for future review.

Besides these examples, the platform includes other improvements in usability, safety, and observability that we will not cover in detail here. Together, they contribute to a smoother day-to-day experience for teams that rely on dynamic configuration.

Conclusions and next steps

Dynamic configuration is a foundational capability of modern infrastructure. It enables fast iteration and rapid incident response, but only when it is equipped with strong safety features and provides a good developer experience. In this post, we shared how we think about a modern dynamic config platform at Airbnb, and how we developed Sitar’s architecture to meet those expectations.

The work is ongoing. As Airbnb’s business grows, we are continuing to refine rollout strategies, improve config testing, invest in observability and smart incident response tooling, and evolve other platform components.

In future posts, we plan to dive deeper into specific areas of the platform, such as how we optimize the Kubernetes sidecar that delivers config updates and how we design the developer experience around config management.

If this type of work interests you check out our open roles.

Acknowledgments

Our progress with Sitar would not have been possible without the support and contributions of many people. We’d like to thank Craig Sosin, Nikolaj Nielsen, Daniel Fagnan, Alex Edwards, Xian Gao, Nick Morgan, Carolina Calderon, Hanfei Lin, Joyce Li, Yunong Liu, Alex Berghage, Brian Wolfe, Yann Ramin, Denis Sheahan, Richa Khandelwal, Swetha Vaidy, Abhishek Parmar, Adam Kocoloski, Adam Miskiewicz, and all the other engineers and teams at Airbnb who joined design reviews and offered valuable feedback, as this work would not have been possible without them.

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Safeguarding Dynamic Configuration Changes at Scale was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anna Sulkina has always been a traveler, and we’re lucky her travels have brought her to Airbnb. Anna is a Senior Director of Engineering, and she’s responsible for Application & Cloud infrastructure. She brings over two decades of industry experience to Airbnb, including work spanning the stack from the frontend to the backend to the plumbing that makes everything come together. Anna is a mother, a passionate trail runner, and an accomplished leader. Here’s Anna’s story in her own words.

Discovering a passion after the Soviet Union

I grew up in Eastern Ukraine, and the year I was graduating from high school, the Soviet Union collapsed. Despite the political turmoil, it was an interesting time to get into technology, and I have my brother to thank for that.

I was always a nerdy kid, at school and at home, and my older brother really stoked that curiosity. He was studying computer hardware in Moscow, and he’d bring home computer parts to play with. I still remember the first computer he’d assembled, which required a cassette player to load programs. Only after many minutes of buzzing and clicking would the computer finally whirr to life.

Thinking back, that was really my first inspiration to work in technology. Seeing the inner workings of this new thing, a computer, and watching how the parts came together to form a whole — that’s what made me realize I wanted to work with computers, too.

Of course, I didn’t know the Soviet Union would end, which made studying in Moscow impossible. But technology was still my future.

English: Harder than any programming language

I got my start learning programming in a local Ukrainian university, and after four years of studying, I immigrated to America.

When I arrived, I knew how to program, and I knew how to write and read English, but I couldn’t communicate well. I took ESL classes at a community college and, in parallel, enrolled in Berkeley Extension classes to advance my C++ knowledge and learn Java, which was still very new at the time.

Throughout my first couple of jobs, I was more likely to run into challenges with the English language rather than with programming languages.

My first job was in computer hardware diagnostics at a tiny company with only five engineers, where we communicated directly with hardware manufacturers. This was right before the dot-com bubble burst.

I almost didn’t get the job, though. The interview process for this job included a written portion that tested my knowledge of key computer science terms before getting to solve the coding problems. Given my prior education, I knew all the terms, but I ran out of time because the language gap slowed me down. Luckily, my interviewer happened to be taking the same Java class at Berkeley, and when I explained what happened, he gave me the chance to come back. I finished the test, got the job, and the rest is history.

In subsequent jobs, I transitioned fully from C++ to Java, which became my primary programming language for many years. I eventually got the hang of speaking English more confidently, but for a while, it still felt like Russian was my first language, Java was my second, and English was only my third.

Going deeper in the stack and taking on leadership roles

At various times, my career often felt all over the place. But looking back, I see a trajectory I wasn’t aware of at the time. I started with a brief stint in hardware diagnostics, but after that, I worked in the frontend and, over time, descended the software stack from frontend to backend to the deeper infrastructure I work with today.

Parallel to this trajectory down the stack was an upward trajectory in responsibility. Leadership wasn’t an obvious path for me at first — I had to be pitched multiple times — but the more I tried it, the more interesting and enjoyable it felt.

When I worked at Caymas Systems, a telecom startup, my manager was quick to recognize my leadership potential. He was really encouraging, but even more encouraging was witnessing the difference between teams with good leaders and those without.

After Caymas Systems, I worked at Comcast, where I eventually switched from an IC to an engineering manager. Once I experienced the joys of coaching people, building cool software together, and developing high-performing teams, I knew this was the path I wanted to take.

Fail whales and distributed systems

This path took me through a formative time in my career: the almost nine years I spent working at Twitter. I began as a first-line manager and, over time, worked through some of Twitter’s biggest events, including the “fail whale” era and the Ellen DeGeneres “selfie that broke Twitter” moment.

This was an exciting time. I was working at the heart of Twitter’s tech stack, supporting teams that powered its consumer and revenue verticals. This is where I grew into a senior manager and, eventually, a director. Looking back over nearly a decade of work, two major lessons stand out: one technical and one cultural.

The technical lesson was about failure — namely, its inevitability.

Over my tenure, the Twitter stack transitioned from a monolith to a microservices architecture. This resulted in a set of robust, high-scale, low-latency distributed systems, and it was here that I learned that, when building resilient distributed systems, you need to design for failure, not hope to avoid it.

I often think back to How Complex Systems Fail — the more complex a system, the more likely it is to fail. I remembered that lesson every time we were called to the Twitter command center to deal with an incident; it was all hands on deck until everything was back online.

The cultural lesson was about adoption and what it takes to fuel great ideas.

Today, almost two-thirds of enterprises use GraphQL in production, but in its early days, it was a new, largely untested idea. During a hack week, a couple of engineers laid the groundwork for using this technology at Twitter. I worked closely with them and bootstrapped the team that eventually built Twitter’s GraphQL API, replacing the legacy REST services.

I still think about this experience today. It required convincing leadership and building consensus across numerous teams and stakeholders, but once we did, the payoff was significant: this one technical choice accelerated the velocity of product feature teams across the company.

Why I picked Airbnb

When Airbnb reached out in 2022, I realized my time at Twitter was coming to a close. By that point, my organization was well-run and high-performing — a success, but also a sign that I was ready for my next adventure.

Airbnb immediately stood out because the company offered, for the first time in my career, a true alignment between my personal and technical interests. I love traveling, and I have been a long-time Airbnb guest since 2013. I had always wanted to work for a company that built a product I truly cared about, and this was my chance.

I only got more excited when I learned about the people and teams I’d be working with. The Developer Platform organization, which was responsible for supporting all of Airbnb’s engineers, faced challenges I’d seen before. There was a lot of good work happening in silos, and folks were longing for a clear strategy and direction. Also, I saw an opportunity to not only improve developer experience but also build trust with the rest of the engineers and stakeholders.

So, I started at the beginning. We focused on setting up the organization, coaching leadership, and building internal alignment within the team, as well as external alignment across all the teams we supported. Fundamental questions like “Why are we here together?” and “Where are we going?” all had to be answered.

After a year or two of this work, we had a high-performing team with a clear strategy and strong execution, consistently delivering business value and improving the developer experience and productivity at Airbnb. Even more importantly, we earned the rest of the engineers’ trust, and we enabled our technical teams to perform better.

We saw this reflected in the bi-annual DevX surveys (which we built out), and the results showed overall developer satisfaction increasing about 10% year over year during my time on the team.

Solving new problems while working from anywhere

Today, I’m Senior Director of Engineering for Application & Cloud Infrastructure, which includes compute, networking, core services, and the GraphQL application platform. Our mission is to deliver reliable, secure, and efficient platforms for building, operating, and scaling applications, services, and workloads at Airbnb.

My primary users are still the engineers at Airbnb. When they need to compute, they don’t wrangle AWS themselves — we provide a layer of abstraction that helps them use low-level infrastructure. Similarly, if they need authorization, authentication, configuration management, and a host of other services, they come to us rather than starting from scratch.

I’m excited to come to work every day because of the people I get to work with and the opportunities we face together. The culture is excellent, the people are smart and collaborative, and the engineers we support appreciate the work we do.

The setup is empowering, too, and as you solve problems, you can grow and expand to tackle bigger problems that span teams and organizations. Add in the ability to work from anywhere, and for me, it feels like the sky is the limit.

As I look back on my career, and really, my entire life, I tend to see it now through the lens of long-distance trail running — a major hobby of mine.

After working at a startup, having twins, and running my first marathon, I felt like I could do anything. At work and on the trails, I think about how to prepare for the journeys ahead and how to maintain a pace that allows me and the people around me to thrive in the long run. Recovery is necessary, but so is strategy, drive, discipline, and finding the people who will go with you as well as cheer you on along the way.

I’m happy this path, as unpredictable as it has been, has taken me to Airbnb. Airbnb is in that ideal position between a startup and a long-established company. The systems and workflows are mature, but there are still many interesting problems to solve and opportunities to pursue. If that’s of interest to you, I encourage you to check out openings at Airbnb.

My Journey to Airbnb — Anna Sulkina was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Public school to PhD

The story of Airbnb’s Head Economist for Policy and Director of Data Science involves geology, co-teaching with a Nobel Prize winner, and CSI. (No, not the hit TV franchise.)

Peter Coles was born and raised in Milwaukee, Wisconsin. He studied math at Princeton, earned his PhD in economics at Stanford, and taught at Harvard Business School before joining eBay and becoming a Data Science leader at Airbnb.

As you’ll see from his story, Peter has a deep interest in how marketplaces work. By transitioning from academia to the business world, he not only gets to study first-hand data about millions of guests and hosts, but also to influence product and policy decisions. And he still gets to hang out with academics. Check out all the research Peter and his team are doing here.

My fascination with marketplaces goes back a long time.

Sometime around second grade in Milwaukee, Wisconsin (where I grew up), my friends and I had the great idea to run a rock stand. It was like a lemonade stand, but instead we would sell rocks. Rocks we found in the street. Neighborhood kids could find their own rocks and sell them, and we’d take 25%. Nobody got any sales. Fortunately I’ve learned a bit more about marketplaces since then — more about that shortly.

From kindergarten through high school, I was a public school kid. My parents valued education, and helping others — my father was a doctor, my mother a nutritionist — and those are values I still hold dear.

While I played soccer and tennis and was moderately social, this period was probably best defined by an obsession with competitions. Math, Chess, Science Olympiad, Quiz Bowl, Academic Decathlon, puzzle races with my younger brother — there was hardly a nerdy competition offered where I didn’t compete. Time well spent? Let’s just say I missed a lot of high school parties while studying to become the five-time Wisconsin State Rocks, Minerals, and Fossil Identification champion — so you can be the judge.

By the time college came around, it seemed time to rebel. I wanted to be done with the nerdy stuff. I applied and was accepted to Princeton, and started studying ancient history. After one semester and (in my view) an underappreciated essay on the Hittites, I was back to majoring in math. At least I was good at that! I figured I could work on practical skills later.

After graduating, I accepted a fellowship in Germany and continued to Stanford for a PhD in economics — a somewhat more applied science, though I focused on game theory, at the intersection of math and strategy. I had the good fortune there to be the second graduate student of Jon Levin, now the President of Stanford University, who taught me the importance of simplification in research — even when the subject matter itself is complex.

Even while in this still-theoretical space, I kept my feet on the ground — or at least on the pedals. During a monthlong break in my classes in Germany, I biked around Europe, crashing with friends and family members of my classmates — people I had never met before staying with them. In a sense, I was prototyping Airbnb well before it existed!

My time in graduate school was Silicon Valley in the 2000s, after the dot-com crash, so tech was in the midst of a renaissance. Many of my friends were at growing companies like Google and Amazon. It was very tempting to stay in California to be a part of this, but I ended up with one more stop in academia.

Markets in theory, markets in practice

Harvard Business School, known for its focus on managerial science, was perhaps the most compelling place in the academic world that would allow me to stay close to the tech industry. I got a double stroke of good fortune: not only was I offered an assistant professorship there, but I also got to co-teach with Al Roth, a founder of the field of Market Design. Al is still an important mentor to me, and later won a Nobel Prize!

In my time researching and teaching graduate students, I was exposed to many examples of market design, conducting research on the topic of “Matching”; that is, mechanisms to pair users from two groups, often when price cannot be used to clear the market. This covered strategy of participants, signaling in markets, and I even had a chance to improve the market for PhD economists. I also wrote a number of case studies, including on Zillow, Microsoft, Craigslist, and more. The teaching and writing was a lot of fun, but I also came to realize I wasn’t a fit for academia in the long term. My attention span was too short to dedicate most of my time to research papers (and especially peer reviews), but I was enormously appreciative of this phase of my career.

By this point it was 2013, and two simultaneous and interrelated phenomena were exploding in tech: mobile and the sharing economy. It was a perfect time to head back west and finally enter the tech world.

I landed at eBay, which for a student of marketplaces was an ideal company: just about everything is for sale, and it was ripe for market design. Steve Tadelis, a mentor from my Stanford days, had created one of the first economics teams inside a tech company, which I took over when Steve left. At the same time, eBay was getting on the data science train — this was before every company had a DS team — and my group joined another to form eBay’s Data Labs. One of my favorite projects there was a project called “What’s it Worth” (which I worked on with Airbnb colleague Dean Chen), where we developed a methodology for determining the fair market value of items. Some hands-on practical work, some modeling — this was just what I was hoping for.

In 2015 Riley Newman, one of Airbnb’s first employees and then its Head of Data Science, presented an even more enticing opportunity. The Airbnb platform was growing quickly, and for the first time attracting substantial regulatory attention. They needed an economics team to partner with the growing policy team, to jointly address the question of Airbnb’s relationship to cities. This was a new way for me to apply economics. I was all in.

Addressing Airbnb’s critical questions with data

When I think back to my eight and a half years so far at Airbnb, I view this as entailing three “phases.” In the first, I worked to address economic questions by establishing a global team of data scientists and economists to analyze the relationship of short-term rentals to the world.

Meanwhile, as Airbnb continued to grow, execs were asking big questions that couldn’t be answered by any specific data science team. They needed a group with visibility across the whole organization. So in this second phase, Jackson Wang and I founded a team called Central Strategy & Insights, or CSI.

The acronym was no coincidence: we saw ourselves as forensic investigators, piecing together stories as we collected evidence. One important period of CSI’s work addressed changes brought on by the pandemic — in particular a major adjustment in where guests were looking to stay, and the supply we’d need to accommodate them. We also led the company’s business reviews, and generated analyses to describe the business to shareholders ahead of the IPO.

My third phase at Airbnb started a lot like the first, but supersized: developing models to inform a well-considered approach to policy considerations, this time as travel rebounded after the pandemic and governments were no longer fully occupied with a public health emergency. Our newly expanded group of economics PhDs and analysts also came up with ways to evaluate Airbnb’s impact on guests, hosts, and society, including via our US Economic Impact Report.

A great balance of academic and applied science

Almost all of my first several years at Airbnb had been internally facing. That’s changed in recent times, as we’ve spun up and expanded a program to collaborate with academic researchers to analyze Airbnb’s data and improve the experience for users.

The first step was to figure out how to collaborate with external researchers, while respecting privacy and legal limitations. Collaboration interest then came quickly. We have now published well-received papers with professors from MIT, Berkeley, Stanford, UCLA, NYU and more, with others in progress. One paper I wrote with colleagues and academic partners develops foundations for what “quality” means in platforms, from an economic perspective.

We’ve also launched a monthly seminar where we invite our academic collaborators to discuss research with Airbnb data scientists and technologists. Developing research is great, but there’s nothing like live discussions to cross-pollinate and foster ideas. This builds on a strong collaborative learning tradition at Airbnb, with internal classes and reading groups to grow our skills and keep up with tech developments.

Alongside engaging with academia, I’m so excited my data science colleagues and I have a mandate to be innovative and proactive. We have the space and encouragement to work on big ideas, even if they might take a year or two to prove out — and perhaps more importantly, even if some of the ideas fail. But nothing is more important than the people. I am proud of the students, scientists, and even professors I have hired here over the years, and love seeing them grow and find success.

A license to tackle big topics, continual education, research on the product as well as its relationship to the outside world, amazing colleagues, and a direct connection to academia all make Airbnb a unique place to be a market designer, economist, and data scientist. Whether or not you spent your free time in eighth grade studying rocks.

If you want to learn more about the research happening at Airbnb you can read our published papers here. If this type of work interests you, check out our open roles.

My Journey to Airbnb: Peter Coles was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By: Gerum Haile, Bo Shi, Yujia Liu, Yanwei Bai, Bo Yuan, Rory MacQueen, Yixia Mao

Across the more than 220 global markets that Airbnb operates in, cards are the primary way that guests pay for stays, experiences, and services. However, to help make our platform accessible to more people, reduce friction at checkout, and drive more adoption, we introduced trusted, locally preferred payment methods — called local payment methods or LPMs. By offering and supporting these payment methods, Airbnb enables guests everywhere to choose what works best for them.

In this blog post, we’ll discuss the implementation details behind our Pay as a Local initiative, which allowed us to launch 20+ local payment methods across multiple markets in just over one year.

LPMs: What they are, why they matter, and our discovery and selection process

Local payment methods go beyond traditional cards and include:

• Country or region-specific digital wallets (such as M-Pesa or MTN, MoMo)
• Online bank transfers (such as Online Banking Czech, Online Banking Slovakia)
• Real-time or instant bank payments (such as PIX, UPI)
• Local payment schemes (such as EFTPOS, Cartes Bancaires)

By embracing LPMs, Airbnb helps make travel more inclusive and seamless for people around the world. LPMs help the platform to:

• Boost conversion and bookings by offering guests familiar, trusted payment options.
• Unlock new markets where credit card usage is low or non-existent.
• Build accessibility for guests without credit cards or traditional banking access.

Through our research on local payment methods (LPMs), we identified over 300 unique payment options worldwide. For the initial phase of the LPM initiative, we used a structured qualification framework to select which local payment methods we would support. We evaluated the top 75 travel markets and selected the top one to two payment methods per market — excluding those without a clear travel use case — and arrived at a shortlist of just over 20 LPMs best suited for integration into our payment platform.

Background on Airbnb’s payment platform

Airbnb’s payments platform is designed to decouple payment logic from the core business (i.e., stays, experiences, and services), allowing for greater flexibility and scalability. The platform efficiently coordinates both guest pay-ins and host payouts by working with regulated payment service providers and financial partners.

Beyond payment processing, the system also supports robust payment trust and compliance functions.

Modernization

As part of a multi-year replatforming initiative for our payments architecture called Payments LTA (long-term architecture), we shifted from a monolithic system to a capability-oriented services system structured by domains, using a domain-driven decomposition approach. This modernization approach reduced our time to market, increased reusability and extensibility, and empowered greater team autonomy.

The core payment domain delivers essential capabilities for pay-in, payout, and payment intermediation. It consists of multiple subdomains, including Pay-in, Payout, Transaction Fulfillment, Processing, Wallet & instruments, Ledger, Incentives & Stored Value, Issuing, and Settlement & Reconciliation.

Replatforming as an enabler for local payment method expansion

The processing subdomain enables integration with third-party payment service providers (PSPs) and supports API and file-based vendor integration, as well as switching and routing capabilities. As part of our replatforming initiative, we adopted a connector and plugin-based architecture for onboarding new third-party payment service providers. This strategy has significantly reduced the time required to integrate new PSPs in different markets.

During this replatforming effort, we also introduced Multi-Step Transactions (MST): a processor-agnostic framework that supports payment flows completed across multiple stages. MST defines a PSP-agnostic transaction language to describe the intermediate steps required in a payment, such as submitting supplemental data or handling dynamic interactions. These steps, called Actions, can include:

• Redirects
• Strong customer authentication (SCA) frictions (challenges, fingerprinting)
• Payment method — specific flows

When a PSP indicates that an additional user action is required, its vendor plugin normalizes the request into an ActionPayload and returns it with a transaction intent status of ACTION_REQUIRED. This architecture ensures consistent handling of complex, multi-step payment experiences across diverse PSPs and markets.

LPM integration architecture

While our modernized payment platform laid the foundation for enabling LPMs, these payment methods come with a unique set of challenges. Many local methods require users to complete transactions in third-party wallet apps. This introduces complexity in app switching, session hand-off, and synchronization between Airbnb and external digital wallets.

Each local payment vendor also exposes different APIs and behaviors across charge, refund, and settlement flows, making integration and standardization difficult.

Technical approach

We analyzed the end-to-end behavior of our 20+ LPMs, and identified three foundational payment flows that capture the full spectrum of user and system interactions. By distilling LPM behaviors into these standardized payment flow archetypes, we established a unified framework for integration:

1. Redirect flow: Guests are redirected to a third-party site or app to complete the payment, then return to Airbnb to finalize their booking (e.g., Naver Pay, GoPay, FPX).
2. Async flow: Guests complete payment externally after receiving a prompt (such as a QR code or push notification), and Airbnb receives payment confirmation asynchronously via webhooks (e.g., Pix, MB Way, Blik).
3. Direct flow: Guests enter their payment credentials directly within Airbnb’s interface, allowing real-time processing similar to traditional card payments (e.g., Carte Bancaires, Apple Pay).

This standardized approach has enabled significant reusability across integrations and substantially reduced the engineering effort required to support new payment methods.

Asynchronous payment orchestration

Since many guests complete payments through external providers, we redesigned our payment orchestration — building on top of MST — to support payment flows that require user actions outside Airbnb (redirect flows and async flows).

For redirect flows, where guests complete the payment on a third-party app or website:

• Airbnb’s payments platform sends a charge request to the local payment vendor, whose response includes a redirectUrl.
• Our platform redirects the user to the external app or website to complete the payment.
• Once the payment is successfully completed, the user is redirected back to Airbnb with a result token. Airbnb’s payments platform then uses this token to securely confirm and finalize the payment with the local processor.

For async flows (which typically involve scanning a QR code):

• Airbnb’s payments platform sends a charge request to the local payment vendor, whose response includes a qrCodeData.
• The checkout page displays the QR code for the user to scan and complete the payment in their wallet app.
• After the payment succeeds, the vendor sends a webhook notification to Airbnb’s payments platform, which updates the payment status to success and confirms the user’s order.

Naver Pay: Redirect To Naver Pay Website

Naver Pay is one of the fastest-growing digital payment methods in South Korea. As of early 2025, it has reached over 30.6 million active users, representing approximately 60% of the South Korean population. Enabling Naver Pay in the South Korean market not only helps deliver a more seamless and familiar payment experience for local guests, but also expands Airbnb’s reach to new users who prefer using Naver Pay as their primary payment method.

Pix: Scan A QR Code

Pix is an instant payment system developed by the Central Bank of Brazil, enabling 24/7 real-time money transfers through methods such as QR codes or Pix keys. Its adoption has been extraordinary — by late 2024, more than 76% of Brazil’s population was using Pix, making it the country’s most popular payment method, surpassing cash, credit, and debit cards. In 2024 alone, Pix processed over BRL 26.4 trillion (approximately USD 4.6 trillion) in transaction volume, underscoring its pivotal role in Brazil’s digital payment ecosystem.

Config-driven payment method integration

Airbnb embraced a config-driven approach, powered by a central YAML-based Payment Method Config that acts as a single source of truth for flows, eligibility, input fields, refund rules, and more. Instead of scattering payment method logic across the frontend, backend, and various services, we consolidate all relevant details in this config. Both core payment services and frontend experiences dynamically reference this single source of truth, ensuring consistency for eligibility checks, UI rendering, and business rules. This unified approach dramatically reduces duplication, manual updates, and errors across the stack, making integration and maintenance faster and more reliable.

These configs also drive automated code generation for backend services using code generation tools, producing Java classes, DTOs, enums, schema, and integration scaffolding. As a result, integrating or updating a payment method is largely declarative — just a config change. This streamlines launches from months to weeks and makes ongoing maintenance far simpler.

Payment widget

Our payment widget — the payment method UI embedded into the checkout page — includes the list of available payment methods and handles the user’s inputs. Local payment methods often require specialized input forms (such as CPF for Pix) and have unique country/currency eligibility.

Rather than hardcoding forms and rules into the client, we centralize both form-field specification and eligibility checks in the backend. Servers send configuration payloads to clients defining exactly which fields to collect, which validation rules to apply, and which payment options to render. This empowers the frontend to dynamically adapt UI and validation for each payment method, accelerating launches and keeping user experiences fresh without frequent client releases.

For example, Pix in Brazil requires the guest’s first name, last name, and CPF (tax ID), which we collect and transmit as required to complete the payment.

Below is a diagram illustrating how dynamic payment method configurations are delivered from the backend to the frontend, enabling tailored checkout presentations for each payment method.

Building confidence through better testability

Testing local payment methods can be difficult, because developers often don’t have access to local wallets. Yet with such a broad range of payment methods and complex flows, comprehensive testing is essential to prevent regressions and ensure seamless functionality.

To address this, we enhanced Airbnb’s in-house Payment Service Provider (PSP) Emulator, enabling realistic simulation of PSP interactions for both redirect and asynchronous payment methods. The Emulator allows developers to test end-to-end payment scenarios without relying on unstable (or nonexistent) PSP sandboxes. For redirect payments, the Emulator provides a simple UI mirroring PSP acquirer pages, allowing testers to explicitly approve or decline transactions for precise scenario control. For async methods, it returns QR code details and automatically schedules webhook emission tasks upon receiving a /payments request — delivering a complete, reliable testing environment across diverse LPMs.

Scaling observability for local payment methods

Maintaining high reliability and availability is critical for Airbnb’s global payment system. As we expand to support many new local payment methods, we face increasing complexity: greater dependencies on external PSPs and wide variations in payment behaviors. For example, a real-time card payment and a redirect flow like Naver Pay follow completely different technical paths. That diversity makes observability difficult — a single “payment success rate” may represent card health well, but say little about an asynchronous LPM. Without proper visibility, regressions can go unnoticed until they affect real users. As dozens of new LPMs go live, observability has become the foundation of reliability.

To address this, we built a centralized monitoring framework that unifies metrics across all layers, from client to PSP. When launching a new LPM, onboarding now requires a single config change; add the method name, and metrics begin streaming automatically:

• Client metrics — user-level flow health from clients
• Payment backend metrics — API-level metrics for payment flows
• PSP metrics — API-level visibility between Airbnb and the PSP
• Webhook metrics — async completion status for redirect methods or refunds

We have also standardized the alerting rules across our platform’s Client, Backend, PSP, and Webhook layers using composite alerts and anomaly detection. Each alert follows a consistent pattern (failure count, rate, time window), e.g., “Naver Pay resume failures > 5 and failure rate > 20% in 30 minutes.” This design minimizes false positives during low-traffic periods.

This framework scales effectively, providing end-to-end visibility from user click to PSP confirmation. It enables engineers to trace issues in minutes rather than hours, whether those issues were caused by internal changes or external outages. By turning observability into a shared, automated layer, we were able to strengthen the backbone of payment reliability while accelerating the rollout of new LPMs worldwide.

Impact

The Pay as a Local initiative delivered significant business and technical impact:

• Meaningful booking uplift: We observed meaningful uplift in bookings and new users in markets where we launched local payment methods
• Faster integrations: Reduced integration time significantly through reusable flows and config-driven automation.
• Stronger reliability: Improved observability for early outage detection, standardized testing to prevent regressions, and streamlined vendor escalation and on-call processes for global resilience.

Conclusion

Supporting local payment methods helps Airbnb to stay competitive and relevant in the global travel industry. These payment options help improve checkout conversion, drive adoption, and unlock new growth opportunities.

This post outlined how the Airbnb payment platform has evolved to support local payment methods at scale — through asynchronous payment orchestration, config-driven onboarding, centralized observability, and robust testability. Together, these capabilities enable faster integrations, lower maintenance overhead, and offer a more seamless, localized checkout experience for guests worldwide.

As Airbnb continues to expand globally, our payments platform will keep evolving with the same principles of extensibility, reliability, and scalability, ensuring that guests everywhere can pay confidently, using the methods they know and trust.

Acknowledgments

We had many people at Airbnb contributing to this big rearchitecture, but countless thanks to Mini Atwal, Ashish Singla, Musaab At-Taras,Linmin Yang, Yong Rhyu, Yohannes Tsegay, Livar Cunha, Praveena Subrahmanyam, Steve Ickes, Vijaykumar Borkar, Vibhu Ramani, Aashna Jain, Abhishek Ghosh, Abhishek Patel, Adithya Tammavarapu, Akai Hsieh, Akash Budhia, Amar Parkash, Amee Mewada, Ankita Balakrushan Tate, Bharath Kumar Chandramouli, Bo Shi, Bo Yuan. Callum Li. Carlos Townsend Pico, Chanakya Daparthy, Charles Tang, Cibi Pari, Cindy Jaimez, Cindy Shi, Dan Yo, Daniela Nobre, Danielle Zegelstein, David Cordoba, David Drinan, Dawei Wang, Dechuan Xu, Denise Francisco, Denny Liang, Dimi Matcovschi, Divya Verma, Feifeng Yang, Gabriel Siqueira, Sunny Wallia, Prashant Jamlakar, Daniel Kriske, Giovanni Iniguez, Haojie Zhang, Haokun Chen, Haoti Zhong, Harriet Russell, Harshit Gupta, Henrique Moreira Indio do Brasil, Ishan Ishan, Jenny Shen, Jerroid Marks, Jiafang Jiang, Joey Yin, Jon Chew, Karen Kuo, Katie Turley, Letian Zhang, Maneesh Lall, Manish Singhal, Maria Daneri, Mark Jang, Mengfei Ren, Michelle Desiderio, Mohit Dhawan, Nam Kim, Nerea Ruiz Alvarez, Nikita Kapoor, Oliver Zhang, Omer Faruk Gul, Pallavi Sharma, Prateek Sri, Rae Huang, Rohit Krishnan Dandayudham, Rory MacQueen, Ruize Liu, Sam Bitter, Sam Tang, Saran Singh. Sardana Sai Anil, Serdar Yildirim, Shwetha Saibanna, Silvia Crespo Sanchez, Simon Xia, Stella Dong, Stella Su, Stephanie Leung, Steve Cao, Sumit Ranjan, Tay Rauch, Thanigaivelan Manickavelu, Tiffany Selby, Toland Hon, Trish Burgess,Vishal Garg, Vivian Lue, Vyom Rastogi, William Betz, Xi Wen, Xing Xing, Xuanxuan Wu, Yangguang Li, Yanwei Bai, Yeung Song, Yixia Mao, Yujia Liu. Yun Cho, Zhenhui Zhu, Ziyun Ye

****************

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Pay As a Local was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How Airbnb hardened Mussel, our key-value store, with smarter traffic controls to stay fast and reliable during traffic spikes.

By Shravan Gaonkar, Casey Getz, Wonhee Cho

Introduction

Every request lookup on Airbnb, from stays, experiences, and services search to customer support inquiries ultimately hits Mussel, our multi-tenant key-value store for derived data. Mussel operates as a proxy service, deployed as a fleet of stateless dispatchers — each a Kubernetes pod. On a typical day, this fleet handles millions of predictable point and range reads. During peak events, however, it must absorb several-fold higher volume, terabyte-scale bulk uploads, and sudden bursts from automated bots or DDoS attacks. Its ability to reliably serve this volatile mix of traffic is therefore critical to both the Airbnb user experience and the stability of the many services that power our platform.

Given Mussel’s traffic volume and its role in core Airbnb flows, quality of service (QoS) is one of the product’s defining features. The first-generation QoS system was primarily an isolation tool. It relied on a Redis-backed counter, client quota based rate-limiter, that checked a caller’s requests per second (QPS) against a configurable fixed quota. The goal was to prevent a single misbehaving client from overwhelming the service and causing a complete outage. For this purpose, it was simple and effective.

However, as the service matured, our goal shifted from merely preventing meltdowns to maximizing goodput — that is, getting the most useful work done without degrading performance. A system of fixed, manually configured quotas can’t achieve this, as it can’t adapt in real time to shifting traffic patterns, new query shapes, or sudden threats like a DDoS attack. A truly effective QoS system needs to be adaptive, automatically exerting prioritized backpressure when it senses the system has reached its useful capacity.

To better match our QoS system to the realities of online traffic and maximize goodput, over time we evolved it to add several new layers.

• Resource-aware rate control (RARC): Charges each request in request units (RU) that reflect rows, bytes, and latency, not just counts.
• Load shedding with criticality tiers: Guarantees that high-priority traffic (e.g., customer support, trust and safety) stays responsive when capacity evaporates.
• Hot-key detection & DDoS mitigation: Detects skewed access patterns in real time and then shields the backend — whether the surge is legitimate or a DDoS burst — by caching or coalescing the duplicate requests before they reach the storage layer.

What follows is an engineer’s view of how these layers were designed, deployed, and battle-tested, and why the same ideas may apply to any multi-tenant system that has outgrown simple QPS limits.

Background: Life with Client Quota Rate Limiter

When Mussel launched, rate-limiting was entirely handled via simple QPS rate-limiting using a Redis-based distributed counter service. Each caller received a static, per-minute quota, and the dispatcher incremented a Redis key for every incoming request. If the key’s value exceeded the caller’s quota, the dispatcher returned an HTTP 429. The design was simple, predictable, and easy to operate.

Two architectural details made this feasible. First, Mussel and its storage engine were tightly coupled; backend effort correlated reasonably well with the number of calls at the front door. Second, the traffic mix was modest in size and variety, so a single global limit per caller rarely caused trouble.

As adoption grew, two limitations became clear.

1. Cost variance: A one-row lookup and a 100,000-row scan were treated equally, even though their load on the backend differed by orders of magnitude. The system couldn’t distinguish high-value cheap work from low-value expensive work.
2. Traffic skew: Per-caller rate limits provided isolation at the client level, but were blind to the data’s access pattern. When a single key became “hot” — for example, a popular listing accessed by thousands of different callers simultaneously — the aggregate traffic could overwhelm the underlying storage shard, even if each individual caller remained within its quota. This created a localized bottleneck that degraded performance for the entire cluster, impacting clients requesting completely unrelated data. Isolation by caller was insufficient to prevent this kind of resource contention.

Addressing these gaps meant shifting from a request-counting mindset to a resource-accounting mindset and designing controls that reflect the real cost of each operation.

Resource-aware rate control

A fair quota system must account for the real work a request imposes on the storage layer. Resource-aware rate control (RARC) meets this need by charging operations in request units (RU) rather than raw requests per second.

A request unit blends four observable factors: fixed per-call overhead, rows processed, payload bytes, and — crucially — latency. Latency captures effects that rows and bytes alone miss: two one-megabyte reads can differ greatly in cost if one hits cache and the other triggers disk. In practice, we use a linear model. For both reads and writes, the cost is:

RU_read = 1 + w_r × bytes_read + w_l × latency_ms
RU_write = 6 + w_b × bytes_written / 4096 bytes + w_l × latency_msWeight factors w_r, w_b, and w_l come from load-test calibration 
based on the compute, network and disk I/O. 
bytes_read, bytes_written and latency is measured per request

Although approximate, the formula separates operations whose surface metrics look similar yet load the backend very differently.

Each dispatcher continues to rely on rate-limiter for distributed counting, but the counter now represents request-unit tokens instead of raw QPS. At the start of every epoch, the dispatcher adds the caller’s static RU quota to a local token bucket and immediately debits that bucket by the RU cost of each incoming request. When the bucket is empty, the request is rejected with HTTP 419. Because all dispatchers follow the same procedure and epochs are short, their buckets remain closely aligned without additional coordination.

Adaptive protection is handled in the separate load-shedding layer; backend latency influences which traffic is dropped or delayed, not the size of the periodic RU refill. This keeps rate accounting straightforward — static quotas expressed in request units — while still reacting quickly when the storage layer shows signs of stress.

Load shedding: Staying healthy when capacity evaporates or develops hotspots

Rate limits based on request units excel at smoothing normal traffic, but they adjust on a scale of seconds. When the workload shifts faster — a bot floods a key, a shard stalls, or a batch job begins a full-table scan — those seconds are enough for queues to balloon and service-level objectives to slip. To bridge this reaction-time gap, Mussel uses a load-shedding safety net that combines three real-time signals: (1) traffic criticality, (2) a latency ratio, and (3) a CoDel-inspired queueing policy.

The latency ratio is a ratio that serves as a real-time indicator of stress on the system stress. Each dispatcher computes this ratio by dividing the long-term p95 latency by the short-term p95 latency. A stable system has a ratio near 1.0; a value dropping towards 0.3 indicates that latency is rising sharply. When that threshold is crossed, the dispatcher temporarily increases the RU cost applied to a designated client class so that its token bucket drains faster and the request rate naturally backs off. If the ratio keeps falling, the same penalty can be expanded to additional classes until latency returns to a safe range.

The estimate uses the constant-memory P² algorithm [1], requiring no raw sample storage or cross-node coordination.

The Control-Delay (CoDel) thread pool tackles the second hazard: queue buildup inside the dispatcher itself [2]. It monitors the time a request waits in the queue. If that sojourn time proves the system is already saturated, the request fails early, freeing up memory and threads for higher-priority work. An optional latency penalty can also be applied to RU accounting, charging more for queries from callers that persistently trigger the latency ratio.

Together, these layers — criticality, a real-time latency ratio, and adaptive queueing — form a shield that lets guest-facing traffic ride out backend hiccups. In practice, this system has cut recovery times by about half and keeps dispatchers stable without human intervention.

Hot-key detection and DDoS defence

Request-unit limits and load shedding keep client usage fair, but they cannot stop a stampede of identical reads aimed at one record. Imagine a listing that hits the front page of a major news outlet: tens of thousands of guests refresh their browser, all asking for the same key. A misconfigured crawler — or a deliberate botnet — can generate the same access pattern, only faster. The result is shard overload, a full dispatcher queue, and rising latency for unrelated work.

Mussel neutralises this amplification with a three-stephot-key defence layer: real-time detection, local caching, and request coalescing.

Real-time detection in constant space

Every dispatcher streams incoming keys into an in-memory top-k counter. The counter is a variant of the Space-Saving algorithm [2] popularized in Brian Hayes’s “Britney Spears Problem” essay [4]. In just a few megabytes, it tracks approximate hit counts, maintains a frequency-ordered heap, and surfaces the hottest keys in real time in each individual dispatcher.

Local caching and request coalescing

When a key crosses the hot threshold, the dispatcher serves it from a process-local LRU cache. Entries expire after roughly three seconds, so they vanish as soon as demand cools; no global cache is required. A cache miss can still arrive multiple times in the same millisecond, so the dispatcher tracks in-flight reads for hot keys. New arrivals attach to the pending future; the first backend response then fans out to all waiters. In most cases only one request per hot key per dispatcher pod ever reaches the storage layer.

Impact in production

In a controlled DDoS drill that targeted a small set of keys at ≈ million-QPS scale, the hot-key layer collapsed the burst to a trickle — each dispatcher forwarded only an occasional request, well below the capacity of any individual shard — so the backend never felt the surge.

Retrospective and key takeaways

The journey from a single QPS counter to a layered, cost-aware QoS stack has reshaped how Mussel handles traffic and, just as importantly, how engineers think about fairness and resilience. A few themes surface when we look back across the stages described above.

The first is the value of early, visible impact. The initial release of request-unit accounting went live well before load shedding or hot-key defence. Soon after deployment it automatically throttled a caller whose range scans had been quietly inflating cluster latency. That early win validated the concept and built momentum for the deeper changes that followed.

A second lesson is to prefer to keep control loops local. All the key signals — P² latency quantiles, the Space-Saving top-k counter, and CoDel queue delay — run entirely inside each dispatcher. Because no cross-node coordination is required, the system scales linearly and continues to protect capacity even if the control plane is itself under stress.

Third, effective protection works ontwo different time-scales. Per-call RU pricing catches micro-spikes; the latency ratio and CoDel queue thresholds respond to macro slow-downs. Neither mechanism alone would have kept latency flat during the last controlled DDoS drill, but in concert they absorbed the shock and recovered within seconds.

Finally, QoS is a living system. Traffic patterns evolve, back-end capabilities improve, and new workloads appear. Planned next steps include database-native resource groups and automatic quota tuning from thirty-day usage curves. The principles that guided this project — measure true cost, react locally and quickly, layer defences — form a durable template, but the implementation will continue to grow with the platform it protects.

Does this type of work interest you? We’re hiring, check out open roles here.

 References

1. Raj Jain and Imrich Chlamtac. 1985. The P² algorithm for dynamic calculation of quantiles and histograms without storing observations. Communications of the ACM, 28(10), 1076–1085. https://doi.org/10.1145/4372.4378
2. Erik D. Demaine, Alejandro López-Ortiz, and J. Ian Munro. 2002. Frequency estimation of internet packet streams with limited space. In Algorithms — ESA 2002: 10th Annual European Symposium, Rome, Italy, September 17–21, 2002. Rolf H. Möhring and Rajeev Raman (Eds.). Lecture Notes in Computer Science, Vol. 2461. Springer, 348–360.
3. Kathleen M. Nichols and Van Jacobson. 2012. Controlling queue delay. Communications of the ACM, 55(7), 42–50. https://doi.org/10.1145/2209249.2209264
4. Brian Hayes. 2008. Computing science: The Britney Spears problem. American Scientist, 96(4), 274–279. https://www.americanscientist.org/article/the-britney-spears-problem

By Shravan Gaonkar, Chandramouli Rangarajan, Yanhan Zhang

How we completely rearchitected Mussel, our storage engine for derived data, and lessons learned from the migration from Mussel V1 to V2.

Airbnb’s core key-value store, internally known as Mussel, bridges offline and online workloads, providing highly scalable bulk load capabilities combined with single-digit millisecond reads.

Since first writing about Mussel in a 2022 blog post, we have completely deprecated the storage backend of the original system (what we now call Mussel v1) and have replaced it with a NewSQL backend which we are referring to as Mussel v2. Mussel v2 has been running successfully in production for a year, and we wanted to share why we undertook this rearchitecture, what the challenges were, and what benefits we got from it.

Why rearchitect

Mussel v1 reliably supported Airbnb for years, but new requirements — real-time fraud checks, instant personalization, dynamic pricing, and massive data — demand a platform that combines real-time streaming with bulk ingestion, all while being easy to manage.

Key Challenges with v1

Mussel v2 solves a number of issues with v1, delivering a scalable, cloud-native key-value store with predictable performance and minimal operational overhead.

• Operational complexity: Scaling or replacing nodes required multi-step Chef scripts on EC2; v2 uses Kubernetes manifests and automated rollouts, reducing hours of manual work to minutes.
• Capacity & hotspots: Static hash partitioning sometimes overloaded nodes, leading to latency spikes. V2’s dynamic range sharding and presplitting keep reads fast (p99 < 25ms), even for 100TB+ tables.
• Consistency flexibility: v1 offered limited consistency control. v2 lets teams choose between immediate or eventual consistency based on their SLA needs.
• Cost & Transparency: Resource usage in v1 was opaque. v2 adds namespace tenancy, quota enforcement, and dashboards, providing cost visibility and control.

New architecture

Mussel v2 is a complete re-architecture addressing v1’s operational and scalability challenges. It’s designed to be automated, maintainable, and scalable, while ensuring feature parity and an easy migration for 100+ existing user cases.

Dispatcher

In Mussel v2, the Dispatcher is a stateless, horizontally-scalable Kubernetes service that replaces the tightly coupled, protocol-specific design of v1. It translates client API calls into backend queries/mutations, supports dual-write and shadow-read modes for migration, manages retries and rate limits, and integrates with Airbnb’s service mesh for security and service discovery.

Reads are simplified: Each dataname maps to a logical table, enabling optimized point lookups, range/prefix queries, and stale reads from local replicas to reduce latency. Dynamic throttling and prioritization maintain performance under changing traffic.

Writes are persisted in Kafka for durability first, with the Replayer and Write Dispatcher applying them in order to the backend. This event-driven model absorbs bursts, ensures consistency, and removes v1’s operational overhead. Kafka also underpins upgrades, bootstrapping, and migrations until CDC and snapshotting mature.

The architecture suits derived data and replay-heavy use cases today, with a long-term goal of shifting ingestion and replication fully to the distributed backend database to bring down latency and simplify operations.

Bulk load
Bulk load remains essential for moving large datasets from offline warehouses into Mussel for low-latency queries. v2 preserves v1 semantics, supporting both “merge” (add to existing tables) and “replace” (swap datasets) semantics.

To maintain a familiar interface, v2 keeps the existing Airflow-based onboarding and transforms warehouse data into a standardized format, uploading to S3 for ingestion. Airflow is an open-source platform for authoring, scheduling, and monitoring data pipelines. Created at Airbnb, it lets users define workflows in code as directed acyclic graphs (DAGs), enabling quick iteration and easy orchestration of tasks for data engineers and scientists worldwide.

A stateless controller orchestrates jobs, while a distributed, stateful worker fleet (Kubernetes StatefulSets) performs parallel ingestion, loading records from S3 into tables. Optimizations — like deduplication for replace jobs, delta merges, and insert-on-duplicate-key-ignore — ensure high throughput and efficient writes at Airbnb scale.

TTL

Automated data expiration (TTL) can help support data governance goals and storage efficiency. In v1, expiration relied on the storage engine’s compaction cycle, which struggled at scale.

Mussel v2 introduces a topology-aware expiration service that shards data namespaces into range-based subtasks processed concurrently by multiple workers. Expired records are scanned and deleted in parallel, minimizing sweep time for large datasets. Subtasks are scheduled to limit impact on live queries, and write-heavy tables use max-version enforcement with targeted deletes to maintain performance and data hygiene.

These enhancements provide the same retention functionality as v1 but with far greater efficiency, transparency, and scalability, meeting Airbnb’s modern data platform demands and enabling future use cases.

The migration process

Challenge

Mussel stores vast amounts of data and serves thousands of tables across a wide array of Airbnb services, sustaining mission-critical read and write traffic at high scale. Given the criticality of Mussel to Airbnb’s online traffic, our migration goal was straightforward but challenging: Move all data and traffic from Mussel v1 to v2 with zero data loss and no impact on availability to our customers.

Process

We adopted a blue/green migration strategy, but with notable complexities. Mussel v1 didn’t provide table-level snapshots or CDC streams, which are standard in many datastores. To bridge this gap, we developed a custom migration pipeline capable of bootstrapping tables to v2, selected by usage patterns and risk profiles. Once bootstrapped, dual writes were enabled on a per-table basis to keep v2 in sync as the migration progressed.

The migration itself followed several distinct stages:

• Blue Zone: All traffic initially flowed to v1 (“Blue”). This provided a stable baseline as we migrated data behind the scenes.
• Shadowing (Green): Once tables were bootstrapped, v2 (“Green”) began shadowing v1 — handling reads/writes in parallel, but only v1 responded. This allowed us to check v2’s correctness and performance without risk.
• Reverse: After building confidence, v2 took over active traffic while v1 remained on standby. We built automatic circuit breakers and fallback logic: If v2 showed elevated error rates or lagged behind v1, we could instantly return traffic to v1 or revert to shadowing.
• Cutover: When v2 passed all checks, we completed the cutover on a dataname-by-dataname basis, with Kafka serving as a robust intermediary for write reliability throughout.

To further de-risk the process, migration was performed one table at a time. Every step was reversible and could be fine-tuned per table or group of tables based on their risk profile. This granular, staged approach allowed for rapid iteration, safe rollbacks, and continuous progress without impacting the business.

Migration pipeline

As described in our previous blog post, the v1 architecture uses Kafka as a replication log — data is first written to Kafka, then consumed by the v1 backend. During the data migration to v2, we leveraged the same Kafka stream to maintain eventual consistency between v1 and v2.

To migrate any given table from v1 to v2, we built a custom pipeline consisting of the following steps:

1. Source data sampling: We download backup data from v1, extract the relevant tables, and sample the data to understand its distribution.
2. Create pre-split table on v2: Based on the sampling results, we create a corresponding v2 table with a pre-defined shard layout to minimize data reshuffling during migration.
3. Bootstrap: This is the most time-consuming step, taking hours or even days depending on table size. To bootstrap efficiently, we use Kubernetes StatefulSets to persist local state and periodically checkpoint progress.
4. Checksum verification: We verify that all data from the v1 backup has been correctly ingested into v2.
5. Catch-up: We apply any lagging messages that accumulated in Kafka during the bootstrap phase.
6. Dual writes: At this stage, both v1 and v2 consume from the same Kafka topic. We ensure eventual consistency between the two, with replication lag typically within tens of milliseconds.

Once data migration is complete and we enter dual write mode, we can begin the read traffic migration phase. During this phase, our dispatcher can be dynamically configured to serve read requests for specific tables from v1, while sending shadow requests to v2 for consistency checks. We then gradually shift to serving reads from v2, accompanied by reverse shadow requests to v1 for consistency checks, which also enables quick fallback to v1 responses if v2 becomes unstable. Eventually, we fully transition to serving all read traffic from v2.

Lessons learned

Several key insights emerged from this migration:

• Consistency complexity: Migrating from an eventually consistent (v1) to a strongly consistent (v2) backend introduced new challenges, particularly around write conflicts. Addressing these required features like write deduplication, hotkey blocking, and lazy write repair — sometimes trading off storage cost or read performance.
• Presplitting is critical: As we shifted from hash-based (v1) to range-based partitioning (v2), inserting large consecutive data could cause hotspots and disrupt our v2 backend. To prevent this, we needed to accurately sample the v1 data and presplit it into multiple shards based on v2’s topology, ensuring balanced ingestion traffic across backend nodes during data migration.
• Query model adjustments: v2 doesn’t push down range filters as effectively, requiring us to implement client-side pagination for prefix and range queries.
• Freshness vs. cost: Different use cases required different tradeoffs. Some prioritized data freshness and used primary replicas for the latest reads, while others leveraged secondary replicas to balance staleness with cost and performance.
• Kafka’s role: Kafka’s proven stable p99 millisecond latency made it an invaluable part of our migration process.
• Building in flexibility: Customer retries and routine bulk jobs provided a safety net for the rare inconsistencies, and our migration design allowed for per-table stage assignments and instant reversibility — key for managing risk at scale.

As a result, we migrated more than a petabyte of data across thousands of tables with zero downtime or data loss, thanks to a blue/green rollout, dual-write pipeline, and automated fallbacks — so the product teams could keep shipping features while the engine under them evolved.

Conclusion and next steps

What sets Mussel v2 apart is the way it fuses capabilities that are usually confined to separate, specialized systems. In our deployment of Mussel V2, we observe that this system can simultaneously

1. ingest tens of terabytes in bulk data upload,
2. sustain 100 k+ streaming writes per second in the same cluster, and
3. keep p99 reads under 25 ms

— all while giving callers a simple dial to toggle stale reads on a per-namespace basis. By pairing a NewSQL backend with a Kubernetes-native control plane, Mussel v2 delivers the elasticity of object storage, the responsiveness of a low-latency cache, and the operability of modern service meshes — rolled into one platform. Engineers no longer need to stitch together a cache, a queue, and a datastore to hit their SLAs; Mussel provides those guarantees out of the box, letting teams focus on product innovation instead of data plumbing.

Looking ahead, we’ll be sharing deeper insights into how we’re evolving quality of service (QoS) management within Mussel, now orchestrated cleanly from the Dispatcher layer. We’ll also describe our journey in optimizing bulk loading at scale — unlocking new performance and reliability wins for complex data pipelines. If you’re passionate about building large-scale distributed systems and want to help shape the future of data infrastructure at Airbnb, take a look at our Careers page — we’re always looking for talented engineers to join us on this mission.

References

1. https://medium.com/airbnb-engineering/mussel-airbnbs-key-value-store-for-derived-data-406b9fa1b296

At Airbnb, we recently completed migrating our largest repo, the JVM monorepo, to Bazel. This repo contains tens of millions of lines of Java, Kotlin, and Scala code that power the vast array of backend services and data pipelines behind airbnb.com.

Migration in numbers (4.5 years of work):

• Build CSAT: 38% → 68%
• 3–5x faster local build and test times
• 2–3x faster IntelliJ syncs
• 2–3x faster deploys to the development environment

In this blog post, we’ll discuss the why, share some highlights on the how, and finish off with key learnings.

Why Bazel?

Before the migration, our JVM monorepo used Gradle as its build system. We decided to migrate toBazel because it offered three key advantages: speed, reliability, and a uniform build infrastructure layer.

Speed

Bazel’s cacheable, portable actions allow us to scale performance with remote execution

In 2021, builds of large services often took >20 minutes locally and pre-merge CI p90 was 35 minutes.

Building with Gradle was near its limit. We had already vertically scaled to high-end AWS machines on CI and remote development machines for developers of large services. In CI, we also used heuristics to split project builds and tests across multiple machines. However, this was inefficient, because of machine underutilization and duplication of shared tasks.

Bazel remote execution allowed us to scale to thousands of parallel actions. This was far more efficient than our sharding heuristics. Remote build execution (RBE) workers are also short-lived, which results in better machine utilization and cost efficiency. In addition, Build without the Bytes allows downloading only a subset of files, greatly reducing download volume (in Gradle, every cached artifact needs to be downloaded). Finally and most importantly, local builds are significantly faster thanks to RBE.

In addition, Gradle configuration of some large projects often took minutes due to it being single-threaded. Bazel analysis, in contrast, runs in parallel, in part because its configuration language, Starlark, is constrained to be side-effect-free.

Reliability

Bazel’s hermeticity ensures reliable, repeatable builds

Gradle tasks have access to the full file system, which can lead to serious unintended consequences at scale. One example we ran into was when a developer updated a task to clean up recent files in the /tmp/ directory. This created a race condition with other Gradle tasks that used the /tmp/ directory and caused CI to fail when thousands of Gradle tasks had to be rerun.

Bazel solves this issue with sandboxing, which ensures that only specified inputs are available to a build action. If a file isn’t declared as an input, it simply doesn’t exist in the sandboxed environment.

Gradle tasks also implicitly depend on the machine’s resources. Gradle builds run on local machines and CI machines of different sizes. This can lead to resource contention when a task is run on a smaller machine or when the cache is cold and thousands of tasks are run on the same machine.

Remote build execution (RBE) solves this by running actions in identical containers with strict resource limits. We also configured both local and CI builds to use RBE, which greatly reduces environment differences.

Shared infrastructure

Airbnb currently has a collection of language- and platform-specific repos, such as web, iOS, Python, and Go, all of which are now on Bazel. Unifying on Bazel enables a uniform build infrastructure layer across repos, which includes:

• Remote caching
• Remote build execution
• Affected targets calculation
• Instrumentation & logging from the Build Event Protocol

How did we migrate?

Proof of concept

As a first milestone, we wanted to show that we could build a service and run its unit tests in Bazel. Because this was a proof of concept, we wanted to minimize disruption to engineers. Therefore, the Bazel build co-existed with Gradle. As a result, developers could choose between using Gradle or Bazel locally.

We needed to prove that developers would choose to opt in to using Bazel over Gradle. It wasn’t enough for Bazel to be faster, developers had to willingly opt in to using Bazel.

For this proof of concept, we chose Airbnb’s GraphQL monolith platform, Viaduct, which had the following important properties:

1. It was one of Airbnb’s largest and most complex services. If we could migrate Viaduct to Bazel, then we could likely migrate the rest of the monorepo.
2. Slow builds were a major pain point, so Bazel could have a large impact.
3. Viaduct has 300 product engineers modifying its code every month, so improving Viaduct’s build speed would be a substantial productivity win.
4. Because of (2) and (3) above, Viaduct’s core infrastructure team was eager to partner with us.

To achieve a working Viaduct build with Bazel, we did two things. First, we had to port much of Viaduct’s build logic from Gradle to Bazel. Second, because we decided to maintain co-existing builds and the Gradle build graph was still changing, we decided to build an automated build file generator (which we’ll cover in detail in a separate section).

Importantly, even though we were able to locally build the service 2–4x faster with Bazel, many developers did not yet want to switch.

In talking with the service’s owners, we discovered a number of missing integrations and bugs. It took us an additional few months to address these pain points, after which Viaduct developers willingly switched from Gradle to Bazel.

Scaling builds and tests with Bazel

The proof of concept showed that Bazel was superior to Gradle for one of Airbnb’s largest services and a large audience of developers. Now we wanted to scale it to the rest of Airbnb’s JVM monorepo.

We decided to scale breadth-first, getting all of the repo compiling and testing in Bazel. Again, to minimize disruption, Bazel builds co-existed with Gradle, which had two important benefits.

First, developers could still use Bazel for local development and get most of its benefits even though their code was still built with Gradle for deployment. Second, we could always disable Bazel if it was negatively impacting developers. For example, when Bazel infrastructure like the remote cache or remote execution cluster experienced an incident, we could and did disable Bazel, letting users fall back to Gradle.

However, a major downside was that both Gradle and Bazel build graphs had to be maintained. Manually maintaining a Bazel build graph would have degraded the developer experience. As a result, we invested further in automated build file generation, so that developers didn’t need to manually maintain Bazel build files.

Automated build file generation

For our build file generator, we were heavily inspired by Gazelle, which generates Bazel build files by parsing source files to build a dependency graph.

Although we considered extending Gazelle to support JVM languages, we had very strict performance requirements and needed to handle dependency cycles. This ultimately led us to build our own automated build file generator.

Because we had to maintain co-existing build graphs, we needed to run the build file generator on every commit before merging into mainline. This meant it had to run as fast as possible to not significantly degrade the developer experience. To achieve this, we implemented external caching to speed up the automated build file generation.

Similar to Gazelle, the build file generator parses Java, Kotlin, and Scala source files for package and import statements and symbol declarations to build a file-level dependency graph.

In CI, we publish a cached index of the repository at each mainline commit. When a user runs sync-configs, it downloads this cache and only re-scans directories which have changed since the merge base. This greatly improves performance for the common case where users only modify a small set of files.

In addition, with this build file generator, we were able to support a more fine-grained build graph, which resulted in >10x more Bazel targets than Gradle projects. This enabled faster builds through more parallel builds and less cache invalidation. However, one challenge of moving to a more fine-grained build graph is the possibility of introducing compilation cycles; sync-configs is able to detect this automatically and merge compilation units when necessary.

Even after the migration, the build file generator remains in use. It improves the developer experience by automatically fixing build file configurations and removing unused dependencies. In contrast, when we were on Gradle, users manually maintained ~4,500 Gradle files, which led to unused dependencies, a bloated dependency graph, and slower builds with fewer cache hits.

Porting build logic

In addition to JVM source files, our repo has a large amount of build logic such as code generation owned by multiple teams. These often took the form of Gradle plugins.

Because we now had automated build file generation, when porting the build logic, we also had to integrate it with the build file generator. Also, because we had more granular targets, a single line of Gradle config now might need to apply to 10+ Bazel build files.

As a result, our build file generator architecture supported plugins similar to Gazelle extensions. These plugins were triggered by the presence of specific files such as Thrift or GraphQL files. These plugins could also generate new build targets such as codegen actions.

In some cases, the Gradle logic was manually added as a one-off or not easily inferred from the file structure. As a result, we also supported generator directives similar to Gazelle, such as adding dependencies or setting attributes.

Initially, our team ported much of this build logic ourselves with minimal help from service owners. As Bazel adoption grew, owners of complex build logic were incentivized to migrate to Bazel, because it was faster and more reliable than Gradle. In the process, they often wrote their own build file generator plugins, highlighting the extensibility of our generator.

Third-party library multi-version support

Another major issue we hit on the road to 100% compilation and testing with Bazel was multiple versions of the same third-party library.

Initially, we specified a single version of each library. The build file generator would add dependencies from this universe.

However, in Gradle, each sub-project within the monorepo could use different versions of third-party libraries. As a result, when compiling against a single version, compilation or testing could fail with a missing symbol.

To bring multi-version support to our Bazel system, we built tooling to generate multiple maven_install rules and added a custom aspect to resolve conflicts at the target level.

Multiple versions of Guava in Bazel before we added conflict resolution

Once we had this capability, we systematically synced library versions from Gradle so that each build target’s classpath more closely matched its Gradle counterpart.

To learn more about our approach, see our BazelCon 2022 talk. Since giving this talk, we have made improvements like moving the resolution to analysis-time for better IDE support and adding more user-friendly tooling for updating libraries.

Migrating the deploys

As we got the vast majority of projects building and passing unit tests with Bazel in CI, we began to focus on migrating deployments. The Bazel-built `.jar`s were not identical to the Gradle-built `.jar`s. As a result, we needed a strategy to ensure the deployments were safe. We started with services.

Services

To verify the correctness of deploying services with Bazel, we used startup and integration tests. Of ~700 services, ~100 encountered startup or integration test failures. The majority of failures were missing dependencies that were loaded via Java reflection, usually from config files or other files. As a result, we were able to fix a number of these issues by parsing files for classes that would be loaded via reflection, and then adding the required dependency.

Another major source of errors was differing library versions, which could lead to missing symbol errors at runtime. In Gradle, users manually specified dependencies and their versions. However, in Bazel, build files were generated from source and dependencies were inferred from import statements, which didn’t specify the version. We solved many of these errors forcing third-party library versions to match those of the Gradle project.

After taking into account reflected classes and syncing versions, only a single-digit percentage of services hit production runtime issues that required more in-depth manual work to fix.

Data pipelines and other projects

In addition to services, we had 450 data pipelines and ~50 other projects that were deployed to either a Spark cluster or a Flink runtime.

Similar to services, we were able to catch a number of issues using tests. In particular, data pipelines on Airbnb’s data engineering paved path have CI tests that run a small version of the Spark pipeline locally. For these ~400 paved-path pipelines, after passing CI tests, only about 3% had production issues at runtime. As a result, we were able to very quickly migrate the paved-path pipelines.

As with services, we had a few remaining deployables that were a bit more bespoke and had to be individually deployed and monitored to verify correctness.

What did we learn?

Customer partnership

Early in the migration, we identified key pilot services that had large opportunities for impact and an appetite to invest in migrating to a new build system. For example, Viaduct had complex build logic leading to slow builds and reliability issues. In addition, many developers contributed to the service, so improving their builds had a large impact on Airbnb’s developer experience.

Partnering with pilot teams was incredibly valuable. They were early adopters and made significant contributions in the form of reporting and debugging issues, profiling performance bottlenecks, and suggesting features. The pilot teams also became advocates and provided internal support, helping motivate the rest of the Airbnb developer community.

Dangers of premature optimization

This migration took 4.5 years. With the benefit of hindsight we think we could have drastically improved the migration timeline if we had migrated first, before improving.

Although increasing build granularity improved build times, it increased the time to migrate. Specifically, increased build granularity greatly increased the number of configuration files, making it much harder to manage configurations manually. This forced more functionality into automated build file generation, which increased its complexity.

If we had migrated first and then optimized the build granularity, we believe we could have migrated sooner, enabling users to get benefits from Bazel sooner and reducing the time spent maintaining two co-existing builds.

Similarly, build granularity also made it harder to match deploy `.jar`s between Gradle and Bazel. This led to spending more time testing deployments and fixing runtime issues.

On a more positive note, we accelerated the migration by deciding to support multiple third-party library versions and implementing version resolution. This enabled us to sync versions from Gradle to Bazel, which fixed a large number of build and runtime issues.

Towards the end, one major takeaway in the migration was, by default, we should try to imitate what was there before. In our case, deviating from Gradle usually added technical risk, and should be carefully considered, especially its downstream consequences.

As engineers, we often want to improve things. However, during migrations, improvements can have non-obvious consequences and potentially significantly slow down the migration.

🏁 Conclusion

After 4.5 years, we fully migrated Airbnb’s biggest repo from Gradle to Bazel, achieving:

• Build CSAT: 38% → 68%
• 3–5x faster local build and test times
• 2–3x faster IntelliJ syncs
• 2–3x faster deploys to the development environment

Finally, now that several Airbnb repos are on Bazel, we’re able to share common infrastructure such as remote build caching, remote build execution, affected targets calculation, and more.

Interested in helping us solve problems like these at Airbnb? Learn more about our open engineering roles here.

Introduction

Traditionally, organizations have deployed databases on costly, high-end standalone servers using sharding for scaling as a strategy. As data demands grew, the limitations of this strategy became increasingly evident with increasingly longer and more complex maintenance projects.

Increasingly distributed horizontally scalable databases are not uncommon and many of them are open source. However, running these databases reliably in the cloud with high availability, low latency and scalability, all at a reasonable cost is a problem many companies are trying to solve.

We chose an innovative strategy of deploying a distributed database cluster across multiple Kubernetes clusters in a cloud environment. Although currently an uncommon design pattern due to its complexity, this strategy allowed us to achieve target system reliability and operability.

In this post, we’ll share how we overcame challenges and the best practices we’ve developed for this strategy and we believe these best practices should be applicable to any other strongly consistent, distributed storage systems.

Managing Databases on Kubernetes

Earlier this year, we integrated an open source horizontally scalable, distributed SQL database into our infrastructure.

While Kubernetes is a great tool for running stateless services, the use of Kubernetes for stateful services — like databases — is challenging, particularly around node replacement and upgrades.

Since Kubernetes lacks knowledge of data distribution across nodes, each node replacement requires careful data handling to prevent data quorum loss and service disruption, this includes copying the data before replacing a node.

At Airbnb, we opted to attach storage volumes to nodes using AWS EBS, this allows quick volume reattachment to new virtual machines upon node replacement. Thanks to Kubernetes’ Persistent Volume Claims (PVC), this reattachment happens automatically. In addition we need to allow time for a new storage node to catch up with the cluster’s current state before moving to the next node replacement. For this, we rely on the custom k8s operator, which allows us to customize various Kubernetes operations according to specifics of the application.

Coordinating Node Replacement

Node replacements occur for various reasons, from AWS instance retirement to Kubernetes upgrades or configuration changes. To address these cases, we categorize node replacement events into three groups:

1. Database-initiated events: Such as config changes or version upgrades.
2. Proactive infrastructure events: Like instance retirements or node upgrades.
3. Unplanned infrastructure failures: Such as a node becoming unresponsive.

To safely manage node replacements for database-initiated events, we implemented a a custom check in the k8s-operator that verifies that all nodes are up and running before deleting any pod.

In order to serialize it with the second group initiated by infrastructure, we implemented an admission hook in k8s to intercept pod eviction. This admission hook rejects any attempt to evict the pod, but assigns a custom annotation on the pod which our customer database k8s-operator watches and acts on to safely delete the pod serializing it with any database-initiated node replacements described above.

Node replacements due to unplanned infrastructure failure events like hardware failure, can’t be coordinated. But we can still improve availability by ensuring that any node replacement event from the first two groups will be blocked until the failed hardware is replaced.

In our infrastructure the k8s operator handles both proactive and infrastructure-triggered node replacements, maintaining data consistency in the presence of node replacements and ensuring that unplanned events don’t impact ongoing maintenance.

Kubernetes Upgrades

Regular Kubernetes upgrades are essential but can be high-risk operations, especially for databases. Cloud managed Kubernetes might not offer rollbacks once the control plane is upgraded, posing a potential disaster recovery challenge if something goes wrong. While our approach involves using self-managed Kubernetes clusters, which does allow rolling back the control plane, a bad Kubernetes upgrade could still cause service disruption till rollback is completed.

Ensuring Fault Tolerance with Multiple Kubernetes clusters

At Airbnb, we think the best way to achieve high regional availability is to deploy each database across three independent Kubernetes clusters, each within a different AWS availability zone (AZ). AWS uses availability zones not just for independent power, networking, and connectivity, but they also do rollouts zone by zone. Our Kubernetes cluster alignment with AWS AZ also means that any underlying infrastructure issues or bad deployments have a limited blast radius as they are restricted to a single AZ. Internally, we also deploy a new configuration or a new database version to a part of the logical cluster running in a single Kubernetes cluster in one AZ first.

While this setup adds complexity, it significantly boosts availability by limiting the blast radius of any issues stemming from faulty deployments at every layer — whether database, Kubernetes, or AWS infrastructure.

For instance, recently, a faulty config deployment in our infrastructure abruptly terminated all VMs of a specific type in our staging Kubernetes cluster, deleting most of the query layer pods. However, since the disruption was isolated to a single Kubernetes cluster, two-thirds of our query layer nodes remained operational, preventing any impact.

We also overprovision our database clusters to ensure that, even if an entire AZ, Kubernetes cluster, or all storage nodes within a zone goes down, we still have sufficient capacity to handle traffic.

Leveraging AWS EBS for Reliability and Latency Handling

EBS offers two key benefits for our deployment: rapid reattachment during node replacements and superior durability compared to local disks. With EBS, we confidently run a highly available cluster using only three replicas, maintaining reliability without needing additional redundancy.

However, EBS can occasionally experience tail latency spikes, with p99 latency reaching up to 1 second. To mitigate this, we implemented a storage read timeout session variable, allowing queries to transparently retry against other storage nodes during EBS latency spikes. By default the database we use sends all requests and retries to the leader. To enable retries on storage nodes with healthy EBS, we have to allow reads from both leader and replica reads, but prefer the closest one for the original request. This brings the added benefit of reduced latency and no cross-AZ network costs, as we have a replica in each AZ. Finally, for use cases that permit it, we leverage stale reads feature, enabling reads to be served independently by the replica without requiring synchronous calls to the leader, which may be experiencing an EBS latency spike at the time of the read.

Conclusion: Exploring Open Source Databases on Kubernetes

Our journey running a distributed database on Kubernetes has empowered us to achieve high availability, low latency, scalability, and lower maintenance costs. By leveraging the operator pattern, multi-cluster deployments, AWS EBS, and stale reads, we’ve demonstrated that even open source distributed storage systems can thrive in cloud environments.

We already operate several database clusters in production in the described setup, with the largest one handling 3M QPS across 150 storage nodes, storing over 300+ TB of data spread across 4M internal shards. All this with 99.95% availability thanks to techniques described in this post.

For other companies considering to run open-source databases on Kubernetes, the opportunities are immense. Embrace the challenge, run open-source databases to shape these tools for enterprise use. The future of scalable, reliable data management in the cloud lies in collaboration and open-source innovation — now is the time to lead and participate.

Acknowledgments

Thanks to Abhishek Parmar, Brian Wolfe, Chen Ding, Daniel Low, Hao Luo, Xiaomou Wang for collaboration and Shylaja Ramachandra for editing.

New techniques we’re using at Airbnb to improve and maintain performance of SwiftUI features at scale

By Cal Stephens, Miguel Jimenez

Airbnb first adopted SwiftUI in 2022, starting with individual components and later expanding to entire screens and features. We’ve seen major improvements to engineers’ productivity thanks to its declarative, flexible, and composable architecture. However, adopting SwiftUI has brought new challenges related to performance. For example, there are many common code patterns in SwiftUI that can be inefficient, and many small papercuts can add up to a large cumulative performance hit. To begin addressing some of these issues at scale, we’ve created new tooling for proactively identifying these cases and statically validating correctness.

SwiftUI feature architecture at Airbnb

We’ve been leveraging declarative UI patterns at Airbnb for many years, using our UIKit-based Epoxy library and unidirectional data flow systems. When adopting SwiftUI in our screen layer, we decided to continue using our existing unidirectional data flow library. This simplified the process of incrementally adopting SwiftUI within our large codebase, and we find it improves the quality and maintainability of features.

However, we noticed that SwiftUI features using our unidirectional data flow library didn’t perform as well as we expected, and it wasn’t immediately obvious to us what the problem was. Understanding SwiftUI’s performance characteristics is an important requirement for building performant and outside of the “standard” SwiftUI toolbox.

Understanding SwiftUI view diffing

When working with declarative UI systems like SwiftUI, it’s important to ensure the framework knows which views need to be re-evaluated and re-rendered when the state of the screen changes. Changes are detected by diffing the view’s stored properties any time its parent is updated. Ideally the view’s body will only be re-evaluated when its properties actually change:

However, this behavior is not always the reality (more on why in a moment). Unnecessary view body evaluations hurt performance by performing unnecessary work.

How do you know how often a view’s body is re-evaluated in a real app? An easy way to visualize this is with a modifier that applies a random color to the view every time it’s rendered. When testing this on various views in our app’s most performance-sensitive screens, we quickly found that many views were re-evaluated and re-rendered more often than necessary:

The SwiftUI view diffing algorithm

SwiftUI’s built-in diffing algorithm is often overlooked and not officially documented, but it has a huge impact on performance. To determine if a view’s body needs to be re-evaluated, SwiftUI uses a reflection-based diffing algorithm to compare each of the view’s stored properties:

1. If a type is Equatable, SwiftUI compares the old and new values using the type’s Equatable conformance. Otherwise:
2. SwiftUI compares value types (e.g., structs) by recursively comparing each instance property.
3. SwiftUI compares reference types (e.g., classes) using reference identity.
4. SwiftUI attempts to compare closures by identity. However, most non-trivial closures cannot be compared reliably.

If all of the view’s properties compare as equal to the previous value, then the body isn’t re-evalulated and the content isn’t re-rendered. Values using SwiftUI property wrappers like @State and @Environment don’t participate in this diffing algorithm, and instead trigger view updates through different mechanisms.

When reviewing different views in our codebase, we found several common patterns that confounded SwiftUI’s diffing algorithm:

1. Some types are inherently not supported, like closures.
2. Simple data types stored on the view may be unexpectedly compared by reference instead of by value.

Here’s an example SwiftUI view with properties that interact poorly with the diffing algorithm:

struct MyView: View {
  /// A generated data model that is a struct with value semantics,
  /// but is copy-on-write and wraps an internal reference type.
  /// Compared by reference, not by value, which could cause unwanted body evaluations.
  let dataModel: CopyOnWriteDataModel/// Other miscellaneous properties used by the view. Typically structs, but sometimes a class.
  /// Unexpected comparisons by reference could cause unwanted body evaluations.
  let requestState: MyFeatureRequestState/// An action handler for this view, part of our unidirectional data flow library. 
  /// Wraps a closure that routes the action to the screen's action handler.
  /// Closures almost always compare as not-equal, and typically cause unwanted body evaluations. 
  let handler: Handler<MyViewAction>var body: some View { ... }
}

If a view contains any value that isn’t diffable, the entire view becomes non-diffable. Preventing this in a scalable way is almost impossible with existing tools. This finding also reveals the performance issue caused by our unidirectional data flow library: action handling is closure-based, but SwiftUI can’t diff closures!

In some cases, like with the action handlers from our unidirectional data flow library, making the value diffable would require large, invasive, and potentially undesirable architecture changes. Even in simpler cases, this process is still time consuming, and there’s no easy way to prevent a regression from creeping in later on. This is a big obstacle when trying to improve and maintain performance at scale in large codebases with many different contributors.

Controlling SwiftUI view diffing

Fortunately, we have another option: If a view conforms to Equatable, SwiftUI will diff it using its Equatable conformance instead of using the default reflection-based diffing algorithm.

The advantage of this approach is that it lets us selectively decide which properties should be compared when diffing our view. In our case, we know that the handler object doesn’t affect the content or identity of our view. We only want our view to be re-evalulated and re-rendered when the dataModel and requestState values are updated. We can express that with a custom Equatable implementation:

// An Equatable conformance that makes the above SwiftUI view diffable.
extension MyView: Equatable {
  static func ==(lhs: MyView, rhs: MyView) -> Bool {
    lhs.dataModel == rhs.dataModel
      && lhs.requestState == rhs.requestState
      // Intentionally not comparing handler, which isn't Equatable.
  }
}

However:

1. This is a lot of additional boilerplate for engineers to write, especially for views with lots of properties.
2. Writing and maintaining a custom conformance is error-prone. You can easily forget to update the Equatable conformance when adding new properties later, which would cause bugs.

So, instead of manually writing and maintaining Equatable conformances, we created a new @Equatable macro that generates conformances for us.

// A sample SwiftUI view that has adopted @Equatable
// and is now guaranteed to be diffable.
@Equatable
struct MyView: View {
  // Simple data types must be Equatable, or the build will fail.
  let dataModel: CopyOnWriteDataModel
  let requestState: MyFeatureRequestState// Types that aren't Equatable can be excluded from the
  // generated Equatable conformance using @SkipEquatable,
  // as long as they don’t affect the output of the view body.
  @SkipEquatable let handler: Handler<MyViewAction>var body: some View { ... }
}

The @Equatable macro generates an Equatable implementation that compares all of the view’s stored instance properties, excluding properties with SwiftUI property wrappers like@State and @Environemnt that trigger view updates through other mechanisms. Properties that aren’t Equatable and don’t affect the output of the view body can be marked with @SkipEquatable to exclude them from the generated implementation. This allows us to continue using the closure-based action handlers from our unidirectional data flow library without impacting the SwiftUI diffing process!

After adopting the @Equatable macro on a view, that view is guaranteed to be diffable. If an engineer adds a non-Equatable property later, the build will fail, highlighting a potential regression in the diffing behavior. This effectively makes the @Equatable macro a sophisticated linter — which is really valuable for scaling these performance improvements in a codebase with many components and many contributors, since it makes it less likely for regressions to slip in later.

Managing the size of view bodies

Another essential aspect of SwiftUI diffing is understanding that SwiftUI can only diff proper View structs. Any other code, such as computed properties or helper functions that generate a SwiftUI view, cannot be diffed.

Consider the following example:

// Complex SwiftUI views are often simplified by
// splitting the view body into separate computed properties.
struct MyScreen: View {
  /// The unidirectional data flow state store for this feature.
  @ObservedObject var store: StateStore<MyState, MyAction>var body: some View {
    VStack {
      headerSection
      actionCardSection
    }
  }private var headerSection: some View {
    Text(store.state.titleString)
      .textStyle(.title)
  }private var actionCardSection: some View {
    VStack {
      Image(store.state.cardSelected ? "enabled" : "disabled")
      Text("This is a selectable card")
    }
    .strokedCard(.roundedRect_mediumCornerRadius_12)
    .scaleEffectButton(action: {
      store.handle(.cardTapped) 
    })
  }
}

This is a common way to organize complex view bodies, since it makes the code easier to read and maintain. However, at runtime, SwiftUI effectively inlines the views returned from the properties into the main view body, as if we instead wrote:

// At runtime, computed properties are no different
// from just having a single, large view body!
struct MyScreen: View {
  @ObservedObject var store: StateStore<MyState, MyAction>// Re-evaluated every time the state of the screen is updated.
  var body: some View {
    Text(store.state.titleString)
      .textStyle(.title)VStack {
      Image(store.state.cardSelected ? "enabled" : "disabled")
      Text("This is a selectable card")
    }
    .strokedCard(.roundedRect_mediumCornerRadius_12)
    .scaleEffectButton(action: {
      store.handle(.cardTapped) 
    })
  }
}

Since all of this code is part of the same view body, all of it will be re-evaluated when any part of the screen’s state changes. While this specific example is simple, as the view grows larger and more complicated, re-evaluating it will become more expensive. Eventually there would be a large amount of unnecessary work happening on every screen update, hurting performance.

To improve performance, we can implement the layout code in separate SwiftUI views. This allows SwiftUI to properly diff each child view, only re-evaluating their bodies when necessary:

struct MyScreen: View {
  @ObservedObject var store: StateStore<MyState, MyAction>var body: some View {
    VStack {
      HeaderSection(title: store.state.titleString)
      CardSection(
       isCardSelected: store.state.isCardSelected,
       handler: store.handler)
    }
  }
}/// Only re-evaluated and re-rendered when the title property changes.
@Equatable
struct HeaderSection: View {
  let title: Stringvar body: some View {
    Text(title)
      .textStyle(.title)
  }
}/// Only re-evaluated and re-rendered when the isCardSelected property changes.
@Equatable
struct CardSection: View {
  let isCardSelected: Bool
  @SkipEquatable let handler: Handler<MyAction>var body: some View {
    VStack {
      Image(store.state.isCardSelected ? "enabled" : "disabled")
      Text("This is a selectable card")
    }
    .strokedCard(.roundedRect_mediumCornerRadius_12)
    .scaleEffectButton(action: {
      handler.handle(.cardTapped) 
    })
  }
}

By breaking the view into smaller, diffable pieces, SwiftUI can efficiently update only the parts of the view that actually changed. This approach helps maintain performance as a feature grows more complex.

View body complexity lint rule

Large, complex views aren’t always obvious during development. Easily available metrics like total line count aren’t a good proxy for complexity. To help engineers know when it’s time to refactor a view into smaller, diffable pieces, we created a custom SwiftLint rule that parses the view body using SwiftSyntax and measures its complexity. We defined the view complexity metric as a value that increases every time you compose views using computed properties, functions, or closures. With this rule we automatically trigger an alert in Xcode when a view is getting too complex. (The complexity limit is configurable, and we currently allow a maximum complexity level of 10.)

Conclusion

With an understanding of how SwiftUI view diffing works, we can use an @Equatable macro to ensure view bodies are only re-evaluated when the values inside views actually change, break views into smaller parts for faster re-evaluation, and encourage developers to refactor views before they get too large and complex.

Applying these three techniques to SwiftUI views in our app has led to a large reduction in unnecessary view re-evaluation and re-renders. Revisiting the examples from earlier, you see far fewer re-renders in the search bar and filter panel:

Using results from our page performance score system, we’ve found that adopting these techniques in our most complicated SwiftUI screens really does improve performance for our users. For example, we reduced scroll hitches by15% on our main Search screen by adopting @Equatable on its most important views, and breaking apart large view bodies into smaller diffable pieces. These techniques also give us the flexibility to use a feature architecture that best suits our needs without compromising performance or imposing burdensome limitations (e.g., completely avoiding closures in SwiftUI views).

Of course, these techniques aren’t a silver bullet. It’s not necessary for all SwiftUI features to use them, and these techniques by themselves aren’t enough to guarantee great performance. However, understanding how and why they work serves as a valuable foundation for building performant SwiftUI features, and makes it easier to spot and avoid problematic patterns in your own code.

If you’re interested in joining us on our quest to make the best iOS app in the App Store, please see our careers page for open iOS roles.

Comprehensive Load Testing with Load Generator, Dependency Mocker, Traffic Collector, and More

Authors: Chenhao Yang, Haoyue Wang, Xiaoya Wei, Zay Guan, Yaolin Chen and Fei Yuan

System-level load testing is crucial for reliability and efficiency. It identifies bottlenecks, evaluates capacity for peak traffic, establishes performance baselines, and detects errors. At a company of Airbnb’s size and complexity, we’ve learned that load testing needs to be robust, flexible, and decentralized. This requires the right set of tools to enable engineering teams to do self-service load tests that integrate seamlessly with CI.

Impulse is one of our internal load-testing-as-a-service frameworks. It provides tools that can generate synthetic loads, mock dependencies, and collect traffic data from production environments. In this blog post, we’ll share how Impulse is architected to minimize manual effort, seamlessly integrate with our observability stack, and empower teams to proactively address potential issues.

Architecture

Impulse is a comprehensive load testing framework that allows service owners to conduct context-aware load tests, mock dependencies, and collect traffic data to ensure the system’s performance under various conditions. It includes the following components:

1. Load generator to generate context-aware requests on the fly, for testing different scenarios with synthetic or collected traffic.
2. Dependency mocker to mock the downstream responses with latency, so that the load testing on the service under test (SUT) doesn’t need to involve certain dependent services. This is especially crucial when the dependencies are vendor services that don’t support load testing, or if the team wants to regression load test their service during day-to-day deployment without affecting downstreams.
3. Traffic collector to collect both the upstream and downstream traffic from the production environment, and then apply the resulting data to the test environment.
4. Testing API generator to wrap asynchronous workflows into synchronous API calls for load testing.

Each of these four tools are independent, allowing service owners the flexibility to select one or more components for their load testing needs.

Load generator

Context aware

When load testing, requests made to the SUT often require some information from the previous response or need to be sent in a specific order. For example, if an update API needs to provide an entity_id to update, we must ensure the entity already exists in the testing environment context.

Our load generator tool allows users to write arbitrary testing logic in Java or Kotlin and launch containers to run these tests at scale against the SUT. Why write code instead of DSL/configuration logic?

• Flexibility: Programming languages are more expressive than DSL and can better support complex contextual scenarios.
• Reusability: The same testing code can be used in other tests, e.g., integration tests.
• Developer proficiency: Low/no learning curve to onboard, don’t need to learn how to write testing logic.
• Developer experience: IDE support, testing, debugging, etc.

Here is an example of synthetic context-aware test case:

class HelloWorldLoadGenerator : LoadGenerator {
   override suspend fun run() {
       val createdEntity = sutApiClient.create(CreateRequest(name="foo", ...)).data// request with id from previous response (context)
       val updateResponse = sutApiClient.update(UpdateRequest(id=createdEntity.id, name="bar"))// ... other operations// clean up
       sutApiClient.delete(DeleteRequest(id=createdEntity.id))
   }
}

Decentralized

The load generator is decentralized and containerized, which means each time a load test is triggered, a set of new containers will be created to run the test. This design has several benefits:

• Isolation: Load testing runs between different services are isolated from each other, eliminating any interference.
• Scalability: The number of containers can be scaled up or down according to the traffic requirements.
• Cost efficiency: The containers are short-lived, as they only exist during the load testing run.

What’s more, as our services are cloud based, a subtle point is that the Impulse framework will evenly distribute the workers among all our data centers, and the load will be emitted evenly from all the workers. Impulse’s load generator ensures the overall trigger per second (TPS) is as configured. Based on this, we can better leverage the locality settings in load balancers, which can better mimic the real traffic distribution in production.

Execution

The load generator is designed to be executed in the CI/CD pipeline, which means we can trigger load testing automatically. Developers can configure the testing spec in multiple phases, e.g., a warm up phase, a steady state phase, a peak phase, etc. Each phase can be configured with:

• Test cases to run
• TPS (trigger per second) of each test case
• Test duration

Dependency mocker

Impulse is a decentralized framework where each service has its own dependency mocker. This can eliminate interference between services and reduce communication costs. Each dependency mocker is an out-of-process service, which means the SUT behaves just as it does in production. We run the mockers in separate instances to avoid any impact on the performance of the SUT. The mock servers are all short lived — they only start before tests run and shut down afterwards to save costs and maintenance effort. The response latency and exceptions are configurable and the number of mocker instances can be adjusted on demand to support large amounts of traffic.

Other noteworthy features:

• You can selectively stub some of the dependencies. Currently, stubbing is supported for HTTP JSON, Airbnb Thrift, and Airbnb GraphQL dependencies.
• The dependency mockers support use cases beyond load testing. For instance, integration tests often rely on other services or third-party API calls, which may not guarantee a stable testing environment or might only support ideal scenarios. Dependency mockers can address this by offering predefined responses or exceptions to fully test those flows.

Impulse supports two options for generating mock responses:

1. Synthetic response: The response is generated by user logic, as in integration testing; the difference is that the response comes from a remote (out-of-process) server with simulated latency.
   - Similar to the load generator, the logic is written in Java/Kotlin code and contains request matching and response generation.
   - Latency can be simulated using p95/p99 metrics.
2. Replay response: The response is replayed from the production downstream recording, supported by the traffic collector component.

Here is an example of a synthetic response with latency in Kotlin:

downstreamsMocking.every(
      thriftRequest<FooRequest>().having { it.message == "hello" }
    ).returns { request ->
      ThriftDownstream.Response.thriftEncoded(
        HttpStatus.OK,
        FooResponse.builder.reply("${request.message} world").build()
      )
    }.with {
      delay = latencyFromP95(p95=500.miliseconds, min=200.miliseconds, max=2000.miliseconds)
    }

Traffic collector

The traffic collector component is designed to capture both upstream and downstream traffic, along with the relationships between them. This approach allows Impulse to accurately replay production traffic during load testing, avoiding inconsistencies in downstream data or behavior. By replicating downstream responses — including production-like latency and errors — via the dependency mocker, the system ensures high-fidelity load testing. As a result, services in the testing environment behave identically to those in production, enabling more realistic and reliable performance evaluations.

Testing API generator

We rely heavily on event-driven, asynchronous workflows that are critical to our business operations. These include processing events from a message queue (MQ) and executing delayed jobs. Most of the MQ events/jobs are emitted from synchronous flows (e.g., API calls), so theoretically they can be covered by API load testing. However, the real world is more complex. These asynchronous flows often involve long chains of event and job emissions originating from various sources, making it difficult to replicate and test them accurately using only API-based methods.

To address this, the testing API generator component creates HTTP APIs during the CI stage according to the event or job schema. These APIs act as wrappers around the underlying asynchronous flows and are registered exclusively in the testing environment. This setup enables load testing tools — such as load generators — to send traffic to these synthetic APIs, allowing asynchronous flows to be exercised as if they were synchronous. As a result, it’s possible to perform targeted, realistic load testing on asynchronous logic that would otherwise be hard to simulate.

The goal of the testing API generator is to help developers identify performance bottlenecks and potential issues in their async flow implementations and under high traffic conditions. It does this by enabling direct load testing of async flows without involving middleware components like MQs. The rationale is that developers typically aim to evaluate the behavior of their own logic, not the middleware, which is usually already well-tested. By bypassing these components, this approach simplifies the load testing process and empowers developers to independently manage and execute their own tests.

Integration with other testing frameworks

Airbnb emphasizes product quality, utilizing versatile testing frameworks that cover integration and API tests across development, staging, and production environments, and integrate smoothly into CI/CD pipelines. The modular design of Impulse facilitates its integration with these frameworks, offering systematic service testing.

Conclusion

In this blog post, we shared how Impulse and its four core components help developers perform self-service load testing at Airbnb. As of this writing, Impulse has been implemented in several customer support backend services and is currently under review with different teams across the company who are planning to leverage Impulse to conduct load testing.

We’ve received a lot of good feedback in the process. For example: “Impulse helps us to identify and address potential issues in our service. During testing, it detected an ApiClientThreadToolExhaustionException caused by thread pool pressure. Additionally, it alerted us about occasional timeout errors in client API calls during service deployments. Impulse helped us identify high memory usage in the main service container, enabling us to fine-tune the memory allocation and optimize our service’s resource usage. Highly recommend utilizing Impulse as an integral part of the development and testing processes.”

Acknowledgments

Thanks to Jeremy Werner, Yashar Mehdad, Raj Rajagopal, Claire Cheng, Tim L., Wei Ji, Jay Wu, Brian Wallace for support on the Impulse project.

Does this type of work interest you? Check out our open roles here.

A look into how Airbnb uses speech recognition, intent detection, and language models to understand users and assist agents more effectively.

By Yuanpei Cao, Heng Ji, Elaine Liu, Peng Wang, and Tiantian Zhang

At Airbnb, we aim to provide a smooth, intuitive, and helpful community support experience, whether it’s helping a guest navigate a booking change or helping a host with a listing issue. While our Help Center and customer support chatbot helps resolve many inquiries efficiently, some users prefer the immediacy of a voice conversation with a support representative. To make these interactions faster and more effective, we’ve significantly improved our Interactive Voice Response (IVR) system via machine learning.

Over the years, Airbnb has invested in conversational AI to enhance customer support. In our previous blog posts Task-Oriented Conversational AI in Airbnb Customer Support and Using Chatbots to Provide Faster COVID-19 Community Support, we explored how AI-driven chatbots streamline guest and host interactions through automated messaging. This post explains how we extend that work to voice-based support, leveraging machine learning to improve real-time phone interactions with our intelligent IVR system.

We’ll take you through the end-to-end IVR journey, the key machine learning components that power it, and how we designed a system that delivers faster, more human-like, and more intuitive voice support for our community.

Reimagining the voice support journey

Traditional IVR systems often rely on rigid menu trees, requiring callers to press buttons and navigate pre-set paths. Instead, we designed an adaptive, conversational IVR that listens, understands, and responds in real time. Here’s normally what happens when a caller reaches out to Airbnb support:

1. Call and greeting: IVR picks up and prompts, “In a few sentences, please tell us why you’re calling today.”
2. Automated speech recognition (ASR): The caller’s response is transcribed with Airbnb-specific ASR. For example, if a caller says, “I need to request a refund for my reservation,” ASR accurately converts this speech into text, preserving key domain-specific terms.
3. Understanding intent: A Contact Reason Detection model classifies the issue into a category like cancellations, refunds, account issues, etc.
4. Decision-making: If self-service is possible, the system retrieves and sends a relevant help article or an intelligent workflow via SMS or app notification. If the caller explicitly requests agent support or the issue requires human intervention, the call is routed to a customer support agent with relevant details attached.
5. Clarifying response: A Paraphrasing model generates a summary of the user intent, which IVR shares with the user before delivering the solution. This ensures that users understand the context of the resource they receive. Continuing our example, the system would respond, “I understand your issue is regarding a refund request. We have sent you a link to resources about this topic. Follow the instructions to find answers. If you need to speak with an agent, press 0 to be connected to our customer service representative.” The underscored Paraphrasing component enhances engagement by bridging the gap between system-generated responses and user comprehension, making the self-service experience more intuitive.
6. Resolution or escalation: The caller receives an SMS or app notification with a direct link to a relevant Airbnb Help Center article. If further assistance is needed, they can press 0 to connect with a customer service representative.

By moving away from rigid menus to natural language understanding, we allow guests and hosts to express their issues in their own words, helping to increase satisfaction and resolution efficiency.

Breaking down our ML-powered IVR system

1. Automated speech recognition (ASR): transcribing with precision

In a voice-driven support system, achieving high transcription accuracy is essential, particularly in noisy phone environments where speech can be unclear. General speech recognition models often struggle with Airbnb-specific terminology, leading to errors like misinterpreting “listing” as “lifting” or “help with my stay” as “happy Christmas Day.” These inaccuracies create challenges in understanding user intent and impact downstream processes.

To enhance ASR accuracy, we transitioned from a generic high-quality pretrained model to one specifically adapted for noisy phone audio. Additionally, we introduced a domain-specific phrase list optimization that ensures Airbnb terms are properly recognized. Based on a sample of hundreds of clips, this significantly reduced the word error rate (WER) from 33% to approximately 10%. The reduced WER significantly enhanced the accuracy of downstream help article recommendations, increasing user engagement, improving customer NPS among users who interacted with the ASR menu, while reducing reliance on human agents and lowering customer service handling time.

2. Contact Reason prediction: understanding the why

After transcribing the caller’s statements, the next step involves identifying their intent. We accomplished this by creating a detailed Contact Reason taxonomy that categorizes all potential Airbnb inquiries, as elaborated in “T-LEAF: Taxonomy Learning and EvaluAtion Framework.” We then use an intent detection model to classify calls into a Contact Reason category, ensuring each inquiry is handled appropriately. For example, if a caller mentions “I haven’t received my refund yet,” the model predicts the Contact Reason as Missing Refund and forwards it to the relevant downstream components.

In production, we deploy the Issue Detection Service to host the intent detection models, running them in parallel to achieve optimal scalability, flexibility, and efficiency. Parallel computing ensures that intent detection latency remains under 50ms on average, making the process imperceptible to IVR users and ensuring a seamless real-time experience. The detected intent is then analyzed within the IVR workflow to determine the next action, whether it’s guiding the user through a self-service resolution or escalating directly to a human agent.

Occasionally, callers prefer to speak directly with a human agent instead of describing their issues, using terms like “agent” or “escalation.” For such scenarios, we use a different intent detection model to recognize when a caller wants to escalate to a human agent. If this intent is detected, the IVR system honors the caller’s request and routes the call to the suitable support team.

3. Help article retrieval: delivering the right information

Many common Airbnb issues can be quickly resolved by providing clear and relevant educational information. To help provide useful information to users and minimize the need for human customer support, we use the Help Article Retrieval and Ranking system. This advanced system automatically identifies the issue in a user’s inquiry and delivers the most relevant help article link via SMS text message and Airbnb app notification. Our process incorporates two machine learning stages.

Semantic retrieval and ranking: We index Airbnb Help Article embeddings into a vector database, enabling efficient retrieval of up to 30 relevant articles per user query using cosine similarity, typically within 60ms. An LLM-based ranking model then re-ranks these retrieved articles, with the top-ranked article directly presented to users via IVR channels. This dual-stage system not only powers IVR interactions but also supports our customer support chatbot and Help Center search. Across these platforms, its effectiveness is continuously evaluated using metrics like Precision@N, facilitating ongoing improvements and refinements.

4. Paraphrasing model: enhancing user understanding

A key challenge in IVR-based customer support is ensuring users clearly understand the resolution before receiving help article links, as they typically lack visibility into the article’s contents or title. To address this, we implemented a lightweight paraphrasing approach leveraging a curated set of standardized summaries.

UX writers created concise and clear paraphrases for common Airbnb scenarios. During online serving, user inquiries are mapped to these curated summaries via nearest-neighbor matching based on text embedding similarity. We calibrated a similarity threshold to ensure high-quality matches. Manual evaluation of end-to-end model outputs confirmed precision exceeding 90%.

The outcome was a finite-state solution delivering the most appropriate paraphrased IVR prompt before presenting a help article link. For example, if a caller states, “I need to cancel my reservation and request a refund,” the model generates a response like “I understand your issue is about a refund request” before sending the retrieved help article link.

Integrating this model ensures users receive clear, contextually relevant summaries prior to accessing help articles. In an experiment targeting English hosts who contacted customer support, we found that presenting a paraphrased summary before sending the article link increases user engagement with article content, resulting in improvement in self-resolution rates, helping to reduce the need for direct customer support assistance.

Conclusion

By combining Automated Speech Recognition and Contact Reason Detection systems with a help article retrieval system, and a paraphrasing model, we have created an IVR system that streamlines support interactions and improves user satisfaction. Our solution enables callers to describe issues naturally, reduces dependency on human agents for common inquiries, and provides instant, relevant support through self-service. When human assistance is necessary, the system ensures a smooth transition by routing users to the right agent with essential context.

Interested in working at Airbnb? Check out our open roles.

Acknowledgements

Thanks to Zhenyu Zhao, Mia Zhao, Wayne Zhang, Lucca Siaudzionis, Lulu Chen, Sukey Xu, Floria Wan, Michael Zhou, Can Yang, Yaolin Chen, Shuaihu Wang, Huifan Qu, Ming Shang,Yu Jiang, Wanting Chen, Elena Zhao, Shanna Su, Cassie Cao, Hao Wang, Haoran Zhu, Xirui Liu, Ying Tan, Xiaohan Zeng, Xiaoyu Meng, Gavin Li, Gaurav Rai, Hemanth Kolla, Ihor Hordiienko, Matheus Scharf, and Stepan Sydoruk who helped bring this vision to life. Also thanks to Paige Schwartz, Stephanie Chu, Neal Cohen, Becky Ajuonuma, Iman Saleh, Dani Normanm, Javier Salido, and Lauren Mackevich for the review and editing.

Thanks to Jeremy Werner, Joy Zhang, Claire Cheng, Yashar Mehdad, Shuohao Zhang, Shawn Yan, Kelvin Xiong, Michael Lubavin, Teng Wang, Wei Ji, and Chenhao Yang’s leadership support on building conversational AI products at Airbnb.

Our journey in applying embedding-based retrieval techniques to build an accurate and scalable candidate retrieval system for Airbnb Homes search

Authors: Mustafa (Moose) Abdool, Soumyadip Banerjee, Karen Ouyang, Do-Kyum Kim, Moutupsi Paul, Xiaowei Liu, Bin Xu, Tracy Yu, Hui Gao, Yangbo Zhu, Huiji Gao, Liwei He, Sanjeev Katariya

Introduction

Search plays a crucial role in helping Airbnb guests find the perfect stay. The goal of Airbnb Search is to surface the most relevant listings for each user’s query — but with millions of available homes, that’s no easy task. It’s especially difficult when searches include large geographic areas (like California or France) or high-demand destinations (like Paris or London). Recent innovations — such as flexible date search, which allows guests to explore stays without fixed check-in and check-out dates — have added yet another layer of complexity to ranking and finding the right results.

To tackle these challenges, we need a system that can retrieve relevant homes while also being scalable enough (in terms of latency and compute) to handle queries with a large candidate count. In this blog post, we share our journey in building Airbnb’s first-ever Embedding-Based Retrieval (EBR) search system. The goal of this system is to narrow down the initial set of eligible homes into a smaller pool, which can then be scored by more compute-intensive machine learning models later in the search ranking process.

Figure 1: The general stages and scale for the various types of ranking models used in Airbnb Search

We’ll explore three key challenges in building this EBR system: (1) constructing training data, (2) designing the model architecture, and (3) developing an online serving strategy using Approximate Nearest Neighbor (ANN) solutions.

Training Data Construction

The first step in building our EBR system was training a machine learning model to map both homes and de-identified search queries into numerical vectors. To achieve this, we built a training data pipeline (Figure 3) that leveraged contrastive learning — a strategy that involves identifying pairs of positive- and negative-labeled homes for a given query. During training, the model learns to map a query, a positive home, and a negative home into a numerical vector, such that the similarity between the query and the positive home is much higher than the similarity between the query and the negative home.

To construct these pairs, we devised a sampling method based on user trips. This was an important design decision, since users on Airbnb generally undergo a multi-stage search journey. Data shows that before making a final booking, users tend to perform multiple searches and take various actions — such as clicking into a home’s details, reading reviews, or adding a home to a wishlist. As such, it was crucial to develop a strategy that captures this entire multi-stage journey and accounts for the diverse types of listings a user might explore.

Diving deeper, we first grouped all historical queries of users who made bookings, using key query parameters such as location, number of guests, and length of stay — our definition of a “trip.” For each trip, we analyzed all searches performed by the user, with the final booked listing as the positive label. To construct (positive, negative) pairs, we paired this booked listing with other homes the user had seen but not booked. Negative labels were selected from homes the user encountered in search results, along with those they had interacted with more intentfully — such as by wishlisting — but ultimately did not book. This choice of negative labels was key: Randomly sampling homes made the problem too easy and resulted in poor model performance.

Figure 2: Example of constructing (positive, negative) pairs for a given user journey. The booked home is always treated as a positive. Negatives are selected from homes that appeared in the search result (and were potentially interacted with) but that the user did not end up booking.

Figure 3: Example of overall data pipeline used to construct training data for the EBR model.

Model Architecture

The model architecture followed a traditional two-tower network design. One tower (the listing tower) processes features about the home listing itself — such as historical engagement, amenities, and guest capacity. The other tower (the query tower) processes features related to the search query — such as the geographic search location, number of guests, and length of stay. Together, these towers generate the embeddings for home listings and search queries, respectively.

A key design decision here was choosing features such that the listing tower could be computed offline on a daily basis. This enabled us to pre-compute the home embeddings in a daily batch job, significantly reducing online latency, since only the query tower had to be evaluated in real-time for incoming search requests.

Figure 4: Two-tower architecture as used in the EBR model. Note that the listing tower is computed offline daily for all homes.

Online Serving

The final step in building our EBR system was choosing the infrastructure for online serving. We explored a number of approximate nearest neighbor (ANN) solutions and narrowed them down to two main candidates: inverted file index (IVF) and hierarchical navigable small worlds (HNSW). While HNSW performed slightly better in terms of evaluation metrics — using recall as our main evaluation metric — we ultimately found that IVF offered the best trade-off between speed and performance.

The core reason for this is the high volume of real-time updates per second for Airbnb home listings, as pricing and availability data is frequently updated. This caused the memory footprint of the HNSW index to grow too large. In addition, most Airbnb searches include filters, especially geographic filters. We found that parallel retrieval with HNSW alongside filters resulted in poor latency performance.

In contrast, the IVF solution, where listings are clustered beforehand, only required storing cluster centroids and cluster assignments within our search index. At serving time, we simply retrieve listings from the top clusters by treating the cluster assignments as a standard search filter, making integration with our existing search system quite straightforward.

Figure 5: Overall serving flow using IVF. Homes are clustered beforehand and, during online serving, homes are retrieved from the closest clusters to the query embedding.

In this approach, our choice of similarity function in the EBR model itself ended up having interesting implications. We explored both dot product and Euclidean distance; while both performed similarly from a model perspective, using Euclidean distance produced much more balanced clusters on average. This was a key insight, as the quality of IVF retrieval is highly sensitive to cluster size uniformity: If one cluster had too many homes, it would greatly reduce the discriminative power of our retrieval system.

We hypothesize that this imbalance arises with dot product similarity because it inherently only considers the direction of feature vectors while ignoring their magnitudes — whereas many of our underlying features are based on historical counts, making magnitude an important factor.

Figure 6: Example of the distribution of cluster sizes when using dot product vs. Euclidean distance as a similarity measure. We found that Euclidean distance produced much more balanced cluster sizes.

Results

The EBR system described in this post was fully launched in both Search and Email Marketing production and led to a statistically-significant gain in overall bookings when A/B tested. Notably, the bookings lift from this new retrieval system was on par with some of the largest machine learning improvements to our search ranking in the past two years.

The key improvement over the baseline was that our EBR system effectively incorporated query context, allowing homes to be ranked more accurately during retrieval. This ultimately helped us display more relevant results to users, especially for queries with a high number of eligible results.

Acknowledgments

We would like to especially thank the entire Search and Knowledge Infrastructure & ML Infrastructure org (led by Yi Li) and Marketing Technology org (led by Michael Kinoti) for their great collaborations throughout this project!

by: Charles Covey-Brandt

Airbnb recently completed our first large-scale, LLM-driven code migration, updating nearly 3.5K React component test files from Enzyme to use React Testing Library (RTL) instead. We’d originally estimated this would take 1.5 years of engineering time to do by hand, but — using a combination of frontier models and robust automation — we finished the entire migration in just 6 weeks.

In this blog post, we’ll highlight the unique challenges we faced migrating from Enzyme to RTL, how LLMs excel at solving this particular type of challenge, and how we structured our migration tooling to run an LLM-driven migration at scale.

Background

In 2020, Airbnb adopted React Testing Library (RTL) for all new React component test development, marking our first steps away from Enzyme. Although Enzyme had served us well since 2015, it was designed for earlier versions of React, and the framework’s deep access to component internals no longer aligned with modern React testing practices.

However, because of the fundamental differences between these frameworks, we couldn’t easily swap out one for the other (read more about the differences here). We also couldn’t just delete the Enzyme files, as analysis showed this would create significant gaps in our code coverage. To complete this migration, we needed an automated way to refactor test files from Enzyme to RTL while preserving the intent of the original tests and their code coverage.

How We Did It

In mid-2023, an Airbnb hackathon team demonstrated that large language models could successfully convert hundreds of Enzyme files to RTL in just a few days.

Building on this promising result, in 2024 we developed a scalable pipeline for an LLM-driven migration. We broke the migration into discrete, per-file steps that we could parallelize, and configurable retry loops, and significantly expanded our prompts with additional context. Finally, we performed breadth-first prompt tuning for the long tail of complex files.

1. File Validation and Refactor Steps

We started by breaking down the migration into a series of automated validation and refactor steps. Think of it like a production pipeline: each file moves through stages of validation, and when a check fails, we bring in the LLM to fix it.

We modeled this flow like a state machine, moving the file to the next state only after validation on the previous state passed:

This step-based approach provided a solid foundation for our automation pipeline. It enabled us to track progress, improve failure rates for specific steps, and rerun files or steps when needed. The step-based approach also made it simple to run migrations on hundreds of files concurrently, which was critical for both quickly migrating simple files, and chipping away at the long tail of files later in the migration.

2. Retry Loops & Dynamic Prompting

Early on in the migration, we experimented with different prompt engineering strategies to improve our per-file migration success rate. However, building on the stepped approach, we found the most effective route to improve outcomes was simply brute force: retry steps multiple times until they passed or we reached a limit. We updated our steps to use dynamic prompts for each retry, giving the validation errors and the most recent version of the file to the LLM, and built a loop runner that ran each step up to a configurable number of attempts.

With this simple retry loop, we found we could successfully migrate a large number of our simple-to-medium complexity test files, with some finishing successfully after a few retries, and most by 10 attempts.

3. Increasing the Context

For test files up to a certain complexity, just increasing our retry attempts worked well. However, to handle files with intricate test state setups or excessive indirection, we found the best approach was to push as much relevant context as possible into our prompts.

By the end of the migration, our prompts had expanded to anywhere between 40,000 to 100,000 tokens, pulling in as many as 50 related files, a whole host of manually written few-shot examples, as well as examples of existing, well-written, passing test files from within the same project.

Each prompt included:

• The source code of the component under test
• The test file we were migrating
• Validation failures for the step
• Related tests from the same directory (maintaining team-specific patterns)
• General migration guidelines and common solutions

Here’s how that looked in practice (significantly trimmed down for readability):

// Code example shows a trimmed down version of a prompt 
// including the raw source code from related files, imports, 
// examples, the component source itself, and the test file to migrate.const prompt = [
  'Convert this Enzyme test to React Testing Library:',
  `SIBLING TESTS:\n${siblingTestFilesSourceCode}`,
  `RTL EXAMPLES:\n${reactTestingLibraryExamples}`,
  `IMPORTS:\n${nearestImportSourceCode}`,
  `COMPONENT SOURCE:\n${componentFileSourceCode}`,
  `TEST TO MIGRATE:\n${testFileSourceCode}`,
].join('\n\n');

This rich context approach proved highly effective for these more complex files — the LLM could better understand team-specific patterns, common testing approaches, and the overall architecture of the codebase.

We should note that, although we did some prompt engineering at this step, the main success driver we saw was choosing the right related files (finding nearby files, good example files from the same project, filtering the dependencies for files that were relevant to the component, etc.), rather than getting the prompt engineering perfect.

After building and testing our migration scripts with retries and rich contexts, when we ran our first bulk run, we successfully migrated 75% of our target files in just four hours.

4. From 75% to 97%: Systematic Improvement

That 75% success rate was really exciting to get to, but it still left us with nearly 900 files failing our step-based validation criteria. To tackle this long tail, we needed a systematic way to understand where remaining files were getting stuck and improve our migration scripts to address these issues. We also wanted to do this breadth first to aggressively chip away at our remaining files without getting stuck on the most difficult migration cases.

To do this, we built two features into our migration tooling.

First, we built a simple system to give us visibility into common issues our scripts were facing by stamping files with an automatically-generated comment to record the status of each migration step. Here’s what that code comment looked like:

// MIGRATION STATUS: {"enyzme":"done","jest":{"passed":8,"failed":2,"total":10,"skipped":0,"successRate":80},"eslint":"pending","tsc":"pending",}

And second, we added the ability to easily re-run single files or path patterns, filtered by the specific step they were stuck on:

$ llm-bulk-migration --step=fix-jest --match=project-abc/**

Using these two features, we could quickly run a feedback loop to improve our prompts and tooling:

1. Run all remaining failing files to find common issues the LLM is getting stuck on
2. Select a sample of files (5 to 10) that exemplify a common issue
3. Update our prompts and scripts to address that issue
4. Re-run against the sample of failing files to validate our fix
5. Repeat by running against all remaining files again

After running this “sample, tune, sweep” loop for 4 days, we had pushed our completed files from 75% to 97% of the total files, and had just under 100 files remaining. By this point, we had retried many of these long tail files anywhere between 50 to 100 times, and it seemed we were pushing into a ceiling of what we could fix via automation. Rather than invest in more tuning, we opted to manually fix the remaining files, working from the baseline (failing) refactors to reduce the time to get those files over the finish line.

Results and Impact

With the validation and refactor pipeline, retry loops, and expanded context in place, we were able to automatically migrate 75% of our target files in 4 hours.

After four days of prompt and script refinement using the “sample, tune, and sweep” strategy, we reached 97% of the 3.5K original Enzyme files.

And for the remaining 3% of files that didn’t complete through automation, our scripts provided a great baseline for manual intervention, allowing us to complete the migration for those remaining files in another week of work.

Most importantly, we were able to replace Enzyme while maintaining original test intent and our overall code coverage. And even with high retry counts on the long tail of the migration, the total cost — including LLM API usage and six weeks of engineering time — proved far more efficient than our original manual migration estimate.

What’s Next

This migration underscores the power of LLMs for large-scale code transformation. We plan to expand this approach, develop more sophisticated migration tools, and explore new applications of LLM-powered automation to enhance developer productivity.

Want to help shape the future of developer tools? We’re hiring engineers who love solving complex problems at scale. Check out our careers page to learn more.

****************

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

How Airbnb is adapting ranking for our map interface.

Malay Haldar, Hongwei Zhang, Kedar Bellare Sherry Chen

Search is the core mechanism that connects guests with Hosts at Airbnb. Results from a guest’s search for listings are displayed through two interfaces: (1) as a list of rectangular cards that contain the listing image, price, rating, and other details on it, referred to as list-results and (2) as oval pins on a map showing the listing price, called map-results. Since its inception, the core of the ranking algorithm that powered both these interfaces was the same — ordering listings by their booking probabilities and selecting the top listings for display.

But some of the basic assumptions underlying ranking, built for a world where search results are presented as lists, simply break down for maps.

What Is Different About Maps?

The central concept that drives ranking for list-results is that user attention decays starting from the top of the list, going down towards the bottom. A plot of rank vs click-through rates in Figure 1 illustrates this concept. X-axis represents the rank of listings in search results. Y-axis represents the click-through rate (CTR) for listings at the particular rank.

To maximize the connections between guests and Hosts, the ranking algorithm sorts listings by their booking probabilities based on a number of factors and sequentially assigns their position in the list-results. This often means that the larger a listing’s booking probability, the more attention it receives from searchers.

But in map-results, listings are scattered as pins over an area (see Figure 2). There is no ranked list, and there is no decay of user attention by ranking position. Therefore, for listings that are shown on the map, the strategy of sorting by booking probabilities is no longer applicable.

Uniform User Attention

To adapt ranking to the map interface, we look at new ways of modeling user attention flow across a map. We start with the most straightforward assumption that user attention is spread equally across the map pins. User attention is a very precious commodity and most searchers only click through a few map pins (see Figure 3). A large number of pins on the map means those limited clicks may miss discovering the best options available. Conversely, limiting the number of pins to the topmost choices increases the probability of the searcher finding something suitable, but runs the risk of removing their preferred choice.

We test this hypothesis, controlled by a parameter . The parameter serves as an upper bound on the ratio of the highest booking probability vs the lowest booking probability when selecting the map pins. The bounds set by the parameter controls the booking probability of the listings behind the map pins. The more restricted the bounds, the higher the average booking probability of the listings presented as map pins. Figure 4 summarizes the results from A/B testing a range of parameters.

The reduction in the average impressions to discovery metric in Figure 4 denotes the fewer number of map pins a searcher has to process before clicking the listing that they eventually book. Similarly, the reduction in average clicks to discovery shows the fewer number of map pins a searcher has to click through to find the listing they booked.

Launching the restricted version resulted in one of the largest bookings improvement in Airbnb ranking history. More importantly, the gains were not only for bookings, but for quality bookings. This could be seen by the increase in trips that resulted in 5-star rating after the stay from the treatment group, in comparison to trips from the control group.

Tiered User Attention

In our next iteration of modeling user attention, we separate the map pins into two tiers. The listings with the highest booking probabilities are displayed as regular oval pins with price. Listings with comparatively lower booking probabilities are displayed as smaller ovals without price, referred to as mini-pins (Figure 5). By design, mini-pins draw less user attention, with click-through rates about 8x less than regular pins.

This comes in handy particularly for searches on desktop where 18 results are shown in a grid on the left, each of them requiring a map pin on the right (Figure 6).

The number of map pins is fixed in this case, and limiting them, as we did in the previous section, is not an option. Creating the two tiers prioritizes user attention towards the map pins with the highest probabilities of getting booked. Figure 7 shows the results of testing the idea through an online A/B experiment.

Discounted User Attention

In our final iteration, we refine our understanding of how user attention is distributed over the map by plotting the click-through rate of map pins located at different coordinates on the map. Figure 8 shows these plots for the mobile (top) and the desktop apps (bottom).

To maximize the chances that a searcher will discover the listings with the highest booking probabilities, we design an algorithm that re-centers the map such that the listings with the highest booking probabilities appear closer to the center. The steps of this algorithm are illustrated in Figure 9, where a range of potential coordinates are evaluated and the one which is closer to the listings with the highest booking probabilities is chosen as the new center.

When tested in an online A/B experiment, the algorithm improved uncancelled bookings by 0.27%. We also observed a reduction of 1.5% in map moves, indicating less effort from the searchers to use the map.

Conclusion

Users interact with maps in a way that’s fundamentally different from interacting with items in a list. By modeling the user interaction with maps in a progressively sophisticated manner, we were able to improve the user experience for guests in the real world. However, the current approach has a challenge that remains unsolved: how can we represent the full range of available listings on the map? This is part of our future work. A more in-depth discussion of the topics covered here, along with technical details, is presented in our research paper that was published at the KDD ’24 conference. We welcome all feedback and suggestions.

If this type of work interests you, we encourage you to apply for an open position today.

Airbnb had a large presence at the 2024 KDD conference hosted in Barcelona, Spain. Our Data Scientist and Engineers presented on topics like Deep Learning & Search Ranking, Online Experimentation & Measurement, Product Quality & Customer Journey, and Two-sided Marketplaces. This blog post summarizes our contributions to KDD for 2024 and provides access to the academic papers presented during the conference.

Authors: Huiji Gao, Peter Coles, Carolina Barcenas, Sanjeev Katariya

KDD (Knowledge and Data Mining) is one of the most prestigious global conferences in data mining and machine learning. Hosted annually by a special interest group of the Association for Computing Machinery (ACM), it’s where attendees learn about some of the most ground-breaking AI developments in data mining, machine learning, knowledge discovery, and large-scale data analytics.

This year, the 30th KDD conference was held at Barcelona, Spain, attracting thousands of researchers and scientists from academia and industry. Various companies contributed to and attended the conference including Google, Meta, Apple, Amazon, Airbnb, Pinterest, LinkedIn, Booking, Expedia, ByteDance etc. There were 151 Applied Data Science (ADS) track papers and 411 Research track papers accepted, 34 tutorials, and 30 workshops.

Airbnb had a significant presence at KDD 2024 with three full ADS track papers (acceptance rate under 20%), one workshop, and seven workshop papers and invited talks accepted into the main conference proceedings. The topics of our work spanned Deep learning & Search Ranking, Online Experimentation & Measurement, Causal Inference & Machine Learning, and Two-sided Marketplaces.

In this blog post, we will summarize our teams’ contributions and share highlights from an exciting week-long conference with research and industry talks, workshops, panel discussions, and more.

Deep Learning and Search Ranking

Intelligent search ranking — the process of accurately matching a guest with a listing based on their preference, a listing’s features, and additional search context — still remains a nuanced challenge that researchers are constantly trying to solve.

Making optimal guest-host matches has remained an issue in a two-sided marketplace for a variety of reasons — the timespan of guest searches (ranging between days and weeks), unpredictable host behavior and ratings (the potential for hosts to cancel a booking or receive low ratings), and limited understanding of guest preference across multiple interfaces. We published several papers addressing the issue of search ranking as part of our presence at KDD.

Learning to Rank for Maps at Airbnb

Airbnb brings together hosts who rent listings to prospective guests from around the globe. Results from a guest’s search for listings are displayed primarily through two interfaces: (1) as a list of rectangular cards that contain on them the listing image, price, rating, and other details, referred to as list-results, and (2) as oval pins on a map showing the listing price, called map-results. Both these interfaces, since their inception, have used the same ranking algorithm that orders listings by their booking probabilities and selects the top listings for display.

However, some of the basic assumptions underlying ranking are built for a world where search results are presented as lists and simply break down for map-results. In this work, we rebuilt ranking for maps by revising the mathematical foundations of how users interact with map search results. Our iterative and experiment-driven approach led us through a path full of twists and turns, ending in a unified theory for the two interfaces.

Our journey shows how assumptions taken for granted when designing machine learning algorithms may not apply equally across all user interfaces, and how they can be adapted. The net impact was one of the largest improvements in user experience for Airbnb which we discuss as a series of experimental validations. The work introduced in this paper is merely the beginning of future exciting research projects, such as making learning to rank unbiased for map-results and demarcating the map pins to direct the user attention towards more relevant ones.

Multi-objective Learning to Rank by Model Distillation

In online marketplaces, the objective of search ranking is not only on optimizing purchasing or conversion rate (primary objective), but also the purchase outcomes (secondary objectives), e.g. order cancellation, review rating, customer service inquiries, platform long term growth. To balance these primary and secondary objectives, several multi-objective learning to rank approaches have been widely studied

Traditional approaches in industrial search and recommender systems encounter challenges such as expensive parameter tuning that leads to sub-optimal solutions, suffering from imbalanced data sparsity issues, and lack of compatibility with ad-hoc objectives. In this work, we propose a distillation-based ranking solution for multi-objective ranking, which optimizes the end-to-end ranking system at Airbnb across multiple ranking models on different objectives, along with various considerations to optimize training and serving efficiency that meets industry standards.

Compared with traditional approaches, the proposed solution not only significantly meets and increases the primary objective of conversion by a large margin, but also addresses the secondary objective constraints while improving model stability. Furthermore, we demonstrated the proposed system could be further simplified by model self-distillation. We also did additional simulations to show that this approach could help us efficiently inject ad-hoc non-differentiable business objectives into the ranking system, while enabling us to balance our optimization objectives.

Online Experimentation and Measurement

Online experimentation (e.g., A/B testing) is a common way for organizations like Airbnb to make data-driven decisions. But high variance is frequently a challenge. For example, it’s hard to prove that a change in our search UX will drive value because bookings can be infrequent and depend on a large number of interactions over a long period of time.

Metric Decomposition in A/B Tests

More than a decade ago, CUPED (Controlled Experiments Utilizing Pre-Experiment Data) mainstreamed the idea of variance reduction leveraging pre-experiment covariates. Since its introduction, it has been implemented, extended, and modernized by major online experimentation platforms. Despite the wide adoption, it is known by practitioners that the variance reduction rate from CUPED, utilizing pre-experimental data, varies case by case and has a theoretical limit. In theory, CUPED can be extended to augment a treatment effect estimator utilizing in-experiment data, but practical guidance on how to construct such an augmentation is lacking.

In this work, we fill this gap by proposing a new direction for sensitivity improvement via treatment effect augmentation, whereby a target metric of interest is decomposed into

two or more components in an attempt to isolate those with high signal and low noise from those with low signal and high noise. We show through theory, simulation, and empirical examples that if such a decomposition exists (or can be engineered), sensitivity may be increased via approximately null augmentation (in a frequentist setting) and reduced posterior variance (in a Bayesian setting).

We provide three real world applications demonstrating different flavors of metric decomposition. These applications illustrate the gain in agility metric decomposition yields relative to an un-decomposed analysis, indicating both empirically and theoretically the value of this practice in both frequentist and Bayesian settings. An important extension to this work would be to next consider sample size determination in both the frequentist or Bayesian contexts; while a boost in sensitivity typically means less data is required for a given analysis, a methodology that determines the smallest sample size required to control various operating characteristics in this context would be of practical value.

Two-sided Marketplace Optimization

Airbnb employees hosted a workshop on Two-sided Marketplace Optimization: Search, Pricing, Matching & Growth. This workshop brought practitioners of two-sided marketplaces together and discussed the evolution of content ranking, recommendation systems, and data mining when solving for producers and consumers on these platforms.

Two-sided marketplaces have recently emerged as viable business models for many real-world applications. They model transactions as a network with two distinct types of participants: one type to represent the supply and another the demand of a specific good. Traditionally, research related to online marketplaces focused on how to better satisfy demand. But with two-sided marketplaces, there is more nuance at play. Modern global examples, like Airbnb, operate platforms where users provide services; users may be hosts,or guests. Such platforms must develop models that address all their users’ needs and goals at scale. Machine learning-powered methods and algorithms are essential in every aspect of such complex, internet-scale-sized, two-sided marketplaces.

Airbnb is a community based on connection and belonging–we strive to connect people and places. Our contributions to this workshop showcase the work we’re doing to support this mission by optimizing guest experiences, finding equilibrium spots for listing prices, reducing the incidence of poor interactions (and customer support costs as a side effect), detecting when operational staff should follow up on activity at scale, and more.

Guest Intention Modeling for Personalization

Airbnb has transformed the way people travel by offering unique and personalized stays in destinations worldwide. To provide a seamless and tailored experience, understanding user intent plays an important role.

However, limited user data and unpredictable guest behavior can make it difficult to understand the essential intent from guests on listings from hosts. Our work shows how we approach this challenging problem. We describe how we apply a deep learning approach to predict difficult-to-infer details for a user’s travel plan, such as the next destination and travel dates. The framework analyzes high-level information from users’ in-app browsing history, booking history, search queries, and other engagement signals, and produces multiple user intent signals.

Marketing emails, flexible travel search (e.g., for “Europe in the summer”), and recommendations on the app home page are three guest interactions that benefit from correct intention modeling. Hosts also benefit, since a clear understanding of guest demand can help them optimize listings to increase satisfaction and bookings.

Guest Demand Understanding

Hosts can find it difficult to correctly price their listings in two-sided marketplaces serviced by end users. Most hosts are not professional hospitality workers, and would benefit from access to data and advice on how guests see their listings and how they compare to other listings in their neighborhood. We constantly look for ways to give guidance on how hosts can optimally price their listings. The same information can then be used to help guests find their ideal stay.

In our paper, we presented an example of how this problem can be solved in general.

As illustrated above, both demand and supply change over time, influencing the equilibrium price for a property at a specific point. A historical optimum (such as A above) has to be adjusted to find the current optimum (point C). It is difficult to run experiments since any large-scale experiment we might run will cause the environment to change in complex ways. We tackle this problem by combining economic modeling with causal inference techniques. We segment guests and estimate how price-sensitive each guest segment is, and fine-tune them with empirical data from small targeted experiments and larger-scale natural ones, which are used to adjust estimates for the price sensitivity of each guest segment. Hosts can then use the models’ output to make informed tradeoffs between higher occupancy and higher nightly rates.

Listing Embedding for Host-side Products

In order to facilitate the matching of listings and guests, Airbnb provides numerous products and services to both hosts and guests. Many of these tools are based on the ability to compare listings, i.e. finding similar listings or listings that may be viewed as equivalent substitutes. Our work presents a study on the application and learning of listing embeddings in Airbnb’s two-sided marketplace. Specifically, we discuss the architecture and training of a neural network embedding model using guest side engagement data, which is then applied to host-side product surfaces. We address the key technical challenges we encountered, including the formulation of negative training examples, correction of training data sampling bias, and the scaling and speeding up training with the help of in-model caching. Additionally, we discuss our comprehensive approach to evaluation, which ranges from in-batch metrics and vocabulary-based evaluation to the properties of similar listings. Finally, we share our insights from utilizing listing embeddings in Airbnb products, such as host calendar similar listings.

Customer Support Optimization in Search Ranking

As of the date of the paper, Airbnb had more than 7.7 million listings from more than 5 million hosts worldwide. Airbnb is investing both in rapid growth and in making sure that the booking experience is pleasant for hosts and guests. It would, however, be ideal to avoid poor experiences in the first place. Our work highlights how we prevent poor experiences without significantly reducing growth.

We use the mass of accumulated support data at Airbnb to model the probability that, if the current user were to book a listing, they would require CS support. Our model discovered multiple features about the searcher, home, and hosts that accurately predict CS requirements. For example, same-day bookings tend to require more support, and a responsive host tends to reduce support needs. So, if a guest chooses a same-day booking, matching them with a highly responsive host can lead to a better experience overall. We incorporate the output of our CS support model in search result rankings; booked homes will sometimes rank lower if we predict a booking will lead to a negative experience.

LLM Pretraining using Activity Logs

It’s often important to follow up with users after they’ve had a long series of interactions with a two-sided marketplace to help make sure that their experiences are of high quality. When user interactions meet certain business criteria, operations agents create tickets to follow up with them. For example, user retention and reactivation agents might review user activity logs and decide to follow up with the user, to encourage them to re-engage with the platform.

We propose transforming structured data (activity logs) into a more manageable text format and then leveraging modern language models (i.e., BERT) to pretrain a large language model based on user activities. We then performed fine-tuning on the model using historical data about which users were followed up with and checked its predictions. Our work demonstrates the large language model trained on pre-processed activity can successfully identify when a user should be followed up with, at an experimentally significant rate. Our preliminary results suggest that our framework may outperform by 80% the average precision of a similar model that was designed relying heavily on feature engineering.

Product Quality and Customer Journey Optimization

Typically, product quality is evaluated based on structured data. Customer ratings, types of support issues, resolution times, and other factors are used as a proxy for how someone booking on Airbnb might value a listing. This kind of data has limitations — more popular listings have more data, often users don’t leave feedback, and feedback is usually biased towards the positive (users with negative experiences tend to churn and not give feedback).

In the Workshop on Causal Inference and Machine Learning in Practice, we highlighted an example of how we push the boundaries of product quality assessment techniques and applications, mixing traditional casual inference with cutting-edge machine learning research. In our work “Understanding Product Quality with Unstructured Data: An Application of LLMs and Embeddings at Airbnb”, we presented how an approach based on text embeddings and LLMs can be combined with approaches based on structured data to significantly improve product quality evaluations. We generate text embeddings on a mix of listing and review texts, then cluster the embeddings based on rebooking and churn rates. Once we have clear clusters, we extract keywords from the original data, and use these keywords to calculate a listing quality score, based on their similarity to the keyword list.

In addition, we were invited to give a talk on Quality Foundations at Airbnb, at KDD’s 3rd Workshop on End-End Customer Journey Optimization. It’s often hard to differentiate the quality of customer experiences using simple review ratings, in part due to the tightness of their distribution. In this talk, we present an alternative notion of quality based on customer revealed preference: did a customer return to use the platform again after their experience? We describe how a metric — Guest Return Propensity (GRP) — leverages this concept and can differentiate quality, capture platform externalities, and predict future returns.

In practice, this measure may not be suited to many common business use cases due to its lagging nature and an inability to easily explain why it has changed. We describe a quality measurement system that builds on the conceptual foundation of GRP by modeling it as an outcome of upstream realized quality signals. These signals — from sources like reviews and customer support — are weighted by their impact on return propensity and mapped to a quality taxonomy to aid in explainability. The resulting score is capable of finely differentiating the quality of customer experiences, aiding tradeoff decisions, and providing timely insights.

Conclusion

The 2024 edition of KDD was an amazing opportunity for data scientists and machine learning engineers from across the globe and industry, government, and academia, to connect and exchange learnings and discoveries. We were honored to have the opportunity to share some of our knowledge and techniques, generalizing what we have been learning when we apply machine learning to problems we see at Airbnb. We continue to focus on improving our customers’ experience and growing our business, and the information we’ve shared has been crucial to our success. We’re excited to continue learning from peers and contribute our work back to our community. We eagerly await advancements and improvements that might come about as others build upon the work we’ve shared.

Below, you’ll find a complete list of the talks and papers shared in this article along with the team members who contributed. If this type of work interests you, we encourage you to apply for an open position today.

List of papers and talks

Learning to Rank for Maps at Airbnb (link)

Authors: Malay Haldar, Hongwei Zhang, Kedar Bellare, Sherry Chen, Soumyadip Banerjee, Xiaotang Wang, Mustafa Abdool, Huiji Gao, Pavan Tapadia, Liwei He, Sanjeev Katariya

Multi-objective Learning to Rank by Model Distillation (link)

Authors: Jie Tang, Huiji Gao, Liwei He, Sanjeev Katariya

Metric Decomposition in A/B Tests (link)

Authors: Alex Deng (former employee at Airbnb), Luke Hagar (University of Waterloo), Nathaniel T. Stevens (University of Waterloo), Tatiana Xifara (Airbnb), Amit Gandhi (University of Pennsylvania)

Understanding Guest Preferences and Optimizing Two-sided Marketplaces: Airbnb as an Example (link)

Authors: Yufei Wu, Daniel Shmierer

Predicting Potential Customer Support Needs and Optimizing Search Ranking in a Two-Sided Marketplace (link)

Authors: Do-kyum Kim, Han Zhao, Huiji Gao, Liwei He, Malay Haldar, Sanjeev Katariya

​​Understanding User Booking Intent at Airbnb (link)

Authors: Xiaowei Liu, Weiwei Guo, Jie Tang, Sherry Chen, Huiji Gao, Liwei He, Pavan Tapadia, Sanjeev Katariya

Can Language Models Accelerate Prototyping for Non-Language Data? Classification & Summarization of Activity Logs as Text (link)

Authors: José González-Brenes

Learning and Applying Airbnb Listing Embeddings in Two-Sided Marketplace (link)

Authors: Siarhei Bykau, Dekun Zou

Understanding Product Quality with Unstructured Data: An Application of LLMs and Embeddings at Airbnb (link)

Authors: Jikun Zhu, Zhiying Gu, Brad Li, Linsha Chen

Invited Talk: Quality Foundations at Airbnb

Speakers: Peter Coles, Mike Egesdal

Vijaya Kaza is the Chief Security Officer and Head of Engineering for Trust and Safety at Airbnb. She leads teams responsible for developing the technology (Platforms, tools and AI models), to safeguard the Airbnb community, as well as for securing Airbnb’s infrastructure and information assets. She is also the executive co-sponsor of Airbnb Tech’s Diversity Council.

Here’s Vijaya’s story of how she got to Airbnb, in her own words.

Straight shot to science and engineering

I grew up in a modest, multi-generational family in India with 30 to 40 family members under one roof on any given day. As the oldest child in that house, I was expected to excel academically and set an example for the other children to follow.

In our culture back then, being “good at school” was synonymous with shining in science and math. As luck would have it, I had a strong affinity for those subjects and enjoyed studying and diving deep into them. I followed a natural path that combined math and science, studying engineering in college, and got a bachelor’s and two master’s in electrical engineering. This foundation paved the way for my future work in technology.

Stumbling into cybersecurity

After college, I landed my first job as a Software Engineer at Cisco. However, my entry into the security field was accidental. I simply followed a manager I liked who was moving into a new security business unit, fell in love with Security and never looked back after that! There was no grand plan or calculated career strategy.

After 17 years at Cisco leading product development for a $1B security product portfolio, I headed to FireEye, another well-known name in the Cybersecurity space. There I had the responsibility for helping the company transition from on-prem to a cloud/SaaS business model, and growing the revenue of their cloud security portfolio. That role gave me the experience of working on different areas of security, as well as leading both Product Management and Engineering in a General Manager capacity.

Next, I led Product Development at Lookout, a startup in San Francisco focused on mobile security. At the time, it was a consumer security company pivoting to building for enterprise customers. I didn’t know it then, but that glimpse of consumer security was a great primer for my eventual role at Airbnb.

Cybersecurity is an ever-changing domain with constant innovation and I’ve thoroughly enjoyed working and learning in such a dynamic space. Each major technological transformation — from cloud to mobile to AI — brings novel security challenges to solve for businesses and end users.

An unexpected opportunity at Airbnb

When I was initially approached for the Chief Security Officer (CSO) role at Airbnb, I was taken aback. I had always worked in the field doing engineering and product development work, so I wasn’t sure about this role. But after meeting Ari Balogh, Airbnb’s Chief Technology Officer (CTO) for an informal coffee chat, we really hit it off and I was thoroughly impressed by his vision to transform the engineering and technology organization within the company. Ari shared his philosophy of focusing on the craft of engineering and that really resonated with me. The prospect of molding Airbnb’s engineering culture was very enticing.

Turns out Airbnb was also looking for someone to lead engineering for its Trust and Safety organization. Given my background with a blend of engineering and security domain expertise, I ended up taking on both of these roles. This unique opportunity allowed me to go back to my roots leading engineering teams while taking advantage of my security experience in the capacity of a CSO.

When I joined the company in 2019, I was struck by Airbnb’s dedication and effort to deliver a positive user experience. Our attention and design focus that go into helping guests and hosts have a seamless experience is unlike anything I’ve experienced before.

Why Airbnb?

I joined Airbnb because of the opportunity to have an outsized impact — Airbnb’s almost 6,000 employees serve millions of people worldwide. Our mission-driven approach, powered by our founders’ unmatched passion and commitment to doing good through initiatives like Airbnb.org really resonated with me. I’ve been continuously impressed by the caliber of talented, caring people here who are united by Airbnb’s vision. It’s rare to find a company that so effectively combines technical excellence with social consciousness.

Two teams, one mission

While both the Trust and Safety and Security teams share the common mission of safeguarding users and the platform, the actual techniques, threats, and focus areas are completely different. Trust is at the core of our business and we’re deeply focused on providing our guests and hosts peace of mind as they live, work, travel and host on Airbnb. We build technology to help lower safety and privacy risks for our community.

For example, our innovative reservation screening technology aims to help reduce the risk of disruptive parties on Airbnb globally by taking steps to identify higher-risk reservations and potentially prevent these bookings from being made.

On the cybersecurity front, we focus on securing Airbnb’s assets, our data, our employees, and our infrastructure. We implement robust security controls and threat detection capabilities to safeguard Airbnb’s internal resources.

Embracing the improv mindset

Outside of work, I’ve pursued some hobbies that have surprisingly imparted invaluable leadership lessons. A few years ago, I decided to try improv comedy, something that’s entirely outside my wheelhouse. I had never done anything close to theater or acting in my entire life. What started as a fun experiment quickly became a passion project and I progressed through the levels and eventually performed live in front of an audience with friends and family in attendance.

It may seem unrelated at first, but the very nature of improv is profoundly relevant for leadership. You’re constantly put on the spot and need to think on your feet, responding to new questions and scenarios in the moment. Improv trains this vital skill of processing information in real time and formulating a coherent, compelling reaction.

Keep a steady head

Over any career, including mine, there are inevitable professional setbacks, disappointments, and annoyances along the way. The key is to not make much of these short-term hurdles; it’s how you respond that matters most. The less you agonize over the bumps in the road, the better. Maintain your focus, keep one foot in front of the other, and persist forward undeterred. I learned these lessons early, having taken on leadership roles from a young age as the eldest child in a large household. My advice is to keep a steady head, maintain perspective, and plow forward with conviction.

We’re currently expanding the Airbnb team and hiring for several roles. Check out our open positions here.

How Airbnb leverages machine learning and reinforcement learning techniques to solve a unique information retrieval task in order to provide guests with unique, affordable, and differentiated accommodations around the world.

By: Dillon Davis, Huiji Gao, Thomas Legrand, Weiwei Guo, Malay Haldar, Alex Deng, Han Zhao, Liwei He, Sanjeev Katariya

Introduction

Airbnb has transformed the way people travel around the globe. As Airbnb’s inventory spans diverse locations and property types, providing guests with relevant options in their search results has become increasingly complex. In this blog post, we’ll discuss shifting from using simple heuristics to advanced machine learning and reinforcement learning techniques to transform what we call location retrieval in order to address this challenge.

The Challenge of Location Retrieval

Guests typically start searching by entering a destination in the search bar and expect the most relevant results to be surfaced. These destinations can be countries, states, cities, neighborhoods, streets, addresses, or points of interest. Unlike traditional travel accommodations, Airbnb listings are spread across different neighborhoods and surrounding areas. For example, a family searching for a vacation rental in San Francisco might find better options in nearby cities like Daly City, where there are larger single-family homes. Thus, the system needs to account for not just the searched location but also nearby areas that might offer better options for the guest. This is evidenced by the locations of booked listings when searching for San Francisco shown below.

Given Airbnb’s scale, we cannot rank every listing for every search. This presented a challenge to create a system that dynamically infers a relevant map area for a query. This system, known as location retrieval, needed to balance including a wide variety of listings to appeal to all guests’ needs while still being relevant to the query. Our search ranking models can then efficiently rank the subset of our inventory that is within the relevant map area and surface the most relevant inventory to our guests. This system and more is outlined below

Starting with Heuristics: The Cold Start Problem

Initially, Airbnb relied on heuristics to define map areas based on the type of search. For example, if a guest searched for a country, the system would use administrative boundaries to filter listings within that country. If they searched for a city, the system would create a 25-mile radius around the city center to retrieve listings.

Improving these heuristics proved to be profoundly impactful. One such example is the introduction of a log scale parameterized smooth function to compute an expansion factor for the diagonal size of the administrative bounds of the searched destination. We applied this for very precise locations like addresses, buildings, and POI’s resulting in a 0.35% increase in uncancelled bookers on the platform when tested in an online A/B experiment against the baseline heuristics. Figures below demonstrate how search results for a building in Ibiza, Spain improved dramatically with this heuristic by surfacing significantly more and higher quality inventory.

These heuristics were simple and worked well enough to start, but they had limitations. They couldn’t differentiate between different types of searches (e.g., a family looking for a large home versus a solo traveler looking for a small apartment), and they didn’t adapt well to new data as Airbnb’s inventory and guest preferences evolved.

Exploring Statistics to Help Improve Location Retrieval

With more data available over time from these intuition based heuristics, we thought there might be a way to take advantage of this historical user booking behavior to improve location retrieval. We built a dataset for each travel destination that recorded where guests booked listings when searching for that destination. Based on this data, the system could create retrieval map areas that included 96% of the nearest booked listings for a given destination.

We tested these newly constructed retrieval map areas in lieu of the intuition based heuristics outlined above based on the hypothesis that it would provide guests a more bookable selection of inventory. While this statistical approach was more aligned with guest booking behavior, it still had limitations. It treated all searches for a location the same, regardless of specific search parameters like group size or travel dates. This uniform approach meant that some guests might not see the best listings for their particular needs. As a result, this statistics based method had no detectable increase in uncancelled bookers on the platform when tested against the heuristics outlined above in an online A/B experiment. This led us to believe that location retrieval may require more advanced techniques such as machine learning.

Advancing to Machine Learning

Instead of only relying on past booking data, the new system could learn from various search parameters, such as the number of guests and stay duration. By analyzing this data, a model could predict more relevant map areas for each search, rather than applying a one-size-fits-all approach.

For example, a group of ten travelers searching for a San Francisco vacation rental might prefer larger homes in the suburbs, while solo travelers might prioritize central locations. The machine learning model could distinguish between these different preferences and adjust the retrieval map areas accordingly, providing more tailored results.

We constructed our machine learning model in the following manner. This is a result of three iterations that introduced the machine learning model, expanded its feature set, and expanded search attribution. The architecture is depicted in the figure below.

1. Training Examples: Searches issued by a booker by entering a destination in the search bar or manipulating the map that contained the booked listing in their search results on the same day or one day before the booking. We discard any bookings that are canceled 7 days after booking.
2. Training Features: We derive features directly from the search request such as location name, stay length, number of guests, price filters, location country, etc. There are 9 continuous features and 19 categorical features in total.
3. Training Labels: The latitude and longitude coordinates of the booked listing attributed to the search
4. Architecture: A two layer neural network of size 256 was chosen in order to have more flexibility for loss formulation compared to traditional regression and decision tree based approaches.
5. Model Output: 4 floats that define the latitude and longitude offsets from the center latitude and longitude coordinates of the searched destination that represent the relevant map area.
6. Loss: Trained to predict map areas that contain their associated booked listing while minimizing the size of the predicted map area and the occurrence of predictions that cannot construct a valid rectangular map area.

The machine learning system increased the recall of booked listings (i.e., how often the system retrieved a listing that was eventually booked) by 7.12% and reduced the size of the retrieval map area by 40.83%. It had a cumulative impact of +1.8% in uncancelled bookers on the platform. The initial model was evaluated against the baseline and each subsequent model iteration was evaluated against the preceding outgoing model.

Figures below demonstrate how search results for a specific street in Lima, Peru improved dramatically with the model by surfacing results that are much closer to the searched street.

Before

After

Exploring New Frontiers with Reinforcement Learning

While machine learning improved the system’s ability to differentiate search results, there was still room for improvement, particularly in learning whether locations that had never been surfaced before were relevant to guests for a search. To address this, Airbnb introduced reinforcement learning to the location retrieval process.

Reinforcement learning allowed the system to continuously learn from guest interactions by surfacing new areas for a given destination and adjusting the retrieval map area based on guest booking behavior. This approach, known as a contextual multi-armed bandit problem, involved balancing exploration (surfacing new locations) with exploitation (surfacing previous successful locations). The system could actively experiment with different retrieval map areas learning from guest bookings to refine its predictions.

Applying a contextual multi-armed bandit traditionally requires defining an active contextual estimator, a method for uncertainty estimation, and an exploration strategy. We took the following approach given product constraints, system constraints, and the nature of our model formulation. The architecture is depicted in the figure below.

1. Active contextual estimation: We employed our existing machine learning model for location retrieval retrained on a daily basis to regularly learn from any new bookings data that we collect while surfacing previously unshown locations.
2. Uncertainty estimation: We modified our model architecture with a random dropout layer to generate 32 unique predictions for a given search (Monte Carlo Dropout). This allows us to measure the mean and standard deviation of our prediction while minimizing negative impact to system performance and changes to our existing model formulation.
3. Exploration Strategy: We compute an upper confidence bound using the mean and standard deviation of our prediction in order to construct larger retrieval map areas based on the model’s confidence in its prediction for the search.

This system successfully explored more for less-traveled locations where it was less confident and explored less for locations that are often searched and booked. For example, pictured below are the mean (inner) and upper confidence bound (outer) estimates of retrieval map areas for San Francisco, CA (left) and Smith Mountain Lake, Virginia (right). San Francisco is searched almost 25x more than Smith Mountain Lake with proportionately more bookings as well. As a result, the model is more confident in its retrieval map area estimate for San Francisco vs Smith Mountain Lake resulting in 2–3x less exploration for San Francisco queries vs Smith Mountain Lake.

The reinforcement learning system was also tested against the outgoing machine learning model in online A/B experiments showing a cumulative 0.51% increase in uncanceled bookers and 0.71% increase in 5 star trip rate over two iterations that introduced reinforcement learning and optimized scoring of the more complex model.

Conclusion: A Transformative Journey

Airbnb’s journey from simple heuristics to sophisticated machine learning and reinforcement learning models demonstrates the power of data-driven approaches in transforming complex systems. By continually iterating and improving its location retrieval process, Airbnb has not only enhanced the relevance of its search results but also helped guests experience more 5 star trips.

This transformation cumulatively results in a 2.66% increase in uncanceled bookers — a major achievement for a company operating at Airbnb’s scale. More details can be found in our technical paper. As Airbnb continues to innovate, we are continuously evaluating and introducing more advanced features and retrieval mechanisms like retrieving with complex polygons . These will further refine and enhance the search experience for millions of guests worldwide.

If this type of work interests you, check out some of our related positions and more at Careers at Airbnb!

****************

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Airbnb made it easy to bring data/AI ideas to life through a platform for prototyping web applications.

By: Dan Miller

Introduction

Trustworthy data has always been a part of Airbnb’s technical DNA. However, it is challenging for our data scientists and ML practitioners to bring data- and AI-powered product ideas to life in a way that resonates with our design-focused leadership. Slide decks with screenshots, design documents with plots, and even Figmas are insufficient to capture ideas that need to be experienced in order to be understood. This was especially true as large language models (LLMs) took the world by storm, since they are typically used interactively in chat interfaces.

In this blog post, we’ll focus on Sandcastle, an Airbnb-internal prototyping platform that enables data scientists, engineers, and even product managers to bring data/AI ideas to life as internal web applications for our design and product teams. Through Sandcastle, hundreds of individuals can be “cereal entrepreneurs” — empowered to directly iterate on and share their ideas. We’ll talk through common industry challenges involved in sharing web applications internally, give an overview of how Airbnb solved these challenges by building on top of its existing cloud infrastructure, and showcase the scale of our results.

Challenges

Imagine a data scientist is working on a typical data science problem at Airbnb: optimizing the positive milestones guests reach along their user journey, visualizing that journey, or improving explainability and statistical power in mathematically challenging scenarios like company-wide launches without A/B, or measuring brand perception. The data scientist has a brilliant LLM-powered idea. They want to demonstrate the capability their idea exposes in an interactive way, ideally one that can easily “go viral” with non-technical stakeholders. Standing between the idea and stakeholders are several challenges.

Leadership & non-technical stakeholders will not want to run a Jupyter notebook, but they can click around in a UI and try out different input assumptions, choose different techniques, and deep-dive into outputs.

Data scientists are most comfortable writing Python code, and are quite unfamiliar with the world of modern web development (TypeScript, React, etc.). How can they capture their idea in an interactive application, even in their own development environment? Traditionally, this is done by collaborating with a frontend engineering team, but that brings its own set of challenges. Engineering bandwidth is typically limited, so prototyping new ideas must go through lengthy planning and prioritization cycles. Worse, it is nearly impossible for data scientists to iterate on the science behind their ideas, since any change must go through reprioritization and implementation.

Suppose we can surmount the challenge of capturing an idea in a locally-run interactive web application. How do we package and share it in a way that other data scientists can easily reproduce using standard infrastructure?

How can a data science organization handle infrastructure, networking with other parts of Airbnb’s complex tech stack, authentication so their apps don’t leak sensitive data, and storage for any temporary or intermediate data. How can they create easily shareable “handles” for their web applications that can easily go viral internally?

Sandcastle

Airbnb’s solution to the challenges above is called Sandcastle. It brings together Onebrain: Airbnb’s packaging framework for data science / prototyping code, kube-gen: Airbnb’s infrastructure for generated Kubernetes configuration, and OneTouch: Airbnb’s infrastructure layer for dynamically scaled Kubernetes clusters. Sandcastle is accessible for data scientists, software developers, and even product managers, whether their preferred language is Python, TypeScript, R, or something else. We have had team members use Sandcastle to go from “idea” to “live internal app” in less than an hour.

Onebrain

The open source ecosystem solves our first challenge, interactivity. Frameworks like Streamlit, Dash, and FastAPI, make it a delight for non-frontend developers to get an application up and running in their own development environment. Onebrain solves the second challenge: how to package a working set of code in a reproducible manner. We presented on Onebrain in detail at KDD 2023 but include a brief summary here. Onebrain assumes you arrange your code in “projects”: collections of arbitrary source code around a onebrain.yml file which looks like below.

name: youridea
version: 1.2.3
description: Example Sandcastle app
authors: ['Jane Doe <jane.doe@airbnb.email>']build_enabled: trueentry_points:
  main:
    type: shell
    command: streamlit run app.py --server.port {{port}}
    parameters:
      port: {type: int, default: 8880}env:
  python:
    pip: {streamlit: ==1.34.0}

This “project file” includes metadata like name, version, authorship, along with a collection of command line entry points that may run shell scripts, Python code, etc. and an environment specification directing which Python and R packages are needed to run. A developer may run “brain run” in the same directory as their project file for interactive development. Onebrain is integrated with Airbnb’s continuous integration, so every commit of the project will be published to our snapshot service. The snapshot service is a lightweight mechanism for storing immutable copies of source code that may be easily downloaded from anywhere else in Airbnb’s tech stack. Services may invoke

brain run youridea --port 9877

to resolve the latest snapshot of the project, bootstrap any dependencies, and invoke the parameterized shell command. This decouples rapid iteration on application logic with slower CI/CD against the service configuration we’ll talk about below.

kube-gen

Cloud infrastructure is challenging to configure correctly, especially for data scientists. Fortunately, Airbnb has built a code-generation layer on top of Kubernetes called kube-gen, which handles most of authentication, tracing, and cross-service communication for you. Sandcastle further simplifies things by using kube-gen hooks to generate all but one service configuration file on the developer’s behalf during build. The kube-gen configuration for a typical application would include environment-specific service parameters, Kubernetes app + container configuration, Spinnaker™ pipeline definitions, and configuration for Airbnb’s network proxy. Sandcastle generates sensible defaults for all of that configuration on-the-fly, so that all an app developer needs to write is a simple container configuration file like below. Multiple developers have raised support threads because the configuration was so simple, they thought they were making a mistake!

name: sandcastle-youridea
image: {{ .Env.Params.pythonImage }}command:
  - brain
  - download-and-run
  - youridea
  - --port
  - {{ .Env.Params.port }}resources: {{ ToInlineYaml .Env.Params.containerResources }}

The file above allows an app developer to configure which Onebrain project to run, which port it exposes a process on, and customize the underlying Docker image and CPU+RAM resources if necessary.

Within 10–15 minutes of checking in a file like above, the app will be live at an easily shareable URL like https://youridea.airbnb.proxy/ , where it can be shared with anyone at the company who has a working corporate login. Sandcastle also handles “identity propagation” from visiting users to the underlying data warehouse infrastructure, to ensure that applications respect user permissions around accessing sensitive metrics and tables.

Replicating Sandcastle

Product ideas powered by data and AI are best developed through rapid iteration on shareable, lightweight live prototypes, instead of static proposals. There are multiple challenges to facilitating the creation of secure internal prototypes. Open source frameworks like Streamlit and Dash help, but aren’t enough: you also need a hosting platform. It doesn’t make sense to open source Sandcastle, because the answers to “how does my service talk to others” or “how does authentication work” are so different across company infrastructures. Instead, any company can use Sandcastle’s approach as a recipe: 1) Application: adapt open source web application frameworks to their bespoke tech stack with 2) Hosting platform: that handles authentication, networking and provides shareable links.

Here is a quick summary of the things you’ll need to think about if you hope to build a “Sandcastle” for your own company:

• Open source web application framework(s): At Airbnb we largely use Streamlit for data science prototyping, with a bit of FastAPI and React for more bespoke prototypes. Prioritize ease of development (especially hot reload), a rich ecosystem of open source components, and performant UIs via caching.
• Packaging system: a way of publishing snapshots of “data/AI prototype code” from DS/ML development environments to somewhere consumable from elsewhere in your tech stack. At Airbnb we use Onebrain, but there are many paid public alternatives.
• Reproducible runs of DS/ML code: this should include Python / Conda environment management. Airbnb uses Onebrain for this as well, but you may consider pip.

In addition, you’ll need prototyping-friendly solutions for the three pillars of cloud computing:

• Compute: spin up a remote hosting environment with little or ideally no complicated infrastructure configuration required.
• Storage: access to ephemeral storage for caching and, more importantly, access to your company’s data warehouse infrastructure so prototypes can query your offline data.
• Networking: an authentication proxy that allows internal users to access prototypes, ideally via easily memorable domains like appname.yourproxy.io, and passes along user information so prototypes can pass visitor credentials through to the data warehouse or other services. Also, read-only access to other internal services so prototypes can query live data.

Build with a view towards “going viral”, and you’ll end up with a larger internal audience than you expect, especially if your platform is deliberately flexible. This allows their developers to focus on leveraging the rich open source prototyping ecosystem. More importantly, key stakeholders will be able to directly experience data/AI ideas at an early stage.

Conclusion

Sandcastle unlocked fast and easy deployment and iteration of new ideas, especially in the data and ML (including LLMs, generative AI) spaces. For the first time, data scientists and PMs are able to directly iterate on interactive versions of their ideas, without needing lengthy cycles for prioritization with an engineering team.

Airbnb’s data science, engineering, and product management community developed over 175 live prototypes in the last year, 6 of which were used for high-impact use cases. These were visited by over 3.5k unique internal visitors across over 69k distinct active days. Hundreds of internal users a week visit one of our many internal prototypes to directly interact with them. This led to an ongoing cultural shift from using decks / docs to using live prototypes

If this type of work interests you, check out some of our related positions:

• Staff data scientist — Algorithms, Trust
• Senior Staff Software Engineer, AI for Developer Productivity
• … and more at Careers at Airbnb!

You can also learn more about data science and AI at Airbnb by checking out Airbnb at KDD 2023, Airbnb Brandometer: Powering Brand Perception Measurement on Social Media Data with AI, and Chronon, Airbnb’s ML Feature Platform, Is Now Open Source.

Acknowledgments

Thanks to:

• Chris C Williams, Erik Iverson, Mike Dodge, Patrick Srail — for being early adopters whose feedback was critical in shaping Sandcastle’s evolution.
• Alex Deng, Carolina Barcenas, Navin Sivanandam — for their leadership support.

A deep dive into the streaming aspect of the Lambda architecture framework that optimizes how data is consumed from system-of-record data stores and updates secondary read-optimized stores at Airbnb.

Overview

In our previous blog post we introduced the motivation and high-level architecture of Riverbed. As a recap, Riverbed is a part of Airbnb’s tech stack designed to streamline and optimize how data is consumed from system-of-record data stores and update secondary read-optimized stores. The framework is built around the concept of ‘materialized views’ — denormalized representations of data that can be queried in a predictable, efficient manner. The primary goal of Riverbed is to improve scalability, enable more efficient data fetching patterns, and provide enhanced filtering and search capabilities for a better user experience. It achieves this by keeping the read-optimized store up-to-date with the system-of-record data stores, and by making it easier for developers to build and manage pipelines that stitch together data from various data sources.

In this blog post, we will delve deeper into the streaming aspect of the Lambda architecture framework. We’ll discuss step by step its critical components and explain how it constructs and sinks the materialized view from the Change Data Capture (CDC) events of various online data sources. Specifically, we’ll take a closer look at the join transformation within the Notification Pipeline, illustrating how we designed a DAG-like data structure to efficiently join different data sources together in a memory-efficient manner.

To make the framework and its components easier to understand, let’s begin with a simplified example of a Riverbed pipeline definition:

{
  Review {
    id @documentId
    reviewUser {
      id
      firstName
      lastName
    }
  }
}

Riverbed provides a declarative schema-based interface for customers to define Riverbed pipelines. From the sample definition above, a Riverbed pipeline is configured to integrate data sources from the Review and User entities, generating Riverbed sink documents with the review ID as the materialized view document ID.

Based on this definition, Riverbed generates two types of streaming pipelines:

• Source Pipelines: Two pipelines consume CDC events from the Review and User tables respectively and publish Apache Kafka® events known as notification events, indicating which documents need to be refreshed.
• Notification Pipeline: This pipeline consumes the notification events published by the source pipelines and constructs materialized view documents to be written into sink stores.

Now, let us delve deeper into these two types of pipelines.

Source Pipeline

Picture 1 shows the Source Pipeline as the first component in Riverbed. It is an auto-generated pipeline that listens to changes in system-of-record data sources. When changes occur, the Source Pipeline constructs NotificationEvents and emits them onto the Notification Kafka® topic to notify the Notification Pipeline on which documents should be refreshed. In the event-driven architecture of Riverbed, the Source Pipeline acts as the initial trigger for real-time updates in the read-optimized store. It not only ensures that the mutations in the underlying data sources are appropriately captured and communicated to the Notification Pipeline for subsequent processing, but also is the key solution for the concurrency and versioning issues in the framework.

While the emphasis of this blog post is the Notification Pipeline, a detailed exploration of the Source Pipeline — especially its critical role in maintaining real-time data consistency and its interaction with Notification Pipelines — will be discussed in the next blog post of this series.

Notification Pipeline

The Notification Pipeline is the core component of the Riverbed framework. It consumes Notification events, then queries dependent data sources and stitches together “documents” that are written into a read-optimized sink to support a materialized view. A notification event is processed by the following operations:

• Ingestion: For every change to a data source that the Read-Optimized Store is dependent on, we must re-index all affected documents to ensure freshness of data. In this step, Notification Pipeline consumes Notification events from Kafka® and deserializes them into objects that simply contain the document ID and primary source ID.
• Join: Based on these deserialized objects, Notification Pipeline queries various data stores to fetch all data sources that are necessary for building the materialized view.
• Stitch: This step models the join results from various data sources into a comprehensive Java Pojo called StitchModel, so that engineers can perform further customized data processing on it.
• Operate: In this step, a chain of various operators including filter, map, flatMap, etc, containing product-specific business logic can be applied to the StitchModel to convert it into the final document structure that will be stored in the index.
• Sink: As the last step, documents can be drained into various data sinks to refresh the materialized views.

Among these operations, Join, Stitch and Sink are the most important as well as the most complicated ones. In the following sections, we will dive deeper into their design.

Data Source Join

One of the most crucial and intricate operations in Riverbed’s Notification Pipeline is the Join operation. A Join operation starts from the primary source ID and then fetches data for all data sources associated with the materialized view based on their relationship.

JoinConditionsDag

In Riverbed, we use JoinConditionsDag, a Directed Acyclic Graph, to store the relationship metadata among data sources, where each node represents one unique data source and each edge represents the join condition between two data sources. In the Notification Pipelines, JoinConditionsDag’s root node is always a metadata node for the notification event which contains the document ID and the primary source ID. The join condition connecting to the notification event node reflects the join condition to query the primary source. Below is a sample JoinConditionsDag defining the join relationship between the primary source Listing and some of its related data sources:

Given notification events are used to indicate which document needs to be refreshed and does not contain any source data, Notification Pipeline joins data sources starting from the primary source ID provided by the Notification event. Guided by the JoinConditionsDag, when the Notification Pipeline processes a Notification event containing the primarySourceId, it queries the Listing table to fetch Listing data where the id matches primarySourceId. Subsequently, leveraging this Listing data, it queries the ListingDescription and Room tables to retrieve listing descriptions and rooms data, respectively, where the listingId equals id of Listing. In a similar manner, RoomAmenity data is obtained with roomId matching the id of the Room data.

JoinResultsDag

Now, we have the JoinConditionsDag guiding the Notification Pipeline to fetch all data sources. However, the question arises: how can we efficiently store the query results? One straightforward option is to flatten all the joined results into a table-like structure. Yet, this approach can consume a significant amount of memory, especially when performing joins with high cardinality. To optimize memory usage, we designed another DAG-like data structure named JoinResultsDag.

There are two major components in a JoinResultsDag. Cell is the atomic container for a data record. Each cell maintains its own successor relationships by mapping successor data source aliases to the CellGroups. CellGroup is the container to store the joined records from one data source. Each data source table record is stored in each Cell.

As mentioned above, the biggest difference and the advantage of using a DAG-based data structure instead of using the traditional flat join table is that it can efficiently store a large amount of join result data especially when there is a 1:M or M:N join relationship between data sources. For example, we have one pipeline to create materialized views for Airbnb Listings with information about all their Listing rooms, which also have lots of room amenities. If we use the traditional flat join table, it will look like the following table.

Obviously, storing joined results using a flat table structure demands extensive resources for both storage and processing. In contrast, JoinResultsDag effectively mitigates data duplication by allowing multiple successor nodes to refer back to the same ancestor nodes.

Now with JoinConditionsDag representing the relationship among all data sources and JoinResultsDag storing all the results, joins can be performed in Riverbed roughly as follows:

Starting from the NotificationEvent, Riverbed first initializes a JoinResultsDag with the deserialized Notification event as root. Then guided by the JoinConditionsDag and following a depth-first-search traverse, it visits the data store of each source, queries data based on the join conditions defined on the JoinConditionsDag edges, encapsulates the query results rows inside each Cell and then continues fetching the data of its dependencies until finished visiting all data sources.

Stitching of Data

With the joined results now stored in JoinResultsDag, an additional operation is necessary to transform these varied data pieces into a more usable and functional model. This enables engineers to apply their custom operators, mapping the data onto their specifically designed Sink Document. We refer to this process as the Stitch Operation, resulting in what is known as the StitchModel.

The StitchModel, a Java POJO derived from the custom pipeline definition, serves as the intermediate data model that not only contains the actual data but also contains useful metadata about the event such as document ID, version, mutation source, etc.

After the StitchModel metadata is generated, with the help of the JoinResultsDag, the Stitch operation is more straightforward. It maps the JoinResultsDag into a JSON model with the same structure and then converts the JSON model into the custom defined Java POJO utilizing the GSON library.

Sink data

The final stage in Riverbed’s Notification Pipeline is to write documents into data sinks. In Riverbed, sinks define where the processed data, now in the form of documents, will be ingested after the preceding operations are completed. Riverbed allows for multiple sinks, including Apache Hive(™) and Kafka®, so the same data can be ingested into multiple storage locations if required. This flexibility is a key advantage of the Notification Pipeline, enabling it to cater to a wide variety of use cases.

Riverbed writes documents into data sinks via their write APIs. For the best performance, it encapsulates a collection of documents into the API request and then makes use of the batched write API of each data sink to update multiple documents efficiently.

Summary

In conclusion, we’ve navigated the critical steps of Riverbed’s streaming system within the Lambda architecture framework, focusing on the construction of materialized views from CDC events. Our highlight on the join transformation within the Notification Pipeline showcased a DAG-like structure for efficient and memory-conscious data joining. This discussion has shed light on the architectural approach to constructing materialized views in streaming and introduced innovative data structure designs for optimizing streaming data joins. Looking ahead, we will delve deeper into the Source Pipeline of the streaming system and explore the batch system of Riverbed, continuing our journey through advanced data architecture solutions.

If this kind of work sounds appealing to you, check out our open roles — we’re hiring!

An Important Foundation For Security, Privacy, and Compliance at Airbnb

By: Sam Kim, Alex Klimov, Woody Zhou, Sylvia Tomiyama, Aniket Arondekar, Ansuman Acharya

Introduction

Airbnb is built on trust. One key way we maintain trust with our community is by ensuring that personal data is handled with care, in a manner that meets security, privacy, and compliance requirements. Understanding where and what personal data exists is foundational to this.

Over the past several years, we’ve built our own data classification system that adapts to the needs of our data ecosystem, to streamline our processes, and further unlock our ability to protect the data entrusted to Airbnb. This was made possible by many teams working closely to achieve this overarching, shared objective. Information Security, Privacy, Data Governance, Legal, and Engineering collaborated to tackle this problem holistically to produce a unified data identification and classification strategy across all data stores.

In this blog, we will shed light on the complexities of how data classification works at Airbnb, what measurements we set to assess the quality, performance, and accuracy of the systems involved, and the important considerations when building a data classification system. We hope to share insights for others that are facing similar challenges and to provide a framework for how data classification systems can be built at scale.

The Complexities of Data Classification at Airbnb

Data classification is the process of identifying where data exists and then organizing, detecting, and annotating that data based on a taxonomy. At Airbnb, we have established a Personal Data Taxonomy Council to define the taxonomy for personal data and to refine it over time. This taxonomy breaks down personal data into various data elements that are relevant for our ecosystem such as email address, physical address, and guest names. Once data is annotated with its applicable personal data element(s), various enforcement systems use these annotations to ensure personal data is handled according to our Security and Privacy policies. In this blog post, we will focus primarily on the data classification workflow and not each type of enforcement use case.

The workflow can be classified into three pillars:

• Catalog: What data do we have?
• Detection: What data do we suspect is personal data?
• Reconciliation: Which classification do we choose?

Let’s dig deeper into how each of these form the backbone of data classification.

Catalog

Cataloging involves building a dynamic, accurate, and scalable system to first identify where data exists and then organize the whole inventory. Cataloging is akin to mapping the data landscape or organizing a library. It involves dynamically discovering new data, enriching it with metadata from various sources, and manually inputting information. This process is crucial for enforcing data policies, accurately classifying data, and assigning it to the correct owners.

• Automated and Dynamic Discovery: Automation makes the cataloging process scalable and efficient. For the variety of data stores that Airbnb uses, such as production and analytical databases, object stores, and cloud storage, our catalogs connect to them and dynamically fetch the full inventory of data. Either through stream or batch processing, they dynamically update to reflect new and changed data. This ensures the catalog is a reliable and accurate source of truth.
• Complexity and Diversity in Data Sources: The challenge of cataloging stems from the variety and complexity of data sources, including different formats and locations. Our cataloging systems fetch metadata in several ways: through direct API calls or by crawling schemas in formats like thrift, JSON, yaml, or config files, accommodating the diverse nature of modern data storage.

For search and discovery, many of our data entities are surfaced in the data management platform, Metis. This helps the data owners quickly answer questions such as which data contains personal data, who owns the data, and which controls are in place.

Detection

For personal data detection, we use the in-house automated detection service in our Data Protection Platform which was built to protect data in compliance with global regulations and security requirements. As our own taxonomy grows, we have expanded our capabilities and made the service easily extensible to detect all other types of personal data elements and personal Airbnb IDs.

Detection engine

For each data entity stored in the catalogs, scanning jobs are scheduled through a message queue, which then samples data and runs through our list of classifiers. Recognizing the need for periodic classifier updates, the detection engine was designed for simplicity and flexibility. Since its inception, our detection engine has upgraded to include additional steps and adopted the approach of configuration-driven development. The majority of the logic of the detection engine has been rewritten as simpler configurations to increase the speed of iterating on existing classifiers, improve testing, and enable quick development of new features.

The detection engine can be seen as a pipeline, which involves the scanner, validator, and thresholding.

Scanner: The scanner classifies personal data using metadata and content, employing methods like regex for emails and keyword lists for cities, and advanced machine learning models for complex data types requiring contextual understanding.

Validator: Sampled data matching a scanner undergoes a customizable validation step to enhance classifier accuracy, verifying details like latitude/longitude ranges or custom ciphertexts from encryption services.

Thresholding: To reduce noise, thresholding is applied before storing results, varying by data structure type (e.g., matched rows vs. findings in a document) and set based on historical data frequency and criticality.

With the revamped pipeline, this has resulted in a significant decrease in false positive findings and reduced the burden on data owners to verify every result, which has historically impeded their productivity.

Reconciliation

Not every detection surfaced may be correct or, in other cases, more context may be required. Therefore, we employ a human-in-the-loop strategy: where data owners confirm the classifications. This step is critical in ensuring these classifications are correct before any data policies are automatically enforced to protect our data.

Automated Notifications

For compliance, we have an automated notification system that issues tickets whenever personal data is detected. These get surfaced to the appropriate data or service owners with strict SLAs.

For data entities that have schemas defined in code, such as transactional tables from production services (online), Amazon S3 buckets, or tables that are exported to our data warehouse, we assist the developers by automatically creating code changes that update their table schemas with the detected personal data elements.

Enforcing resolution

To enforce resolution on these tickets, tables are automatically access controlled in the data warehouse when the tickets are not resolved within SLA. Additionally reviews are conducted to ensure our classifications are correct for data where its handling requirements apply.

Tracking actions taken on these tickets for when personal data is detected has been important to assess the quality of our data classification flow and to keep an audit trail of past detections. It also highlights points of friction developers face when resolving these tickets. The investments we have made in this area have continued to improve the process and reduce the time needed to resolve tickets each year.

Assessing Quality of a Data Classification System

Because of the complexity of the system and its sub-components, this presented a unique challenge as we strive to define what quality means for the entire system. To build with the long-term in mind, we evaluated how well our entire data classification system functions as a whole.

We’ve set up measurements to assess quality of our data classification in three categories:

Recall: This measures our coverage and ability to not miss where personal data may exist, crucial for protecting the stored personal data. We assess recall through:

• Number of data entities integrated in the data classification system
• Volume of personal data that exists from all different sources
• Types of personal data being annotated and automatically detected against our taxonomy

Precision: This evaluates the accuracy of our data classifications, vital for data owners tagging their data. High precision minimizes tagging friction. Precision is measured by:

• Tracking false positive rates of classifiers for each type of personal data
• Tracking ticket resolutions made by data owners, which also aids in understanding nuanced classification cases

Speed: This gauges the efficiency of identifying and classifying personal data, aiming to minimize compliance risks. Speed is measured by:

• Time it takes for the detection engine for scanning new data entities
• Time it takes for data owners to reconcile classifications and resolve tickets
• The frequency of data tagging at creation by data owners

These measurements ensure our data classification system is effective, accurate, and efficient, safeguarding our personal data.

Considerations for Building a Data Classification System

It is important to be aware of issues that may be present with the outlined approach in general. Below are some challenges that we’ve considered when building a data classification system:

• Post-Processing Classification: The outlined approach mostly relies on post-processing classification, which means that schema information is added after data has been collected and stored. In a modern data world where data and metadata are constantly changing, post-processing cannot catch up with data evolution.
• Inconsistent Classifications: Data generally flows from online to offline through ETL (extract, transform, and load) processes, and then reverse ETLing back to the online world. However, data classification that is performed independently in both worlds can lead to inconsistent classifications.
• Waste of Process Cost: Duplicate annotations can be made for the same data in the online and offline domains, which might result in increased costs for data classification processes.

To address these challenges, we describe the process of “shifting left” with data classification and how we started to push developers to annotate their data at the beginning of the data lifecycle.

Shifting Left

Instead of thinking about governance and data classification as an activity that happens post-hoc, we’ve started to embed the annotation process directly into data schemas as they are being created and updated. This enables us to:

• Shift Classification from Data to Schema: The schema annotation process takes place earlier in the data lifecycle at the point of data collection. This keeps annotations updated as data evolves and ensures data is annotated before collection and consumption, allowing for immediate policy enforcement.
• Shift Classification from Offline to Online: Traditionally done offline, data classification is now integrated into production services, ensuring data is structured and formatted correctly from the start. Leveraging data lineage information enables automated annotation, reducing the need for manual effort and lowering process costs.
• Shift from Data Steward to Data Owner: Oftentimes, stewardship, or the responsible management and oversight, of the data is conducted by people who are downstream of data creation, such as data consumers or governance professionals. This change shifts stewardship to the data producers, merging the roles of data steward and data owner. This empowers the team that owns the data to manage it more effectively and scale operations.

Schema Annotation Enforcement

Focusing on our most crucial online data, we have started executing on shifting left by directly integrating with our internal schema definition language that is known for its annotation capabilities. We now mandate that developers include personal data annotations at the source when creating new data models, providing guidance on accurate tagging. This requirement is enforced through checks that run in our CI/CD pipelines which:

• Automatically suggest data elements: Based on the schema’s metadata, we automatically detect the data elements for all fields defined in the schema with our detection service.
• Validate data elements: Annotations are validated against our own taxonomy and schemas are enforced and all fields are annotated, even when it is not considered personal.
• Warn about downstream impact: We notify data owners when annotations can impact downstream services such as offline data pipelines and direct them to the proper resources for handling.

While shifting left has significantly helped to push classifying data earlier and increase coverage of schema annotations, this does not discount the importance of the rest of the classification process. Classifications that happen post-process are necessary for instance in cases where data storages that do not include well-defined schemas. Therefore, continued investments are still needed in detection and reconciliation to cover areas that cannot be shifted left and to verify annotations that may have already been made by owners as a second layer of protection.

Conclusion/Lessons Learned

The Airbnb data classification framework has been successful in advancing data management, security, and privacy. Reflecting on the journey, it has offered invaluable insights that have shaped our methodologies. Key takeaways include:

• Adopting a unified strategy for classifying online and offline personal data to streamline processes
• Implementing a ‘Shift Left’ approach to engage with data owners early in the development cycle
• Addressing classification uncertainties through clear guidelines and decision-making
• Enhancing education and training initiatives for data owners and consumers

As the data landscape continues to rapidly change, these lessons will guide future data classification efforts and ensure continued trust and protection of customer data.

Acknowledgements

Our data classification strategy has evolved over many years and we’ve been able to quickly adapt and iterate thanks to our decision to build an in-house solution. Security, privacy, and compliance are of utmost importance at Airbnb, and this work would not be possible without the contribution of many of our cross-functional partners and leaders.

This includes, but are not limited to: Bill Meng, Aravind Selvan, Juan Tamayo, Xiao Chen, Pinyao Guo, Wendy Jin, Liam McInerney, Pat Moynahan, Gabriel Gejman, Marc Blanchou, Brendon Lynch, and many others.

If this type of work interests you, check out some of our related positions at Careers at Airbnb or check out more resources in the Airbnb Tech Blog!

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

Airbnb’s Use of A New Flink platform evolved from Apache Hadoop® Yarn

Introduction

At Airbnb, Apache Flink was introduced in 2018 as a supplementary solution for stream processing. It ran alongside Apache Spark™ Streaming for several years before transitioning to become the primary stream processing platform. In this blog post, we will delve into the evolution of Flink architecture at Airbnb and compare our prior Hadoop Yarn platform with the current Kubernetes-based architecture. Additionally, we will discuss the efforts undertaken throughout the migration process and explore the challenges that arose during this journey. In the end we will summarize the impact, learnings along the way and future plans.

Architecture Evolution

The evolution of Airbnb’s streaming processing architecture based on Apache Flink can be categorized into three distinct phases:

Phase One: Flink jobs operated on Hadoop Yarn with Apache Airflow serving as the job scheduler.

Around 2018, several teams at Airbnb adopted Flink as their streaming processing engine, mainly due to its superior low-latency capabilities compared to Spark Streaming. During this period, Flink jobs were running on Hadoop Yarn, and Airflow was employed as the workflow manager for task scheduling and dependency management.

The selection of Airflow as the workflow manager was largely influenced by its widespread use in addressing various job scheduling needs, as there were no other user-friendly open-source alternatives readily available at that time. Each team was responsible for handling their Airflow Directed Acyclic Graphs (DAGs), job source code, and the requisite dependency JARs. Typically, Flink JAR files were locally built before deployment to Amazon S3.

The architecture catered to our requirements during that period with a limited range of use cases.

From 2019 onwards, Apache Flink gained significant traction at Airbnb, replacing Spark Streaming as the primary stream processing platform. With the scaling in usage of Flink we encountered various challenges and limitations in this architecture. To begin with, Airflow’s batch-oriented design, relying on polling intervals, did not match Airbnb’s needs, and we experienced significant delays in job start and failure recovery, often causing SLA violations for low-latency use cases. Airflow also caused a singleton issue as duplicate job submissions occasionally occur due to race conditions among Airflow workers and user operations not following expected patterns. Besides, Airflow’s Directed Acyclic Graph (DAG) structure is complex and does not function well with some of Airbnb’s streaming use cases. We also encountered engineering context mismatch in this architecture: product engineers might find themselves unfamiliar with Apache Airflow and Hadoop, resulting in a steep learning curve when setting up new Apache Flink jobs.

To tackle the above technical and operational challenges, we started to explore new possibilities. Our initial step involved replacing Airflow with a customized lightweight streaming job scheduler, marking the inception of Phase Two.

Phase Two: Flink jobs operated on Hadoop Yarn, with a lightweight streaming job scheduler.

At a high level, Airflow was replaced by a lightweight streaming job scheduler operating on Kubernetes. The job scheduler contains a master node and a pool of worker nodes:

The master node is responsible for managing the metadata of all Flink jobs and ensuring the proper life cycle of each worker node. This includes tasks such as parsing user-provided job configurations, synchronizing metadata and job statuses with Apache Zookeeper™, and ensuring that worker nodes consistently maintain their expected states.

A worker node is responsible for handling the dependencies and life cycle of a single Flink job. Workers package the necessary dependencies, submit the Flink job to Hadoop Yarn, continuously monitor its status, and in the event of a failure, it triggers an immediate restart.

The Phase 2 design resulted in faster turnaround time and reduced downtime during job restarts. It also resolved single point of failure issues with Zookeeper.

As usage of Flink grew, we encountered new challenges in Phase Two:

• Lack of CI/CD: Flink developers had to devise their own version control strategies.
• Absence of native secrets management: There is no vanilla secrets management on Hadoop Yarn.
• Limited resource and dependency isolation: Each supported Flink version had to be manually preinstalled on the Yarn cluster. While Yarn’s resource queues could provide some level of resource isolation, job-level isolation was absent.
• Service Discovery complexity: As more use cases were onboarded, each potentially requiring access to various internal Airbnb services, configuring service access on Yarn proved to be cumbersome. It forced a binary choice between enabling service access for the entire cluster or none at all.
• Monitoring and debugging challenges: Managing and maintaining the logging pipeline and SSH access became non-trivial tasks on a multi-tenant Yarn cluster.
• Ongoing complexity and dependencies: Although the Flink job scheduler was lightweight compared to Airflow, it introduced additional complexities.

Phase Three (current state): Flink jobs run on Kubernetes, and the job scheduler is eliminated.

Deploying Flink on Kubernetes allows direct Flink deployment on a running Kubernetes cluster. With this integration we can explore enabling efficient autoscaling and the Kubernetes operator to simplify the management of Flink jobs and clusters.

Flink on Kubernetes offers several advantages over Hadoop Yarn addressing the above challenges:

• Developer experience: Standardized by integrating with the existing CI/CD systems.
• Secrets Management: With Flink on Kubernetes, each Flink job can securely store its own secrets within the pods. This provides a more secure way to manage sensitive information.
• Isolated Environment: Jobs running on Flink on Kubernetes benefit from isolation at both the resource and dependency levels. Each job can run on its dedicated Flink version if supported by its image, allowing for better management of dependencies.
• Enhanced Monitoring: Integration with Airbnb’s pre-defined logging and metric sidecars on Kubernetes simplifies setup and improves monitoring. This enables detailed insights into individual pods and rate limiting for logging per pod, making it easier to track and troubleshoot issues.
• Service Discovery: Flink jobs now adhere to Airbnb’s standardized approach for service discovery, using the cluster mesh. This ensures consistent and reliable communication between services.
• Simplified SSH access: Users with the appropriate permissions can now SSH into the Flink pod without the need for an SSH tunnel. This provides greater flexibility and control over SSH permissions per job.

Additionally, we’ve observed an increasing level of Kubernetes support and adoption within the Flink community, which increased our confidence in running Flink on Kubernetes.

It’s worth mentioning that Kubernetes brings its own risks and limitations. For instance, a single Flink task manager failover can lead to the pause of the entire job process. This can pose issues in scenarios with frequent node rotations within Kubernetes and large jobs deployed with hundreds of task managers. For context, node rotation on Kubernetes is performed to ensure the operability and stability of the cluster. It involves replacing existing nodes with new ones, typically with updated configurations or to perform maintenance tasks, with the goals of applying host configuration changes, maintaining node balance and enhancing operational efficiency. In comparison, node rotations on Yarn occur less frequently, so the impact on job availability is less significant. We will explore how we are mitigating these challenges in the Future Work section.

Components Deep Dive

Below is an overview of our current architecture:

To provide a better understanding of the system, below is a deep dive of the five primary components, as well as how users interact with them when setting up a new Flink job:

• Job configurations: This serves as an abstraction layer over Kubernetes and CI/CD components, providing Flink users with a simplified interface for creating Flink application templates. It shields users from the complexities of the underlying Kubernetes infrastructure. Flink users define the core specifications of their Flink job via a configuration file. This includes critical information like the entrypoint class name, job parallelism, and the necessary ingress services and sinks.

• Image management: This component involves the pre-construction of Flink base images, which are bundled with essential dependencies required to access Airbnb resources. These images are stored in Amazon Elastic Container Registry and can be readily deployed with user Jars or further customized to meet specific user needs.

• CI/CD: By introducing a few customizations to support Flink’s stateful deployment, we’ve integrated Flink with our existing CI/CD system, providing a standardized version control and continuous delivery experience. Flink jobs are deployed within Kubernetes, each residing in its distinct namespace to ensure isolation and effective administration.

• Flink portal: an API service that offers essential features for managing the states of Flink jobs. These features include stopping a Flink job with a savepoint and querying completed checkpoints on Amazon S3. Additionally, it provides a self-service UI portal, enabling users to monitor and check the status of their jobs. Users also gain access to critical job state management functionalities, empowering them to either initiate the job from a bootstrapped savepoint or resume it from a previous checkpoint.

• Flink job runtime: Each Flink job is deployed as an independent application cluster on Kubernetes. To ensure fault tolerance and state storage, Zookeeper, ETCD, and Amazon S3 are utilized. Additionally, pre-configured sidecar containers accompany the Flink containers to provide support for critical functions such as logging, metrics, DNS, and more. A service mesh is employed to facilitate communication between Flink jobs and other microservices.

Impact

Improved Developer Velocity

Onboarding Flink jobs is faster, where our developers noted that it takes hours instead of days, and developers can focus more on their application logic.

Improvement in Flink Job Availability and Latency

The architecture of Flink on Kubernetes improves job availability and scheduling latency by eliminating certain components of the Flink client and job scheduler found in Flink on Yarn.

Cost Savings in Infrastructure

The streamlining of Flink infrastructure complexity and the removal of certain components, such as the job scheduler, have resulted in cost savings in our infrastructure. Additionally, by running Flink jobs on a shared Kubernetes cluster at Airbnb, we could potentially improve the overall cost efficiency of our company’s infrastructure.

Future Work

Improvement in Job Availability

In the Flink world, node rotations in Kubernetes can cause job restarts and result in downtime. While Flink itself can recover from job restarts without data loss, the potential downtime and availability impact may be unfavorable for highly latency-sensitive applications. To address this, there are a few approaches we are evaluating.

1. Reducing the number of node rotations to minimize job restarts.
2. Faster job recovery.

Enable Job Autoscaling

With the introduction of Reactive Mode in Flink 1.13, users can dynamically adjust the parallelism of their jobs without the need for a job restart. This job auto scaling feature can enhance job stability and cost efficiency. In the future we could enable autoscaling for Flink Kubernetes workloads by leveraging system metrics (such as CPU usage) and Flink metrics (such as backpressure), to determine the appropriate parallelism.

Flink Kubernetes Operator

The Flink Kubernetes Operator utilizes Custom Resources and functions as a controller to manage the entire production lifecycle of Flink applications. By leveraging the operator, we can streamline the operation and deployment processes for Flink jobs. It provides better control over deployment and lifecycle of jobs, and an out of box solution for autoscaling and auto tuning.

Conclusion

To summarize, the migration of Airbnb’s streaming processing architecture based on Apache Flink from Hadoop Yarn to Kubernetes has been a significant milestone in enhancing our streaming data processing capabilities. This transition has resulted in a more streamlined and user-friendly experience for Flink developers. By overcoming challenges that were complex to address on Yarn, we have laid the foundation for more efficient and effective streaming data processing.

As we look ahead, we are committed to further refining our approach and resolving any remaining challenges. We are enthusiastic about the ongoing growth and potential of Apache Flink within our company, and we anticipate continued innovation and improvement in the future.

If this kind of work sounds appealing to you, check out our open roles — we’re hiring!

Appreciations

The Flink on Kubernetes platform would not have been possible without cross-functional and cross-org collaborators as well as leadership support. They include, but are not limited to: Jingwei Lu, Long Zhang, Daniel Low, Weibo He, Zack Loebel-Begelman, Justin Cunningham, Adam Kocoloski, Liyin Tang and Nathan Towery.

Special thanks to the broader Airbnb data community members who provided input or aid to the implementation team throughout the design, development, and launch phases.

We also want to thank Wei Hou and Xu Zhang for their support in authoring this post during their time at Airbnb.

****************

Apache Spark™, Apache Airflow™, and Apache ZooKeeper™ are trademarks of The Apache Software Foundation.

Apache Flink® and Apache Hadoop® are registered trademarks of The Apache Software Foundation.

Kubernetes® is a registered trademark of The Linux Foundation.

Amazon S3 and AWS are trademarks of Amazon.com, Inc. or its affiliates.

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

By Tiantian Zhang, Shuai Shao (Shawn)

Introduction

At Airbnb, we have developed Brandometer, a state-of-the-art natural language understanding (NLU) technique for understanding brand perception based on social media data.

Brand perception refers to the general feelings and experiences of customers with a company. Quantitatively, measuring brand perception is an extremely challenging task. Traditionally, we rely on customer surveys to find out what customers think about a company. The downsides of such a qualitative study is the bias in sampling and the limitation in data scale. Social media data, on the other hand, is the largest consumer database where users share their experiences and is the ideal complementary consumer data to capture brand perceptions.

Compared to traditional approaches to extract concurrency and count-based top relevant topics, Brandometer learns word embeddings and utilizes embedding distances to measure relatedness of brand perceptions (e.g., ‘belonging’, ‘connected’, ‘reliable’). Word embedding represents words in the form of real-valued vectors, and it performs well in reserving semantic meanings and relatedness of words. Word embeddings obtained from deep neural networks are arguably the most popular and evolutionary approaches in NLU. We explored a variety of word embedding models, from quintessential algorithms Word2Vec and FastText, to the latest language model DeBERTa, and compared them in terms of generating reliable brand perception scores.

For concepts represented as words, we use similarity between its embedding and that of “Airbnb” to measure how important the concept is with respect to the Airbnb brand, which is named as Perception Score. Brand Perception is defined as Cosine Similarity between Airbnb and the specific keyword:

where

In this blog post, we will introduce how we process and understand social media data, capture brand perceptions via deep learning and how to ‘convert’ the cosine similarities to calibrated Brandometer metrics. We will also share the insights derived from Brandometer metrics.

Brandometer Methodology

Problem Setup and Data

In order to measure brand perception on social media, we assessedall Airbnb related mentions from 19 platforms (e.g., X — formerly known as Twitter, Facebook, Reddit, etc) and generated word embeddings with state-of-the-art models.

In order to use Social media data to generate meaningful word embeddings for the purpose of measuring brand perception, we conquered two challenges:

• Quality: Social media posts are mostly user-generated with varying content such as status sharing and reviews, and can be very noisy.
• Quantity: Social media post sparsity is another challenge. Considering that it typically requires some time for social media users to generate data in response to certain activities and events, a monthly rolling window maintains a good balance of promptness and detectability. Our monthly dataset is relatively small (around 20 million words) as compared to a typical dataset used to train good quality word embeddings (e.g., about 100 billion words for Google News Word2Vec model). Warm-start from pre-trained models didn’t help since the in-domain data barely moved the learned embeddings.

We developed multiple data cleaning processes to improve data quality. At the same time, we innovated the modeling techniques to mitigate the impact on word embedding quality due to data quantity and quality.

In addition to data, we explored and compared multiple word embedding training techniques with the goal to generate reliable brand perception scores.

Word2Vec

Word2Vec is by far the simplest and most widely used word embedding model since 2013. We started with building CBOW-based Word2Vec models using Gensim. Word2Vec produced decent in-domain word embeddings, and more importantly, the concept of analogies. In our domain-specific word embeddings, we are able to capture analogies in the Airbnb domain, such as “host” — “provide” + “guest” ~= “need”, “city” — “mall” + “nature” ~= “park”.

FastText

FastText takes into account the internal structure of words, and is more robust to out-of-vocabulary words and smaller datasets. Moreover, as inspired by Sense2Vec, we associate words with sentiments (i.e., POSITIVE, NEGATIVE, NEUTRAL), which forms brand perception concepts on the sentiment levels.

DeBERTa

Recent progress in transformer-based language models (e.g., BERT) has significantly improved the performance of NLU tasks with the advantage of generating contextualized word embeddings. We developed DeBERTa based word embeddings, which works better with smaller dataset and pays more attention to surrounding context via disentangled attention mechanisms. We trained everything from scratch (including tokenizer) using Transformers, and the concatenated last attention layer embeddings resulted in the best word embeddings for our case.

Brand Perception Score Stabilization and Calibration

The variability of word embeddings has been widely studied (Borah, 2021). The causes range from the underlying stochastic nature of deep learning models (e.g., random initialization of word embeddings, embedding training which leads to local optimum for global optimization criteria) to the quantity and quality changes of data corpus across time.

With Brandometer, we need to reduce the variability in embedding distances to generate stable time series tracking. Stable embedding distances helped preserve the inherent patterns and structures present in the time series data, and hence it contributes to better predictability of the tracking process. Additionally, it made the tracking process more robust to noisy fluctuations. We studied the influential factors and took the following steps to reduce:

1. Score averaging over repetitive training with bootstrap sampling
2. Rank-based perception score

Score averaging over repetitive training with upsampling

For each month’s data, we trained N models with the same hyper-parameters, and took the average of N perception scores as the final score for each concept. Meanwhile, we did upsampling to make sure that each model iterated on an equal number of data points across months.

We defined variability as:

where

CosSim(w) refers to the cosine similarity based perception score defined in Eq. 1, A refers to the algorithm, M refers to the time window (i.e. month), V refers to the vocabulary and |V| is the vocabulary size, and n refers to the number of repetitively trained models.

As N approaches 30, the score variability values converge and settle within a narrow interval. Hence, we picked N = 30 for all.

Rank-based perception score

Based on Maria Antoniak’s work, we used the overlap between nearest neighbors to measure the stability of word embeddings, since the relative distances matter more than the absolute distance values in downstream tasks. Therefore, we also developed rank-based scores, which shows greater stability as compared to similarity-based scores.

For each word, we first ranked them in descending order of cosine similarity via Eq. 1. The rank-based similarity score is then computed as 1/rank(w) where w∈V. More relevant concepts will have higher rank-based perception scores.

The score variability is defined the same as Variability(A, M, V) in Eq. 2 except that

where RankSim(w) refers to the rank based perception score. With rank-based scores, when N approaches to 30, the score variability values converge to a much narrower interval especially for DeBERTa.

Selection of Score Output by Designed Metrics

One challenge of this project was that we didn’t have a simple and ultimate way to conclude which score output was better since there is no objective ‘truth’ of brand perception. Instead, we defined a new metric to learn some characteristics of the score.

Average Variance Across Different Period (AVADP)

• We first picked the group of top relevant brand perceptions for Airbnb: ‘host,’ ‘vacation,’ ‘rental,’ ‘love,’ ‘stay,’ ‘home,’ ‘booking,’ ‘travel,’ ‘guest’.
• Higher value indicates more fluctuations across different periods — likely a bad thing, because the selected brand perception is assumed to be relatively stable and hence should not vary too much month by month.

We checked these statistics on the calibrated results as shown above. We can see that the ranked-based score is the winner as compared to similarity-based scores:

• Lower AVADP: More fluctuations than the non-ranked across a different period — likely a good thing, because the selected brand perception is assumed to be relatively stable and hence should not vary too much month by month.

Use Cases of Brandometer

Though we set out to solve the problem of brand measurement, we believe use cases can go above and beyond:

Use Cases Deep Dive

Industry Analysis: Top Brand Perception among Key Players [Monthly Top Perception]

With top perceptions such as “Stay” and “Home,” Airbnb provides a brand image of “belonging”, echoing our mission statement and unique supply inventory, while other companies have “Rental,” “Room,” “Booking,” a description of functionality, not human sensation.

Top Emerging Perception reveals major events discussed online [Monthly Top Perception]

The Top 10 Perceptions are generally stable month to month. The top standing perceptions include

• Home, Host, Stay, Travel, Guest, Rental, etc.

Meanwhile, we use Brandometer to monitor emerging perceptions that jump to the top list, which may reflect major events associated with the brand or user preference changes.

Major Campaign Monitor (Time Series Tracking)

Businesses create campaigns to promote products and expand the brand image. We were able to capture a perception change on one specific Brand Theme after a related campaign.

These use cases are just the beginning. Essentially, this is an innovative way of gathering massive online input as we learn the needs and perception of the community. We will constantly reflect on how we leverage these insights to continually improve the Airbnb experience for our community.

Next Steps

Airbnb’s innovative Brandometer has already demonstrated success in capturing brand perception from social media data. There are several directions for future improvement:

• Better content segmentation for clearer and more concise insights.
• Develop more metrics reflecting social media brand perception.
• Enhance data foundation, not just Airbnb, but other companies in the same market segment to get more comprehensive insights.

If this kind of work sounds appealing to you, check out our open roles — we’re hiring!

Acknowledgments

Thanks to Mia Zhao, Bo Zeng, Cassie Cao for contributing the best ideas on improving and landing Airbnb Brandometer. Thanks to Jon Young, Narin Leininger, Allison Frelinger for the support of social media data consolidation. Thanks to Linsha Chen, Sam Barrows, Hannah Jeton, and Irina Azu who provide feedback and suggestions. Thanks to Lianghao Li, Kelvin Xiong, Nathan Triplett, Joy Zhang, Andy Yasutake for reviewing and polishing the blog post content and all the great suggestions. Thank Joy Zhang, Tina Su, Andy Yasutake for leadership support!

Special thanks to Joy Zhang, who initiated the idea, for all the inspiring conversations, continuous guidance and support!

Airbnb Brandometer: Powering Brand Perception Measurement on Social Media Data with AI was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By: Eli Hart, Ben Schwab, and Yvonne Wong

Trio is Airbnb’s framework for Jetpack Compose screen architecture in Android. It’s built on top of Mavericks, Airbnb’s open source state management library for Jetpack. In this blog post series, we’ve been breaking down how Trio works to help explain our design decisions, in the hopes that other teams might benefit from aspects of our approach.

We recommend starting with Part 1, about Trio’s architecture, and then reading Part 2, about how navigation works in Trio, before you dive into this article. In this third and final part of our series, we’ll discuss how Props in Trio allow for simplified, type-safe communication between ViewModels. We’ll also share an update on the current adoption of Trio at Airbnb and what’s next.

Trio Props

To better understand Props, let’s look at an example of a simple Message Inbox screen, composed of two Trios side by side. There is a List Trio on the left, showing inbox messages, and a Details Trio on the right, showing the full text of a selected message.

The two Trios are wrapped by a parent screen, which is responsible for instantiating the two children, passing along data to them, and positioning them in the UI. As you might recall from Part 2, Trios can be stored in State; the parent’s State includes both the message data as well as the child Trios.

data class ParentState(
  val inboxMessages: List<Message>,
  val selectedMessage: Message?,
  val messageListScreen: Trio<ListProps>,
  val messageDetailScreen: Trio<DetailsProps>,
} : MavericksState

The parent’s UI decides how to display the children, which it accesses from the State. With Compose UI, it’s easy to apply custom layout logic: we show the screens side by side when the device is in landscape mode, and in portrait we show only a single screen, depending on whether a message has been selected.

@Composable 
override fun TrioRenderScope.Content(state: ParentState) {
  if (LocalConfiguration.current.orientation == ORIENTATION_LANDSCAPE) {
    Row(Modifier.fillMaxSize()) {
      ShowTrio(state.listScreen, modifier = Modifier.weight(1f))
      ShowTrio(state.detailScreen)
    }
  } else {
    if (state.selectedMessage == null) {
      ShowTrio(state.listScreen)
    } else {
      BackHandler { viewModel.clearMessageSelection() }
      ShowTrio(state.detailScreen)
    }
  }
}

Both child screens need access to the latest message state so they know which content to show. We can provide this with Props!

Props are a collection of Kotlin properties, held in a data class and passed to a Trio by its parent.

Unlike Arguments, Props can change over time, allowing a parent to provide updated data as needed throughout the lifetime of the Trio. Props can include Lambda expressions, allowing a screen to communicate back to its parent.

A child Trio can only be shown in a parent that supports its Props type. This ensures compile-time correctness for navigation and communication between Trios.

Defining Props

Let’s see how Props are used to pass message data from the parent Trio to the List and Details Trios. When a parent defines child Trios in its State, it must include the type of Props that those children require. For our example, the List and Details screen each have their own unique Props.

The List screen needs to know the list of all Messages and whether one is selected. It also needs to be able to call back to the parent to tell it when a new message has been selected.

data class ListProps(
  val selectedMessage: Message?,
  val inboxMessages: List<Message>,
  val onMessageSelected: (Message) -> Unit,
)

The Details screen just needs to know which message to display.

data class DetailProps(
  val selectedMessage: Message?
)

The parent ViewModel holds the child instances in its State, and is responsible for passing the latest Props value to the children.

Passing Props

So, how does a parent Trio pass Props to its child? In its init block it must use the launchChildInitializer function — this function uses a lambda to select a Trio instance from the State, specifying which Trio is being targeted.

class ParentViewModel: TrioViewModel {

  init {
    launchChildInitializer({ messageListScreen }) { state ->
      ListProps(
        state.selectedMessage,
        state.inboxMessages,
        ::showMessageDetails
      )
    }

   launchChildInitializer({ detailScreen }) { state ->
      DetailProps(state.selectedMessage)
   }
  }

  fun showMessageDetails(message: Message?) ...
}

The second lambda argument receives a State value and returns a new Props instance to pass to the child. This function manages the lifecycle of the child, initializing it with a flow of Props when it is first created, and destroying it if it is ever removed from the parent’s state.

The lambda to rebuild Props is re-invoked every time the Parent’s state changes, and any new value of Props is passed along to the child through its flow.

A common pattern we use is to include function references in the Props, which point to functions on the parent ViewModel. This allows the child to call back to the parent for event handling. In the example above we do this with the showMessageDetails function. Props can also be used to pass along complex dependencies, which forms a dependency graph scoped to the parent.

Note that we cannot pass Props to a Trio when it is created, like we do with Args. This is because Trios must be able to be restored after process death, and so the Trio class, as well as the Args used to create it, are Parcelable. Since Props can contain lambdas and other arbitrary objects that cannot be safely serialized, we must use the above pattern to establish a flow of Props from parent to child that can be reestablished even after process recreation. Navigation and inter-screen communication would be a lot simpler if we didn’t have to handle process recreation!

Using Props

In order for a child Trio to use Props data in its UI, it first needs to be copied to State.

Child ViewModels override the function updateStateFromPropsChange to specify how to incorporate Prop values into State. The function is invoked every time the value of Props changes, and the new State value is updated on the ViewModel. This is how children stay up-to-date with the latest data from their parent.

class ListViewModel : TrioViewModel<ListProps, ListState> {

  override fun updateStateFromPropsChange(
    newProps: ListProps,
    thisState: ListState
  ): ListState {
    return thisState.copy(
      inboxMessages = newProps.inboxMessages,
      selectedMessage = newProps.selectedMessage
    )
  }

  fun onMessageSelected(message: Message) {
    props.onMessageSelected(message)
  }
}

For non-state values in Props, such as dependencies or callbacks, the ViewModel can access the latest Props value at any time via the props property. For example, we do this in the onMessageSelected function in the sample code above. The List UI will invoke this function when a message is selected, and the event will be propagated to the parent through Props.

There were a lot of complexities when implementing Props — for example, when handling edge cases around the Trio lifecycle and restoring state after process death. However, the internals of Trio hide most of the complexity from the end user. Overall, having an opinionated, codified system with type safety for how Compose screens communicate has helped improve standardization and productivity across our Android engineering team.

Standardizing Screen Flow Props

One of the most common UI patterns at Airbnb is to coordinate a stack of screens. These screens may share some common data, and follow similar navigation patterns such as pushing, popping, and removing all the screens of the backstack in tandem.

Earlier, we showed how a Trio can manage a list of children in its State to accomplish this, but it’s tedious to do that manually. To help, Trio provides a standard “screen flow” implementation, which consists of a parent ScreenFlow Trio and related child Trio screens. The parent ScreenFlow automatically manages child transactions, and renders the top child in its UI. It also broadcasts a custom Props class to its children, giving access to shared state and navigation functions.

Consider building a Todo app that has a TodoList screen, a TaskScreen, and an EditTaskScreen. These screens can all share a single network request that returns a TodoList model. In Trio terms, the TodoList data model could be the Props for these three screens.

To manage these screens we use ScreenFlow infrastructure to create a TodoScreenFlow Trio. Its state extends ScreenFlowState and overrides a childScreenTransaction property to hold the transactions. In this example, the flow’s State was initialized to start with the TodoListScreen, so it will be rendered first. The flow’s State object also acts as the source of truth for other shared state, such as the TodoList data model.

data class TodoFlowState(
  @PersistState
  override val childScreenTransactions: List<ScreenTransaction<TodoFlowProps>> = listOf(
    ScreenTransaction(Router.TodoListScreen.createFullPaneTrio(NoArgs))
  ),
  // shared state
  val todoListQuery: TodoList?,
) : ScreenFlowState<TodoFlowState, TodoFlowProps>

This state is private to the TodoScreenFlow. However, the flow defines Props to share the TodoList data model, callbacks like a reloadList lambda, and a NavController with its children.

data class TodoFlowProps(
  val navController: NavController<TodoFlowProps>,
  val todoListQuery: TodoList?,
  val reloadList: () -> Unit,
)

The NavController prop can be used by the children screens to push another sibling screen in the flow. The ScreenFlowViewModel base class implements this NavController interface, managing the complexity of integrating the navigation actions into the screen flow’s state.

interface NavController<PropsT>(
   fun push(router: TrioRouter<*, in PropsT>)
   fun pop()
)

Lastly, the navigation and shared state is wired into a flow of Props when the TodoScreenFlowViewModel overrides createFlowProps. This function will be invoked anytime the internal state of TodoScreenFlowViewModel changes, meaning any update to TodoList model will be propagated to the children screens.

class TodoScreenFlowViewModel(
  initializer: Initializer<NavPopProps, TodoFlowState>
) : ScreenFlowViewModel<NavPopProps, TodoFlowProps, TodoFlowState>(initializer) {

  override fun createFlowProps(
    state: TodoFlowState,
    props: NavPopProps
  ): TodoFlowProps {
    return TodoFlowProps(
      navController = this,
      state.todoListQuery,
      ::reloadList,
    )
  }
}

Inside one of the children screen’s ViewModels, we can see that it will receive the shared Props:

class TodoListViewModel(
  initializer: Initializer<TodoFlowProps, TodoListState>
) : TrioViewModel<TodoFlowProps, TodoListState>(initializer) {

  override fun updateStateFromPropsChange(
    newProps: TodoFlowProps, 
    thisState: TodoTaskState
  ): TodoTaskState {
      // Incorporate the shared data model into this Trio’s private state passed to its UI:
      return thisState.copy(todoListQuery = newProps.todoListQuery)
  }

  fun navigateToTodoTask(task: TodoTask) {
    this.props.navController.push(Router.TodoTaskScreen, TodoTaskArgs(task.id))
  }
}

In navigateToTodoTask, the NavController prepared by the flow parent is used to safely navigate to the next screen in the flow (guaranteeing it will receive the shared TodoFlowProps). Internally, the NavController updates the ScreenFlow’s childScreenTransactions triggering the ScreenFlow infra to provide the shared TodoFlowProps to the new screen, and render the new screen.

Trio’s Success at Airbnb

Development history and launch

We started designing Trio in late 2021, with the first Trio screens seeing production traffic in mid 2022.

As of March 2024, we now have over 230 Trio screens with significant production traffic at Airbnb.

From surveying our developers, we’ve heard that many of them enjoy the overall Trio experience; they like having clear and opinionated patterns and are happy to be in a pure Compose environment. As one developer put it, “Props was a huge plus by allowing multiple screens to share callbacks, which simplified some of my code logic a lot.” Another said, “Trio makes you unlearn bad habits and adopt best practices that work for Airbnb based on our past learnings.” Overall, our team reports faster development cycles and cleaner code. “It makes Android development faster and more enjoyable,” is how one engineer summed it up.

Dev Tooling

To support our engineers, we have invested in IDE tooling with an in-house Android Studio Plugin. It includes a Trio Generation tool that creates all of the files and boilerplate for a new Trio, including routing, mocks, and tests.

The tool helps the user choose which Arguments and Props to use, and helps with other customization such as setting up custom Flows. It also allows us to embed educational experiences to help newcomers ramp up with Trio.

One piece of feedback we heard from engineers was that it was tedious to change a Trio’s Args or Props types, since they are used across many different files.

We leveraged our IDE plugin to provide a tool to automatically change these values, making this workflow much faster.

Our team leans heavily on tooling like this, and we’ve found it to be very effective in improving the experience of engineers at Airbnb. We’ve adopted Compose Multiplatform for our Plugin UI development which we believe made building powerful developer tooling more feasible and enjoyable.

Conclusion

Overall, with more than 230 of our production screens implemented as Trios, Trio’s organic adoption at Airbnb has proven that many of our bets and design choices were worth the tradeoffs.

One change we are anticipating, though, is to incorporate shared element transitions between screens once the Compose framework provides APIs to support that functionality. When Compose APIs for this are available, we’ll likely have to redesign our navigation APIs accordingly.

Thanks for following along with the work we’ve been doing at Airbnb. Our Android Platform team works on a variety of complex and interesting projects like Trio, and we’re excited to share more in the future.

If this kind of work sounds appealing to you, check out our open roles — we’re hiring!

Introducing Trio | Part III was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By: Varant Zanoyan, Nikhil Simha Raprolu

Airbnb is happy to announce that Chronon, our ML Feature Platform, is now open source.

We’re excited to be making this announcement along with our partners at Stripe, who are early adopters and co-maintainers of the project.

This blog post covers the main motivation and functionality of Chronon. For an overview of core concepts in Chronon, please see this previous post.

Background

We built Chronon to relieve a common pain point for ML practitioners: they were spending the majority of their time managing the data that powers their models rather than on modeling itself.

Prior to Chronon, practitioners would use one of the following two approaches:

1. Replicate offline-online: ML practitioners train the model with data from the data warehouse, then figure out ways to replicate those features in the online environment. The benefit of this approach is that it allows practitioners to utilize the full data warehouse, both the data sources and powerful tools for large-scale data transformation. The downside is that this leaves no clear way to serve model features for online inference, resulting in inconsistencies and label leakage that severely affect model performance.
2. Log and wait: ML practitioners start with the data that is available in the online serving environment from which the model inference will run. They log relevant features to the data warehouse. Once enough data has accumulated, they train the model on the logs, and serve with the same data. The benefit of this approach is that consistency is guaranteed and leakage is unlikely. However the major drawback is that it can result in long wait times, hindering the ability to respond quickly to changing user behavior.

The Chronon approach allows for the best of both worlds. Chronon requires ML practitioners to define their features only once, powering both offline flows for model training as well as online flows for model inference. Additionally, Chronon offers powerful tooling for feature chaining, observability and data quality, and feature sharing and management.

How It Works

Below we explore the main components that power most of Chronon’s functionality using a simple example derived from the quickstart guide. You can follow that guide to run this example.

Let’s assume that we’re a large online retailer, and we’ve detected a fraud vector based on users making purchases and later returning items. We want to train a model to predict whether a given transaction is likely to result in a fraudulent return. We will call this model each time a user starts the checkout flow.

Defining Features

Purchases Data: We can aggregate the purchases log data to the user level to give us a view into this user’s previous activity on our platform. Specifically, we can compute SUMs, COUNTs and AVERAGEs of their previous purchase amounts over various time windows.

source = Source(
    events=EventSource(
        table="data.purchases", # This points to the log table in the warehouse with historical purchase events, updated in batch daily
        topic="events/purchases", # The streaming source topic
        query=Query(
            selects=select("user_id","purchase_price"), # Select the fields we care about
            time_column="ts") # The event time
    ))

window_sizes = [Window(length=day, timeUnit=TimeUnit.DAYS) for day in [3, 14, 30]] # Define some window sizes to use below

v1 = GroupBy(
    sources=[source],
    keys=["user_id"], # We are aggregating by user
    online=True,
    aggregations=[Aggregation(
            input_column="purchase_price",
            operation=Operation.SUM,
            windows=window_sizes
        ), # The sum of purchases prices in various windows
        Aggregation(
            input_column="purchase_price",
            operation=Operation.COUNT,
            windows=window_sizes
        ), # The count of purchases in various windows
        Aggregation(
            input_column="purchase_price",
            operation=Operation.AVERAGE,
            windows=window_sizes
        ), # The average purchases by user in various windows
        Aggregation(
            input_column="purchase_price",
            operation=Operation.LAST_K(10),
        ), # The last 10 purchase prices aggregated as a list
    ],
)

This creates a `GroupBy` which transforms the `purchases` event data into useful features by aggregating various fields over various time windows, with `user_id` as a primary key.

This transforms raw purchases log data into useful features at the user level.

User Data: Turning User data into features is a littler simpler, primarily because we don’t have to worry about performing aggregations. In this case, the primary key of the source data is the same as the primary key of the feature, so we can simply extract column values rather than perform aggregations over rows:

source = Source(
    entities=EntitySource(
        snapshotTable="data.users", # This points to a table that contains daily snapshots of all users
        query=Query(
            selects=select("user_id","account_created_ds","email_verified"), # Select the fields we care about
        )
    ))

v1 = GroupBy(
    sources=[source],
    keys=["user_id"], # Primary key is the same as the primary key for the source table
    aggregations=None, # In this case, there are no aggregations or windows to define
    online=True,
) 

This creates a `GroupBy` which extracts dimensions from the `data.users` table for use as features, with `user_id` as a primary key.

Joining these features together: Next, we need to combine the previously defined features into a single view that can be both backfilled for model training and served online as a complete vector for model inference. We can achieve this using the Join API.

For our use case, it’s very important that features are computed as of the correct timestamp. Because our model runs when the checkout flow begins, we want to use the corresponding timestamp in our backfill, such that feature values for model training logically match what the model will see in online inference.

Here’s what the definition would look like. Note that it combines our previously defined features in the right_parts portion of the API (along with another feature set called returns).

source = Source(
    events=EventSource(
        table="data.checkouts", 
        query=Query(
            selects=select("user_id"), # The primary key used to join various GroupBys together
            time_column="ts",
            ) # The event time used to compute feature values as-of
    ))

v1 = Join(  
    left=source,
    right_parts=[JoinPart(group_by=group_by) for group_by in [purchases_v1, returns_v1, users]] # Include the three GroupBys
)

Backfills/Offline Computation

The first thing that a user would likely do with the above Join definition is run a backfill with it to produce historical feature values for model training. Chronon performs this backfill with a few key benefits:

1. Point-in-time accuracy: Notice the source that is used as the “left” side of the join above. It is built on top of the “data.checkouts” source, which includes a “ts” timestamp on each row that corresponds to the logical time of that particular checkout. Every feature computation is guaranteed to be window-accurate as of that timestamp. So for the one-month sum of previous user purchases, every row will be computed for the user as of the timestamp provided by the left-hand source.
2. Skew handling: Chronon’s backfill algorithms are optimized for handling highly skewed datasets, avoiding frustrating OOMs and hanging jobs.
3. Computational efficiency optimizations: Chronon is able to bake in a number of optimizations directly into the backend, reducing compute time and cost.

Online Computation

Chronon abstracts away a lot of complexity for online feature computation. In the above examples, it would compute features based on whether the feature is a batch feature or a streaming feature.

Batch features (for example, the User features above)

Because the User features are built on top of a batch table, Chronon will simply run a daily batch job to compute the new feature values as new data lands in the batch data store and upload them to the online KV store for serving.

Streaming features (for example, the Purchases features above)

The Purchases features are built on a source that includes a streaming component, as indicated by the inclusion of a “topic” in the source. In this case, Chronon will still run a batch upload in addition to a streaming job for real time updates. The batch jobs is responsible for:

1. Seeding the values: For long windows, it wouldn’t be practical to rewind the stream and play back all raw events.
2. Compressing “the middle of the window” and providing tail accuracy: For precise window accuracy, we need raw events at both the head and the tail of the window.

The streaming job then writes updates to the KV store to keep feature values up to date at fetch time.

Online Serving / Fetch API

Chronon offers an API to fetch features with low latency. We can either fetch values for individual GroupBys (i.e. the Users or Purchases features defined above) or for a Join. Here’s an example of what one such request and response for a Join would look like:

// Fetching all features for user=123
Map<String, String> keyMap = new HashMap<>();
keyMap.put("user", "123")
Fetcher.fetch_join(new Request("quickstart_training_set_v1", keyMap));
// Sample response (map of feature name to value)
'{"purchase_price_avg_3d":14.2341, "purchase_price_avg_14d":11.89352, ...}'

Java code that fetches all features for user 123. The return type is a map of feature name to feature value.

The above example uses the Java client. There is also a Scala client and a Python CLI tool for easy testing and debugging:

run.py --mode=fetch -k '{"user_id":123}' -n quickstart/training_set -t join

> {"purchase_price_avg_3d":14.2341, "purchase_price_avg_14d":11.89352, ...}

Utilizes the run.py CLI tool to make the same fetch request as the Java code above. run.py is a convenient way to quickly test Chronon workflows like fetching.

Another option is to wrap these APIs into a service and make requests via a REST endpoint. This approach is used within Airbnb for fetching features in non-Java environments such as Ruby.

Online-Offline Consistency

Chronon not only helps online-offline accuracy, it also offers a way to measure it. The measurement pipeline starts with the logs of the online fetch requests. These logs include the primary keys and timestamp of the request, along with the fetched feature values. Chronon then passes the keys and timestamps to a Join backfill as the left side, asking the compute engine to backfill the feature values. It then compares the backfilled values to actual fetched values to measure consistency.

What’s Next?

Open source is just the first step in an exciting journey that we look forward to taking with our partners at Stripe and the broader community.

Our vision is to create a platform that enables ML practitioners to make the best possible decisions about how to leverage their data and makes enacting those decisions as easy as possible. Here are some questions that we’re currently using to inform our roadmap:

How much further can we lower the cost of iteration and computation?

Chronon is already built for the scale of data processed by large companies such as Airbnb and Stripe. However, there are always further optimizations that we can make to our compute engine, both to reduce the compute cost and the “time cost” of creating and experimenting with new features.

How much easier can we make authoring a new feature?

Feature engineering is the process by which humans express their domain knowledge to create signals that the model can leverage. Chronon could integrate NLP to allow ML practitioners to express these feature ideas in natural language and generate working feature definition code as a starting point for their iteration.

Lowering the technical bar to feature creation would in turn open the door to new kinds of collaboration between ML practitioners and partners who have valuable domain expertise.

Can we improve the way models are maintained?

Changing user behavior can cause shifts in model performance because the data that the model was trained on no longer applies to the current situation. We imagine a platform that can detect these shifts and create a strategy to address them early and proactively, either by retraining, adding new features, modifying existing features, or some combination of the above.

Can the platform itself become an intelligent agent that helps ML practitioners build and deploy the best possible models?

The more metadata that we gather into the platform layer, the more powerful it can become as a general ML assistant.

We mentioned the goal of creating a platform that can automatically run experiments with new data to identify ways to improve models. Such a platform might also help with data management by allowing ML practitioners to ask questions such as “What kinds of features tend to be most useful when modeling this use case?” or “What data sources might help me create features that capture signal about this target?” A platform that could answer these types of questions represents the next level of intelligent automation.

Getting Started

Here are some resources to help you get started or to evaluate if Chronon is a good fit for your team.

• Check out the project on Github, the Chronon website, and especially the quickstart guide.
• Drop into our community Discord channel. The Airbnb and Stripe teams are excited to chat with you about how Chronon might fit into your stack.

Interested in this type of work? Check out our open roles here — we’re hiring.

Acknowledgements

Sponsors: Henry Saputra Yi Li Jack Song

Contributors: Pengyu Hou Cristian Figueroa Haozhen Ding Sophie Wang Vamsee Yarlagadda Haichun Chen Donghan Zhang Hao Cen Yuli Han Evgenii Shapiro Patrick Yoon

Chronon, Airbnb’s ML Feature Platform, Is Now Open Source was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By: Eli Hart, Ben Schwab, and Yvonne Wong

In the previous post in this series, we introduced you to Trio, Airbnb’s framework for Jetpack Compose screen architecture in Android. Some of the advantages of Trio include:

• Guarantees type safety when communicating across module boundaries in complex apps
• Codifies expectations about how ViewModels are used and shared, and what interfaces look like between screens
• Allows for stable screenshot and UI tests and simple navigation testing
• Compatible with Mavericks, Airbnb’s open source state management library for Jetpack (Trio is built on top of Mavericks)

If you need a refresher on Trio or are learning about this framework for the first time, start with Part 1. It provides an overview of why we built Trio when transitioning to Compose from a Fragments-based architecture. Part 1 also explains the core framework concepts like the Trio class and UI class.

In this post, we’ll build upon what we’ve shared so far and dive into how navigation works in Trio. As you’ll see, we designed Trio to make navigation simpler and easier to test, especially for large, modularized applications.

Navigating with Trio

A unique approach in our design is that Trios are stored in the ViewModel’s State, right alongside all other data that a Screen exposes to the UI. For example, a common use case is to store a list of Trios to represent a stack of screens.

data class ParentState(
  @PersistState val trioStack: List<Trio>
) : MavericksState

The PersistState annotation is a mechanism of Mavericks that automatically saves and restores parcelable State values across process death, so the navigation state is preserved. A compile time validation ensures that Trio values in State classes are annotated like this so that their state is always saved correctly.

The ViewModel controls this state, and can expose functions to push a new screen or pop off a screen. Since the ViewModel has direct control over the list of Trios, it can also easily perform more complex navigation changes such as reordering screens, dropping multiple screens, or clearing all screens. This makes navigation extremely flexible.

class ParentViewModel : TrioViewModel {
  fun pushScreen(trio: Trio) = setState {
    copy(trioStack = trioStack + trio)
  }

  fun pop() = setState {
    copy(trioStack = trioStack.dropLast(1))
  }
}

The Parent Trio’s UI accesses the Trio list from State and chooses how and where to place the Trios. We can implement a screen flow by showing the latest Trio in the stack.

@Composable
override fun TrioRenderScope.Content(state: ParentState) {
  ShowTrio(state.trioStack.last())
}

Coordinating Navigation

Why store Trios in State? Alternative approaches might use a navigator object in the Compose UI. However, representing the application’s navigation graph in State allows the ViewModel to update its data and navigation in a single place. This can be extremely helpful when we need to delay making a navigation change until after an asynchronous action, like a network request, completes. We could not do this easily with Fragments and found that with Trio’s approach, our navigation becomes simpler, more explicit, and more easily testable.

This example shows how the ViewModel can handle a “save and exit” call from the UI by launching a suspending network request in a coroutine. Once the request completes, we can pop the screen by updating the Trio stack in State. We can also atomically modify other values in the state at the same time, perhaps based on the result of the network request. This easily guarantees that navigation and ViewModel state stay in sync.

class CounterViewModel : TrioViewModel {

  fun saveAndExit() = viewModelScope.launch {
    val success = performSaveRequest()

    setState {
      copy(
        trioStack = trioStack.dropLast(1),
        success = success
      )
    }
  }
}

As the navigation stack becomes more complex, application UI hierarchy gets modeled by a chain of ViewModels and their States. As the state is rendered, it creates a corresponding Compose UI hierarchy.

A Trio can represent an arbitrary UI element of any size, including nested screens and sections, while providing a backing state and a mechanism to communicate with other Trios in the hierarchy.

There are two additional nice benefits of modeling the hierarchy in ViewModel state like this. One is that it becomes simple to specify custom navigation scenarios when setting up testing — we can easily create whatever navigation states we want for our tests.

Another benefit is that since the navigation hierarchy is decoupled from the Compose UI, we can pre-load Trios that we anticipate needing, just by initializing their ViewModels ahead of time. This has made it significantly simpler for us to optimize performance through preloading screens.

Mavericks State typically holds simple data classes, and not complex objects like a Trio, which have a lifecycle. However, we find that the benefits this approach brings are well worth the extra complexity.

Managing Activities

Ideally, an application with Trio would use just a single activity, following the standard application architecture recommendation from Google. However, especially for interop purposes, Trios will sometimes need to start new activity intents. Traditionally, this isn’t done from a ViewModel because ViewModels should not contain Activity references, since they outlive the Activity lifecycle; however, in order to maintain our paradigm of doing all navigation in the ViewModel, Trio makes an exception.

During initialization, the Trio ViewModel is given a Flow of Activity via its initializer. This Flow provides the current activity that the ViewModel is attached to, and null when it is detached, such as during activity recreation. Trio internals manage the Flow to guarantee that it is up to date and the activity is not leaked.

When needed, a ViewModel can access the next non-null activity value via the awaitActivity suspend function. For example, we can use it to start a new activity after a network request completes.

class ViewModelInitializer<S : MavericksState>(
  val initialState: S,
  internal val activityFlow: Flow<Activity?>,
  ...
)

class CounterViewModel(
  initializer: ViewModelInitializer
) : TrioViewModel {

  fun saveAndOpenNextPage() = viewModelScope.launch {
    performSaveRequest()
    awaitActivity().startActivity()
  }
}

The awaitActivity function is provided by the TrioViewModel as a convenient way to get the next value in the activity flow.

suspend fun awaitActivity(): ComponentActivity {
  return initializer.activityFlow.filterNotNull().first()
}

While a bit unorthodox, this pattern allows activity-based navigation to also be collocated with other business logic in the ViewModel.

Modularization Structure

Properly modularizing a large code base is a problem that many applications face. At Airbnb, we’ve split our codebase into over 2000 modules to allow faster build speeds and explicit ownership boundaries. To support this, we’ve built an in house navigation system that decouples feature modules. It was originally created to support Fragments and Activities, and was later expanded to integrate with Trio, helping us to solve the general problem of navigation at scale in a large application.

In our project structure, each module has a specific type, indicated by its prefix and suffix, which defines its purpose and enforces a set of rules about which other modules it can depend on.

Feature modules, prefixed with “feat”, contain our Trio screens; each screen in the app might live in its own separate module. To prevent circular dependencies and improve build speeds, we do not allow feature modules to depend on each other.

This means that one feature cannot directly instantiate another. Instead, each feature module has a corresponding navigation module, suffixed with “nav”, which defines a router to its feature. To avoid a circular dependency, the router and its destination Trio are associated with Dagger multibinding.

In this simple example, we have a counter feature and a decimal feature. The counter feature can open the decimal feature to modify the decimal count, so the counter module needs to depend on the decimal navigation module.

Routing

The navigation module is small. It contains only a Routers class with nested Router objects corresponding to each Trio in the feature module.

// In feat.decimal.nav
@Plugin(pluginPoint = RoutersPluginPoint::class)
class DecimalRouters : RouterDeclarations() {

  @Parcelize
  data class DecimalArgs(val count: Double) : Parcelable

  object DecimalScreen 
    : TrioRouter<DecimalArgs, NavigationProps, NoResult>
}

A Router object is parameterized with the types that define the Trio’s public interface: the Arguments to instantiate it, the Props that it uses for active communication, and if desired, the Result that the Trio returns.

Arguments is a data class, often including primitive data indicating starting values for a screen.

Importantly, the Routers class is annotated with @Plugin to declare that it should be added to the Routers PluginPoint. This annotation is part of an internal KSP processor that we use for dependency injection, but it essentially just generates the boilerplate code to set up a Dagger multibinding set. The result is that each Routers class is added to a set, which we can access from the Dagger graph at runtime.

On the corresponding Trio class in the feature module, we use the @TrioRouter annotation to specify which Router the Trio maps to. Our KSP processor matches these at compile time, and generates code that we can use at runtime to find the Trio destination for each Router.

// In feat.decimal
@TrioRouter(DecimalRouters.DecimalScreen::class)
class DecimalScreen(
  initializer: Initializer<DecimalArgs, ...>
) : Trio<DecimalArgs, NavigationProps, ...>

The processor validates at compile time that the Arguments and Props on the Router match the types on the Trio, and that each Router has a single corresponding destination. This guarantees runtime type safety in our navigation system.

Router Usage

Instead of manually instantiating Trios, we let the Router do it for us. The Router ensures that the proper type of Arguments is provided, looks up the matching Trio class in the Dagger graph, creates the initializer class to wrap the arguments, and finally, uses reflection to invoke the Trio’s constructor.

This functionality is accessible through a createTrio function on the router, which we can invoke from the ViewModel. This allows us to easily create a new instance of a Trio, and push it onto our Trio stack. In the following example, the Props instance allows the Trio to call back to its parent to perform this push; we’ll explore Props in detail in Part 3 of this series.

class CounterViewModel : TrioViewModel {

  fun showDecimal(count: Double) {
    val trio = DecimalRouters.DecimalScreen.createTrio(DecimalArgs(count))
    props.pushScreen(trio)
  }
}

If we want to instead start a Trio in a new activity, the Router also provides a function to create an intent for a new activity that wraps the Trio instance; we can then start it from the ViewModel using Trio’s activity mechanism, as discussed earlier.

class CounterViewModel : TrioViewModel {

  fun showDecimal(count: Double) = viewModelScope.launch {
    val activity = awaitActivity()
    val intent = DecimalRouters.DecimalScreen
                    .newIntent(activity, DecimalArgs(count))
    
    activity.startActivity(intent)
  }
}

When a Trio is started in a new activity, we simply need to extract the Parcelable Trio instance from the intent, and show it at the root of the Activity’s content.

class TrioActivity : ComponentActivity() {
  override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)

    val trio = intent.parseTrio()
    setContent {
      ShowTrio(trio)
    }
  }
}

We can also start activities for a result by defining a Result type on the router.

class DecimalRouters : RouterDeclarations() {

  data class DecimalResult(val count: Double)

  object DecimalScreen : TrioRouter<DecimalArgs, …, DecimalResult>
}

In this case, the ViewModel contains a “launcher” property, which is used to start the new activity.

class CounterViewModel : TrioViewModel {

  val decimalLauncher = DecimalScreen.createResultLauncher { result ->
    setState {
      copy(count = result.count)
    }
  }

  fun showDecimal(count: Double) {
    decimalLauncher.startActivityForResult(DecimalArgs(count))
  }
}

For example, if the user adjusts the decimals on the decimal screen, we could return the new count to update our state in the counter. The lambda argument to the launcher allows us to handle the result when the decimal screen returns, which we can then use to update the state. This furthers our goal of centralizing all navigation in the ViewModel, while guaranteeing type safety.

Our Router system offers other nice features in addition to modularization, like interceptor chains in the Router resolution providing intermediary screens before showing the final Trio destination. We use this to redirect users to the login page when required, and also to show a loading page if a dynamic feature needs to be downloaded first.

Fragment Interop

Making Trio screens interoperable with our existing Fragment screens was very important to us. Our migration to Trio is a years-long effort, and Trios and Fragments need to easily coexist.

Our approach to interoperability is twofold. First, if a Fragment and Trio don’t need to dynamically share information while created (i.e., they only take initial arguments and return a result), then it is easiest to start a new activity when transitioning between a Fragment and a Trio. Both architecture types can be easily started in a new activity with Arguments, and can optionally return a result when finished, so it is very easy to navigate between them this way.

Alternatively, if a Trio and Fragment screen need to share data between themselves while the screens are both active (i.e., the equivalent of Props with Trio), or they need to share complex data that is too large to pass with Arguments, then the Trio can be nested within an “Interop Fragment”, and the two Fragments can be shown in the same activity. The Fragments can communicate via a shared ViewModel, similar to how Fragments normally share ViewModels with Mavericks.

Our Router object makes it easy to create and show a Trio from another Fragment, with a single function call:

class LegacyFragment : MavericksFragment {

  fun showTrioScreen() {        
    showFragment(
      CounterRouters
             .CounterScreen
             .newInteropFragment(SharedCounterViewModelPropsAdapter::class)
    )
  }
}

The Router creates a shell Fragment and renders the Trio inside of it. An optional adapter class, the SharedCounterViewModelPropsAdapter in the above example, can be passed to the Fragment to specify how the Trio will communicate with Mavericks ViewModels used by other Fragments in the activity. This adapter allows the Trio to specify which ViewModels it wants to access, and creates a StateFlow that converts those ViewModel states into the Props class that the Trio consumes.

class SharedCounterViewModelPropsAdapter : LegacyViewModelPropsAdapter<SharedCounterScreenProps> {
    
override suspend fun createPropsStateFlow(
  legacyViewModelProvider: LegacyViewModelProvider,
  navController: NavController<SharedCounterScreenProps>,
  scope: CoroutineScope
): StateFlow<SharedCounterScreenProps> {
         
  // Look up an activity view model
  val sharedCounterViewModel: SharedCounterViewModel = legacyViewModelProvider.getActivityViewModel()
         
  // You can look up multiple view models if necessary
  val fragmentClickViewModel: SharedCounterViewModel = legacyViewModelProvider.requireExistingViewModel(viewModelKey = {
    SharedCounterViewModelKeys.fragmentOnlyCounterKey
  })

  // Combine state updates into Props for the Trio, 
  // and return as a StateFlow. This will be invoked anytime
  // any state flow has a new state object.
  return combine(sharedCounterViewModel.stateFlow, fragmentClickViewModel.stateFlow) { sharedState, fragmentState ->
            SharedCounterScreenProps(
                navController = navController,
                sharedClickCount = sharedState.count,
                fragmentClickCount = fragmentState.count,
                increaseSharedCount = {
                    sharedCounterViewModel.increaseCounter()
                }
            )
    }.stateIn(scope)
  }
}

Conclusion

In this article, we discussed how navigation works in Trio. We use some unique approaches, such as our custom routing system, providing access to activities in a ViewModel, and storing Trios in the ViewModel State to achieve our goals of modularization, interoperability, and making it simpler to reason about navigation logic.

Stay tuned for Part 3, where we will explain how Trio’s Props enable dynamic communication between screens.

And if this sounds like the kind of challenge you love working on, check out open roles — we’re hiring!

Introducing Trio | Part II was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

By: Eli Hart, Ben Schwab, Yvonne Wong

At Airbnb, we have developed an Android framework for Jetpack Compose screen architecture, which we call Trio. Trio is built on our open-source library Mavericks, which it leverages to maintain both navigation and application state within the ViewModel.

Airbnb began development of Trio more than two years ago, and has been using it in production for over a year and a half. It is powering a significant portion of our production screens in Airbnb’s Android app, and has enabled our engineers to create features in 100% Compose UI.

In this blog post series, we will look at how Mavericks can be used in modern, Compose based applications. We will discuss the challenges of Compose-based architecture and how Trio has attempted to solve them. This will include an exploration of concepts such as:

• Type-safe navigation between feature modules
• Storing navigation state in a ViewModel
• Communication between Compose-based screens, including opening screens for results and two-way communication between screens
• Compile-time validation of navigation and communication interfaces
• Developer tools created to support Trio workflows

This series is split into three parts. Part 1 (this blog post) covers Trio’s high-level architecture. Stay tuned for Part 2, which will detail Trio’s navigation system, and Part 3, which will examine how Trio uses Props for communication between screens.

Background on Mavericks

To understand Trio’s architecture, it’s important to know the basics of Mavericks, which Trio is built on top of. Airbnb originally open sourced Mavericks in 2018 to simplify and standardize how state is managed in a Jetpack ViewModel. Check out this post from the initial Mavericks (“MvRx”) launch for a deeper dive.

Used in virtually all the hundreds of screens in Airbnb’s Android app (and by many other companies too!), Mavericks is a state management library that is decoupled from the UI, and can be used with any UI system. The core concept is that screen UI is modeled as a function of state. This ensures that even the most complex screen can be rendered in a way that’s thread safe, independent of the order of events leading up to it, and easy to reason about and test.

To achieve this, Mavericks enforces the pattern that all data exposed by the ViewModel must be contained within a single MavericksState data class. In a simple Counter example, the state would contain the current count.

data class CounterState(
  val count: Int = 0
) : MavericksState

State properties can only be updated in the ViewModel via calls to setState. The setState function takes a “reducer” lambda, which, given a previous state, outputs a new state. We can use a reducer to increment the count by simply adding 1 to the previous value.

class CounterViewModel : MavericksViewModel<CounterState>(...) {
  fun incrementCount() {
    setState {
      // this = previous state
      this.copy(count = count + 1)
    }
  }
}

The base MavericksViewModel enqueues all calls to setState and runs them serially in a background thread. This guarantees thread safety when changes are made in multiple places at once, and ensures that changes to multiple properties in the state are atomic, so the UI never sees a state that is only partially updated.

MavericksViewModel exposes state changes via a coroutine Flow property. When paired with reactive UI, like Compose, we can collect the latest state value and guarantee that the UI is updated with every state change.

counterViewModel.stateFlow.collectAsState().count

This unidirectional cycle can be visualized with the following diagram:

Challenges with Fragment-based architecture

While Mavericks works well for state management, we were still experiencing some challenges with Android UI development, stemming from the fact that we were using a Fragment-based architecture integrated with Mavericks. With this approach, ViewModels are mainly scoped to the Activity and shared between Fragments via injection. Fragment views are updated by state changes from the ViewModel, and call back to the ViewModel to make state changes. The Fragment Manager manages navigation independently when Fragments need to be pushed or popped.

Due to this architecture, we were running up against some ongoing difficulties, which became the motivation for building Trio.

1. Scoping — Sharing ViewModels between multiple Fragments relies on the implicit injection of the ViewModel. Thus, it isn’t clear which Fragment is responsible for creating the Activity ViewModel originally, or for providing the initial arguments to it.
2. Communication — It’s difficult to share data between Fragments directly and with type safety. Again, because ViewModels are injected, it’s hard to have them communicate directly, and we don’t have good control over the ordering of their creation.
3. Navigation — Navigation is done via the Fragment Manager and must happen in the Fragment. However, state changes are done in the ViewModel. This leads to synchronization problems between ViewModel and navigation states. It’s hard to coordinate if-then scenarios like making a navigation call only after updating a state value in the ViewModel.
4. Testability — It’s difficult to isolate the UI for testing because it is wrapped in the Fragment. Screenshot tests are prone to flakiness and a lot of indirection is required for mocking the ViewModel state, because ViewModels are injected into the Fragment with property delegates.
5. Reactivity — Mavericks provides a unidirectional state flow to the View, which is helpful for consistency and testing, but the View system doesn’t lend itself well to reactive updates to state changes, and it can be difficult or inefficient to update the view incrementally on each state change.

While some of these problems could have been mitigated by using a better Fragment based architecture, we found that Fragments were overall too limiting with Compose and decided to move away from them entirely.

Why we built Trio

In 2021, our team began to explore adopting Jetpack Compose and completely transitioning away from Fragments. By fully embracing Compose, we could better prepare ourselves for future Android developments and eliminate years of accumulated tech debt.

Continuing to use Mavericks was important to us because we have a large amount of internal experience with it, and we didn’t want to further complicate an architectural migration by also changing our state management approach. We saw an opportunity to rethink how Mavericks could support a modern Android application, and address problems we encountered with our previous architecture

With Fragments, we struggled to guarantee type safe communication between screens at runtime. We wanted to be able to codify the expectations about how ViewModels are used and shared, and what interfaces look like between screens.

We also didn’t feel our needs were fully met by the Jetpack Navigation component, especially given our heavily modularized code base and large app. The Navigation component is not type safe, requires defining the navigation graph in a single place, and doesn’t allow us to co-locate state in our ViewModel. We looked for a new architecture that could provide better type safety and modularization support.

Finally, we wanted an architecture that would improve testability, such as more stable screenshot and UI tests, and simpler navigation testing.

We considered the open source libraries Workflow and RIBs, but opted not to use them because they were not Compose-first and were not compatible with Mavericks and our other pre-existing internal frameworks.

Given these requirements, our decision was to develop our own solution, which we named Trio.

Trio Architecture

Trio is an opinionated framework for building features. It helps us to define and manage boundaries and state in Compose UI. Trio also standardizes how state is hoisted from Compose UI and how events are handled, enforcing unidirectional data flow with Mavericks. The design was inspired by Square’s Workflow library; Trio differs in that it was designed specifically for Compose and uses Mavericks ViewModels for managing state and events.

Self-contained blocks are called “Trios”, named for the three main classes they contain. Each Trio has its own ViewModel, State, and UI, and can communicate with and be nested in other Trios. The following diagram represents how these components work together. The ViewModel makes changes to state via Mavericks reducers, the UI receives the latest state value to render, and events are routed back to the ViewModel for further state updates.

If you’re already familiar with Mavericks this pattern should look very similar! The ViewModel and State usage is very similar to what we did with Fragments. What’s new is how we embed the ViewModels in Compose UI and add Routing and Props based communication via Trio.

Trios are nested to form custom, flexible navigation hierarchies. “Parent” Trios create child Trios with initial arguments through a Router, and store those children in their State. The parent can then communicate dynamically with its children through a flow of Props, which provide data, dependencies, and functional callbacks.

The framework helps us to guarantee type safety when navigating and communicating between Trios, especially across module boundaries.

Each Trio can be tested individually by instantiating it with mocked arguments, State, and Props. Coupled with Compose’s state-based rendering and Maverick’s immutable state patterns, this provides controlled and deterministic testing environments.

The Trio Class

Creating a new Trio implementation requires subclassing the Trio base class. The Trio class is typed to define Args, Props, State, ViewModel, and UI; this allows us to guarantee type-safe navigation and inter-screen communication.

class CounterScreen : Trio<
  CounterArgs, 
  CounterProps, 
  CounterState,     
  CounterViewModel, 
  CounterUI
>

A Trio is created with either an initial set of arguments or an initial state, which are wrapped in a sealed class called the Initializer. In production, the Initializer will only contain Args passed from another screen, but in development we can seed the Initializer with mock state so that the screen can be loaded standalone, independent of the normal navigation hierarchy.

class CounterScreen(
  initializer: Initializer<CounterArgs, CounterState>
) 

Then, in our subclass body, we define how we want to create our State, ViewModel, and UI, given the starting values of Args and Props.

Args and Props both provide input data, with the difference being that Args are static while Props are dynamic. Args guarantee the stability of static information, such as IDs used to start a screen, while Props allow us to subscribe to data that may change over time.

override fun createInitialState(args: CounterArgs, props:  CounterProps) {
  return CounterState(args.count)
}

Trio provides an initializer to create a new ViewModel instance, passing necessary information like the Trio’s unique ID, a Flow of Props, and a reference to the parent Activity. Dependencies from the application’s dependency graph can also be also passed to the ViewModel through its constructor.

override fun createViewModel(
  initializer: Initializer<CounterProps, CounterState>
) {
  return CounterViewModel(initializer)
}

Finally, the UI class wraps the composable code used to render the Trio. The UI class receives a flow of the latest State from the ViewModel, and also uses the ViewModel reference to call back to it when handling UI events.

override fun createUI(viewModel: CounterViewModel ): CounterUI {
  return CounterUI(viewModel)
}

We like that grouping all of these factory functions in the Trio class makes it explicit how each class is created, and standardizes where to look to understand dependencies. However, it can also feel like boilerplate. As an improvement, we often use reflection to create the UI class, and we use assisted inject to automate creation of the ViewModel with Dagger dependencies.

The resulting Trio declaration as a whole looks like this:

class CounterScreen(
  initializer: Initializer<CounterArgs, CounterState>
) : Trio<
  CounterArgs, 
  CounterProps, 
  CounterState,     
  CounterViewModel, 
  CounterUI
>(initializer) {

  override fun createInitialState(CounterArgs, CounterProps) {
    return CounterState(args.count)
  }
}

The UI Class

The Trio’s UI class implements a single Composable function named “Content”, which determines the UI that the Trio shows. Additionally, the Content function has a “TrioRenderScope” receiver type. This is a Compose animation scope that allows us to customize the Trio’s animations when it is displayed.

class CounterUI(
  override val viewModel: CounterViewModel
) : UI<CounterState, CounterViewModel> {
   
  @Composable
  override fun TrioRenderScope.Content(state: CounterState) {
    Column {
      TopAppBar()
      Button(
        text = state.count,
        modifier = Modifier.clickable {
          viewModel.incrementCount()
        }
      )
      ...
    }
  }
}

The Content function is recomposed every time the State from the ViewModel changes. The UI directs all UI events, such as clicks, back to the ViewModel for handling.

This design enforces unidirectional data flow, and testing the UI is easy because it is decoupled from the logic of state changes and event handling. It also standardizes how Compose state is hoisted for consistency across screens, while removing the boilerplate of setting up access to the ViewModel’s state flow.

Rendering a Trio

Given a Trio instance, we can render it by invoking its Content function, which uses the previously mentioned factory functions to create initial values of the ViewModel, State, and UI. The state flow is collected from the ViewModel and passed to the UI’s Content function. The UI is wrapped in a Box to respect the constraints and modifier of the caller.

@Composable
internal fun TrioRenderScope.Content(modifier: Modifier = Modifier) {
  key(trioId) {
    val activity = LocalContext.current as ComponentActivity

    val viewModel = remember {
      getOrCreateViewModel(activity)
    }

    val ui = remember { createUI(viewModel) }

    val state = viewModel.stateFlow
                    .collectAsState(viewModel.currentState).value

    Box(propagateMinConstraints = true, modifier = modifier) {
      ui.Content(state = state)
    }
  }
}

To enable customizing entry and exit animations, the Content function also uses a TrioRenderScope receiver; this wraps an implementation of Compose’s AnimatedVisibilityScope which displays the Content. A helper function is used to coordinate this.

@Composable
fun ShowTrio(trio: Trio, modifier: Modifier) {
  AnimatedVisibility(
    visible = true,
    enter = EnterTransition.None,
    exit = ExitTransition.None
  ) {
    val animationScope = TrioRenderScopeImpl(this)
    trio.Content(modifier, animationScope)
  }
}

In practice, the actual implementation of Trio.Content is quite a bit more complex because of additional tooling and edge cases we want to support — such as tracking the Trio’s lifecycle, managing saved state, and mocking the ViewModel when shown within a screenshot test or IDE preview.

Conclusion

In this introduction to Trio we discussed Airbnb’s background with Mavericks and Fragments, and why we built Trio to transition to a Jetpack Compose-based architecture. We presented an overview of Trio’s architecture, and looked at core components such as the Trio class and UI class.

In upcoming articles, we will continue this three-part series by detailing how navigation works with Trio, and how Trio’s Props allow dynamic communication between screens. And if this work sounds interesting to you, check out open roles at Airbnb!

Introducing Trio | Part I was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

KDD (Knowledge and Data Mining) is a flagship conference in data science research. Hosted annually by a special interest group of the Association for Computing Machinery (ACM), it’s where you’ll learn about some of the most ground-breaking developments in data mining, knowledge discovery, and large-scale data analytics.

Airbnb had a significant presence at KDD 2023 with two papers accepted into the main conference proceedings and 11 talks and presentations. In this blog post, we’ll summarize our team’s contributions and share highlights from an exciting week of workshops, panel discussions, and more.

Deep learning and search ranking

Even though search ranking is a problem that researchers have been working on for decades, there are still many nuances to explore. For example, at Airbnb, guests are typically searching over a period of days or weeks, not minutes. And being a two-way marketplace, there are factors like the potential for hosts to cancel the booking that we’d like to account for in ranking.

Optimizing Airbnb Search Journey with Multi-task Learning, our paper accepted at KDD 2023, presents Journey Ranker, a new multi-task deep learning model. The core insight here is that for this kind of long-term search task, we want to optimize for intermediate steps in the user journey.

CAPTION: The Journey Ranker base module assists guests in reaching positive milestones. There is also a Twiddler module that assists guests in avoiding negative milestones. The modules work off a shared feature representation of listing and guest context, and their output scores are combined.

Because of its modular design, Journey Ranker can be used wheneverthere are positive or negative milestones to consider. We’ve implemented it in different Airbnb search and other products to drive improvements in business metrics.

We also co-presented a tutorial on Data-Centric AI (DCAI). DCAI is a fast-growing field in deep learning, because as model design matures, innovation is being driven by data. We shared DCAI best practices and trends for developing training data, developing inference data, maintaining data, and creating benchmarks, with many examples from working with LLMs.

Online experimentation and measurement

Online experimentation (e.g., A/B testing) is a common way for organizations like Airbnb to make data-driven decisions. But high variance is frequently a challenge. For example, it’s hard to prove that a change in our search UX will drive value when bookings are infrequent and depend on a large number of interactions over a long period of time.

Our paper Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes presents two new methods for variance reduction that rely exclusively on in-experiment data:

1. A framework for a model-based leading indicator metric that continually estimates progress toward a delayed binary outcome.
2. A counterfactual treatment exposure index that quantifies the amount a user is impacted by the treatment.

In testing, both methods achieved a variance reduction of 50% or more. These techniques have greatly improved our experimentation efficiency and impact.

With more than 50% variance reduction, the new model-based leading indicator metric (listing-view utility, on the right) aligns with the target uncancelled booking metric much better than other indicators such as listing-view with dates (on the left).

Another interesting challenge in online experimentation is avoiding interference bias, which can happen when you have competition between your A/B test subjects. Airbnb presented a keynote talk on this topic at KDD’s 2nd Workshop on Decision Intelligence and Analytics for Online Marketplaces. As an example, if you ran an A/B test where group B saw lower booking prices, they might “cannibalize” the bookings from group A. There are two imperfect solutions: clustering (isolating the options for participants) and switchbacks (grouping participants by time intervals).

Also at the workshop, we presented the paper The Price is Right: Removing A/B Test Bias in a Marketplace of Expirable Goods. This discusses the problem of lead-day bias: where items like concert tickets, air travel, and Airbnb bookings vary in price based on the distance from their expiration date. This can wreak havoc on A/B tests, and in the paper we present several mitigation techniques, such as limited rollout, smart overlapping of experiments, and Heterogeneous Treatment Effect (HTE) remixed estimator to correct for bias and accelerate R&D process.

Along with limited rollout and smart overlapping of experiments, HTE-remixed estimator can provide sufficiently robust estimation of the long-term experiment impact from the short-term result and significantly shorten the experiment run-time.

Causal inference for marketing and user journey optimization

In marketing, the million-dollar question is how much should you spend per channel? This can be reframed as a causal inference problem: how many incremental conversions does each channel drive?

When we look at marketing activities across Nielsen’s Designated Marketing Areas (DMAs) we find moderate to strong correlation across channels. This makes it hard to isolate the impact of one channel from another. In fact, when we include the correlated channels in the same regression, the coefficients flip signs for most channels, a clear sign of multicollinearity.

Existing solutions to multicollinearity, such as shrinkage estimators, principal component analysis, and partial linear regression, are particularly helpful for prediction problems but work less well for our use case where we need to maintain business interpretability while isolating causality. Our approach, described in the paper Hierarchical Clustering as a Novel Solution to Multicollinearity, is to hierarchically cluster DMAs based on their similarity in marketing impressions over time. With such clustering, cross-channel correlation dropped by up to 43% and the channel coefficients no longer flip signs.

Not only does our method provide an intuitive and effective solution to multicollinearity, it also circumvents the need for complex transformation and preserves the interpretability of the data and the results throughout, empowering broad applications to causal inference problems.

We presented this paper at the new KDD workshop, Causal Inference and Machine Learning in Practice: Use cases for Product, Brand, Policy, and beyond. Airbnb’s Totte Harinen co-organized this workshop, which strongly resonated with KDD’s audience — it had 12 papers and four invited talks from 37 authors in 14 institutions.

In addition, we were invited to present two talks and one poster at KDD’s 2nd Workshop on End-End Customer Journey Optimization, and joined the workshop’s panel discussion. One of these talks covered CLV (customer lifetime value) modeling. At Airbnb, we want to grow our brand and community by growing all users. Our CLV ecosystem applies two frameworks:

1. The value of Airbnb customers. We use traditional ML approaches along with research into more customer-lifecycle-focused architectures (i.e. HMMs). We augment this with demand-supply incrementality modeling to properly account for guest andhost contributions to value.
2. The value growth that Airbnb delivers to customers. By accounting for long-term incremental effects of booking on Airbnb along with incremental contributions from marketing and attribution strategies, we can measure incremental changes in CLV and optimize towards them.

Causal inference can also be applied to search. At the CJ workshop, we presented our paper Low Inventory State: Identifying Under-Served Queries for Airbnb Search, which explored the problem of searches that return a low number of results. Whether or not that number is “too low” and will deter a guest from booking depends on search parameters and intent to book. For a given search query, we can use causal inference to determine the incremental effect of an additional result on the probability of booking. Our model outperforms non-causal methods and can assist with supply management as well.

Finally, our poster discussed how we measure the effects of national TV advertising campaigns. We analyzed TV exposure data and demographic data with data on Airbnb onsite behavior using a third-party identity graph. We were able to resolve disparate datasets to a unique identifier and model individual households.

We use propensity score matching to estimate TV effects, and then scale these estimates to a nationally-representative population. We leverage this data to provide tactical insights for marketing and understand how long TV effects take to decay.

The plot above (from simulated study for illustration) shows the results of an analysis for a TV campaign from August — October. We can see that the TV campaign was effective at increasing bookings for households that saw an Airbnb TV ad and was more effective for one subgroup (red line) than the other subgroup.

Data science and analytics infra

How can you achieve science at scale in a medium-to-large engineering organization? At the KDD’s 2nd Workshop on Applied Machine Learning Management, we shared Airbnb’s solution for data science reproducibility and reuse, Onebrain. The core of Onebrain is a coding standard for configuring data science projects entirely in YAML. Onebrain’s backend abstracts away CI/CD, configuration/dependency management, and command-line parsing. Since it’s “just code,” Onebrain projects can be checked into a version-controlled repo, and any repo can be a Onebrain repo.

User interaction with Onebrain happens through a CLI. With a single command, anyone can use an existing project as a template for their own work, or generate a one-click URL to spin up a server and run the project. Usage is growing fast with over 200 distinct projects and over 500 users at Airbnb within just a year.

While most of our research focuses on high-order data use-cases like models, data capture is essential as it’s the starting point for any analysis. Event logging libraries typically capture actions on and impressions of app components (buttons, sections, pages). But with this level of granularity, it can be difficult to abstract out user behavior, measure the total time spent on a surface, or understand the context surrounding an action.

At the 2nd Workshop on End-End Customer Journey Optimization, we spoke about a new type of client-side event called Sessions. Part of Airbnb’s client-side logging solution, Sessions provide a way to track user context and behaviors within the Airbnb product. Unlike traditional time-based sessions used in web analytics, these Sessions can be tied to various aspects of the Airbnb user experience. For example, they can be tied to specific surfaces like the checkout page, API calls used for observability, or even internal states of the app that abstract away complex UI components. The flexibility of Sessions allows us to capture a wide range of user interactions and better understand their journey throughout our platform.

Conclusion

KDD is an amazing opportunity for data scientists from around the world, and across industry and academia, to come together and exchange learnings and discoveries. We were honored to be invited to share techniques we’ve developed through applied research at Airbnb. The strategies and insights we presented at KDD have been essential to improving Airbnb’s platform, business, and user experience. We’re constantly motivated by innovations happening around us, and we’re thrilled to give back to the community and eager to see what kinds of new applications and advancements may come about as a result.

At the bottom of the page, you’ll find a complete list of the talks and papers shared in this article along with the team members who contributed. If you can see yourself on our team, we encourage you to apply for an open position today.

List of papers and talks

Optimizing Airbnb Search Journey with Multi-task Learning [link]

Authors: Chun How Tan, Austin Chan, Malay Haldar, Jie Tang, Xin Liu, Mustafa Abdool, Huiji Gao, Liwei He, Sanjeev Katariya

Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes [link]

Authors: Alex Deng, Michelle Du, Anna Matlin, Qing Zhang

Beyond the Simple A/B test: Mitigating Interference Bias at Airbnb

Speaker: Ruben Lobel

The Price is Right: Removing A/B Test Bias in a Marketplace of Expirable Goods [link]

Author: Thu Le, Alex Deng

Unveiling the Guest & Host Journey: Session-Based Instrumentation on Airbnb Platform

Speaker: Shant Torosean

Devoted to Long-Term Adventure: Growing Airbnb Through Measuring Customer Lifetime Value

Speaker: Sean O’Donell, Jason Cai, Linsha Chen

Low Inventory State: Identifying Under-Served Queries for Airbnb Search [link]

Author: Toma Gulea, Bradley Turnbull

Measuring TV Campaigns at Airbnb

Speaker: Adam Maidman, Sam Barrows

Tutorial: Data-Centric AI [link]

Presenter: Daochen Zha, Huiji Gao

Hierarchical Clustering As a Novel Solution to the Notorious: Multicollinearity Problem in Observational Causal Inference [link]

Authors: Yufei Wu, Zhiying Gu, Alex Deng, Jacob Zhu, Linsha Chen

Onebrain — Microprojects for Data Science [link]

Authors: Daniel Miller, Alex Deng, Narek Amirbekian, Navin Sivanandam, Rodolfo Carboni

How we championed the CRM CI/CD framework integrating Salesforce DX, GIT, BUILDKITE and Vlocity for an enhanced, efficient and continuous delivery with high software quality.

By: Sharda Kumari Pramod Gavade

Introduction

The CRM platform offers a robust suite of functionalities for building scalable applications with minimal reliance on complex coding. However, managing and deploying code and configurations within this ecosystem can be challenging, and the constantly evolving nature of the platform presents an extra layer of complexity. This can lead to slow deployment times, difficulty in balancing code and configuration (e.g. Apex classes and triggers vs. validation rules, page layouts), and managing multiple environments, among other issues.

To address these challenges, at Airbnb we have developed a resilient DevOps framework specifically tailored to the CRM platform. The framework automates the process of moving code and configuration into production for developers, system administrators in addition to low-code users such as business analysts building dashboards.

The Challenges

The CRM Platform is a versatile ecosystem with powerful functionalities, but managing code and configurations within it can be challenging. With complex metadata and multiple environments, deploying changes efficiently is difficult. Additionally, the evolving nature of the platform requires a proactive approach.

Deployment times: Deploying changes can be a time-consuming process, especially for extensive applications. This can impact the speed of delivery causing dissatisfaction among developers and stakeholders alike.

Balancing code and configuration: The CRM platform permits the development of applications via both programmatic code (using Apex, Visualforce and Lightning Web Components) and low-code development methodologies (using tools like App Builder and Flow Builder). However, effectively managing a combination of code and configuration based development poses a formidable challenge.

Managing multiple environments: CRM developers typically engage in individual environments (sandbox or scratch orgs) to build and test their code prior to moving it to higher-level environments. However, administering multiple environments across different teams can become increasingly intricate and time-consuming.

Complexity of file metadata: CRM is an intricate platform with different types of metadata (including, but not limited to, Apex classes, triggers, Lightning components, flows) that necessitate meticulous management during the development and deployment process.

Keeping up with changes: As a cloud-based platform, CRM undergoes frequent changes, with new features and updates being released regularly. Keeping up with these changes and ensuring their non-disruptive integration with existing applications can be a significant challenge.

The Solution

At Airbnb, we have developed a resilient DevOps framework tailored specifically to the CRM platform that integrates Salesforce DX, Git, and Buildkite. Our approach facilitates the inclusion of all stakeholders, including developers to system administrators and low-code users, into the development and deployment process, thereby optimizing the DevOps solution for all personas involved.

The CRM DevOps lifecycle comprises the following critical environments that are necessary for the effective deployment of code:

• Developer(DEV) — The Developers utilize the DEV environment to build and refine the code.
• Integration(SIT) — SIT ensures seamless integration with other systems.
• Quality Assurance(QA) — QA verifies the software’s functionality.
• Full copy(STAGING) — Staging provides a realistic setting for training and user acceptance.
• Pre-release — Pre-release serves as a controlled hosting platform before the code goes live.
• Hotfix — Hotfix enables swift resolution of urgent production issues.
• Prod — Production instance that houses all live traffic and data.

Each of these environments is associated with a corresponding branch in the Git version control system and is seamlessly connected via a Buildkite DevOps pipeline. After successfully passing through the peer review process by developers, Buildkite jobs are triggered which utilize Salesforce DX to deploy the code into the target sandbox.

The DevOps journey kicks off with the refreshing of individual developer instances within the CRM from the integration (SIT) environment to import changes made by other developers. Following this, a feature branch is created in Git from the integration branch, facilitating development and unit testing in individual environments. Developers can work efficiently in their own development environments while ensuring seamless integration and collaboration across the board.

Once new functionality has been implemented, the development phase is complete. Code undergoes a series of rigorous quality assurance procedures before moving on to the SIT environment. One of the procedures employed is static code analysis, which ensures adherence to coding standards and best practices. Additionally, pull requests are subjected to extensive review and approval procedures to maintain code quality. Before being promoted, the code is finally deployed to the SIT environment for integration testing, where system integrations are validated.

After successful testing, changes are promoted to the QA environment for functional and regression testing, which includes automation testing using Provar. QA environment is used for testing a feature end to end between the integrating systems. Automation testing scripts are run in this environment to ensure a bug free product. While code reviews make sure high coding standards are in place, QA and Automation testing ensures a well functioning solution.

The next step in the DevOps journey is the staging environment, where user acceptance testing and performance testing take place. PTPaaS sandbox is used for performance and load testing, which is a pilot offering specifically designed for performance testing. This sandbox is connected to the staging branch to ensure seamless execution of performance testing. With the final steps of validation complete, changes that originated as changes in someone’s feature branch can now be promoted to production and used by everyone.

In contrast to normal feature development, we need to be able to push urgent fixes. This flow takes a different route. Fixes originates in an environment dedicated for Hotfixes which is a replica of Prod (called Hotfix). A fix is worked on in the hotfix environment, tested and validated by QA and then pushed to Prod via Staging. Hotfixes circumvent the SIT environment. Once promoted to production, fixes are then back propagated to the lowest developer sandbox in the chain.

A Release time block (Deployment window) is determined for the code push and is agreed upon by all stakeholders for the code push. Having a dedicated time for deployment helps with setting expectations with the end users on the new features being released. One of the most significant challenges of CRM deployments is prolonged deployment times, which is further compounded with sizable implementations and longer test executions. Changes can sometimes take upto 90 min to be deployed. To circumvent this challenge, our framework takes a proactive approach by performing a build validation against the Production environment. This validation checks all test class executions and metadata validation in advance, typically more than 24 hours before the deployment window, and enables Quick Deploy in the target Org. During the deployment window, our DevOps automation executes a quick deploy from Buildkite, which helps to detect issues with the build or test runs early on and significantly reduces the actual deployment time. To further minimize deployment times, we have implemented Incremental Deploys, which deploy only the differences between codebases instead of the entire codebase. The previous commit ID is stored in a custom setting in the target org, and Buildkite retrieves it during deployment to commit only the hash set.

The Wins

• Using Quick Deploy reduced the deployment time from an average of 90 mins to 15 mins
• Enabled delta deploys instead of full package.
• Config and Metadata versioning.
• Built in static code analysis in the repository.
• Rollback mechanism.
• Integration test hooks.

Conclusion

Our DevOps implementation has successfully enabled efficient, continuous delivery of high-quality software on the CRM platform. This was achieved by seamlessly integrating Salesforce DX, Git, and Buildkite, thereby establishing a DevOps framework that is optimized for all personas, including developers, admins, and low-code users. As a result, we have witnessed a remarkable decrease in deployment time and a noteworthy improvement in software quality, ultimately leading to the delivery of greater value to our clients.

If this type of work interests you, check out some of our related positions.

Acknowledgments

Special mention to Yiming Yan, Sameer Miraj, Vasu Rampally,Srinath Therampattil, Leah Kennedy and the entire team for helping onboard onto this framework. Special thanks to Sudheer Peddineni for the guidance.

By: Clark Wright

Introduction

These days, as the volume of data collected by companies grows exponentially, we’re all realizing that more data is not always better. In fact, more data, especially if you can’t rely on its quality, can hinder a company by slowing down decision-making or causing poor decisions.

With 1.4 billion cumulative guest arrivals as of year-end 2022, Airbnb’s growth pushed us to an inflection point where diminishing data quality began to hinder our data practitioners. Weekly metric reports were difficult to land on time. Seemingly basic metrics like “Active Listings” relied on a web of upstream dependencies. Conducting meaningful data work required significant institutional knowledge to overcome hidden caveats in our data.

To meet this challenge, we introduced the “Midas” process to certify our data. Starting in 2020, the Midas process, along with the work to re-architect our most critical data models, has brought a dramatic increase in data quality and timeliness to Airbnb’s most critical data. However, achieving the full data quality criteria required by Midas demands significant cross-functional investment to design, develop, validate, and maintain the necessary data assets and documentation.

While this made sense for our most critical data, pursuing such rigorous standards at scale presented challenges. We were approaching a point of diminishing returns on our data quality investments. We had certified our most critical assets, restoring their trustworthiness. However, for all of our uncertified data, which remained the majority of our offline data, we lacked visibility into its quality and didn’t have clear mechanisms for up-leveling it.

How could we scale the hard-fought wins and best practices of Midas across our entire data warehouse?

In this blog post, we share our innovative approach to scoring data quality, Airbnb’s Data Quality Score (“DQ Score”). We’ll cover how we developed the DQ Score, how it’s being used today, and how it will power the next chapter of data quality at Airbnb.

Scaling Data Quality

In 2022, we began exploring ideas for scaling data quality beyond Midas certification. Data producers were requesting a lighter-weight process that could provide some of the quality guardrails of Midas, but with less rigor and time investment. Meanwhile, data consumers continued to fly blind on all data that wasn’t Midas-certified. The brand around Midas-certified data was so strong that consumers started to question whether they should trust any uncertified data. Hesitant to dilute the Midas branding, we wanted to avoid introducing a lightweight version of certification that further stratified our data without truly unlocking long-term scalability.

Considering these challenges, we decided to shift to a data quality strategy that pushed the incentives around data quality directly to data producers and consumers. We made the decision that we could no longer rely on enforcement to scale data quality at Airbnb, and we instead needed to rely on incentivization of both the data producer and consumer.

To fully enable this incentivization approach, we believed it would be paramount to introduce the concept of a data quality score directly tied to data assets.

We identified the following objectives for the score:

• Evolve our understanding of data quality beyond a simple binary definition (certified vs uncertified).
• Align on the input components for assessing data quality.
• Enable full visibility into the quality of our offline data warehouse and individual data assets. This visibility should 1) Create natural incentives for producers to improve the quality of the data they own, and 2) Drive demand for high-quality data from data consumers and enable consumers to decide if the quality is appropriate for their needs.

Composing the Score

Before diving into the nuances of measuring data quality, we drove alignment on the vision by defining our DQ Score guiding principles. With the input of a cross-functional group of data practitioners, we aligned on these guiding principles:

• Full coverage — score can be applied to any in-scope data warehouse data asset
• Automated — collection of inputs that determine the score is 100% automated
• Actionable — score is easy to discover and actionable for both producers and consumers
• Multi-dimensional — score can be decomposed into pillars of data quality
• Evolvable — scoring criteria and their definitions can change over time

While they may seem simple or obvious, establishing these principles was critical as they guided each decision made in developing the score. Questions that otherwise would have derailed progress were mapped back to our principles.

For example, our principles were critical in determining which items from our wishlist of scoring criteria should be considered. There were several inputs that certainly could help us measure quality, but if they could not be automatically measured (Automated), or if they were so convoluted that data practitioners wouldn’t understand what the criterion meant or how it could be improved upon (Actionable), then they were discarded.

We also had a set of input signals that more directly measure quality (Midas certification, data validation, bugs, SLAs, automated DQ checks, etc.), whereas others were more like proxies for quality (e.g., valid ownership, good governance hygiene, the use of paved path tooling). Were the more explicit and direct measurements of quality more valuable than the proxies?

Guided by our principles, we eventually settled on having four dimensions of data quality: Accuracy, Reliability (Timeliness), Stewardship, and Usability. There were several other possible dimensions that we considered, but these four dimensions were the most meaningful and useful to our data practitioners, and made sense as axes of improvement, where we care and are willing to invest in improving our data along these dimensions.

Each dimension could mix implicit and explicit quality indicators, with the key being: Not every data consumer needs to fully understand every individual scoring component, but they’ll understand that a dataset that scores poorly on Reliability and Usability struggles with landing on-time consistently and is difficult to use.

We could also weigh each dimension according to our perception of its importance in determining quality. We considered 1) how many scoring components belonged to each dimension, 2) enabling quick mental math, and 3) which elements our practitioners care about most to allocate 100 total points across the dimensions:

Meanwhile, if desired, the dimensions could be unpacked to get to a more detailed view of data quality issues. For example, the Stewardship dimension scores an asset for quality indicators like whether it’s built on our paved path data engineering tools, its governance hygiene, and whether it meets valid data ownership standards.

Presenting the Score to Practitioners

We knew surfacing the DQ Score in an explorable, actionable format was critical to its adoption and success. Furthermore, we had to surface data quality information directly in the venue where data users already discovered and explored data.

Luckily, we had two existing tools that would make this much easier: Dataportal (Airbnb’s data catalog and exploration UI), and the Unified Metadata Service (UMS). The score itself is computed in a daily offline data pipeline that collects and transforms various metadata elements from our data systems. The final task of the pipeline uploads the score for each data asset into UMS. By ingesting the DQ Score into UMS, we can surface the score and its components alongside every data asset in Dataportal, the starting point for all data discovery and exploration at Airbnb. All that remained was designing its presentation.

One of our goals was to surface the concept of quality to data practitioners with varying expertise and needs. Our user base had fully adopted the certified vs uncertified dynamic, but this was the first time we would be presenting the concept of a spectrum of quality, as well as the criteria used to define quality.

What would be the most interpretable version of a DQ Score? We needed to be able to present a single data quality score that held meaning at quick glance, while also making it possible to explore the score in more detail.

Our final design presents data quality in three ways, each with a different use case in mind:

1. A single, high-level score from 0–100. We assigned categorical thresholds of “Poor”, “Okay”, “Good”, and “Great” based on a profiling analysis of our data warehouse that examined the existing distribution of our DQ score. Best for quick, high-level assessment of a dataset’s overall quality.
2. Dimensional scores, where an asset can score perfectly on Accuracy but low on Reliability. Useful when a particular area of deficiency is not problematic (e.g., the consumer wants the data to be very accurate but is not worried about it landing quickly every day).
3. Full score detail + Steps to improve, where data consumers can see exactly where an asset falls short and data producers can take action to improve an asset’s quality.

All three of these presentations are shown in the screenshots below. The default presentation provides the dimensional scores “Scores per category”, the categorical descriptor of “Poor” along with the 40 points, and steps to improve.

If a user explores the full score details, they can examine the exact quality shortcomings and view informative tooltips providing more detail on the scoring component’s definition and merit.

How the Score Is Being Used Today

For data producers, the score is providing

• Clear, actionable steps to improve the DQ of their assets
• Quantified DQ, measuring their work
• Clear expectations around DQ
• Targets for tech debt clean-up

For data consumers, the DQ Score

• Improves data discoverability
• Serves as a signal of trustworthiness for data (just like how the review system works for Airbnb Guests and Hosts)
• Informs consumers of the exact quality shortcomings so they can be comfortable how they’re using the data
• Enables consumers to seek out and demand data quality

From a data strategy perspective, we are leveraging internal query data combined with the DQ Score to drive DQ efforts across our data warehouse. By considering both the volume and the type of consumption (e.g., whether a particular metric is surfaced in our Executive reporting), we are able to direct data teams to the most impactful data quality improvements. This visibility has been very enlightening for teams who were unaware of their long tail of low-quality assets, and has enabled us to double down on quality investments for heavy-lift data models that power a significant share of our data consumption.

Finally, by developing the DQ Score, we were able to provide uniform guidance to our data producers on producing high-quality, albeit uncertified assets. The DQ Score has not replaced certification (e.g., only Midas-certified data can achieve a DQ Score > 90). We continue to certify our most critical subset of data, and believe the use cases for these assets will always merit the manual validation, rigor, and upkeep of certification. But for everything else, the DQ Score reinforces and scales the principles of Midas across our warehouse.

What’s Next

We’re excited about now being able to measure and observe quantified improvements to our data quality, but we’re just getting started. We recently expanded on the original DQ Score to score our Minerva metrics and dimensions. Similarly, we plan to bring the same concept of a DQ Score to other data assets like our event logs and ML features.

As the requirements and demands against our data continue to evolve, so will our quality expectations. We’ll continue to evolve how we define and measure quality, and with rapid improvement in areas like metadata management and data classification, we anticipate further efficiency and productivity gains for all data practitioners at Airbnb.

Appreciations

The DQ Score would not have been possible without several cross-functional and cross-org collaborators. They include, but are not limited to: Mark Steinbrick, Chitta Shirolkar, Jonathan Parks, Sylvia Tomiyama, Felix Ouk, Jason Flittner, Ying Pan, Logan George, Woody Zhou, Michelle Thomas, and Erik Ritter.

Special thanks to the broader Airbnb data community members who provided input or aid to the implementation team throughout the design, development, and launch phases.

If this type of work interests you, check out some of our related positions.

****************

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

How Airbnb leverages ML/NLP to extract useful information about listings from unstructured text data to power personalized experiences for guests.

By: Hongwei Li and Peng Wang

Introduction

At Airbnb, it’s important for us to gather structured data about listings and better understand the data, so we can help Hosts provide great experiences for guests. For example, guests who work remotely need to know if a listing has a suitable workspace and reliable internet, while guests with children might need items like highchairs and cribs. However, not all listings clearly display these attributes, causing there to be a mismatch between what Hosts listings have and what guests are looking for.

This is just one of many examples of how we can use the unstructured data generated on our platform, including text data that has undergone anonymization steps from various text-based guest interactions with the platform, to extract useful structure data. Instead of relying on Hosts to manually input all the potential listing attributes, which would be tedious given the vast number of attributes guests care and inquire about, we developed a machine learning system called Listing Attribute Extraction Platform (LAEP) for extracting the structure data at scale. Note that the original name of the project is called LATEX (Listing ATtribute EXtraction) and it is cited in our previous tech blog. We have since renamed the project to LAEP.

LAEP automatically extracts structured information, such as listing attributes, directly from the unstructured text data we mentioned above. The attributes collected by LAEP are then integrated into various applications, building Airbnb’s Listing Knowledge. It powers downstream tools like the Attribute Prioritization System (APS) and listing attribute collection system (Eve).

LAEP doesn’t just extract listing attributes, it has the ability to detect different types of entities, such as activities, hospitalities, and points of interest (POI) like famous landmarks. This opens up possibilities for supporting a wide range of product applications. For example, hospitality data can help guests get personalized services during the stay while activity data can help identify and create new categories that guests love.

Prior to LAEP, Airbnb had multiple ways to collect structured information for listings, including the Listing Editors page for Hosts, the Supplementary Review Flow (SRF) for guests, and partnering with third-party vendors. However, these approaches faced several challenges and limitations. For instance, Airbnb minimized the impression of SRF questions in the standard review flow to boost the guest review experience, resulting in reduced data intake from the guest side. Consequently, there has been a growing need to extract listing information from unstructured text data, and LAEP was developed to address the aforementioned issues by automating this data collection process.

The LAEP technology gathers and analyzes anonymous and unstructured text data, enabling many potential applications that can enhance the Airbnb experience for both Hosts and guests.

LAEP Implementation

There are three main components in LAEP:

1. Named Entity Recognition (NER): This component identifies and classifies specific phrases or entities in free text into predefined categories like amenities, places of interest, and facilities. For example, from various sources the phrase “swimming pool” would be detected as an entity with the type “Amenity”.
2. Entity Mapping (EM): Once an entity is detected, EM maps it to standard listing attributes stored in Airbnb’s attribute database (Taxonomy). This allows LAEP to create a comprehensive catalog of Airbnb listings by associating detected entities with their corresponding attributes.
3. Entity Scoring (ES): ES determines the presence of a detected phrase within a listing. It infers whether the attribute mentioned actually exists in the associated listing and provides confidence level.

Below is an illustration of the components within LAEP is as follows:

Named Entity Recognition

There are many off-the-shelf pretrained NER models that can extract general entity categories, but none of them fully supports Airbnb’s use cases. Therefore, we built our own NER models to detect and extract predefined entities important to Airbnb business from free text.

The NER model defines five types of entities (Amenity, Facility, Hospitality, Location features, and Structural details) that are important to Airbnb. We sampled and labeled 30K example texts from six channels, then trained the NER model. For current product use cases, we apply the language detection module and filter out English text only. In the future we may build the multilingual Transformer based NER model to handle non-English content. Text is then split into tokens. NER mode localates entity span, and classifies entity labels by using a convolutional neural network (CNN) framework. The output is a list of detected named entities, in the format of tuples <entity label, start index, end index>. Combining all components together, the NER pipeline is shown in figure 3.

Figure 4 shows an example output from the pipeline with detected entities highlighted.

The labeled dataset was randomly split into training and testing datasets with a 9:1 ratio. After training completes, we evaluate the model performance on the testing dataset across all text channels. The evaluation criteria uses Strict Match which requires correctly identifying the boundary and category of the entity, simultaneously. The model overall performance and each category’s performance are in figure 5.

Entity Mapping

There are many different ways for people to talk about the same thing. For instance, we found over twelve variations for the attribute “lockbox,” such as lock box, lock-box, box for the key, and keybox. Typos like “ket box” are also common due to input from error-prone mobile devices. Therefore, we need to map different variations of named-entities to the standard entity name as defined by the standard taxonomy for downstream applications.

With hundreds of listing attributes but millions of detected phrases in a year, many phrases map to the same attribute (like “lockbox”) while others have no mapping. To address this, we introduce confidence levels for mappings, allowing us to establish rules for cases where mapping cannot be done. A confidence value between 0 and 1 is assigned, and if no mappings exceed the confidence threshold, it is marked as “No Mapping.”

Labeling these mappings becomes challenging when dealing with numerous unique phrases and potential attributes. Typically, labeling involves comparing the semantic similarity between the phrase and each of the 800+ attribute names. To overcome this, we started with unsupervised learning methods to tackle the problem instead of using the supervised learning methods to save significant labeling efforts.

In LAEP, the entity mapping approach involves the following steps:

1. Preprocessing: Both the listing attributes and detected phrases undergo preprocessing techniques such as lowercasing and lemmatization to eliminate unnecessary word variations.
2. Mapping to Word Embeddings: All standard listing attributes are mapped to the word-embedding space using a word2vec model fine-tuned with Airbnb’s text data.
3. Finding Closest Attribute: For a preprocessed detected phrase, the closest listing attribute is determined based on cosine similarity in the word-embedding space. The similarity score serves as the confidence score for the mapping.

As the example in the figure above, the word “Lock-box” is mapped to the embedding space of listing attributes and compared with each attribute. The closest match is found with the attribute “lockbox,” which is identified as the top mapping.

Entity Scoring

After mapping a detected phrase to a standard listing attribute, it’s important to infer metadata about the attribute, such as its existence, usability, and local sentiment. Among these, attribute presence is crucial for the guest experience, especially for the example of amenities like “crib” or “highchair” for guests with infants.

The presence model in LAEP determines if the mapped attribute exists in the listing by performing local text classification. It provides a discrete output (YES, Unknown, NO) indicating attribute presence, accompanied by a confidence score reflecting the level of confidence in the inference.

The label classes are {YES, Unknown, NO}, where Yes means the attribute is present, NO means it’s not present, and Unknown accounts for cases where presence is hard to determine from the text alone (e.g., amenity not present).

To build this text classification model, the ES component employs a fine-tuned BERT model. It analyzes source data, including detected phrases and their local context, to infer attribute existence. The output can then be used in the APS and Eve system to provide recommendations to Hosts, merchandize existing home attributes, or clarify popular listing facilities..

The model architecture (Figure 8) utilizes a pre-trained BERT model with text data from six different sources. The input text is truncated to a maximum length of 512 tokens. Empirical studies suggest that using 65 words around the detected phrase (32 before and 32 after), achieves the best result. The embeddings from the [CLS] token are passed through a fully connected layer, dropout layer, and ReLU linear projection layer to generate a probabilistic vector over the label classes.

Conclusion

In this post, we introduced an end-to-end structural information extraction system within Airbnb, LAEP, to detect phrases of interest from various text data sources, map them into standard listing attribute taxonomies, and then infer the meta information of the attributes from the contextual information in the texts while also having privacy by design controls with the objective to not process personal information. LAEP is applied in downstream applications like APS, and can be leveraged to help our teams find new categories of listings and discover new listing attributes that matter to guests. It helps us to understand Airbnb’s listing better with scale and can power future applications to continue improving the experience of both our Hosts and guests.

If this type of work interests you, check out some of our related positions at Careers at Airbnb!

Acknowledgments

We would like to thank all the people who supported this project — Qianru Ma, Joy Jing, Xiao Li, Brennan Polley, Paolo Massimi, Dean Chen, Guillaume Guy, Lianghao Li, Mia Zhao, Joy Zhang, Usman Abbasi, Pavan Tapadia, Jing Xia, Maggie Jarley and more. Special thanks to Ben Mendeler, Shaowei Su, Alfredo Luque and Tianxiang Chen from the ML-infra team for their generous support and help.

****************

All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.