By Vanessa Johnson, Daniel Falokun, Michael Ingber and Oleksandr Zabiiako

The Games Mobile App introduced dark mode back in October. This seamless implementation was a collaborative effort, requiring careful planning and management of design, cross-platform coordination, and engineering work across both legacy and modern user interface systems. The process to implement dark mode across the app included consideration on the visuals, colors, and icons that are mode specific to maintain legibility and brand consistency. The New York Times Games team is excited to have delivered this feature before the end of 2025, and thrilled to be able to roll out a feature that continues to make our app accessible for all.

Development Work on Mobile

On the development side, the mobile and web engineers synchronized assets which were colors, fonts, and images to ensure that hybrid screens showed the correct styles and colors. This was possible since our design system across mobile was constructed in a way that encouraged consistency and scalability across platforms. With clear and easy to follow designs supported by our internal design system, Android and iOS could ensure consistency on both platforms.

From an Android perspective, it required separate approaches for XML (Extensive Markup Language) based screens which use an imperative, hierarchical approach and Jetpack Compose screens that use a more modern declarative approach. XML and Jetpack Compose are different ways the UI can be constructed. To ensure dark mode compatibility, all features were revisited to make sure they were adapted, especially the additions that were being worked on during the dark mode project. Another aspect of this development was updating external libraries that required making pull requests, creating a version of those changes for testing to make sure it worked in the main project, approval, testing, and merging of the changes in the external library. With these additions, the support of dark mode was a multiteam effort.

From an iOS perspective, supporting dark mode required distinct approaches for UIKit-based screens (including legacy Objective-C views) versus SwiftUI screens. UIKit screens depended on two key callbacks: overrideUserInterfaceStyle, which allows developers to explicitly set a screen’s appearance regardless of system settings, and traitCollectionDidChange, which triggers whenever the system’s appearance preference changes, enabling dynamic updates to UI colors and styles. SwiftUI screens, by contrast, used the @Environment(\.colorScheme) property wrapper along with asset catalogs configured for light and dark variants, providing a more declarative and automatic approach to handling appearance changes.

This distinction was critical because UIKit screens required manual observer pattern implementation to catch appearance changes, while SwiftUI handled these updates natively through its reactive system.

The Games app also integrates several shared New York Times packages that are maintained outside the core repository. To keep dark mode working consistently across all features, we needed to coordinate updates to these external packages. This approach ensured that components such as the login flow and profile avatar selection screen, which rely on these shared packages, would support dark mode alongside the rest of the application.

Why was this coordination necessary?

Dark mode is a system-wide feature that users expect to work seamlessly across the entire app. Without updating these external packages, parts of the experience would break or appear inconsistent, diminishing the overall value of the dark mode feature.

With dark mode now available in The New York Times Games app, our users can play their favorite puzzles any time of the day while reducing eye strain during evening play sessions. Dark Mode is a highly valued feature — players are actively using it, demonstrating strong demand for this capability and validating the effort invested in its implementation across all screens and dependencies.

The design perspective

The first major step was moving from fixed color definitions to a semantic color system based on variables. Instead of using hard-coded colors, we introduced semantic tokens to the library, such as backgrounds, text, strokes and interactive elements. This approach allowed us to support Dark Mode consistently and at scale. Since we revisited the colors we took the opportunity to reorganize our naming convention as well. To support this shift we started from creating a color mapping document. On the left it listed our legacy colors, and on the right it was mapped to its new color variable. For example, what used to be called “Background Primary“ became “$bg-page”. This mapping was essential for keeping both designers and developers aligned.

Once the foundations were in place, we moved on to adapt actual screens. That included updating components and pages, defining rules for how surfaces behave in light and dark contexts, and covering edge cases. There was also a need to create new design tokens that were not needed before. For example, for the bottom sheet in the light theme, the background was always white. But when switching to the dark mode, using dark gray instead of black makes it much better.

Our illustrations also couldn’t simply have their colors inverted. Flipping the colors made it look too high in contrasty. We ended up reviewing each illustration individually and manually adapting it for dark mode. In addition to illustrations, we also revisited in-app animations to ensure they support Dark Mode. For example, new features are introduced through a modal shown when a user opens the app after an update. These modals typically include an animation that highlights the main idea of the feature, and we update this system so that the animations now adapt seamlessly to both light and dark modes.

An essential part of the process was a design review of every screen as it was implemented. Each feature and flow went through VQA (visual design check) to ensure visual consistency. This includes validating all interactions and contextual elements, making sure no assets were missing, miss-colored or behaving unexpectedly in a dark environment. This allows us to catch issues early and deliver great final experience.

Overall, the design work behind Dark Mode became much more than a tech upgrade or just inverting our existing colors.. It was a fully-scale process that brought evolution of our design language, color system and visual assets.

The web perspective

We launched a new Badges feature in September, and a “badge detail” view was built which can be opened in multiple places. It can be opened within each of the game webviews where users can earn badges, including on web/mobile-web/news apps, and it can also be opened from the natively built Me Tab and Trophy Case in the Games Apps. For the latter use case, we built a standalone hybrid webview that serves a single badge detail.

This standalone webview is powered by a set of query parameters. The app passes all dynamic information (which badge it is, when it was earned, the user’s progress towards the next level of the badge, etc) through the query string in the URL and the webview parses that information to render the content.

In order to support the apps’ dark mode, we needed this standalone webview to respect each app’s dark mode setting. The apps already contain many webviews — every non-crossword game is a webview. But each of those games has its own internal dark mode setting, disconnected from the app’s global dark mode setting. This was intentionally designed to give users more fine grained control over which games they prefer in dark or light mode.

To sync dark mode between this standalone webview and the apps, we updated the native to web contract so that the webview would recognize an additional query parameter named “dark”. If the app passes in that query parameter, the webview knows it must render in dark mode. Badge detail is then able to set its theming in a very similar way to most game webviews, like so:

Some extra care needed to be taken to get enough contrast between the white text and the multi-colored radiating shape animation, especially given the blurred effect on the webview’s background. So in dark mode only, we lower the opacity of the radiating shapes to 40%.

In summary, to support dark mode on this webview, we leveraged prior work that enabled dark mode on game webviews. The key change was that instead of giving the user control over the setting within the webview, we allowed each app to pass in that app’s setting via a query parameter.

Takeaways

Dark mode in the apps has been a big milestone for The New York Times and brings great joy to our users. This project showed the complexity and teamwork needed across platforms to bring a high impact project like Dark Mode over the finish line.

Vanessa Johnson is an Android Engineer working on the Games App at The New York Times. Vanessa has tech led multiple projects that include Dark Mode, Strands Archive, and Connections Bot in the Apps. When not at work, Vanessa likes to play pickup basketball and travel to speak at various conferences.

Dan Falokun is an iOS Engineer working on the Games App at The New York Times. Dan has tech led multiple projects that include The Midi Crossword, Strands Archive, Dark Mode and Connections Bot in the Apps. Outside of work, Dan likes to travel and listen to music.

Mike Ingber is a Staff Web Engineer on NYT Games.

Special Thanks to our past interns: Naomy Portillo and Tony Vu for starting the project on mobile, Ihor Shamin, Nazar Novak and Maksym Safronov for helping get dark mode over the finish line on the Android side and Lian Chang, Kathy Lee, Robert Vinluan and William Frohn on the design side.

How The New York Times Games Team Delivered Accessible, Cross-Platform Dark Mode was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing Danny Cassidy, Senior Technical Product Manager, Developer Productivity in Developer Platforms

By Geethu Jacob, Daniel Cassidy, Nikki Larson and Erica Vendetti

“PMs actively and effectively engage with engineering and technology decisions, understanding possibilities and consequences.”

This article is part of a New York Times Open series showcasing the breadth of experiences and backgrounds of our Product Managers.

Engineering can fuel product leadership

For Danny Cassidy, coding was an early creative outlet. As a high school student, he taught himself to code, trying his hand at frontend web development.

“I was very inspired by online communities back then, and how the internet could foster connection for people across the world. So, I first learned how to make extensions on forum software. My first paid gig in high school was building a social networking feature into a site for pet owners so that they could create profile pages for their pets. Those were the days!”

Building on the base of that passion, Danny studied information technology in college. “There’s a definite craft to software development”, he says. “It’s not just technical. You are thinking about solving a problem in an elegant and legible way. I always enjoyed that part of the process.” As a software engineer, Danny focused on building backend APIs and content management systems (CMS) for media brands. Here, Danny learned to understand how engineering solutions fit into the business context. Engineers “need to know how a solution will evolve and scale over time. And their solutions need to be understood and maintainable by a multitude of people throughout all of this.”

In 2015, Danny accepted an engineering job at The Times working on the backend APIs that deliver our digital content from our CMS to our desktop and mobile apps. As an engineer at The New York Times, Danny had the opportunity to work with more stakeholders than before:

“As I started working on more complex, cross-functional projects, I noticed great engineering leaders consistently asking fundamental questions: Why are we building this? Who is it serving? How is it helping the business? This realization inspired me to more fully dedicate my career to understanding and measuring how technology can drive significant business and user impact.”

Danny started to think about a move to Product Management after a conversation with leadership:

“I had no idea there was such a thing as Technical Product Management. It was honestly never a consideration for me. I always thought I’d spend my entire career in engineering. One day, my boss at the time asked me if I was interested in Product Management, and it opened up this whole conversation. I am very grateful that he recognized a shift in the kind of work I was excelling at and raised the question to me.”

Leveraging a technical background to build for engineers

Danny is now a Senior Technical Product Manager in the Delivery Platforms Mission. He is responsible for providing centralized infrastructure and tooling to the New York Times product engineering teams so they can develop more efficiently. “The platform we are building out is very technical and serves many kinds of engineers at The Times, all with different levels of experience, expertise, and needs. I lean on my technical background almost every day to help navigate across technical domains and user personas.”

In his current position, Danny’s background as an engineer enables him to effectively engage with engineering and technology decisions, understanding the possibilities and consequences of each potential path. For a role like Danny’s, “it’s difficult to have strong Product Judgment if you struggle to understand how technical decisions impact your users and the business outcomes you are trying to deliver.”

Danny’s engineering background proved invaluable when his team faced a critical CI/CD pipeline replacement. (Think of a CI/CD pipeline as the “assembly line” for software, making sure updates reach users quickly and reliably!) When a vendor sunset their current tooling, Danny understood firsthand the developer blockers it created for multiple teams. Armed with empathy for the engineers, Danny led a user-centered, data-driven evaluation across The New York Times and Wirecutter. His deep engagement with engineering allowed him to synthesize platform pain points, assess future-fit alternatives, and ultimately align the organization around a long-term CI/CD strategy that successfully balanced business impact, developer experience, and technical feasibility.

Danny’s journey underscores how deep technical roots can strengthen a product manager’s leadership especially in platform work where empathy for engineers, strategic thinking, and business context must align. As you’re considering your next career steps, if you’re passionate about connecting user needs and business outcomes in tech, product management could be your next career step.

And if you’re like Danny, your technical acumen might be a powerful asset to build tools that empower others to do their best work!

The Right Team for the Job: From Engineering to Product: A Superpower for Developer Platform… was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Nikki Larson, Erica Vendetti and Michael Beach

Our mission is simple: We seek the truth and help people understand the world. Accomplishing that mission requires each of us to build trust, seek out different perspectives, collaborate respectfully, and continuously strive to deliver the very best in all we do. But what exactly does that mean for Product Managers (PMs)? How do we earn and maintain trust with tens of millions of global readers in an era of misinformation? Where are new readers finding and developing a daily habit with our original reporting? How can Product unlock growth opportunities for our business? How can we help readers connect with their passions by building products like Games, The Athletic or Cooking? How do we use AI to responsibly advance our mission? How do we scale our systems to support breaking news? PMs at The New York Times solve a variety of problems for our readers, our journalists, other product and functional teams. To meet the evolving needs of our almost 12 million subscribers, we rely on a Product team with a wide range of unique backgrounds.

It takes creativity and expertise from people in every part of the company to fulfil our mission. So it’s not uncommon for our PMs to have started their careers in roles outside of product. To build a world-class Product team, we’ve hired both PMs by trade as well as those who were previously dancers, consultants, entrepreneurs, journalists, strategists, operations specialists, and engineers. Our diverse experiences fuel the analytical rigor, creativity, communication skills, leadership, and execution skills needed to excel in getting to know our users’ needs and leading teams to build solutions to meet them. This post will help introduce you to PMs whose backgrounds outside of Product Management help them excel in their current roles at The New York Times.

One guiding framework, many possible solutions

While our PMs work on products across the entire business, they are all guided by a common set of competences. No matter how different the problem, PMs continually develop their skills in these areas to help guide our teams to solutions and root us in our values.

• Product Strategy: PMs use structured thinking to define and prioritize impactful work that solves the right problems for our users, our business, and our journalistic mission. They analyze trends and develop product vision with empathy for our various users, including readers seeking a range of news and life needs, journalists who use our editorial tools to produce the most compelling and distinct multimedia reports, and engineers to enhance developer productivity.
• Execution: PMs are here to move the business forward by unlocking product-driven growth opportunities. They are resourceful, driven, and resilient, overcoming obstacles with determination and focus to help teams move with urgency, focus, and impact.
• Leadership: PMs are critical for team dynamics, helping to facilitate inclusive, high-performing teams where everyone feels valued. They lead with influence, building trust and communicating clearly to achieve shared goals.
• Cross-Functional Collaboration: PMs foster strong cross-functional relationships, collaborating effectively to set priorities and drive results.
• Technical Acumen: PMs possess a strong understanding of technology, enabling them to effectively engage with engineering decisions, including helping to uncover the possibilities and consequences of technical choices.

The New York Times welcomes PMs of different backgrounds and skills, who add a richer understanding of human experience into the very core of the products they manage. PMs share a deep commitment to the importance of independent journalism, and leverage their unique perspective to make the essential value of our journalism better understood and more useful to more people. This series is meant to help you discover how our Product team maintains the integrity of our mission while navigating the digital transformation around us all. Their stories are a nod to the power of transferable skills and the nature of product leadership at The New York Times.

Ultimately, this series is more than just a showcase of our team. It’s an exploration of how diverse pathways can lead to extraordinary impact and a reflection of our belief that the best products are built by teams who bring the richness of the human experience to the challenge of solving problems. They build solutions for our readers, our journalism, and the world.

Curious about joining our team? Check out our career boards for more information.

Building The New York Times Product Team: The Right Team for the Job was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Eric Chima and Leonardo Quixadá

At The New York Times, we’re all excited to build fresh new experiences that delight our users. Our product managers are driven to find new ways to get our work in front of our audience and build reader engagement. Our engineers are motivated to solve unique technical challenges. And just when you think all that work is on track, breaking news strikes and all of our plans change at once.

With all that going on, who could blame us if our test coverage couldn’t quite keep up?

Like every engineering organization, The Times deals with routine maintenance tasks: updating dependencies, cleaning up old code, maintaining accessibility standards, and, yes, building testing into all of our products. Our engineers are committed to quality, but when you work at the speed of news, there’s always a new issue that needs to be addressed. Fortunately, generative AI has arrived with the promise of tidying up after us, taking care of the busy work, and giving time back to our developers to focus on feature development. But how far can you trust it?

Recently, one of our platform teams used AI tools to build out unit tests across our flagship product, the News site. This was an opportunity for our testing to catch up with our rapid development. Our goal was to improve the reliability of our web app, but also to evaluate AI products and determine how far we could push them to do work in bulk across our codebase. As expected, the agents required strict human supervision, but they improved our efficiency in writing tests and let us quickly expand coverage on some of our most critical code.

AI accelerates, but humans test the tests

Unit tests are a crucial part of the development process. The idea is to divide application code into small pieces and write tests for each, creating guardrails to ensure that changing one piece of the site doesn’t impact the rest. Having comprehensive unit tests makes the site more reliable and lets us roll out major changes (like last year’s React upgrade) more confidently. But like many fast-moving engineering teams, we found our test coverage hadn’t kept pace with our rapid development. When we audited our News site, we found that we only had about 60 percent test coverage across our codebase. To compensate, we had to conduct extensive manual testing before releasing new features.

To us, expanding our test coverage seemed like an ideal application for generative AI. We knew that AI could struggle to deal with code context across a large project, but that’s less of an issue when the agent is writing self-contained unit tests for modular code. The tool only needed to cover each individual file, a task it should be able to replicate reliably without understanding the entire codebase.

We set out with an initial goal of reaching 80 percent test coverage across our entire News monorepo, touching code owned by over a dozen different teams. We initially covered six different projects, taking them from an average of 28% code coverage all the way to 83%. We estimated that AI cut the required work by 70%, reducing the effort for a single project from weeks to hours.

But while AI accelerated our work, it wasn’t reliable or autonomous enough to cover the whole monorepo without human supervision. The tools had to be closely monitored and each test verified. It took enough work that even with AI assistance, our small group wasn’t going to be able to cover everybody else’s code. Instead, we chose a variety of different web projects and key news pages as test cases, and then wrote up instructions that other teams (and now, you, our Open Blog readers!) can use to generate their own unit tests.

The test generation process

Before we could begin, we needed a reliable way to measure test coverage throughout our code base. To do that, our engineers wrote a small package called test-cov that would iterate through a directory and measure the test coverage of each file. That package was key because it not only allowed us to measure our progress but would be repeatedly fed into the AI prompt as it learned which files needed additional tests.

With that in place, we began working with simpler, standalone projects within our monorepo. They allowed us to refine our prompting strategies to consistently achieve our 80% coverage target before moving on to more dense code. Eventually, we advanced all the way to the Story template, the critical page type that displays most of our articles. By that stage, we had distilled our process to a point where we were able to quickly and efficiently improve Story’s code coverage from 53 to 83 percent.

The actual workflow consisted of two loops: a big loop (the “User Loop”) and a smaller one (the “AI Loop”). The User Loop involves creating a chat, feeding it a prompt, and then letting the AI Loop run. Once the AI’s work is done, the user can review the results and trigger the AI Loop again if necessary. Ideally, the User Loop would only need to run once, but that isn’t always the case. The AI Loop involves the AI agent running the test-cov package, identifying coverage gaps, generating new tests, running the tests to validate them, and then running the test-cov package again to see whether more iterations are necessary.

Allowing an LLM to edit production files carries inherent risks. One is that the agent will introduce regressions or security holes into your code. Another is that the AI will “cheat” by editing the source code to conform to existing unit tests, rather than the other way around. To combat that, we introduced the most important instruction in our prompt:

“Do not touch the source code.”

That command was one of two “walls” that confined the AI agent. The other wall was our test-cov package, which ran on every iteration and produced a read-only report. That was another constraint that could not be cheated. Together, the two walls trapped the AI loop in a silo where it could only leave if it hit the goal.

The agent in action

Our early experiments involved a lot of trial and error. Sometimes, the agent would create useful unit tests. Other times, it would hallucinate unknown commands, get stuck in a loop of creating and deleting tests, or just run too many commands and hit our LLM’s limit. Even with our smallest applications, the work was slow going.

To solve this, we asked the agent itself to refine the rules we were feeding into the tool. Our final prompts were roughly seven pages (largely AI-written but reviewed and edited by humans) and included strict guidelines on how the LLM should write and verify its tests. We gave it rules to follow across a wide variety of categories, including:

• Agent role — If you tell the agent that it’s a senior Javascript engineer, it’s more likely to write code like one. See example below.

# Unit Test Coverage Improvement Agent

You are a senior JavaScript/TypeScript engineer specializing in unit testing with Jest and React Testing Library. Your mission is to systematically improve code coverage for JavaScript/TypeScript source files in a monorepo environment.

• Target identification and coverage areas — Coverage goals, what directories to search, what testing frameworks to use, and where to target tests. We focused on things like components, utilities, API calls, etc.
• Test structure and generation — What types of tests to create, where to put them, how to name them, etc.
• Available commands — Tools the agent can use to do its work. That included our test-cov package and the tools necessary to execute and validate tests in our system.
• Success criteria and quality standards — All tests pass without flakiness, tests are maintainable and use standard conventions, and test coverage goals are met.
• Safety guardrails — Focus on quality over quantity, report obstacles rather than making assumptions, and back out after 10 iterations to avoid infinite loops. Most importantly, never modify the source code!
• Error handling — What to do if coverage becomes flaky, test coverage decreases, or the agent becomes stuck on complex logic.
• Reporting template — How the agent should report the results when it’s done.

Below: An example rule for reporting templates

## Reporting Template

```markdown
## Code Coverage Improvement Report

**Project**: [Project Name]
**Date**: [Date]
**Duration**: [Time Spent]

### Summary
- **Files Processed**: X
- **Initial Average Coverage**: Y%
- **Final Average Coverage**: Z%
- **Target Achievement**: [Met/Partially Met/Not Met]

### Detailed Results
| File | Initial | Final | Improvement | Status |
|------|---------|-------|-------------|--------|
| file1.ts | 45% | 85% | +40% | ✅ Target Met |
| file2.ts | 60% | 78% | +18% | ⚠️ Close to Target |

### Challenges Encountered
- [List any significant obstacles]

### Recommendations
- [Suggestions for further improvement]
- [Areas needing manual review]

### Files Requiring Manual Attention
- [List files that couldn't reach 80% threshold]
- [Include specific reasons and suggestions]

Once we had refined the prompt enough, the process was much smoother. We still kept close supervision on the process, but the agent was efficiently creating effective tests, and the rules were reusable across larger projects. We had initially hypothesized that large routes would be more difficult than standalone projects, but even when we graduated to the Story route, which is a very complex codebase, we were able to create the tests we wanted with only minimal effort and human supervision. The time required to add or improve unit tests shrank from weeks to hours.

Limitations and learnings

Despite our success with a longer prompt, we found that it’s important to limit the chat context to only essential information. It’s tempting to drop entire directories or pages of technical documentation into the chat, but this can lead to a phenomenon called context rot. As the tool iterates, it becomes distracted by irrelevant information or calls incorrect tools. In some cases, it will hallucinate into the context and then reference its own hallucinations, causing a feedback loop that poisons the test-generation process.

It was also important to limit the agent’s scope each time we ran it. Although we were eager to scale the project quickly, applying these methods to an entire codebase at once introduces several problems. Iterating over large codebases expands the context, increasing the potential for context rot. And even if everything runs successfully, the resulting commit might be too big for a human to effectively review.

In fact, code reviews had been one of our major early concerns with the project. Because our group didn’t own most of the code we were working on, we would be turning over our generated tests to other Times teams to approve. Would The Times’s Story team accept the AI-generated tests we were adding to their project?

To mitigate those worries, we kept our client teams apprised of our goals and plans throughout the early stages of the project. When the new unit tests were ready, we arranged a formal review with Story engineers to make sure they met their standards. In the end, we were pleased by how well our work was received. The tests were effective and well-written. And, as it turns out, teams are very excited when you automate their grunt work.

What’s next?

When we think about the potential of AI in web development, we usually imagine a brand-new website spun up in seconds. We’ve all seen the demos where AI agents quickly build and deploy a web-based version of Tetris. For teams like ours, though, who are building features on top of extensive existing web infrastructure, the appeal of AI is its potential to manage the busy work associated with large codebases. After the success of the test generation project, we’re experimenting with other ways that AI can make our lives easier. Can we use the same principles to update our old, end-of-life CSS framework? Could they help us migrate separate projects to a common monorepo?

We started on News, but our AI work is already paying dividends across the bundle. Last week, our Games team tried our test-generation process on their latest puzzle game, Pips. When they told us that it turned their unit test epic into a two-point ticket, it validated everything we’d done in this project. We’re hoping these techniques become a standard used throughout the company.

This project was just one focused experiment within a broader set of initiatives exploring how AI can responsibly support software development at The Times. Under the close supervision of our engineers, product managers, and designers, it’s clear that AI tools can meaningfully enhance our productivity and change the way we work.

After all, the more we can automate the mundane, the more we can free up our talented contributors to focus on the things they do best. At least until breaking news strikes again.

Leonardo Quixadá is a Senior Software Engineer at The New York Times, working as a web platform engineer for the The New York Times. He is passionate about using GenAI tools and strong engineering practices to improve developer productivity and the speed at which high-impact journalism reaches millions of readers.

Eric Chima is the Senior Technical Product Manager for the Web Platforms team at The New York Times. His team builds the web foundations that enable Times engineers to create new experiences and ways of delivering our journalism.

How The New York Times is scaling Unit Test Coverage using AI Tools was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Nuoran Chen, Mina Chung, Frannie Ello, Christina Su, and Bella Rinne

A key element of The New York Times Product Design Internship Program is a cohort project where all the interns work as a group to create a deliverable that is presented to the entire product design team. We, the summer 2024 cohort, were tasked with designing a proof-of-concept that brings the internal company museum in the NYC office to remote employees.

The Times Museum is home to multiple artifacts where visitors can freely roam from each display to understand how their stories intertwine. From an official letter authorizing the publication of the leaked Pentagon Papers to physical effects of our war correspondents, each display is unique and speaks to the history behind The Times’ mission to bring independent journalism to all.

Our group began our work with a tour of The Times Museum led by retired journalist and museum curator, David W. Dunlap. On this tour, David guided us through each artifact in chronological order, describing the stories behind each piece and additional facts that were not fully captured with labels. His guided tour was essential for understanding the details behind The New York Times history and journalistic evolution. This tour informed two key requirements for the virtual museum.

1. A guided tour with pathways for autonomous exploration: Visitors should experience the museum in order as it was curated while also being able to explore independently.
2. Highlight important artifacts: We want to reduce the cognitive load for visitors and prominent artifacts so visitors are not overwhelmed when they enter the virtual museum.

With these defined requirements, we began the design process.

We held multiple brainstorming sessions to prioritize features and propose user flows in this virtual museum, but quickly found ourselves at a roadblock. For many weeks, we focused on nailing down our key features without bringing our ideas to visual sketches and designs. Not only did this make it challenging to scope our work, but it made it challenging for us to get feedback from other designers.

We divided responsibilities to each team member: Bella took charge of onboarding to the museum, Nuoran and Mina shared the interactive map and navigation, and Christina and Frannie worked together on the artifact pages. Additionally, each team member rotated responsibility in preparing for our biweekly workshops with designers who gave us feedback on our deliverables.

To bring the New York Times museum to a digital audience, we realized the first challenge was how to display artifacts online. How can users navigate the space and discover artifacts that interest them? With this question in mind, we began a deep dive into inspiration. We explored various museums’ digital websites and then drew from the company’s internal communication platform to ensure our design reflected the brand’s editorial integrity. The early explorations based on these inspirations show the spectrum of displaying artifacts — in a more editorial way or spatial way.

After discussion, we agreed that since the physical museum already exists, our goal was to recreate its sense of physical presence — helping users feel connected to the artifacts and experience the excitement of walking through the space. To achieve this digitally, Nuoran used photogrammetry to stitch together photos of the museum, creating a 3D environment where users can either follow a guided tour or explore freely.

This 3D foundation led us to adopt a spatial UI, which ensures the UI doesn’t distract from the artifacts; it becomes part of the physical space itself. Nuoran and Mina designed unified spatial UI components, such as the floor-plan map and the side panel for filtering and navigation. These navigation methods ensured a cohesive and accessible experience. Upon landing, users enter a panoramic 3D view that visualizes the museum, map and the side rail. This helps users understand where they’re located in the space and helps them efficiently navigate between highlighted artifacts.

Users can feel as if they’re walking through the museum by following visual cues such as locator pins embedded in the 3D space or directional arrows on artifact cards that guide exploration and highlight key artifacts.

To support this system, Bella designed an onboarding flow that introduces users to each component’s functionality before they begin their journey.

The second design focus was creating interactive experiences around the highlighted artifacts. With limited time, we only focused on ten key artifacts that best represent the museum while minimizing visitor decision fatigue. David provided the list of artifacts, and we began exploring ways to present them in the virtual space. Our initial approach used editorial-style layouts with static images and text, but we quickly realized this fell short. These artifacts are powerful touchstones of journalism’s history, and much of their presence is lost on a flat screen. Unlike physical museums where artifacts are behind glass and out of reach, virtual environments offer opportunities for more immersive interaction and deeply engaging storytelling.

Realizing that, we regrouped and had a workshop to brainstorm interactive features that could make key artifacts more vibrant and help tell their stories more effectively. Christina and Frannie tried out different animations to bring these ideas to life. For example, the “Man Walking on the Moon” artifact includes three newspaper front pages, each with a different headline — highlighting how journalists in the analog era adapted to rapidly unfolding events. We stacked these newspapers together so users can click on tags to switch between the headlines, mimicking the experience of flipping through real newspapers. For Barney Darnton’s story, rather than stacking all his related objects together, we chose to display them across a single page representing his role as a World War II correspondent in the Pacific Theater. Users can zoom in on individual items to explore his story in greater depth.

Instead of treating the ten artifacts as separated experiences, we organized them in chronological order under a unifying theme: “Integrity of the Times.” This structure forms the basis of the guided tour we offer users from the start. To make this tour’s storytelling more engaging, we introduced a scavenger hunt element throughout the experience. As users explore each artifact, they can collect clues that unlock the full story behind it which adds a layer of interactivity that makes the journey more educational and gamified.

As we wrapped up this project, our team understood the responsibility of designing a virtual museum and the rich history that comes with it. Throughout our design process, we used real imagery and content from the museum in our sketches, constantly questioning if our ideas were the best way to showcase a specific artifact. Is this interaction too playful for this artifact? Does this artifact need more visuals to tell the story? As product designers, we often think about design consistency at the system-level, but as designers working at The New York Times, it is this kind of editorial thinking that brings our work to life. We ended this project by presenting it to the Product Design team, The Times Brand and R&D teams. Our work went on to earn the 2025 iF Design Award and the 2025 IDEA Bronze Prize, highlighting the strength of our approach and storytelling as recognized by the broader design community.

These awards could not have been won without each member of the team and the strength of our collaboration in this project. Sharing project management responsibilities among this team of designers each step of the way helped with decision-making and keeping us prepared for deadlines. Lastly, for designers looking to hone their process, it can be daunting to jump into designs before everything is answered; however, sometimes visuals are necessary provocations that help a team align on an idea.

A special ‘thank you’ to former product designers Bella Rinne and Christina Su for being a part of this project and contributing to this article. We could not have done this project and won this award without you two!

Nuoran Chen is an associate product designer who has worked on The New York Times’ internal design system and currently designs mobile subscription experiences across different Times products.

Mina Chung is an associate product designer who has worked on the core news home page and currently designs current subscribers’ experience managing and upgrading their digital subscription across different NYT product surfaces.

Frannie Ello is an associate product designer who designs across visual formats and video experiences for the news web and app platforms.

Designing A Digital Times Museum For All was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

How real-time algorithms and causal ML transformed our digital subscription funnel from static paywalls to dynamic, millisecond decision-making

The New York Times became a subscription-first news and lifestyle service with the launch of its paywall in 2011. Since then, our subscription strategy has evolved substantially. Initially, users could access a limited number of free articles per month before they encountered the paywall. In 2019, we began personalizing this number using a Machine Learning (ML) model — The Dynamic Meter. In the past few years, we have replaced this model with real-time algorithms that decide, typically within milliseconds, whether to grant access. These algorithms are tailored to balance and optimize the tradeoff between several business Key Performance Indicators (KPIs), while also allowing us the flexibility to adjust for any business constraints. This article further details the motivation behind these algorithms and their design based upon principles from causal machine learning and multi-objective optimization.

Our subscription funnel

The New York Times has a tiered subscription funnel (Figure 1), consisting of unregistered, registered, and subscribed users. This funnel is designed to provide non-subscribers with limited access to our content, allowing them to discover our offerings. At other times, the content may be blocked by a digital “wall”. We have two types of walls — a registration wall that asks a user to register for a free account or log in, and a paywall that asks a user to subscribe. A large number of users are unregistered — they may be shown either a registration wall or a paywall. Once a user is in the registered state, they can be shown only a paywall.

Optimizing the subscription funnel

Optimizing who sees the registration wall or the paywall — and when — is a very relevant question for our business. While blocking access encourages users to subscribe, we do not want to block all our content because it prevents users from discovering the breadth and depth of our journalism.

Our holistic business strategy is made up of distinct KPIs for registered and unregistered users. For registered users, the business KPIs are two-fold and related to engagement and conversion, similar to our previous model.

For unregistered users, we are interested in subscriptions as well as registrations. Post registration, we can offer tailored experiences to provide users with a more rewarding experience. In addition, we consider bounce rate as a relevant KPI, for better user experience and since it is often relevant for SEO (Search Engine Optimization).

Due to such competing business KPIs, the decision to show a registration wall or a paywall is nuanced. Our goal is to build a smart system driven by ML that can decide when to show a paywall, a registration wall, or allow access. As an example, if a user is highly likely to only register but not subscribe, we might show them a registration wall. If they are not likely to subscribe or register, but might engage, the best decision is to allow access so that they get interested in our content.

Beyond sophisticated machine learning methods, the success of our approach is possible due to a close collaboration with our business leadership in clearly defining business KPIs. The data-driven culture at The Times enables this seamless collaboration. Our shared understanding of the fundamental tradeoffs in our business is crucial for building customized ML models with business constraints.

Machine learning for optimizing the subscription funnel

Machine learning has been a key component of how the Times’ paywall has been operating. We published an article on our previous Dynamic Meter model that used to personalize the number of free articles every registered user could access in a month. This model was implemented at the start of each month as a batch process to balance and maximize KPIs for subscription as well as engagement. While very successful for our strategy, this approach did not allow us to take real-time data, as well as article information, into account for making a paywalling decision.

To leverage the capabilities provided by our Machine Learning Platform, we revamped our algorithms to be real-time and took a holistic approach to apply machine learning in optimizing our subscription funnel. We also apply custom manual rules that can override algorithmic outcomes — for instance, designating certain content as open access for public service.

Modeling

From the perspective of prescriptive ML, different user types have different sets of “actions”. For unregistered users, the actions include (1) showing a registration wall, (2) showing a paywall, or (3) allowing access. For a registered user, the actions are (1) showing a paywall or (2) allowing access. Since the actions, as well as the number and nature of objectives, are different for our two user groups, we decided to build separate prescriptive models. Here, we describe the construction of the models holistically.

For each user type, we have a set of supervised predictors fᵢ(𝐗, a), where i represents a specific objective, such as propensity to subscribe or engage with an article. These predictors take in 𝐗, a vector of features available at inference time, and an action a, such as showing a paywall or allowing access.

The supervised predictors fᵢ(𝐗, a) are trained on data collected by a Randomized Control Trial (RCT), which is set up to randomly take an action with equal probability. The RCT is always on and the collected data from it ensures that the supervised predictors are causal, which means they are able to make predictions about counterfactual actions for any given request. This supervised learning setup — where the action a is used as a feature — is typically referred to as an S-Learner.

Since we have multiple objectives, we take a convex linear combination to construct a single objective using weight factors δᵢ such that ∑ ᵢ δᵢ = 1. These weight factors determine how much we value one objective over the others and are chosen based on certain business constraints that we detail below.

The models then take an action a* as per the policy in Equation 1.

a* is the action that maximizes the combined objectives.

Our modeling approach is schematically shown in Figure 2. Real-time features are ingested into supervised predictors, which predict the objectives for each action. These predictions are then multiplied by their weight factors and combined. Finally, the action is chosen that leads to the largest objective.

Respecting business constraints

The weight factors in our models crucially determine the rates at which registration walls and paywalls are shown, which are often determined by business constraints. For example, for the registered user model, a constraint might be a desired paywall rate in aggregate over a day. For the unregistered user model, we may have multiple constraints, such as those on registration wall rate and paywall rate.

We solve an optimization problem to figure out the weight factors by backtesting on a recent RCT dataset . By applying a trained model on this dataset, we can construct functions rₖ(δ₁, δ₂, …; χ) that return any statistics about the actions that the model takes on the dataset. Our problem is to match rₖ with a business-specified target tₖ by continuously changing the weight factors δᵢ(s). For example, t₁ might represent a paywall rate. Correspondingly, r₁ returns the paywall rate that the model would have achieved if it were applied to the historical RCT dataset.

We pose the above problem as yet another multi-objective optimization problem to minimize the losses in Equation 2, which are the squared errors between the achieved and the target rates.

The Pareto optimization problem for these losses is in Equation 3.

We may solve Equation 3 in its native multi-objective formulation, or combine the individual losses and treat it as a single objective problem. Our investigations showed that the loss landscape is rugged and steep, and we usually have 3–5 parameters to optimize. We tried several derivative-free optimization algorithms like Nelder-Mead simplex algorithm and Bayesian optimization, and found that U-NSGA-III worked the best.

After a sufficiently low loss is achieved, the model is deployed to production with the weight factors that are found. We run this optimization as frequently as every day to adapt to the changing traffic patterns. This approach also allows us the flexibility to respond to business requirements at a rapid pace. Any changes typically involve a simple edit to our configuration files to reflect the modified targets, and the tuning of weight factors is triggered right after.

To anticipate the impact of weight factor changes on outcomes such as subscription rate, we utilize Inverse Probability Weighting (IPW), specifically Hajèk estimation, similar to our previous work. This estimation also helps us inform our stakeholders about the expected impact of changing any business constraints, such as the paywall rate.

The training of the supervised predictors and the tuning of the weight factors operate in a control flow as shown in Figure 3. We only deploy a new model if the optimization loss for respecting business constraints is sufficiently low. Otherwise, our team gets alerted while the previously trained model continues to remain in production and serve traffic.

Performance measurement

To ensure that our productionalized models are performant, we compare their performance against our constantly running RCT. This helps us understand how much better the personalization of the model is as compared to a purely random policy.

Say we are operating with 3 KPIs. From the RCT data, we can develop a 3-dimensional plane with its extreme points being the average of data points where only a specific action was taken (registration wall, paywall, or allowed access). Any intermediate percentages of actions would correspond to a point on this plane. If the model is better than the randomized policy, then, in this space of objectives, the model point lies on the side of the RCT plane where the objectives are increasing. This is schematically shown in Figure 4. The dotted red lines help us define the performance improvement for each KPI over the random policy. As we vary the weight factors to adjust for business constraints, the model point moves around along the Pareto front.

In addition, we also conduct A/B tests to validate any improvements or changes we make to the models as compared to their old counterparts.

Conclusion

The New York Times registration wall and paywall are now driven by dynamic real-time algorithms. These algorithms are inherently causal and learn the efficacy of these walls based on a variety of factors using Randomized Control Trial (RCT) data. They are designed in a customized way to balance the tradeoff between multiple KPIs, while also helping provide an explicit control over business constraints. As a result, these models have achieved a boost in subscription rate and registration rate while maintaining the rates at which registration walls and paywalls are shown.

Rohit Supekar is a Lead Machine Learning Scientist at The New York Times, focusing on algorithmic targeting for access and messaging systems. He is passionate about developing and deploying ML solutions, drawing on his doctoral research in applied mathematics and scientific machine learning.

Many thanks to the Algorithmic Targeting, Machine Learning Platform, and Meter teams for their invaluable contributions to this collaborative effort.

Scaling subscriptions at The New York Times with real-time causal machine learning was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Joel Urena, Jenna Kim, Kenneth Ofosu, Shaka Clark, Raven Adaramola, Dylan Campbell

At least once a day, our players requested a Dark Mode feature for The New York Times Games, particularly those who enjoy playing at night. The bright screens were negatively impacting their satisfaction and hindering accessibility. Our players often wondered why this seemingly simple feature — just a switch from white to black — took so long to deliver.

People may think designing for Dark Mode is about inverting colors from white to black, one of the points players often cited when asking for Dark Mode. But there are numerous factors that make up little decisions when designing and implementing Dark Mode. Not only do we have to consider the color accessibility, we have to make intentional color decisions to preserve the brand of each game and The New York Times Games.

Primer: Design Systems & Feature Discovery

In the Spring of 2023, The New York Times Games was evolving quickly and we found ourselves with a user experience that was vastly different depending on which game you played and where you played it. We began to rectify this problem with the development of a Design Systems strategy. Like most other tech orgs, we hypothesized that if we began approaching product design and front-end development systematically, surely that would orient us toward a more consistent product experience, right? The answer: yes, eventually.

From a business and user impact perspective, the clearest Design Systems opportunity to prioritize first was Dark Mode. However, given the maturity of The New York Times Games digital product, the operational complexity we were tackling was massive. At the time, we had eight live games spread across three surfaces: The New York Times Games apps, the The New York Times news app and on the web. We also had ancillary experience for some of these games in the form of our companion articles (e.g: Spelling Bee Forum, Daily Crossword’s Wordplay column).

What we thought would be a simple color switch became an extensive exercise in paying down years of design debt. We discovered that each game had its own distinct user experience, with different fonts, color palettes, and component styles. Implementing a consistent Dark Mode across everything meant we couldn’t just add a new theme; we had to audit every screen and standardize the design from the ground up. This complexity forced us to make a critical decision: should we spend years paying down this debt, or could we find a way to scope Dark Mode in a way that delivered immediate value to our players? Our solution was to prioritize the user-facing experience over a complete systemic overhaul, giving us a path to delivery while continuing to chip away at the underlying issues.

We made the strategic choice to restrict Dark Mode to the Games app, which allowed us to narrow our technical and design focus to typical mobile and tablet display sizes. This decision also served to distinguish the Games app experience from our other platforms, aligning with our long-standing objective of establishing it as the premier destination for playing our games.

We then began unpacking the remaining scope. Product managers developed a visual to help cross-functional teams get a sense of what work needed to be done to get us to ship our first Dark Mode experience.

Once we understood the work ahead, our next step was to pitch the project to Games leadership for a spot on the roadmap. We publish our games using a hybrid strategy — our apps are a native shell built around our web-based games — which meant we needed to align our web and app teams. Given the competing priorities we faced on the Games App roadmap, this was a hard sell. We ultimately struck a balance between the two teams by aligning on a UX that leveraged our hybrid model to give users the maximum amount of flexibility when determining their theme preferences. Just like with Wordle, every game would have its own Dark Mode setting.

Designing the System

Now that it was approved, we wondered, “what are steps to actually get to Dark Mode on Games? How does engineering come into play as designers are exploring?” We decided to create a strategic chart outlining the high-level steps to the feature: Alignment, Foundation, Exploration, Integration. There were still unanswered questions, such as whether to launch all the Dark Mode features at once or to keep the scope narrow by launching it for each game individually. Creating this outline of the design strategy helped us gain clarity that we need to launch Dark Mode individually, reducing the complexity of the project by breaking things down, a common pitfall when working on design systems. This approach also mitigated risk as it allowed us to go through visual quality assurance for each game.

To ensure Alignment, we presented to the wider team on the importance of investing in design tokens. Because the wider team was so accustomed to using hex values for colors, it was crucial to change the way designers and engineers work with color.

For Foundation work, we conducted an audit of all game surfaces to identify any misalignments or discoveries. The New York Times Games product expanded significantly from its initial focus on The New York Times Crossword to include over eight distinct games. This growth, however, occurred without a unified design system. Consequently, each game surface utilized unique hex values, even when visual similarities were present.This investigation again proved the importance of design systems. To address this problem, the hex values used within our products were assembled into primitive/base tokens which will later be used to create semantic tokens which are more complex, intricate tokens used for more specific use cases. Also, thinking about how to help other designers discover the color they want efficiently in the future and understanding the complexity of our surfaces, we made a decision to establish three separate color component libraries with semantic libraries: Games Home Surfaces, Games Brand, Gameplay Color libraries.

• Games Home colors are general, foundational colors primarily used for utilitarian purposes for text, background, and stroke.
• Game Brand colors consist of design tokens used for brand purposes, such as Spelling Bee Yellow, Connections Purple, The Mini Blue, etc.
• Gameplay Color tokens are color tokens used for gameboards and game interactions.

When the time came for diving into Exploration, we approached each game as a separate project. We held each other accountable by reviewing each others’ design tokens. We discussed and consulted with one another on names for alignment — beginning the development of a ‘shared language’ for the system that we could leverage cross-functionally. We put strong emphasis on playability in gameplay through multiple rounds of prototypes to make sure the game experience of a light mode feels the same as the dark mode. For example, tiles within Connections have a very distinct beige color and do pass the accessibility test on a dark black background. However, it can feel overly bright on a dark background, so we adjusted the saturation of the beige to have a consistent game experience.

As we were getting close to handing off the designs to engineers, we also implemented a way for users to switch preferences within Settings since not all users prefer to have Dark Mode at night or they would prefer to have only certain games in Dark Mode. We wanted to deliver a flexible game experience where users can have a choice and be in control of what mode they want for each game.

Putting the Pieces Together

Our design system was engineered with the primary goal of ensuring consistency and scalability across both our internal and external products, beginning with our games. By providing a unified set of design tokens and reusable components, it bridges the gap between design and development, enabling our team to deliver a cohesive user experience. While still in development, the system is already proven to be a valuable tool for maintaining visual and functional consistency across our products. Collaboration has been at the heart of this effort, with several members of our engineering, product, and design functions working closely to align on requirements, standards and best practices.

The design system is built using React with CSS modules for styling, and it is documented in Storybook to ensure ease of use for all stakeholders. We leverage TypeScript for type safety, Vite as a build tool for fast and efficient development, and Playwright for visual regression testing to maintain quality. Design tokens, such as spacing, colors (used to implement dark mode) and typography are implemented using CSS variables to ensure scalability and flexibility. To enforce code quality and consistency, we use Stylelint and ESLint, integrated with Git hooks via tools like lint-staged and husky. These tools ensure that every contribution adheres to our standards, making the system reliable and maintainable. We then created a React Provider and utilized React hooks, which enabled us to roll out the new palette per game and per surface, to ensure we did not have to add the new feature to all surfaces and games at once. Overall our implementation strategy worked, allowing us to incrementally add color tokens without users noticing the difference. Many of the challenges we faced in the project related to our previous css and styles decisions in our code base.

One of the most rewarding aspects of building the design system has been the collaborative process. Weekly and ad-hoc meetings with engineers, engineering managers, product managers, and designers allowed us to align on requirements, share knowledge, and roadmap future improvements. This has fostered a shared sense of ownership and ensured that the system meets the needs of all users. Looking ahead, our vision is for the design system to become the single source of truth for all design-related information, encompassing both design tokens and shared components. By continuing to iterate and expand, we aim to make the system an indispensable resource for our team and a foundation for scalable, high-quality product development.

Takeaways

Building Dark Mode for The New York Times Games revealed two significant wins. First, what started as a focused effort to improve workflow through design systems evolved into a powerful tool for consistency and scalability across our products. We’re now extending this foundational work to other elements like fonts, spacing, buttons and shared patterns.

Second, the data clearly shows the positive impact of Dark Mode on our players. Users who engage with Dark Mode exhibit higher engagement, playing and completing more puzzles. Our strategic decision to prioritize the user-facing experience allowed us to deliver immediate value while systematically addressing underlying design debt.

Our efforts have delivered a feature that directly addresses our players’ long-standing desire for a more comfortable nighttime solving experience, allowing them to enjoy their favorite puzzles well into the evening.

Special thanks to:

Jennifer Scheerer & Jessica Gerson for design support and leadership. Michael Beach & Blake Spikestein for product support and leadership. William Frohn for motion design. Ian Hipschman, Ashby Thornwell, Lauren Yew, Goran Svorcan, Ihor Shamin, Katie Leavitt and the entire Games App Squad for their partnership. Emily Ngo for the Data Insights and Experimentation Support. Nick Ritenour, Coryn Brown and The New York Times Marketing team for their partnership

The New York Times Games’ Path to Dark Mode was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

Zoom image will be displayed

By Jayne Lee

In The New York Times Cooking, “cards,” or the containers that represent our content, are the first impression of our brand. They’re the window into our recipes and users rely on them to evaluate and choose what to cook.

NYT Cooking users are particularly attracted to our appetizing food photos. In every research session, participants get distracted by a delicious-looking dish while answering the moderator’s questions. People tend to browse with their stomachs first and then see if the recipe specifications meet their personal criteria (For example, do I have enough time to cook this?).

Zoom image will be displayed

A problem we encountered: our beautiful photography was muddied by the page’s gray background. The white card containers sitting on top of the gray page forced your eyes to focus on the container instead of the photo. The visual hierarchy was at odds with the preferred reading and browsing order of our audience: photo first, then recipe information.

Recipe information varies, so the card needs to afford that variability. Using containers forced us to keep our cards the same height, resulting in unnecessary whitespace. Since the cards were so tall, this limited the amount of recipes a user could see at once, increasing the time users scanned for recipes.

We needed to highlight our photography, modernize the format and streamline the recipe evaluation process.

While designing solutions, I realized that I had more questions than answers. Answering each question would have significantly bloated the project scope. Some examples were: What order of recipe information is most helpful when deciding between recipes? Are recipe bylines important? Which presentation of ratings is more effective?

Because the priority was to update the format rather than improve comprehension, and to avoid bloating the scope of the project, I started with design provocations over design specs. Because cards are systematic, they appear across many surfaces of our product. Provocations gave me confidence in my design decisions without having to design every card for every context.

Provocations are high-fidelity designs that may look and feel like design specs, but only gesture at solving a problem. The solutions are rough concepts that haven’t been validated with research and data.

Temporarily solving my questions via provocations gave me enough information to focus on my main objective: highlight the photography and modernize the format. Think of it like Ikea’s approach to building furniture: you nail pieces together just enough for the furniture to be stable temporarily. This way you can make adjustments at the end if pieces are misaligned, before final assembly. I knew that I could return to properly solve the questions later on.

Provocations can inspire more creative and unexpected solutions because they liberate me from the constraints momentarily. To make the provocations useful and not just beautiful rough concepts, I came up with principles that mapped to each provocation.

Codifying the ideas into principles allows the team, especially non-designers, to understand the intention and goals of the new ideas.

Here is an example:

Zoom image will be displayed

• Make our visuals do the heavy-lifting.
  Cooking is a sensorial experience. Highlighting our delicious photography and showing how easy our recipes are to make, will inspire users and help them make decisions faster.
• Our visuals speak the loudest.
  Ensure visuals are displayed at a large enough scale to entice users, while still giving users the ability to choose. Text is supplementary to the visual, not in competition.

These provocations served as a vision for how our cards should evolve over time. They allowed us to see where our cards were headed in the future and reverse-engineer small, conservative tweaks we could make to our product now. The expectation of the first iteration was not to move the needle, but to improve the overall product’s quality of life. We removed the old card containers, changed the page background color, grouped the recipe information together and rounded the corners on our photography.

Zoom image will be displayed

• The old card containers were a fixed height, regardless of the amount of content within it. This created unnecessary whitespace, made our cards taller, and required users to scroll more to see more recipes.
• Removing those meant some recipe information would look disconnected from the card. Grouping all of the information together allows users to scan and make decisions faster.
• We rounded the corners of our photography to signify the clickability affordance, now that the card container was no longer visually present. The rounded corners introduce an overall sense of warmth to the product and visually aligns with the other products within the NYT suite.

Making these small tweaks put the focus more on our photography, captivating more stomachs and eyes. Our early data shows users can now find and cook new recipes at a much faster rate. And our research participants can stay happily distracted.

Provocations helped us define a direction for how our cards would evolve over time. It showed us what could be shipped now and how to continue building toward the future. Shifting to provocations allowed us to be more innovative and set a confident, clear strategy of where to go next.

How we developed a dashboard tool created to help ease the workflow of managing puzzles for our Connections editor.

By Shafik Quoraishee and Wyna Liu

In the game Connections, every puzzle is a meticulously crafted challenge designed to captivate our audience and spark intellectual curiosity. Developing these puzzles can sometimes be a time consuming and intricate task. Each puzzle requires planning, beginning with conceptualizing fresh categories and plausible misleads, followed by testing the combinations for balance and solvability, and concluding with refinement and publication-ready formatting. The process requires both creativity and quality control.

Wyna Liu, the editor of Connections has the responsibility of constructing and reviewing multiple puzzles spanning various dates, ensuring that each board remains consistent, fresh and challenging to our puzzle solvers. This is a challenging endeavor where there isn’t much room for error. In order to address the challenge, we developed the Connections Reference Dashboard — an in company tool aimed at streamlining data management while providing the puzzle editor with an intuitive, aesthetically pleasing interface that enhances the daily workflow.

There were two considerations in developing this tool. Firstly was technical work in handling a dynamically changing payload of puzzle data. We wanted to create a rich and visually resonating interface that was easy to navigate and gave a bit of the feel of the Connections game itself.

Therefore, everything from the board results to the search interface was designed with these ergonomics in mind. We wanted to create a level of tactility to the tool which was reminiscent of and which reduced the number of manual steps needed to cross reference both categories and words in individual boards.

The primary functionality that Wyna was after was the ability to quickly identify words that have appeared in previous Connections boards, as well as their contexts — the categories they were members of, and the other categories that belonged to that board. Connections is a puzzle built around the novel ‘misleads’. A “mislead” in the game refers to the specific way words are presented or combined within a particular puzzle that might tempt a player to form an incorrect group. An example is the word “ARCHER”, which might mislead you to group it with “BOW”, “ARROW”, and “TARGET” (for “archery terms”), when its intended category is actually “TV SHOWS” with words like “LOST” and “FRASIER.” While words and categories can be repeated over time, the misleads ideally should not. With more than 700 puzzles, keeping track of what has run, on what date, and in what context, has been a vital part of the construction workflow.

Previously, there was no comprehensive search view to assist in checking this easily, either in Google Sheets, where the game is constructed, or in our internal admin tool, where the game is published. The dashboard tool provides all the necessary information at-a-glance, which has been an enormous time saver, especially since searching for words that have previously run is done multiple times while constructing each Connections board.

The backend, built with Flask, serves as the cornerstone of the dashboard. It is responsible for fetching puzzle data from external sources, caching it locally, and ensuring that the data is available in real time for the frontend. One of the key components of our backend is the data caching mechanism, which minimizes unnecessary network calls by checking if data for a given date is already available. If it is not, the system fetches the data from the NYT Connections API and caches it on disk. This not only speeds up subsequent requests but also provides redundancy against network issues. For example, consider the function below, which checks for cached data before fetching new data:

if resp.status_code == 200:
data = resp.json()
if data and isinstance(data.get("categories"), list):
puzzle_obj = {}
ALL_PUZZLES[date_str] = puzzle_obj
try:
with open(cache_filename, "w", encoding="utf-8") as f:
json.dump(puzzle_obj, f, ensure_ascii=False, indent=2)
except Exception as e:
print(f"[fetch_puzzle_and_cache] Write error {date_str}: {e}")
return puzzle_obj

In addition to this caching strategy, an auto-fetch mechanism has been implemented to preload upcoming puzzles. This ensures that even future, unpublished boards are available for planning and review. The auto-fetch function calculates a date range that includes several weeks into the future and then iterates through that range to fetch and cache each puzzle.

<script setup>
import { reactive, watch, onMounted } from 'vue';
import axios from 'axios';
const state = reactive({
includeUnpublished: true,
puzzles: [],
loading: false,
});
const fetchPuzzles = async () => {
state.loading = true;
try {
const { data } = await axios.get(‘api’', {
params: {
start: state.dateRange.start,
end: state.dateRange.end,
includeUnpublished: state.includeUnpublished,
},
});
state.puzzles = data.puzzleData;
} catch (error) {
console.error('Error fetching puzzles:', error);
}
state.loading = false;
};
onMounted(fetchPuzzles);
watch(
[() => state.dateRange, () => state.includeUnpublished],
fetchPuzzles
);
</script>

The above snippet is an example of how the front end communicates with the server component we set up, through the data api, and all updates occur seamlessly, and allow the addition of filtering parameters that allow for checking whether the results should contain unpublished boards.

The layout is created using Vue.js, employing a grid system to structure four vertical columns, each containing a heading and a list of related items. Each column is encapsulated as an individual Vue component or dynamically rendered from an array of category objects. Data management typically involves an array of objects, with each object containing a category label (e.g., “CONSUMED”, “ALSO”) and an associated list of terms.

Components such as <CategoryColumn> accept props like title and items, displaying each entry within styled containers. Conditional styling for special cells, such as highlighting “HORSE” in yellow, is managed through props or reactive state, signaling active or selected items.

UnoCSS is utilized for styling, ensuring uniformity and rapid development with concise, utility-first CSS classes. The grid layout leverages CSS Grid or Flexbox, providing clear borders, appropriate padding, and interactive hover states.

On the frontend, the choice of Vue.js significantly contributes to a clean, responsive design. The Vue-based interface is intuitive, adapting smoothly across multiple devices and screen sizes. Its reactive nature ensures immediate reflection of changes from the puzzle editor — such as adjustments in date ranges or toggling puzzle visibility — with no noticeable delay.

<template>
<div class="my-4 p-4 border rounded-lg bg-white shadow">
<input
v-model="searchTerm"
type="text"
placeholder="Search for a word..."
class="w-full p-2 border rounded focus:outline-none focus:ring focus:border-blue-300"
/>
<ul v-if="filteredWords.length" class="mt-4 space-y-2">
<li
v-for="word in filteredWords"
:key="word"
@click="selectWord(word)"
class="cursor-pointer p-2 bg-blue-100 hover:bg-blue-200 rounded"
>
{{ word }} (found in {{ getFrequency(word) }} puzzles)
</li>
</ul>
<div v-else class="mt-4 text-gray-500">No words match your search.</div>
</div>
</template>

Vue’s built-in directives like v-model, v-if, v-for, and @click dramatically simplify the process of building interactive components. In our live word search feature, these directives let us handle input binding, conditional rendering, list generation, and event handling — all in a few lines of clean, declarative markup. This approach reduces boilerplate and eliminates manual DOM manipulation, allowing the puzzle editor to interact with a responsive, real-time interface without the overhead of complex logic or state wiring.

An advancement in the puzzle review process brought about by this dashboard is the ability to look up the construction history of puzzles in terms of how often duplicate words occurred. What was once a time-consuming procedure has been streamlined into an efficient workflow. We are able to, through the backend, look up duplicates through the use of a word-frequency count. This allows one to observe how often puzzles with recurring words are present and the spacing in time of puzzles with duplicate words occur. Take for instance ‘BALL’, which occurred 23 times so far through the history of connections.

We also created a convenient method of looking up the frequency count of all words used in connections in descending order of their usage. This not only provides interesting construction information about connections, but could also lead to interesting statistical analysis, such as separation distance between words over time, least frequency used words, and most frequently used words.

The search itself is multi-facetted with several capabilities. The user can search through all the words in all the boards over the duration and lifetime of Connections through using a basic word search for exact matches, and for words contained as parts of other words through the use of regex search. Not only words on their own can be searched for, but categories and category titles as well, which present different snapshots into how words were previously used and in what combination with other words in specific groupings. This can help with organizing future puzzles and providing insights to new category possibilities.

I was excited to collaborate with Wyna on this opportunity as it was an amazing experience in working with editorial to make useful tools to come to life, and give me as a developer the feeling im building something positive that contributes to the mission of the Games Team bringing joy to our users by helping the creative process (in addition to working on the actual games).

In order to build the tool to be as useful to Wyna as possible, I had to make sure to really work the board to be rapidly prototypable so different combinations of optimal systems could be swapped in and out. Sometimes the optimal product and tool isn’t known until you go through several iterations.

And unsurprisingly to me, the process was not only an incredible learning experience, and an opportunity to work with Wyna, but also a ton of fun, and an opportunity to exercise my skills in tool building, which as a developer I feel compelled to build systems that provide joy to others.

Looking ahead, the architecture of the Connections Reference Dashboard is designed to be future-proof and scalable. The modular approach — where the backend and frontend operate as separate yet integrated components — allows for easy enhancements and the addition of new features over time. This flexibility means that as the needs of the puzzle editor evolve or as new challenges arise, the dashboard can be updated and expanded without disrupting the existing workflow

Happy puzzling!

By Heidi Erwin

Most of my work as a Senior Game Designer at The New York Times is oriented around the design and development of larger puzzle games, but one unexpected and delightful part of my job for the past two years has been writing variety riddles for The New York Times Gameplay newsletter. As someone who loves to see the process behind the scenes of the media I enjoy, I wanted to share the experience of creating Brain Ticklers.

What are Brain Ticklers?

“Brain Ticklers” is inherited from Will Shortz, and how his variety puzzles have run over the years. Will’s variety puzzles are typically word puzzles, and ask solvers to anagram phrases or build words from other words, for instance. In fact, I sometimes catch myself writing “Brian Tickler” by accident in my TODO list; I guess Brian’s a hidden fictional character associated with these puzzles who exists only in my mind.

Brain Ticklers are variety puzzles that could run in print (do not require a digital interactive format to be solved), whether that puzzle asks the solver to use deductive logic, wordplay, lateral thinking, visual analysis, or something else. We run one each week in The New York Times Gameplay newsletter, as well as in other parts of the print paper. Here’s one that ran shortly after my puzzles began running in the newsletter in January of 2023:

Move the following five letters into the grid below, such that you spell two words that form a phrase meaning “personal perspective.” Be creative!

Process Overview

The end-to-end process for creating a Brain Tickler generally involves the following:

• A source of inspiration
• A first draft
• Editing
• A final graphic.

More on each of those steps…

Inspiration

Inspiration could be anywhere! One of my favorite parts of writing these puzzles is that I feel encouraged to look at the world through different lenses when I’m out and about.

Inspiration could come from a sign on the street in the real world (that’s right gamers, I’m touching grass), a format restriction, a puzzle I play online — the world is full of puzzle potential.

Three contexts in which ideas for Brain Ticklers spawn for me are 1. Being out and about interacting with the world, 2. NYT Games team activities that prompt thinking about puzzles, and 3. other media (art, books, games, puzzles).

For example, here’s a Brain Tickler from 2024:

What item might be seen with each of these five shapes?

Solution: A bicycle. They’re all bike rack shapes!

This puzzle was inspired by the bike racks I was seeing on runs around Queens. I started photographing them for reference; you can tell that the puzzle graphic pulls pretty directly from these!

There are several opportunities to participate in new game ideation within The New York Times Games team. One of these is the game jams the team hosts, where people on the team put aside their other work for a couple days to ideate, prototype, and pitch. At one point, some work friends and I pitched a Venn diagram puzzle game during game jam, which did not turn into a full game, but did inspire this Brain Tickler (solution at end):

I’ve also been inspired by the formats of other cool puzzles out in the world. In March 2023, we ran 5 puzzles for a “March Matchsticks” puzzle series (like March Madness). These puzzles riff off of the classic matchstick puzzle format. Here are two from our month of matches (answers at the end of this blog post):

In the puzzle below, 18 matches spell out the word “sled.” Rotate one thing to ‘make friends.’

Editing

Every few weeks or so, when I have anywhere from three to eight new puzzles drafted, I hop on call with our Puzzle Editor Sam Ezersky, where he plays the puzzles in real time.

Watching someone else solve a puzzle in real time is helpful in shaping it further: Sometimes it becomes immediately obvious that the setup of a puzzle is unclear if I observe Sam heading down an unintended path. But on top of that playtester feedback, it’s awesome to witness Sam’s puzzle brain in action.

A recent example: I proposed a Brain Tickler where solvers were asked to untangle letter sequences to reveal four phrases of the format “____ in ____.” Sam took one look at “TJIUMSTE” and said, “Just in time.” This was followed by seeing “WLAIAITIDNYG,” immediately thinking out loud, “Lying in wait? No! Lady in waiting!” and then rapid-fire recognizing “CEDHITIOERF” and “LSOANW” as “Editor in chief” and “Son in law.” Sometimes I wonder if Sam solving a puzzle really says anything about whether the puzzle is fair to the average solver, but fortunately he definitely also has puzzle design sensibilities tailored to a general audience. Reviewing puzzles with Sam is a moment to test the accessibility of a puzzle so we can adjust the framing, presentation, or even concept, as needed.

Here’s another recent puzzle that became more elegant during the editing process.

What I initially presented to Sam:

[Spoiler Warning] The solution is that Marie likes Juliet, because Marie likes words that start with a shortened month name: aprons (Apr), mayonnaise (May), jungles (Jun), and Juliet (Jul).

He noticed that April, May, and June were all consecutive, and offered up “Romeo and Juliet” as an alternate fourth pair to continue the consecutive months using the “Jul” in “Juliet” for July. This is the kind of small adjustment that makes a puzzle that’s mostly solid feel tighter and more elegant.

For this type of puzzle, I include an easter egg where the character’s name hints at the quality of words they like. In this case, Marie also begins with a shortened month name: Mar.

Here are two more of this puzzle format for you to solve.

In terms of graphics, the Brain Tickler graphics are fairly simple, and typically I make them in Figma. For some puzzles, being precise with graphics matters more:

Below is the answer to the puzzle from the start of this piece:

And here are other answers to puzzles in this post:

Rotate the image 180 degrees. It now reads as “pals.”

Shipping fast often means shrinking scope — but accessibility deserves permanence, not compromise. At The New York Times, product managers embedded it into how our teams think, build, and deliver using the power of the three P’s: People, Process and Products.

By Erica Vendetti, Rachel Dixon, David Leininger and Matt Argomaniz

As Product Managers, we’re always trying to deepen how we meet the needs of our users and the future of our business. To deliver the features our users want, we’re always thinking about both who uses our products and how our products are used. For our Platform Product Managers, that can mean thinking about the needs of our colleagues — other Product Managers, Data Analysts, Program Managers, Engineers, and Designers. And for our User-Facing Product Managers, that can mean thinking about our readers. To make products accessible to all, accessibility should be a requirement, not a feature.

But when you build at the speed of news, teams must streamline features as they build. To launch a feature quickly, teams continuously refine the scope of feature development. And defining what’s a feature vs a requirement is a delicate dance for Agile teams. Shifting our teams’ perspective so accessibility is a requirement in all development work is a major undertaking. To become champions of accessibility in tech, we focused on the three P’s: People, Process and Products.

The New York Times aims to foster a culture that enables our mission, business and people to thrive. Finding both the time and opportunity to share knowledge is a top down priority, and the company has several staff-led groups that work to connect and uplift diverse perspectives. Three of these groups — which are invested in making technology more inclusive and accessible — came together to share knowledge and best practices:

• The Product function’s DEI Speaker Series Squad, which organizes discussions around building accessible products and being inclusive product managers,
• Women+ in Tech, which focuses on diversity in the technology, and
• The Accessibility Team, which empowers teams to build world leading digital products that are accessible from the start.

Sometimes, the key to success is simply connecting the right people. By leveraging these internal communities, we can build on existing efforts and share knowledge quickly in multiple spaces. This approach helps us tap into the time people were already dedicating for learning and a large engaged community of folks already interested in inclusive practices.

Making digital resources accessible means taking steps to prevent and eliminate obstacles that limit some users from fully using our digital products and tools. When it comes to technology, that means being able to support diverse abilities — including temporary, situational, and age related disabilities. And, of course, improving accessibility improves the user experience for everyone! If you’ve ever increased the brightness on your screen or used dark mode, you’re benefiting from accessibility tooling.

To share our knowledge about accessibility requirements, we hosted internal talks, certifications, and created self-guided syllabus learning for various levels. In our first talk, we shared four tips to help standardize accessibility across any product. These tips can easily be added as acceptance criteria from Product to Engineering to make accessible features.

In our second talk, we shared how to adapt these tips for Platform teams. This helped us build cohesion across the Product and Engineering functions. We were now effectively bridging gaps between user-interface accessibility and internal tooling accessibility.

In our final session, we organized a self-led certificate program that was open to all staff during Maker Week. Maker Week is an annual event where staff can take a break from their usual work and focus on creative self-directed projects. Our Product experts shared steps for how to perform an accessibility audit for web-based products. Hosting the training at the same time that folks were already building a lot of new tools and applications meant that people also had space to explore accessibility requirements. It was a win-win!

Product: Create Internal Tools to Standardize Accessible Features

Our technology teams have invested in creating an internal Accessibility Guide, a resource maintained by a team of internal specialists who monitor Web Content Accessibility Guidelines (WCAG) standards. This guide outlines how designers, engineers and product managers across all platforms can meet WCAG 2.2 AA standards.

By sharing this guide in all of our learning sessions, we’ve removed one of the biggest barriers — finding the right resources. And because accessibility needs can change with each project, the Product Function’s DEI Speaker Series Squad wanted to make sure there was a sustainable way for people to continue their journey. As folks learn how to use the accessibility audit tool, they can volunteer to become a team and department accessibility lead by joining the Accessibility Champions Program. This internal program has three levels to guide learning from general interest all the way to subject matter experts.

The Product function’s DEI Speaker Series Squad, Women+ In Tech, and the Accessibility Team are among many internal champions aiming to create spaces where curious learners and teams can make technology even more inclusive. This is just our starting point — we’re excited to keep building and sharing more together.

By Felipe Buenaño

As part of our efforts to maintain and improve the functionality and performance of The New York Times core website, we recently upgraded our React library from React 16 into React 18. One of the biggest challenges we faced in the process was transforming our codebase from the Enzyme test utility into the React Testing Library.

There are some major key differences in unit testing between these two libraries. Enzyme generates a string representation of the DOM tree. React objects are accessible through Enzyme methods. On the other hand, React Testing Library generates an actual representation of the DOM tree, where React elements are no longer reachable but are rendered as part of the DOM as a whole.

When making updates to our codebase’s testing library, we needed to ensure that site traffic was steady and there were no disruptions. With millions of visitors reaching out to our site daily, we had to be extra cautious. In order to successfully migrate our testing library, we implemented a systematic approach that updated test files individually in a consistent manner.

Our project had relied on Enzyme since its very beginning in May of 2016. However, given the push by the React platform to shift towards the React Testing Library, we needed to make this transition soon. In addition to migrating hundreds of files within our core codebase, our roadmap had other priorities with higher user impact.

So, how do you move hundreds of files from one testing philosophy to another?

There are three ways:

1. The bulldozer approach
2. The consensus approach
3. The piecemeal approach

The bulldozer approach is a top-down solution that updates all the files in one fell swoop. What that means is opening a single pull request that contains multiple changes across hundreds of files. This is a good solution when working in isolation. But because the code base it’s updated constantly we run the risk of having to resolve conflicts at high frequency. Also, this approach does not invite collaboration with others who may also be interested in helping deprecate Enzyme.

The consensus approach is a highly collaborative approach that requires ownership by engineers for each file. This is a great solution for new or highly visible projects.

This approach requires working on individual files and making strategic, impactful changes continuously until we reach the last corner of the codebase. The strategy here is to start with the most basic file, that is, the file that requires few changes. Then move on to the next most basic file. When these updates are made on a consistent basis i.e. when a change is made daily, momentum will start to build up to a point that making changes becomes part of the daily routine.

We decided to go with the piecemeal approach. It allows you to create momentum and consensus at the same time. This approach has an important requirement: work needs to start at the most basic file, which, in this context, is the file that requires only one or two lines of change, or the file with the least number of tests. Once this file is updated, the first step in the migration process begins. For instance when you need to assert the visibility of an element in React Testing Library vs Enzyme, you must figure out how to retrieve the element then test for visibility.

Here is an example of how that may look like:

Given a react component:

import React from 'react';
function Content() {
return <div>Important content goes here!</div>;
}
export default Content;

This is how unit tests for this code might look like using Enzyme:

import React from 'react';
import Adapter from 'enzyme-adapter-react-16';
import { shallow, configure } from 'enzyme';
import Content from '../Content';
configure({ adapter: new Adapter() });

describe('Content', () => {
test('renders a component', () => {
const content = shallow(<Content />);
const div = content.find('div');
expect(div.text()).toBe('Important content goes here!');
});
});

This is how that same code would look like for React Testing Library:

import React from 'react';
import { render } from '@testing-library/react';
import '@testing-library/jest-dom';
import Content from '../Content';
describe('Content', () => {
test('renders a component', () => {
const content = render(<Content />);
const div = content.container.querySelector('div');
expect(div.innerHTML).toBe('Important content goes here!');
});
});

Notice that aside from the obvious changes around the methods used to access the component, React Testing Library leverages the DOM to access the elements within the component.

Later, when working on a test that requires you to click an element, you would already know how to find an element using the previous test as a template. At this point, you would need to figure out now how to fire the event.

Given the following React component:

import React, { useState } from 'react';
function ButtonContent() {
const [count, setCount] = useState(0);
return (
<div>
<button id="btn" onClick={() => setCount(count + 1)}>
Button count {count}
</button>
</div>
);
}
export default ButtonContent;

This is how unit tests for this code might look like using Enzyme:

import React from 'react';
import Adapter from 'enzyme-adapter-react-16';
import { mount, configure } from 'enzyme';
import ButtonContent from '../ButtonContent';
configure({ adapter: new Adapter() });

 test('how element content changes by updating the state of the component', () => {
const buttonContent = mount(<ButtonContent />);
const button = buttonContent.find('#btn');
button.simulate('click', {
preventDefault() {},
});
expect(button.html()).toContain('Button count 1');
});
});

This is how that same code would look like for React Testing Library:

import React from 'react';
import { render } from '@testing-library/react';
import '@testing-library/jest-dom';
import ButtonContent from '../ButtonContent';
describe('ButtonContent', () => {
test('how element content changes by updating the state of the component', () => {
const buttonContent = render(<ButtonContent />);
const button = buttonContent.container.querySelector('#btn');
button.click();
expect(button.innerHTML).toBe('Button count 1');
});
});

Notice the similarities with the first test. When it comes to this test, you need to focus on clicking the element.

Later when more complexity is required by a test that evaluates visibility, interactivity and network interaction you would use the previous two tests as templates then figure out how to retrieve the response to evaluate it.

Given the following React component:

import React, { useState } from 'react';
function InteractiveContent() {
const [isVisible, setIsVisible] = useState(false);
function loadImage(event) {
const img = event.target.previousElementSibling;
img.src = 'https://placehold.co/400';
img.onload = () => {
setIsVisible(true);
};
}
return (
<div>
<img id="myImg" alt="load image" is-visible={isVisible.toString()} />
<button id="btn" onClick={loadImage}>
Click to show an image
</button>
</div>
);
}
export default InteractiveContent;

This is how unit tests for this code might look like using Enzyme:

import React from 'react';
import Adapter from 'enzyme-adapter-react-16';
import { mount, configure } from 'enzyme';
import InteractiveContent from '../InteractiveContent';
configure({ adapter: new Adapter() });
describe('InteractiveContent', () => {
test('should load an image after a button is clicked', () => {
const interactiveContent = mount(<InteractiveContent />);
const button = interactiveContent.find('#btn');
button.simulate('click', {
preventDefault() {},
});
const image = interactiveContent.find('img');
image.simulate('load');
expect(image.html()).toContain('https://placehold.co/400');
});
});

This is how that same code would look like for React Testing Library:

import React from 'react';
import { render, act } from '@testing-library/react';
import '@testing-library/jest-dom';
import InteractiveContent from '../InteractiveContent';
describe('InteractiveContent', () => {
test('should load an image after a button is clicked', () => {
act(() => {
const interactiveContent = render(<InteractiveContent />);
const button = interactiveContent.container.querySelector('#btn');
button.click();
const image = interactiveContent.container.querySelector('#myImg');
image.onload();
expect(image.src).toBe('https://placehold.co/400');
});
});
});

Although this test is more complex than the first two tests, all you need to do is focus on firing the onload event for the image since you’ve already shown how to access an element and click it from the previous tests.

By the time a couple of files have been updated the differences between Enzyme and React Testing Library become trivial to resolve. Also a clear pattern of work will be established. Every time you make changes to a single test, you will start to identify ways you have updated your code to accommodate the new library. Solving more complex test cases becomes easier once the process has undergone several interactions. As the files become more difficult to resolve, solutions can often be found by looking back at previous work.

Since the updates are small and well-documented in the previous tests, integrating people into this work requires little effort. Because of this ability to onboard new people, consensus is built naturally as people can join in and help contribute to completing the migration.

The piecemeal approach was time-consuming since it required us to work on a single file at the time but allowed us to migrate with minimal disruption. This approach also gave maintainable progress, which meant that no bug was introduced while migrating. Most importantly, though, this process allowed for collaboration. It gave people interested in joining this project a template they could follow, making the process more manageable and less stressful. It allowed us to maintain our development pace for other critical features.

Migrating from Enzyme to React Testing Library was a significant but impactful undertaking for our core web application. We could complete the migration without disrupting our roadmap by choosing a methodical, piecemeal approach. This experience taught us the value of strategic planning and balancing technical debt with ongoing development needs.

By Zhen Yang

Whether on the web or the app, the home page of The New York Times is a crucial gateway, setting the stage for readers’ experiences and guiding them to the most important news of the day. The Times publishes over 250 stories daily, far more than the 50 to 60 stories that can be featured on the home page at a given time. Traditionally, editors have manually selected and programmed which stories appear, when and where, multiple times daily. This manual process presents challenges:

• How can we provide readers a relevant, useful, and fresh experience each time they visit the home page?
• How can we make our editorial curation process more efficient and scalable?
• How do we maximize the reach of each story and expose more stories to our readers?

To address these challenges, The Times has been actively developing and testing editorially-driven algorithms to assist in curating home page content. These algorithms are editorially driven in that a human editor’s judgment or input is incorporated into every aspect of the algorithm — including deciding where on the home page the stories are placed, informing the rankings, and potentially influencing and overriding algorithmic outputs when necessary. From the get-go, we’ve designed algorithmic programming to elevate human curation, not to replace it.

The Times began using algorithms for content recommendations in 2011 but only recently started applying them to home page modules. For years, we only had one algorithmically-powered module, “Smarter Living”, on the home page, and later “Popular in The Times.” Both were positioned relatively low on the page. Three years ago, the formation of a cross-functional team — including newsroom editors, product managers, data scientists, data analysts, and engineers — brought the momentum needed to advance our responsible use of algorithms. Today, nearly half of the home page is programmed with assistance from algorithms that help promote news, features, and sub-brand content, such as The Athletic and Wirecutter (see Figure 1 for the position of algorithmic modules on the home page). Some of these modules, such as the Features Module located at the top right of the home page on the web version, are in highly visible locations. During major news moments, editors can also deploy algorithmic modules to display additional coverage to complement a main module of stories near the top of the page (the topmost news package of Figure 1 is an example of this in action).

Algorithmic programming comprises three steps (Figure 2): (1) Pooling: creating a pool of eligible stories for the specific module; (2) Ranking: sorting stories by a ranking mechanism; and (3) Finishing: applying editorial guardrails and business rules to ensure the final output of stories meets our standards. Editorial judgment is incorporated into all of these steps, in different ways.

To make an algorithmic recommendation, we first need a pool of articles eligible to appear in a given home page module. A pool can be either manually curated by editors or automatically generated via a query based on rules set by the newsroom.

A pool typically includes more stories than the number of slots available in the module, so we need a mechanism to rank them to determine which ones to show first and in what order. While there are various ways to rank stories, the algorithm we frequently use on the home page is a contextual bandit, a reinforcement learning method (see our previous blog post for more information). While in its simplest form, a bandit recommends the same set of engaging articles to all users; the “contextual” version uses additional reader information (e.g., reading history or geographical location) to adjust the recommendations and make the experience more relevant for each reader. For an example of the geo-personalized bandit, see our previous blog post.

To prioritize mission-driven and significant stories, we use several approaches to quantify editorial importance. One approach is having editors assign a rank to each story in the pool, with more recent and newsworthy stories generally considered more important. Another method infers a story’s importance based on its past promotion on the homepage, where stories that remain in prominent positions for longer are rated as more important. Regardless of the approach, editorial importance can be combined with the bandit to ensure that editorial judgment is incorporated into the ranking process, thus prioritizing stories deemed important by the newsroom.

Once we have a ranked list of stories, we make final adjustments based on predetermined rules developed with our newsroom partners before stories are shown to readers. One such intervention we developed is a Pinning function that allows editors to override the algorithm and pin important stories at the top. Other important examples are our “Already-Read” and “Already-Seen” filters, which deprioritize stories that the user has already read or seen a certain number of times (Figure 2). This finishing step ensures that editorial judgment shapes the final output and that we maintain a dynamic and fresh user experience.

The process begins with clearly defining editorial intentions, standards and boundaries as well as reader goals, and then designing algorithms appropriately. To illustrate the process, consider the above-mentioned Features Module (Figure 1): Content in this module is among the most widely-read on the home page. The goal of algorithmic programming for this module is to increase engagement by presenting readers with freshly published features and columns and also to ensure that the most relevant and engaging stories are subsequently displayed.

After several rounds of experimentation and extensive collaboration with editors, we realized that more features had to be built to achieve the intended reader experience and for the newsroom to be comfortable with integrating algorithms into their process for programming the home page. Together, we built and launched the following features, which are cornerstones in accelerating the use of algorithmic programming on the home page:

While pinning is an effective tool for increasing the exposure of important stories, it is also a rather blunt one: a pinned story is shown to all readers until unpinned by an editor. To meet a desire by home page editors for a “softer” and more dynamic solution, we developed an Exposure Boosting capability. While a “boosted” story also initially appears at the top of the module, it gradually moves down the slots over time — at a rate predetermined by editors — until it becomes subject to the algorithm’s bandit again (Figure 3: Exposure Boosting).

Another way to increase the exposure of our stories while ensuring that readers are presented with fresh content is by removing articles that the user has seen several times but has not clicked on — this assumes that the reader is not interested in the story displayed and the algorithm instead shows the next story on the list. When an article is shown to a user, whether they click on it or not, it’s called an impression. This rather rudimentary logic has its drawbacks: Frequent visitors might experience recommendations refreshing too often, causing a disorienting “slot machine” effect. They could also quickly exhaust all recommendations, resulting in a static module. At the same time, infrequent users, who don’t reach the impression limit, might see the same stories on distant subsequent visits, leading to a home page that would feel stale.

These potential issues were especially of concern for high-traffic modules like the Features Module. To address them, we developed a capability called Smart Refreshing. This feature creates a more stable experience for frequent visitors by only increasing the impression counter if a certain amount of time has passed since the last impression. Effectively, impressions occurring less than that amount of time are collapsed into a single impression (Figure 3: Smart Refreshing). For infrequent visitors, Smart Refreshing limits staleness by automatically refreshing recommendations after a set period since their first impression, even if the impression limit was not reached. Home page editors decide on the interval between impressions and the maximum duration a story remains after its initial view based on editorial judgment and A/B testing.

In response to concerns from editors that some stories risk not getting enough exposure under purely algorithmic programming, we developed Exposure Minimums. This capability gives the newsroom the reassurance that all stories (particularly less popular ones) receive a minimum number of impressions on the home page before the algorithm takes over their programming. This guarantee helps set editorial expectations for story exposure and has enabled the rollout of algorithms on prominent sections of the home page, such as the Features Module. Typically, higher minimum values increase story exposure but can interfere with algorithm optimization, reducing overall engagement. To find the right balance between exposure and engagement, the exposure minimum is determined in collaboration with our newsroom partners and through A/B testing.

One blocker we encountered while trying to scale algorithmic programming was the lack of visibility for editors regarding reader experience and story performance. One of the biggest challenges was feedback from the newsroom that editors and reporters couldn’t tell if their stories would appear in an algorithmic module on the home page. With the “Already-Read” filter in place, their stories, which they would have read, wouldn’t show up on the home page.

To address this, our product designer, engineers and data scientists partnered with home page editors to conceptualize and build a browser extension that allows editors to track all the algorithmic modules on the home page, preview different A/B testing variants, and review all the stories that have been selected and eligible for promotion for each module. Our engineers also built a tool that sends automated alerts to editors about changes in algorithmic programming, including new stories added to the pool and any headline or summary updates. Additionally, the Data Science team developed a dashboard to provide near-real-time analytics for stories that were algorithmically programmed.

After rigorously testing each of these new features and getting editors familiar with these tools and concepts, we permanently implemented algorithmic programming for the Features Module in the spring of 2024. This approach not only streamlined the editorial workflow (daily updates to the module were reduced by a third), it also gave stories that had a longer shelf life more time on the home page, lifting overall engagement. Our product colleagues were delighted that powering the Features Module with an algorithm also helped increase engagement with our sub-brands such as Wirecutter and Cooking.

The strong foundation we built by incorporating editorial thinking into our algorithms, coupled with the trust we cultivated, led to more demand from the newsroom for more algorithmic programming tools. Today, editors are also using a tailored set of algorithmic modules to power a secondary set of stories for a topic or news event. These modules are completely self-service for editors, and have been particularly useful during major news events, when the volume of coverage produced often exceeds the amount of real estate on the home page (See Figure 4 for a couple of examples).

Currently, algorithmic programming recommends stories within individual modules on the home page. Next, we want to explore and test reordering modules based on a mix of editorial importance, engagement, and personalization signals. We believe this approach can further improve a reader’s experience and amplify our journalism.

Zhen Yang is a Data Scientist on the Algorithmic Recommendations team at The New York Times. Outside of work, Zhen enjoys playing ping pong, rope adventure, and traveling.

By Ilya Gurevich

As software engineers at The New York Times, we place a high value on page performance, SEO, and keeping up to date with the latest technology. With those priorities in mind, the release of React 18 stood out to us as a significant and tangible leap forward in the ever-expanding world of web development. For our React-based sites, the upgrade promised a performance boost and access to exciting new features. Last winter, we set out to embrace the powers of React 18 on our flagship core news site. Along the way, we encountered some unique peculiarities — both in React and in our own site — that we had to learn to navigate through. In the end, we achieved big performance gains and unlocked a world of future improvements that we’re still exploring.

Before we dive into our process for upgrading, let’s take a look at a few of the major benefits and changes in React 18:

• Smoother Rendering with Concurrent Mode: React 18 introduces Concurrent Mode, a paradigm shift that allows for simultaneous rendering of updates and user interactions. This translates to smoother animations, less screen jank and cumulative layout shift, and a more responsive user experience.
• Automatic Batching and Transitions: To take full advantage of concurrency, React 18 automatically batches state updates within a single render cycle, optimizing performance. It does so by breaking up tasks in the main thread, which is a big shift from prior mechanics, where almost all tasks were synchronously executed. The introduction of new useTransition hooks also allows engineers to ensure that certain states will update without blocking the UI.
• Exciting New Features: React 18 paves the way for exciting functionalities like server-side rendering and streaming updates through react server components and selective hydration, opening doors to innovative UI patterns and faster initial renders.

The performance gains were particularly important to us because they promised significant improvements in our Interaction to Next Paint (INP) scores. INP is a measure of page responsiveness and is the newest Core Web Vital, a set of metrics that Google uses to rank websites in search results. SEO scores are vital for a news organization, and improving our INP scores had been a difficult challenge for us, making the React upgrade a high-priority (and high-stakes) initiative.

1. Removing Deprecated Dependencies

Before we could get started with the migration itself, we needed to remove a deprecated Enzyme testing library that was incompatible with React 18. To do that, we had to manually migrate all of our test files to the more up-to-date library, @testing-library/react. In terms of time commitment, this might have been the biggest piece of the entire project. Enzyme was used in hundreds of test files across our repository, and it required a significant manual effort and dozens of pull requests to fully replace it. We accomplished this effort over the course of several months with incremental pull requests in order to accommodate other product work and avoid developer fatigue. At the end of the effort, we definitely felt like experts in the @testing-library/reactAPI, and we were thankful to move on to the React 18 upgrade itself.

2. Foundation Setting

With the test file migration out of the way, we could begin work on integrating React 18. In order to accomplish this safely, we first started by upgrading all of our major dependencies, types and tests to conform to React 18, without implementing the latest features themselves. This involved simply upgrading everything from @types/react, react-test-renderer, react-dom, and @testing-library to the latest versions in our package.json files across our repository. Upgrading all major dependencies also involved refactoring some test and type definitions to conform to the latest versions as well.

3. Turning on the Engines

Once we felt confident in our package upgrades, we were ready to safely integrate the new functionality of React 18. To turn the features into reality, we needed to utilize the latest APIs: createRoot and hydrateRoot. We have several instances across multiple web servers where we’ve integrated React Hydration, with a set of shared UI components rendered between all of them, so it was important for us to enable React 18 functionality in as many places as we could. At first glance, it looks as simple as changing references from ReactDOM.hydrate to hydrateRoot. But was it really?

Unexpected Challenges

As developers, it’s easy to get overconfident when you hit the “deploy to production” button. Your end-to-end integration and unit tests are passing, you’ve covered QA across various surfaces and devices, and you’re moments away from getting that latest feature out the door. We all felt that way when we initially deployed the latest version of React to The New York Times website. Soon after our initial deployment of the new upgrades, we encountered a problem with some highly trafficked content, namely on a content-type we call “embedded interactives”.

At The New York Times, we use custom embedded interactives rendered server-side with dangerouslySetInnerHTML. These interactives have their own HTML, links, and scripts, running independently of the React tree. This allows editors and journalists to inject one-off, self-contained visual and interactive elements into our pages without having to alter or re-deploy core infrastructure. Embedded interactives are the key to some of our most impactful reporting, but they can also pose real challenges for developers.

A simplified example might look something like this (where script tags will modify the DOM as soon as the page has opened):

const embeddedInteractiveString = `
<div id="server-test">server</div>
<script>
document.addEventListener("DOMContentLoaded", () => {
const serverTestElement = document.getElementById("server-test");
serverTestElement.textContent = "client";
});
</script>
`;
return <div dangerouslySetInnerHTML={{ __html: embeddedInteractiveString }} />;

In this setup, the script modifies the “server-test” element’s content from “server” to “client” after page load. This works because browser-rendered scripts execute before React hydrates the DOM. It’s essentially a “black box,” where we trust the injected HTML and its scripts to behave as intended.

The Hydration Hurdle

Enter React 18, with its stricter hydration mismatch requirements. Under the new rules, any DOM modifications between the initial browser load and client-side hydration trigger a fallback to client-side rendering. In our example, even though the script tag modifies the “server-test” element before hydration, in a hydration mismatch, React will discard the server-rendered content and fall back to client-side rendering, essentially nullifying the script’s impact. In previous versions of React, even if there was a hydration mismatch, the React team opted to leave the version of the DOM in an invalid state as opposed to re-rendering entirely on the client, which is why we didn’t experience any issues in the past.

In practice, what does this mean? Well, when rendering components on the client using the dangerouslySetInnerHTML prop, any bit of HTML containing a <script> tag inside of it will not run due to browser security considerations. This means that any embedded interactive that is re-rendered on the client due to hydration mismatches using the dangerouslySetInnerHTML prop will essentially render as if the javascript had never been executed. In our example above, the text content will change from “server” to “client,” but on a hydration mismatch, it will re-render as “server.” This ended up making some of our embedded interactives look wildly different from the expected render.

Expected:

Actual:

So what do we do?

Given that React 18 was significantly more sensitive to hydration mismatches than React 16, we essentially had two choices in front of us. The first was to fix all potential hydration mismatches in our website. The second was to adapt embedded interactives to re-mount on the client as a fallback should a hydration mismatch occur. This left us in a bit of a dilemma. The New York Times has published millions of articles with hundreds of different components and tens of thousands of custom embedded interactives. Of course we wanted to fix all of our hydration mismatches, but how could we do so safely?

In the end, we decided to tackle both problems at the same time.

We know that script tags, when added via the innerHTML prop (or during a client-side re-render), will not run automatically because of browser security considerations. So how do we get around this? Script tags will only run when manually appended or replaced as a childNode to another element in the DOM. This means that in order to properly run script tags, we must first extract and remove them from the interactive HTML and then append them back into the right location in the embedded interactive HTML when the component re-renders.

// This function replaces script tags in generic html with empty placeholders.
// This allows us to replace the script tag reference in-place later on client-mount with the actual script.
export const addsPlaceholderScript = (scriptText, id, scriptCounter) => {
let replacementToken = '';
let hoistedText = scriptText;
replacementToken = `<script id="${id}-script-${scriptCounter}"></script>`;
hoistedText = hoistedText.replace('<script', `<script id="${id}-script-${scriptCounter}"`);
return {
replacementToken,
hoistedText,
};
};
// This function extracts and removes `<script>` tags from an interactive HTML string
// and returns an object containing:
// - `scriptsToRunOnClient`: An array of script texts to be run on client-mount.
// - `scriptlessHtml`: The modified HTML string with scripts removed with empty script references.
export const extractAndReplace = (html, id) => {
const SCRIPT_REGEX = /<script[\s\S]*?>[\s\S]*?<\/script>/gi;
let lastMatchAdjustment = 0;
let scriptlessHtml = html;
let match;
const scriptsToRunOnClient = [];
let scriptCounter = 0;
while ((match = SCRIPT_REGEX.exec(html))) {
const [matchText] = match;
if (matchText) {
let hoistedText = matchText;
let replacementToken = '';
({ hoistedText, replacementToken } = addsPlaceholderScript(hoistedText, id, scriptCounter));
scriptCounter += 1;
const start = match.index - lastMatchAdjustment;
const end = match.index + matchText.length - lastMatchAdjustment;
scriptlessHtml = `${scriptlessHtml.substring(
0,
start
)}${replacementToken}${scriptlessHtml.substring(end, scriptlessHtml.length)}`;
scriptsToRunOnClient.push(hoistedText);
lastMatchAdjustment += matchText.length - replacementToken.length;
}
}
return {
scriptsToRunOnClient,
scriptlessHtml,
};
};
// Run script on client
const runScript = (clonedScript) => {
const script = document.getElementById(document.getElementById(`${clonedScript.id}`))
script.parentNode.replaceChild(clonedScript, script);
}

You may be asking, why not keep scripts on the server and then re-run them on the client? One reason why this is not possible in some scenarios is that some script tags declare variables globally instead of within a function closure. If you were to pre-render those script tags on the server and then re-run them on the client, you would encounter errors due to redeclaration of global variables, which is not possible.

That initial solution fixed many of our embedded interactives. Unfortunately, not every interactive plays well with arbitrarily-ordered script execution. Here’s where we navigate some nuances:

Script Load Ordering

Some interactive scripts, when appended back to the embedded interactive HTML, must be loaded in the correct order. Previous script execution strategies automatically assumed that all <script> tags had already been declared and pre-rendered on the server. Now that we are stripping out script tags and re-mounting them on the client, some inherent logic based on these principles are going to break. Let’s walk through an example.

<script>
const 
results = document.getElementById("RESULTS_MANIFEST").innerHTML.ELECTION_RESULTS;
// do additional logic with results
</script>
<div>
Interactive DOM Content Goes here
</>div>
<script id="RESULTS_MANIFEST>{"ELECTION_RESULTS": ['result1', 'result2', ....]}</script>

In the scenario above, we have an initial script that searches for another script tag by ID and then utilizes some existing logic based on the innerHTML of the second script tag. In previous iterations, since script tags used to be pre-rendered on the server, there wouldn’t be any issue referencing a script tag by ID as the script tag would be available in the DOM by default.

For optimal interaction, script execution needs to follow a specific order when re-appended to the DOM. This involves:

1. Appending non-functional manifest scripts containing static data first.
2. Executing scripts with src attributes asynchronously next.
3. Finally, appending and executing scripts with vanilla JavaScript in their innerHTML.

This sequencing prevents scripts from referencing each other before they’re properly loaded.

// Parses the provided script tag, returning a priority for sorting.
// Priority 1: for JSON or other metadata content.
// Priority 2: for other vanilla JS or src contents
export const getPriority = template => {
let priority;
try {
JSON.parse(template.innerHTML);
priority = 1;
} catch (err) {
priority = 2;
}
return priority;
};
scripts.sort((a, b) => getPriority(a) - getPriority(b));

After integrating these very fine-tuned — almost surgical — manipulations of our embedded interactive code, we felt that we were able to safely release React 18 into the wild again. While we would never be able to extensively QA nearly 40,000 custom-created embedded interactives, we were able to rely on a few reusable templates that the graphics team often returns to. This let us validate specific behavior within our Svelte or Adobe Illustrator-based embedded interactives. In the long term, we’re committed to squashing our remaining hydration mismatches and achieving complete peace of mind. But in the short term, we were ready to push the “deploy” button again.

Once we released the new features (and spent an hour nervously monitoring internal alerts for any issues), we saw almost immediate performance improvements.

As you can see from this chart, INP scores in the p75 range dropped by roughly 30%!

Before the upgrade, one of our biggest challenges had been the frequent re-renders our news site went through as it loaded pages. That caused a poor user experience (and sub-par INP scores) when the user tried to interact with the still-loading page.

After the React 18 upgrade, our re-renders were cut essentially in half!

These two very visible and important improvements are the direct result of React 18’s automatic batching and concurrency features. This gave us a very clear and positive indication that we were moving in the right direction.

The integration of React 18 has already resulted in significant improvements for us, opening the door to a wealth of previously unavailable possibilities. We are now focused on exploring the potential benefits of new features such as startTransition and React Server Components. Our core intention is to continuously bring our INP scores down and improve overall functionality. However, we’re mindful of questions we still need to answer about these enhancements. For now, our primary commitment is ensuring the stable and reliable performance of the current React version we use.

Based on our results on the news site, we felt confident to pursue upgrades for some of our other sites, where we saw similar performance gains. We were able to get our INP scores out of the “poor” zone before Google’s March deadline, and saw no negative SEO results when it became part of their search algorithm. We like to think that our readers are enjoying the slightly-snappier experience. And our newsroom continues to put out powerful and interesting interactives every day, without having to give their rendering framework a second thought.

By Graciela Smet

My name is Graciela and I’m a senior at Stanford University, studying Product Design and Computer Science (Human-Computer Interaction). Last summer, I was a Product Design intern at The New York Times. The experience taught me a lot about what it’s like to work in a fast paced, growing company. I learned the importance of being surrounded by a culture of curious, collaborative, and smart people as well as the career value of gaining experience within the inner workings of a large organization.

When I started my internship search last year, my criteria was as simple as finding a role related to digital design. I was initially drawn to the Times, however, by what I realized were stronger criteria: I wanted my work to contribute to a cause that I deeply believed in. I’ve been a lifelong Times reader and am incredibly passionate about storytelling, journalism, and the importance of trustworthy spaces on the internet. Also, I’m an avid Spelling Bee and Wordle player (and now Connections, too), so that was a big plus.

I was assigned to work on the Publishing team, and at first I was honestly confused about what that meant. As a reader, I had only been exposed to The Times’s user-facing products, but being on the Publishing team gave me the privilege of seeing the operation from the inside: I learned so much about all the tools we use to publish hundreds of stories every day. I worked primarily on Ink, the in-house design system that’s used on all internal tools and platforms.

Over the summer, my projects included creating centralized Figma resources for new Ink users, writing component documentation, and designing a dashboard to track component health and usage across the teams that incorporate Ink. I had never worked with design systems before. My projects allowed me to experiment with and sharpen this skill set while giving me opportunities to grow more deeply as a designer.

My team was small, but included people with roles that complimented each other: six software engineers, four other product designers, an engineering manager, and a product director. As I learned later, this type of cross-functional team is set up intentionally to help support the tools and platforms that power our journalism. Outside my immediate team, I got to work closely with the other designers who were building tools for the newsroom. These designers immediately took me under their wing. They answered all of my questions, let me see into their design process, and even brought me along to their user interviews with top Times reporters and editors. I learned some fascinating things about how our newsroom operates, including one editor sharing tips on how he keeps a pulse on what’s happening all over the world. All so he can stay on top of his coverage and be ahead of everyone else.

Beyond the cool projects I worked on, my favorite part of the internship was absolutely the people and culture. The internship program included dedicated time to work with and get to know the other five product design interns, and I’m so lucky to have grown so close to such a talented group of like minded individuals. I also got to be a part of Maker Week, the Times’s annual hackathon event, spending time with designers who were working on their stretch projects — some of which have evolved to be part of team roadmaps this year. Throughout the summer, it really felt like anywhere I turned, I could find growth and inspiration.

Continuing my internship into the school year felt like a no brainer. It can be incredibly difficult to find an internship opportunity that’s rewarding in both career and personal growth while giving you a real taste of what it’s like to work in a big company. After accepting my extension offer, I moved to a different team, where I get to work on Moderator, the Times’s comment moderation tool, designing new features and workflows that help surface all the incredible insights that our readers offer by sharing their thoughts on our coverage. Some especially exciting projects included designing the editorial workflow to reintroduce Times Picks in the comments panel and designing more support for the Reporter Reply program, which set The Times apart as a more thoughtful place to discuss the news.

To designers earlier in their career, whether you want to work at The Times or not, I would absolutely recommend taking a step back to identify what exactly you might want from your career and internship experiences. This may be easier said than done, but the real magic happens when the experience aligns with your own priorities and when you’re able to grow in more ways than just career advancement. For me, the values and the culture of The Times were a perfect fit for what I wanted out of my summer, and I couldn’t recommend it more strongly.

By Kathy Yang

Experimentation is an important part of product development at The New York Times — it helps us measure the impact of our changes, ship features safely, and gain insight into user behavior. Experiments are conducted by many different teams working across a diverse portfolio of products and projects, from Wordle to The Times’s Audio app to backend migrations. In the early days, this meant that teams would often develop their own approach to experimentation to address their specific requirements.

Over the past year, we have been on a journey toward standardization. There are many benefits to centralizing experimentation — having a single source of truth, observability into how all experiments impact important metrics, and easier comprehension of results from other teams (not to mention the time saved for individuals who would otherwise maintain their own pipelines!). On the flip side, teams had originally developed their own methodologies and processes for a reason — they were tailored for their unique needs.

At The New York Times, most product experiments use our internal experimentation platform, ABRA, which is short for A/B Reporting and Allocation architecture. Here’s a look at an early version. A lot has changed since those days, and we want to share some of the things we’ve learned on our standardization journey. While we use other types of experimentation, such as contextual multi-armed bandits, this article is focused specifically around the ways we’ve improved our A/B testing processes.

As we all know, experiment results are only as trustworthy as the underlying metrics. A/B testing depends on the availability of high-quality metrics to analyze, and you cannot underestimate the importance of having a proper structure in place to manage these metrics.

We should anticipate that metric definitions are ever-evolving as our understanding of user behavior and business needs change. Thus, metric data and definitions should live outside of the experimentation platform. In ABRA, we have a clear delineation of responsibility for the data sources that feed ABRA’s Stats Engine (more on this in the next section): The platform team manages the exposure and assignment data and experiment configuration information, and analysts on other teams manage their metric data and definitions.

We set things up this way because it is crucial that the people who use these metrics on a day-to-day basis — for reporting, experimentation, modeling and other analytics — are the same ones who are responsible for creating, updating and deprecating them. Whether you are building or buying your Stats Engine, the individuals who use the experimentation metrics should be involved in their governance to help ensure they stay accurate and useful.

ABRA uses both enterprise metrics (think subscriber and revenue metrics), which are provided top-down from our central Enterprise Analytics team, as well as team metrics, which are provided by analysts on individual product teams. This dual-tiered structure ensures that enterprise metrics are defined consistently across all teams, while giving individual teams the freedom to add metrics specific to their domain.

While there are many more advanced techniques to explore, we found that we already began to feel the benefits of standardization once we had the basics down. This included conducting daily t-tests to estimate the difference in means between variants, and generating the point estimate, confidence interval, p-value, and the minimum detectable effect (MDE). At this point, teams that have their metrics hooked up to the platform have somewhere they can go to get automated results for their experiments, saving them from building their own.

Providing standardized reporting can be a catalyst for adopting best practices, but there is a significant cultural aspect to standardization as well. Consistent training, collaboration, and ongoing discussions with teams are essential to changing the way both technical and non-technical people think about experimentation. For example, if a team wasn’t consistently calculating confidence intervals or MDE before, the new results make it possible to quantify uncertainty in a way that was previously much more difficult. This needs to be paired with educating individuals on how to interpret these values, training analysts to conduct power analyses ahead of time, and changing stakeholders’ expectations to anticipate (and even insist on) seeing these values when presented with experiment results.

A lot has been written about ensuring data quality, including checking for sample ratio mismatch and capping skewed metrics. I want to expand on a couple more calculations that are particularly relevant at The Times:

Scaled impact

An increase of +1 page view per week per user can mean different things, depending on the population we’re experimenting on (e.g., all readers vs. international readers). Scaled impact is a useful way to present results when you want to compare experiments on different populations. It represents the estimated global increase we would have seen during the experiment if the treatment variant had been given to 100% of users, compared to if the control variant had been given to 100% of users.

To calculate scaled impact, you first start with the absolute difference in means between the treatment and control variants:

You then scale this up by the estimated number of users who were eligible for the experiment. This can be done by multiplying by the number of users exposed to the experiment n, and then dividing by the percent of users allocated to the experiment a (e.g., if you ran an experiment with a 20% control variant and a 20% treatment variant, a would be 40%):

Calculating impact on log-in rate

When a user is logged in, their activity is associated with their user ID. However, they can also have activity that is not tied to a user ID. For example, visiting the home page and playing Wordle can both be done without logging in or creating an account.

For these behaviors that can be done either while logged-out or logged-in, just looking at the impact on the metric itself may not tell the whole story. If a treatment variant increases (or decreases) users’ likelihood of logging in, they can end up with more (or less) of their activity associated with their user ID, even if their total activity didn’t change. When we see a statistically significant change in the log-in rate metric, this tells us that the results for these metrics may not be reliable, and we should not interpret them.

Even with a robust experimentation platform, there will always be situations that require ad hoc analyses. In these cases, providing lightweight tooling or methodology guidelines can help teams move forward with a common approach. Here are some of the practices we use at The New York Times outside of ABRA:

Analyzing ad hoc metrics

Sometimes, the metrics that analysts need to examine are not in ABRA when they run the experiment. To make it easier to calculate the impact on these metrics after the fact, we have a Python notebook tool that approximates the calculations in ABRA’s Stats Engine. While Analysts need to provide their metrics manually, they can be confident that they’re using the agreed upon analysis methodology.

Quantile metrics

T-tests aren’t suitable for quantile metrics, which are often needed for Core Web Vitals (e.g., 90th percentile First Input Delay). To address this, we have a separate pipeline that uses a bootstrapping technique to construct confidence intervals, which teams can use to check their experiments’ impact on Core Web Vitals. (Shoutout to The Times’s Ads Analytics team for building and scaling this resource!)

Non-inferiority testing

Not every experiment aims to improve metrics; sometimes, we just want to know that they aren’t degraded. In such cases, conducting a non-inferiority test, which tests whether the treatment is not worse than the control by at least some margin, can be more appropriate. Although ABRA was originally built for superiority testing, we have guidelines for adapting workflows to accommodate the non-inferiority approach. For example, even though the calculated p-value may not be relevant in this context, analysts can still rely on the confidence interval and assess whether the lower bound is greater than non-inferiority margin.

Eventually, some of these ad hoc tools might find their way onto the platform’s roadmap. Others might not. The key is to stay close to the practitioners on the ground, so that you develop a deep understanding of the common problems they face. Whether you are building or buying an experimentation platform, this will help you provide your analysts with the tooling and guidelines they need.

Experimentation is a core part of how we drive innovation here at the Times. There is still a lot of exploration to be done to standardize experimentation on all fronts, but we have learned some useful tips along the way. We’ve found establishing proper metric governance, prioritizing a short list of methodology essentials, and addressing some analysis needs off-platform to be important ingredients to success. We hope this sparks some ideas for your own experimentation journey!

By Amir Malek

The New York Times digital subscription offering is a complex service that has evolved significantly since we started selling digital subscriptions in 2011. Over time, we’ve added products beyond News, such as Games, Cooking, Wirecutter, and The Athletic, and made it easier for people to purchase both individual products and the combinations (including the All Access bundle that grants access to all of them) from different sources. Today a user can purchase the All Access bundle (now the primary subscription offering) from our mobile website, desktop website, News app, Games app, or Cooking app.

Moving to a bundle-first subscription strategy across all products made it more challenging to differentiate where subscriptions were originating from. We needed a deeper and more nuanced understanding of where and how users were subscribing, one that was compatible with the many variations of the user journey. The path to subscribing depends heavily on how a user gets to the site, the pages they interact with during their session, whether they’re logged in, and even more granular factors. To make sense of these paths, we had to align on several key aspects — how we defined attribution, how we organized our data, how we tracked our events, and how we mapped the data.

Defining an attribution model

Our objective was to avoid creating fragmented metrics to observe every known variation of this journey. Instead, we wanted to develop a unified model that aligned subscriptions with various onsite and offsite metadata, allowing us to segment subscription performance in a scalable manner and derive valuable metrics that are not dependent on the current user experience.

We started by collaborating with partners in product, strategy and marketing to consider several subscription attribution models — these are models that assess and assign credit to the touch-points responsible for subscription conversion in a user’s journey.

As illustrated in Figure 2, we found first click and last click attribution models to be too narrow in scope, only considering a single touchpoint in the user journey. Position-based and time decay models were too prescriptive and rigid to meet the needs of a rapidly developing user experience. In fact, we wanted to avoid assigning weights to any particular touchpoint so that the resulting data would be serviceable for teams across the company. As a result, we moved forward with a neutral, multi-touch attribution model that allows for custom or even algorithmic attribution logic to be derived later on, making it a flexible option that would scale more easily.

Identifying sources and building the logic

To build the model, we started with selecting a source for subscription data. Growth teams historically rely on front-end subscription events — data captured from user interactions on-site, such as purchasing or updating a subscription. The near real-time availability of this data paired with relevant metadata (like page and session-level identifiers and attributes) makes it a suitable choice for experimentation and analysis. Given that this subscription attribution model had to be exceedingly precise and aligned with enterprise-wide metrics, we had to rely on financial subscriber reporting data instead. Using this data source meant more reliable subscription records, but also limited metadata. We needed to invest additional effort to define and build subscription metadata that would expand on our existing reporting.

To define these dimensions, we collaborated with our cross-functional partners to surface the most important business questions (e.g., What product was a user interacting with before entering the subscription process? Did the user encounter a paywall during their session? How did the user get to the site that day?)

As shown in Figure 3, while no single question would encompass the entirety of the user journey, the ensemble helps paint a clearer picture of how users interact with The New York Times prior to subscribing.

While it was important to build the attribution model using a flexible architecture so that dimensions could be easily modified, a positive byproduct of this exercise was that we settled on several standardized dimensions (e.g. product, channel, platform, etc.) that were universally agreed upon by our stakeholders.

Tracking on-site events

A major component of building these standardized dimensions was not only creating reproducible code, but also ensuring that our onsite event tracking was robust. Prior to this initiative, the impression tracking across various paywalls wasn’t totally consistent —The New York Times Cooking recipe paywalls, article paywalls, and The New York Times Games paywalls all had slightly differing specifications. To build the dimension which would capture whether a user encountered a paywall in their journey would have involved gathering and querying for every distinct paywall impression event on site. So instead of hard coding the classification of these individual events, we first examined our existing impression tracking and worked with engineers to implement a single, unified tracking specification so that any paywall (now or in the future), regardless of where on the site it is rendered, would be more easily identifiable. We then repeated this exercise for other dimensions when necessary, which greatly contributed to the resilience and reliability of the model long-term.

Navigating between front-end and back-end data

The final step was to map our dimensions (such as product, channel and platform) to subscriber records. As with most data environments, this involved matching data across multiple systems such that the records would resolve cleanly and allow for new dimensions to be added later. We tested a variety of matches, and ultimately found that working collaboratively with the engineering team helped tremendously in being able to minimize redundant tracking and ultimately create the best possible join.

Applying the model

After months of scoping, designing and building, the first iteration of the subscription attribution model was complete. Teams from product, strategy, marketing and beyond were now able to leverage the mapped dimensions to more accurately report which areas of the business were driving subscriptions. And collectively, we began to uncover less-evident subscription trends with minimal effort.

From this data came more initiatives, namely pairing the attribution model with our subscriber retention data to learn more about subscriber churn — getting a better understanding of how a user’s subscription journey impacts their future propensity to churn. For the data teams in our subscription conversion group, layering audience metrics on top of subscription cohorts unlocked even more insight into highly-dimensional subscription conversion rates.

We also garnered learnings with respect to planning and executing data projects. One of the most important elements we uncovered was that the build process for data projects could mirror that of product development, where you deliver a minimal viable product and iterate on it in a phased manner. This approach allowed us to rigorously test each element of our pipeline without introducing an overwhelming number of dependencies and complexities. We also learned that the value in improving event tracking on site — as opposed to finding a logical workaround or including additional transformations in the pipeline — is evergreen. Collaborating with engineers to implement more consistent tracking made it so that the data worked for us, rather than us working for it.

Moving forward, this attribution model will serve not only as an effective tool to answer a wide range of business questions, but also as a foundation upon which other models and tools can be developed. Thanks to careful architecting, the model lends itself to future scalability, allowing for seamless expansion and improvement as more business questions arise.

This project is just one of many that showcases why working in the Data and Insights Group at The New York Times is so unique and impactful. As a team, we are not only able to experiment and find new ways to enhance our data capabilities, but we also get to collaborate with a variety of teams and stakeholders who collectively help us make improvements and deliver on business objectives that help reshape and support how we deliver our journalism and other products to the world.

By CJ Robinson

The acquisition of Wordle in 2022 propelled New York Times Games to new heights, with more people playing our games than ever before.

As the team of product data analysts who support Games stakeholders, we had to contend with tens of millions of players flowing into our pipeline each day. New York Times Games users generate data from actions like starting or finishing a game, getting the Spelling Bee pangram, or being tricked by the red herrings in Connections. Previously, we worked with raw data and constructed ad hoc tables to build reports. Without a proper data architecture to support the increased volume of data we were receiving, we could not quickly uncover valuable insights hidden inside. So, we had to reconsider our approach.

To increase analyst speed, democratize our data, and build a comprehensive dashboard suite, we spent months in 2022 building new tables with explicit aggregation layers, resulting in a 3x increase in dashboard usage and quicker data pulls as we began to launch more hit games like Connections. In sharing our process, we want to show the aspects of gameplay we measure and how rewarding it is to work with the data we wrangle each day. We also hope to help other product teams learn how to build a data environment that fits their needs.

Before this project, we commonly pulled insights directly from raw data. Our product team needs consistent information to track metrics like user retention and the impact of new features. The need was especially apparent as Wordle was first integrated into our portfolio and built into the Games app.

Since we did not have a concrete infrastructure, each request from a product manager or Games team member resulted in us having to write long, arduous, and inefficient queries to understand simple trends or usage across our site.

As a team, this led to significant work and the creation of new scheduled jobs to process data. At the time, these jobs were built without a long-term vision, so we contended with inconsistent practices, incomplete aggregations, and accidentally duplicated jobs. We felt unable to move forward meaningfully and were bogged down by what was asked of us. Creating a new data architecture was our solution.

What We Did

As we went through the process of creating a fresh data architecture, here’s what helped us the most:

• Starting with a data review to understand the needs: When we looked through our datasets, we saw opportunities to better curate and manage data flows.
• Creating consistency: We needed a common naming convention and language standards for data, field names, and tables to help ensure that our data pipeline would remain adaptable and evergreen when we add new games or features. We also thought about which levels of data were necessary to answer questions from our stakeholders and to ease our workflows as analysts. In the end, we came up with a proposal for an ideal end state with precise steps to structure the data available.
• Getting buy-in early: Since this was a significant undertaking and would take months to build, we needed to get buy-in from executive leadership. We spoke to leads across Games to ensure they understood the process and would help make this a priority. We also worked with other teams within our Data and Insights Group (DIG) to ensure that we would not create inconsistencies with other data and to solicit any technical or architectural feedback on our approach.
• Prioritizing flexibly: We dedicated 20% of our team’s bandwidth to the project, working in ticketed sprints, which allowed us to be flexible but accountable to deadlines and other requests. We made incremental progress in creating and modeling these tables for several months using GitHub for version control and code review. We also took that moment to improve our overall triaging of analytics requests using a Slack workflow. Instead of one-off DMs to analysts, stakeholders sent all messages to a central channel, which our team lead used to prioritize the work more effectively.
• Communicating expectations: We faced unexpected setbacks throughout the project, whether they were due to our routine responsibilities or data quality issues. We provided updates within our internal analytics team and the larger Games and DIG teams to remedy this. At more significant milestones, we sought feedback and managed expectations when making changes to the initial plan.

After five months, we had the first version of the new data architecture. It included four layers of aggregation that resulted in a clearer summary of relevant data for the team to utilize.

Now, for analyses that might have taken us many lines of code that we would repeat, with a single line, we could easily rewrite queries.

We also designed our underlying tables for each game to be consistent so that we could easily duplicate dashboards for new game launches and display our standard metrics. We published an access-controlled data dashboard for Connections on the launch date. The dashboard was especially useful during the Connections beta phase, where we worked with Games stakeholders to evaluate whether we would continue to invest in Connections based on early gameplay trends.

The dashboards had consistent layouts and metrics. They refreshed daily and loaded quickly, which meant that our product managers and designers could rely on this data to monitor performance each day.

Anomaly detection, or the process of automated alerts when something goes wrong with data, was another goal we wanted to work toward in this process. To do that, we used a third-party data observability tool that allowed us to create fine-tuned and custom monitors that alerted us to two data outages in the first few months after adoption.

Our team continually works to make improvements and changes to our data infrastructure — plus, with every new feature or game that the New York Times Games produces, we update our pipeline! We’re currently migrating this architecture off of proprietary in-house pipelining tech to a cloud-based solution for even more transparency and reliability; an exciting prospect that will help us streamline our data even further! Additionally, we utilize a backlog of issues and requests we find as we use the new data for future tasks.

Ultimately, our goal is to make the data generation, ingestion, processing, and reporting pipeline as effortless and evergreen as possible. Now that the heavy lifting of that effort is over, the team can dedicate more time to finding interesting, fun, and actionable insights to make your Games experience better each day — which is what we want to do most and what makes working at The Times so fulfilling. It was immensely rewarding to work on a project that was intellectually stimulating and also made a large impact on the lives of the internal users of this data (which also includes ourselves).

What would you do with this data at your fingertips?

By Jay Guillermo and Chen Wu

Getting to design the home and story pages for The New York Times web and app is an exciting and unique experience for designers. In July 2023 at Figma’s annual Config Conference in San Francisco, we were honored to have the opportunity to share what’s that like and how we design at the speed of news.

The conference attracted about 8,000 people, with about 500 attending our session in person and more virtually. We discussed the process of designing the home page and live story pages while the news is continuously evolving, and how we meet the needs of the newsroom, our business and our readers. Similar to creating the home and story pages, presenting at the conference was a huge team effort. We worked closely with our colleagues in Design, Product, External Communications, Legal at The Times and our partners at Figma to make it happen, and we are grateful for their help and support.

Here’s a recap of our key takeaways:

Build a Flexible Foundation

When news is your product, it’s inherently complex and constantly changing — you must balance the needs of the newsroom and readers, as well as those of The Times’s business. The products we design and build have to accommodate a wide range of topics and events, from planned, such as Election Days, to unplanned/breaking news, such as natural disasters and wars. Because of the constantly changing world around us, we need to build a stable and flexible foundation for presenting and programming the “News of the Day” — a base that can be used for all instances and events.

By building a solid foundation, we can iterate on this system to meet our ever-changing news needs, as well as new media sizes and content types, such as vertical videos and photographs.

Be Agile

When news is your core product, everything is a work in progress and always changing. Being agile and open-minded goes a long way.

A lot of our work requires evolving the shape of the product and the journalism at the same time, which means that we are not only doing UX and UI design, we are also doing content design, or helping to shape how stories are presented to readers. This means that we need to be flexible and iterative, understand how the newsroom works, how reporters and editors work, what their challenges are, what opportunities they see, and what they are preparing for. We need to understand the basics of various news events and work effectively alongside our reporters and editors as they shape their coverage.

Build Trust With Your Partners

One of the key learnings from our presentation is making sure to build trust with your cross-functional partners and stakeholders. They should be your soundboard and your rock.

We are able to roll out features and adjust designs quickly by having constant and open communication with our engineers. Our engineers are incredibly open minded and flexible when we have to shift priorities due to news events and urgent tasks.

We are also able to roll out features and get buy-in quickly by maintaining a constant and open communication channel with our stakeholders — including newsroom editors, reporters and visual editors. We prioritize making sure we are aligned on our goals and the directions and problems we want to solve, and events we need to be preparing for. Over time, because of the trust we’ve built, we can align quickly and move as fast as the news demands.

We share the mindset of “our work, our responsibility” vs. “my work, my responsibility.” When we all take ownership of a feature and product, we work more closely together, make decisions together, trust each other more, and have more courage to try new ideas.

Don’t Be Afraid to Experiment

It’s important to test, iterate and repeat.

Live, a story format for dynamic coverage of the major news of the day, is an area where we often roll out features quickly as tests and, later, we fully adopt and improve them. Sometimes it takes less than two weeks to get an idea from design sketches, newsroom alignment, engineering, and QA into production.

We seek to keep in mind the long-term vision for how a feature can evolve and be used for other news events, so it’s never a one-time use.

And, of course, not everything we test proves to be working as expected, and it’s OK! The beauty of working at a place like The Times is that we get the space to experiment and try new things. We’re encouraged to test the art of the possible and learn from the insights we gather along the way. Sometimes the things that don’t work are what help us the most in making better products. A spirit that every designer at The Times is encouraged to embody.

Hear the Whole Story

To learn more and get the full experience, watch Jay and Chen’s talk at “Design at the Speed of News.”

By CJ Robinson

The acquisition of Wordle in 2022 propelled New York Times Games to new heights, with more people playing our games than ever before.

As the team of product data analysts who support Games stakeholders, we had to contend with tens of millions of players flowing into our pipeline each day. New York Times Games users generate data from actions like starting or finishing a game, getting the Spelling Bee pangram, or being tricked by the red herrings in Connections. Previously, we worked with raw data and constructed ad hoc tables to build reports. Without a proper data architecture to support the increased volume of data we were receiving, we could not quickly uncover valuable insights hidden inside. So, we had to reconsider our approach.

To increase analyst speed, democratize our data, and build a comprehensive dashboard suite, we spent months in 2022 building new tables with explicit aggregation layers, resulting in a 3x increase in dashboard usage and quicker data pulls as we began to launch more hit games like Connections. In sharing our process, we want to show the aspects of gameplay we measure and how rewarding it is to work with the data we wrangle each day. We also hope to help other product teams learn how to build a data environment that fits their needs.

Before this project, we commonly pulled insights directly from raw data. Our product team needs consistent information to track metrics like user retention and the impact of new features. The need was especially apparent as Wordle was first integrated into our portfolio and built into the Games app.

Since we did not have a concrete infrastructure, each request from a product manager or Games team member resulted in us having to write long, arduous, and inefficient queries to understand simple trends or usage across our site.

As a team, this led to significant work and the creation of new scheduled jobs to process data. At the time, these jobs were built without a long-term vision, so we contended with inconsistent practices, incomplete aggregations, and accidentally duplicated jobs. We felt unable to move forward meaningfully and were bogged down by what was asked of us. Creating a new data architecture was our solution.

As we went through the process of creating a fresh data architecture, here’s what helped us the most:

• Starting with a data review to understand the needs: When we looked through our datasets, we saw opportunities to better curate and manage data flows.
• Creating consistency: We needed a common naming convention and language standards for data, field names, and tables to help ensure that our data pipeline would remain adaptable and evergreen when we add new games or features. We also thought about which levels of data were necessary to answer questions from our stakeholders and to ease our workflows as analysts. In the end, we came up with a proposal for an ideal end state with precise steps to structure the data available.
• Getting buy-in early: Since this was a significant undertaking and would take months to build, we needed to get buy-in from executive leadership. We spoke to leads across Games to ensure they understood the process and would help make this a priority. We also worked with other teams within our Data and Insights Group (DIG) to ensure that we would not create inconsistencies with other data and to solicit any technical or architectural feedback on our approach.
• Prioritizing flexibly: We dedicated 20% of our team’s bandwidth to the project, working in ticketed sprints, which allowed us to be flexible but accountable to deadlines and other requests. We made incremental progress in creating and modeling these tables for several months using GitHub for version control and code review. We also took that moment to improve our overall triaging of analytics requests using a Slack workflow. Instead of one-off DMs to analysts, stakeholders sent all messages to a central channel, which our team lead used to prioritize the work more effectively.
• Communicating expectations: We faced unexpected setbacks throughout the project, whether they were due to our routine responsibilities or data quality issues. We provided updates within our internal analytics team and the larger Games and DIG teams to remedy this. At more significant milestones, we sought feedback and managed expectations when making changes to the initial plan.

After five months, we had the first version of the new data architecture. It included four layers of aggregation that resulted in a clearer summary of relevant data for the team to utilize.

Now, for analyses that might have taken us many lines of code that we would repeat, with a single line, we could easily rewrite queries.

We also designed our underlying tables for each game to be consistent so that we could easily duplicate dashboards for new game launches and display our standard metrics. We published an access-controlled data dashboard for Connections on the launch date. The dashboard was especially useful during the Connections beta phase, where we worked with Games stakeholders to evaluate whether we would continue to invest in Connections based on early gameplay trends.

The dashboards had consistent layouts and metrics. They refreshed daily and loaded quickly, which meant that our product managers and designers could rely on this data to monitor performance each day.

Anomaly detection, or the process of automated alerts when something goes wrong with data, was another goal we wanted to work toward in this process. To do that, we used a third-party data observability tool that allowed us to create fine-tuned and custom monitors that alerted us to two data outages in the first few months after adoption.

Our team continually works to make improvements and changes to our data infrastructure — plus, with every new feature or game that the New York Times Games produces, we update our pipeline! We’re currently migrating this architecture off of proprietary in-house pipelining tech to a cloud-based solution for even more transparency and reliability; an exciting prospect that will help us streamline our data even further! Additionally, we utilize a backlog of issues and requests we find as we use the new data for future tasks.

Ultimately, our goal is to make the data generation, ingestion, processing, and reporting pipeline as effortless and evergreen as possible. Now that the heavy lifting of that effort is over, the team can dedicate more time to finding interesting, fun, and actionable insights to make your Games experience better each day — which is what we want to do most and what makes working at The Times so fulfilling. It was immensely rewarding to work on a project that was intellectually stimulating and also made a large impact on the lives of the internal users of this data (which also includes ourselves).

What would you do with this data at your fingertips?

By Shafik Quoraishee

Introduction

As part of MakerWeek 2023, the New York Times annual hackathon, iOS and Android Mobile engineers explored the ability to write in the New York Times Crosswords app on each respective platform.

As an Android engineer who participated in the experiment, I’m excited to share my platform specific experience implementing On Device ML onto the Android Crosswords.

Note: This exploration is for a future feature, which hasn’t been released yet.

Initial Setup and Requirements

The New York Times Crossword has a custom software keyboard built into the app. When a user types their letter on the keyboard, it appears on that square.

To allow for handwriting, the first thing we needed to do was to ensure that the user could actually enter text manually, via a stylus or their finger. We took each crossword square on both the Mini and the Daily, and we transformed it into a custom component we called ‘SketchBox’. This component captures each stroke made by the user’s finger or stylus as they write on the screen, and is specially designed to listen for touch and drag events to display drawn letter strokes.

After our Sketchbox captured the resultant letter pixels from the canvas, we could then send the data to the machine learning algorithm of our choice.

Pencil Timing

Before we get to the actual handwriting detection, we need to address a subtle but important point.

As a user writes on the Sketchbox, they typically lift their finger or the stylus off the canvas, especially to complete letters like K, A, H, etc. This means we needed to determine when exactly a user was done writing, between each stroke. For example, if they enter the stem of the “K,” if we try to detect what letter this may be as soon as they lift their writing utensil off the canvas, it might be interpreted as an “I”.

So how long do we wait between strokes?

For our initial implementation, we introduced the concept of a mutex-like input locking system. Between each stroke we experimented with values around 500 to 1000 milliseconds depending on certain conditions. We didn’t want to wait too long before unlocking the stylus, otherwise the user input experience would seem degraded and choppy.

This is one of the many complexities we had to consider as we were designing the writing mechanic; and something that will be open for refinement in the future.

Data Preparation, Conditioning and Normalization

Before exploring conversion of images into text, we had to consider our input: letters from various devices with different screen sizes and resolutions.

An essential pre-processing step included getting the simplest form of the data that the algorithm needs for accurate learning. In the case of image data, this means getting rid of non-essential noise, and “frivolous geometry”. We downscaled and binarized the letter data, and then converted 128x128 raw input letters to much smaller, efficient and simplified 28x28 images.

Once we did that, we could finally begin to discuss how we translate the rasterized canvas image data to an actual character that our crossword app understands.

Handwriting recognition is a classic machine learning challenge within Optical Character Recognition (OCR). It has seen substantial advancements over the years, notably with Dr. Yann LeCun’s LeNet-5 architecture in 1998, which significantly improved digit recognition on the Modified National Institute of Standards and Technology (MNIST) dataset. The MNIST dataset contains thousands of variations of the digits 1–9, and is the de facto standard database for digit recognition.

The system we were trying to build looks like this high-level architecture diagram:

The machine learning algorithm we chose was intended to provide the best separation of character data into recognition clusters as idealized below. This way it would be easy for our system to determine if the user wanted, for example, to enter an ‘A’ vs a ‘C’. We explored several other options that I won’t go into detail about here, before settling on the use of a Deep Convolutional Neural Network architecture, which proved up to the task.

Building A Deep Convolutional Network

The Deep-CNN ((Convolutional Neural Network) is the cornerstone of any legitimate modern day image based machine learning system. It is a special kind of neural network that examines sections of image data, and using its learning mechanic, intelligently finds important features that help identify and classify images.

In the example presented above, the neural network takes in an input image of a bird, and detects structures that correspond to different qualitative elements of the bird. It can detect portions of the image that correspond to feathers, eyes, edges of the beak, internal beak geometry, and a whole host of traits that a person would have difficulty designing an algorithm from scratch to do.

Using the most important features, it can determine if it’s looking at a bird, and what kind of bird it is. We can even determine things like what direction the bird is flying in or looking at–provided the network is trained properly. We apply the same principle to letters that a user inputs into our crossword app.

The basic CNN is a combination of the following elemental layers. By intelligently mixing and matching these structures with the proper parameters, there’s very little imagewise that we can’t detect:

• Convolutional layers allow the model to extract features automatically
• Max pooling layers narrow down the features obtained from the Conv Layers
• ReLU layers introduce non-linearity and allow for complex pattern discrimination
• Dropout layers mitigate overfitting of the network to the training data

TensorFlow Lite and Ingesting our Model

In addition to building the model, we also had to find a way to deliver it to our device. We chose to use Tensorflow Lite, a mobile framework used to install Python compiled ML models into an Android or iOS device.

Once we were satisfied with our model, we compiled it to a .tflite file and baked it into our application, then built a shunt for listening to letter writing events coming from the crossword squares. We iterated several times through different models and different configurations until we were satisfied with the final result, which turned into a trained file of only 100K or so–perfect for a mobile application, where space considerations are important.

Digit Recognition

Before approaching the problem of full-blown letter recognition, we decided to start simple and tackle the well studied and foundational problem of numerical digit recognition. The core section of code below provides the basic CNN setup we implemented:

First Digit Based CNN Model Failure: What Went wrong?

Despite abundant training data obtained from the MNIST, our recognition results were poor.

We ascertained the training data was “too perfect”. All the digits were only minor variations of each other, and all of them were mostly in the center of the box.

This is not the way people enter data into a crossword square. People have different handwriting styles and ways of tilting and placing the characters in the squares off center.

To solve this, we needed to employ a well known machine learning technique called Data Augmentation.

Data augmentation automatically generates off-center and distorted versions of our training data, bypassing the need for manual adjustments and skewing. This allows for many variations of our initial data set, including significantly off-centered versions of our characters.

By applying data augmentation techniques, we expanded our dataset from thousands to over 1 million samples with off-center minor shifts, rotations, and scaling — also called affine transformations.

Data Augmented Digit Model Success against Digits

As you can see, the digit recognition has improved significantly on the real crossword — a milestone in our work!

Moving to Full Blown Letter Recognition

Now that we solved digits, the next step was to solve letters. This includes lowercase and capital letters and all of their variations. Instead of just ten digits, we are now dealing with 26 lowercase, 26 capital letters and ten digits, which equates to 62 characters.

We can use the EMNIST dataset (Dataset source: http://arxiv.org/abs/1702.05373), an expanded version of the MNIST set, that includes both letters and digits and even punctuation, to help train the model better.

However, even with the enhanced dataset, the digit specific model isn’t sufficient for our needs. This is intuitively not surprising, because while the digit recognition model was powerful, it is surely not “intelligent” enough for the greatly expanded variation in character structure that is introduced by looking at letters.

Hyper Powering The Model through Parameter Optimization

To increase the power of our model, we added much more depth to our network in the form of several additional layers. We also employed Stratified K-Fold cross-validation to diversify our training by using randomized subsets of our augmented training/validation data.

To ensure that our layers were optimally designed, we employed a randomized parametric search along with enhanced statistical testing, which allowed us to find the optimal hyperparameters for our model. This proved to be an enhancement from our previous strategy of guessing the right parameters.

In our testing phase, we attained an average validation accuracy of ~91% on the Augmented EMNIST dataset. This gave us faith that our model would work.

FINALLY: SUCCESS!

After a long journey of exploring the landscape of ML model building, we finally arrived at a working crossword model–which was exciting. Even though we attained stellar results, there is still much work to do for the complete crossword experience, including dealing with partial letters, and letters that are spaced at irregular intervals.

Conclusion

Implementing handwriting recognition on the Android crossword app was an exciting adventure, even in an experimental context. Aside from handwriting, there’s also the potential for interactive features like “Scribble-to-Erase” detection and the possibility of in-app self-training mechanisms, and a whole host of other doors that On-Device ML in the Games App can open.

Having the opportunity to experiment with new techniques and amplifications to our existing products is a core part of why working at the Times as an engineer is so unique and so worth it. We hope one day to turn this into a feature that amplifies the Games experience for our current users and attracts new subscribers.

Shafik Quoraishee is a Senior Android Engineer on the Games Team at The New York Times. He is an avid machine learning/A.I. Enthusiast. Outside of work, he enjoys playing guitar, writing, and running quirky experiments.

Experimenting with Handwriting Recognition for The New York Times Crossword was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Rimar Villasenor, Grace LaRosa, Bon Champion, Daman Chatha and Lavi Tang

It’s not often that you get to build a new product from the ground up. At The New York Times, cross-disciplinary collaboration is at the heart of many of the company’s digital innovations, and design played a pivotal role in imagining, expressing, and validating the various avenues a to-be-defined product might take.

Most recently, this was demonstrated via the launch of New York Times Audio — an app that embodies our idea on how best to represent journalism in audio form (in other words — “what if The New York Times was just a play button”?).

This new product is home to New York Times produced podcasts and some of the industry’s most iconic shows such as This American Life and Serial, along with select read-aloud articles from partnering publishers. It also includes reporter-narrated articles that give listeners a human, first person account of what went into some of our front page reporting — a unique new format that helps listeners connect to our journalism and get a sense of the story behind the story.

Here is a look at how we worked together with colleagues across the business and newsroom to bring this audio-first experience to life.

At the start of the project, we pulled together a cross-functional team to focus solely on bringing our vision to life: a streaming app dedicated to the world’s best audio journalism. This meant shaping new tools and processes, like building up a new audio-centric design system and CMS, along with immersing ourselves in audio best practices and approaches taken elsewhere to understand where our biggest opportunities were.

As we scaled our team and set these foundations, we had to solve for specific design challenges that varied greatly from the ones tackled by other music and podcast apps.

Our product needed to:

• Ensure that our audio journalism feels timely, evolving by day of the week, even time of day.
• Emphasize the role human editorial and curation plays.
• Create a clarity in what can feel like an overwhelming choice of playlists, titles, and content.
• Meet diverse listener news needs, whether they just wanted to quickly get caught up on the news of the day or go deep in a pressing topic.
• Feel like an extension of the Times brand, while also creating a distinct visual language and interface that caters to listeners over readers.

In order to develop an experience that delivered on these ambitions, our design team invested heavily in discovery sprints, weighing the effectiveness of drastically different approaches and iterating at a rapid pace. And, perhaps most notably, we conducted the majority of these efforts in tandem with our editorial partners.

From the very beginning of our design process, newsroom partners were by our side — sharing what was important to them, highlighting gaps in their toolkit, and informing the direction our concepts took. They participated in brainstorms and co-creation sessions, provided feedback in design critiques, and kept us close to their most important needs when programming and presenting our journalism.

This early, ongoing collaboration demonstrated the value of designing as closely to editorial as possible: with their intimate understanding of our coverage, we were able to create a product experience that was both sensitive to their journalistic approach and innovative for our users. Our working together even opened the door for brand new shows and product features that would eventually be woven into the app’s core offering.

Some examples of this are The Headlines, a short morning show developed exclusively for the app, and the daily playlist: a programmed package of must-listen audio stories that tops our homescreen. Each of these was sparked by a user need (more short listens to fit into my day) or design concept (“the essentials” in audio) and brought to fruition via our partnership. Additionally, we relied heavily on editorial feedback when creating our card system. They helped inform which tools were needed to effectively respond to the news cycle and communicate the appropriate content hierarchy, and evaluated them in context when programming early versions of the app to help us understand what might be missing.

All in all, we’ve forged a unique relationship with the newsroom that has continued to shape our work. Our collaboration helped make us feel like one team — editors informing design iterations and the needs of our designs themselves, and designers pitching concepts that ultimately inspire a new editorial approach.

We spent quite a bit of time on research and testing. Early on, we partnered closely with our insights team to user test several divergent app prototypes, with different architectures, content strategies, and design aesthetics. In parallel, our research leads conducted user interviews to gain a deeper understanding of listening habits and the role audio plays in our audiences’ daily lives.

In each stage of testing, we gathered direct commentary and feedback from real listeners. We connected our feedback form to Slack, giving us real-time updates on what people were saying. It was a constant source of inspiration. Every day, we would see what users were thinking about the product we were building — how they were using the product; what their pain points were; what features they were asking for. Their feedback helped give us a sense of what they cared about, and what they wanted the evolution of NYT Audio to be.

We shaped our priorities and product roadmap around this input, guiding us towards our MVP.

We had a lot of initial conversations at the beginning of the project where we analyzed market research and debated product strategy. These often resulted in many more questions. As designers, we focused on translating insights and ideas into artifacts that would help bring clarity to these meetings, driving us to decisions faster. Artifacts varied from simple diagrams to more complex, visually-rich prototypes.

These prototypes also inspired our editorial team to think through possible content ideas, helping give shape to assets we have in the app today.

One of the most exciting and unique aspects of the prototyping included experimenting with sound. Using music and our own voices, we’ve played around with ways a user might navigate the app using sonic cues.

Rather than just viewing artifacts and prototypes as the end result of our design thinking around a problem, we began using them earlier in the process to inspire conversation, which helped give the work a greater sense of direction.

Given that audio is by nature a format that renders our journalism accessible for low and no vision users, we wanted to ensure that the interface was optimized for VoiceOver. In helping establish a set of design and product principles to guide our work, we prioritized accessibility as one of them. This brought a11y to the front of our process and baked it into our product thinking from the beginning.

With our engineering team, we set out to learn as much as we could. We started by conducting an audit of VoiceOver experiences in other apps, and combed through Apple’s guidelines for building for accessibility. Taking a look at an early build of our app, we identified where accessibility was currently working and where it needed to be enhanced. We explored different ways we might improve the experience to make it seamless to navigate using only audio cues. How might we group cues and actions together to reduce the amount of taps and swipes to a listen?

Another part of the challenge was figuring out how to annotate our designs for VoiceOver. Seeing a gap in our documentation kit, we created a reusable Figma template for preparing work for development. Now a permanent part of our documentation, the template has since been adopted by other teams at the Times.

Moving closer to launch, we organized accessibility-focused user research sessions with no and low vision users to get ongoing feedback for how our optimizations were working, which helped us hone the experience prior to release.

One of our goals was to distinguish NYT Audio visually, while retaining the quality and essence of The New York Times brand. To do so, we worked closely with brand designers, art directors and photo editors that work across various Times products to sharpen and align our visual approach for launch.

Leveraging a core competency of the Times — our photojournalism — we knew we wanted to prioritize images in the experience. To capture the essence of each piece of audio journalism and to pique listeners’ curiosity, we coupled our audio with compelling art direction. Alongside our photo editors, we refined the shapes of our content cards and scrutinized the aspect ratio for images to ensure that the art was given the right amount of visual priority in relation to the other elements on screen. It was important for us to land on a system of containers that elevated imagery and made our photo editors’ workflow seamless.

Given that our content architecture is relatively complex with many different types — from podcasts to narrated articles, longer-form pieces and shorts — we spent a lot of time refining our information design. We looked through everything from iconography to the relationship of typographical elements. Together with our brand team, we went through many iterations of our component library. This manifested in changes to our type system, padding, and even the broader color system we applied across our cards (originally, we were extracting color from album art to apply to the card UI, but refined our approach to be more intentional given the visual noise this created).

We also worked to establish a distinguishable identity while connecting NYT Audio to the rest of the Times product portfolio. The brand team helped define a distinct brand color that sat comfortably within our product bundle, and established usage guidelines around typography in accordance with the greater Times design system.

From an ambitious idea to a real tangible product, designing and helping launch the NYT Audio App was a fun and challenging experience. We truly had a seat at the table alongside our newsroom, brand, product and engineering partners, and there is a direct line from our discovery work to the live experience our users engage with day.

This experience is a reflection of how Design fits into the fabric of the Times. The product development organization here empowers the design function to play a pivotal role in setting direction. We shift across altitudes, from collaborating on research and test design, to prototyping with editors, to partnering with product on defining the big opportunities for our roadmap. We’re excited to continue having an impact as we evolve what the audio product, and our approach to elevating audio journalism, will become in the future.

By Kyelee Fitts and Celia Eddy

With tens of thousands of recipes in our catalog, The New York Times Cooking team faces a challenging task — how can we serve the best recipes for our users, taking into account their varying preferences, including diet, nutrition, cook time, cuisine, and ingredients? Each week, The New York Times Cooking editors manually curate a set of recipes and collections to promote on the homepage and in email newsletters. To enhance our editorial curation, we have been experimenting with personalized recipe recommendations with the goal of making sure our users find the right recipes for them.

In August, we launched a new personalized homepage in The New York Times Cooking mobile app. This new homepage includes both editorially and algorithmically programmed content that is grouped into different carousels. Examples of editorially-curated content include the recipe of the day, recipes from Sam Sifton and Melissa Clark’s newsletter, or editorially-curated collections.

Other carousels are powered by algorithmic recommendations; for example the We Think You’ll Love carousel recommends recipes based on what users have engaged with in the past.

In this post, we’ll focus on how we use machine learning algorithms to deliver recipe recommendations for our users. Different algorithms serve different user needs, and we’ll walk through a few examples of how we apply different algorithms across The New York Times Cooking website and mobile app.

First, we need to define a poolof eligible content to recommend to users. Pools can be manually curated by editors or can be generated via a query of our recipe database, using a set of rules we determine for each recommendation carousel. Next, we use an algorithm to rankthe items in that pool according to a ranking algorithm that we define. This ranking algorithm can be as simple as ordering recipes by popularity, but we can also use reinforcement learning methods or natural language processing techniques. Let’s go into a few different ranking algorithms that we use to power some of our different carousels, each of which fulfills a different user need.

Recipe Recirculation

One tool we often use in our ranking algorithms is text embedding. Embeddings are structured numeric representations of words or documents that enable us to use text as input to machine learning models. Once we represent our recipes as fixed-length vector embeddings, we can take the distance between the vector embeddings for any pair of recipes as a measurement of how similar the recipes are to each other. For example, if we were representing these three recipes as a length-2 vector embedding, we can plot them in 2 dimensions and get an idea of how “far apart” the recipes are:

In practice, these embeddings encompass more than just two values, allowing them to capture the multifaceted nuances of recipes. We use several different parts of the recipe text to generate these embeddings: the title, the description, the ingredients, and the cooking steps.

We’ve experimented with many different methods for embedding recipes. Recently, we upgraded our recipe embeddings to use a pre-trained sentence transformer model that is optimal for measuring similarity between two pieces of text. These models are easy to use, fast to score, and most importantly, outperform other embedding methods in our experiments in terms of engagement with our content.

We use these embeddings directly on the Similar Recipes ribbon (pictured below),a recirculation module that appears below the recipe steps. It directs users to recipes that are similar to the one that they are currently viewing.

Once we generate embeddings for all the recipes we want to consider for the carousel, we compute the similarity to every other recipe in the catalog using a metric called cosine similarity. The cosine similarity varies from -1 to 1, where a number closer to 1 means the recipes are more similar. We can then sort all the recipes by cosine similarity to find the ones that are most similar. On this carousel, we use a weighted combination of the similarity and number of page views recipes have recently received, so the top recipes shown are both similar to the current recipe and fairly popular.

The model we use to power the majority of our algorithmic carousels is a contextual multi-armed banditalgorithm. As a quick recap (see our previous blog post for more information), a bandit is a reinforcement learning algorithm that suggests the best decision (i.e., an article or recipe) among a list of options (in our case, a pool of content that’s eligible to be recommended in a certain location) in order to maximize a reward. We use a modified version of a model that predicts the “reward” (in our case, a user clicking on a piece of recommended content) of showing a particular article or recipe to a user as a linear combination of contextual features plus a term that estimates the uncertainty of the prediction. The bandit balances exploitingknown high-CTR content with exploringnew content, and over time it learns what recommendations to make in order to optimize reader engagement. We have made several modifications to the bandit algorithm that we’re using, including introducing stochasticity and “forgetfulness” to ensure we can model changes in article and recipe CTR over time. Through experimentation, we determined that these model updates improved reader engagement and recommendation freshness.

A bandit with no contextual features will typically recommend the most engaging articles or recipes to all users. This is the algorithm that we currently use to power the Most Popular This Week carousel on the mobile app and the web (pictured below). The pool consists of the recipes that received the most pageviews over the last week. This allows us to show the recipes that have been most popular recently and order them efficiently in terms of what will drive the most engagement on the carousel.

We also have the ability to add features to our bandits to help them learn what content will likely result in a click. One contextual feature we use frequently is something we’ll call the personalization feature, which tells us how similar a recommendation candidate is to the recipes a user has engaged with in the past. To create this feature, we first calculate embeddings for all our recipes using a sentence transformer model, as described above. Then, we use prior engagement data to generate user vectorsfor each of our users. In our case, we have chosen recipe saves as a strong indication of a user’s interest in a recipe, since a user saves a recipe if they intend to cook it or return to it later. For a given user, we take all of the recipes that they have saved, get the embeddings of those recipes, and average them pointwise to create a user history vector. Then, we can measure what recipes are closest to that user history vector according to cosine similarity. We use the cosine similarity score as a contextual feature in the bandit and let it learn how much weight to give the feature as a predictor of clicks.

When we first tested this bandit personalization feature on the We Think You’ll Love carousel, we realized that the same popularrecipes tended to be recommended to the majority of users. In other words, the bandit was not paying as much attention to each individual user’s preferences as much as we’d like for a carousel that’s explicitly personalized. To ensure that our recommendations were aligned with the business goals for this carousel, we decided to make a modification to the bandit reward. Instead of rewarding the bandit on every click it got equally, we rewarded it more if it got a click and the recipe was similar to recipes the user had engaged with in the past. Modifying the reward increased the diversity of the recipes we were showing between users and resulted in more personalized recommendations. We have found via experimentation that a bandit with this personalized reward drove more engagement than our previous approach, which involved ranking recipes based purely on the similarity between candidate recipes and a user’s saved recipes.

Many users are interested in cooking recipes with ingredients that use produce that is seasonal to their areas. To meet this need, we developed a seasonality score to match the seasonal ingredients in different regions in the US with the ingredients in our recipes and scored each recipe based on how seasonal it is in each region at each time of the year. The seasonality score is based on several factors, including the number of seasonal ingredients the recipe contains, the percentage of total ingredients that are seasonal, and whether the recipe contains ingredients that are newlyin season.

This method of ranking recipes by seasonality allows us to build pools of recipes that are locally seasonal in different regions of the US. Instead of just ranking by the seasonality score, we use a bandit on top of the seasonal pool in order to optimize engagement within that locally seasonal pool of recipes.

This algorithm is what powers the In Season Near You carouselon the personalized homepage (pictured below).

Finally, let’s cover dietary needs and preferences. Dietary preferences are a huge aspect of recipe personalization: Users want their New York Times Cooking app to cater to their specific dietary needs. We don’t currently collect data from users on their explicit dietary preferences, so instead we use signals about their preferences based on their activity within the product so we can more effectively provide the personalized recommendations people expect. In the new, more personalized homepage, we introduced Dietary Preference carousels (pictured below).

When our recipe developers and editors create a recipe, they tag it with a variety of tags, including cuisine, main ingredients, and whether it adheres to a specific diet. The dietary preference carousels use contextual bandits to recommend a set of the most popular diet-tagged (in particular, vegetarian, vegan, dairy-free, or gluten-free) recipes for users. These carousels only appear if the users have saved to their recipe box a certain number of those diet-tagged recipes.

Another way that we’ve used dietary preferences is as a contextual feature in our bandits. This is a feature we’ve introduced recently to the We Think You’ll Love carousel. To create this feature, we first count how many recipes are of a particular diet in a user’s saved recipes. We then construct a vector with these counts and normalize them:

# [vegetarian, vegan, gluten-free, dairy-free]# For user “Jane”: diet_vector_counts = [10, 5, 5, 0]diet_vector_normalized = [0.5, 0.25, 0.25, 0]

The bandit feature is the cosine similarity between this dietary preference vector and a vector of the recipe’s diet tags. So for a vegetarian recipe, the recipe diet tag vector is:

recipe_diet_vector = [1, 0, 0, 0]

And the feature that the bandit would use to predict whether or not user Jane clicks on this particular vegetarian recipe is ~0.82. Adding this contextual feature about diet tag similarity enables the bandit to make recipe recommendations that are more in line with users’ implicit dietary preferences.

These were just a couple of examples of how we’re improving our recipe recommendations to better meet reader needs. As we learn more about our users’ preferences, we are continuing to refine our algorithms to improve the personalization within The New York Times Cooking app, working closely with our newsroom editors to ensure that we are always bringing our users a curated, enjoyable Cooking experience.

Happy Cooking!

Every year, The New York Times hosts an annual hackathon event called Maker Week. Technology teams, across engineering, product, design, data and insights and cyber technology, have the opportunity to unleash their creativity and work together to explore new products, ideas, and technologies.

It is a chance for Times employees to collaborate with their colleagues, either within their immediate teams or cross functionally, to build an app, tool, analysis, business plan, or anything else. Additionally, teams host peer-to-peer talks and interactive workshops focused on hot topics throughout the week, such as Generative AI, Data Visualizations, and Engineering hacks, just to name a few.

Monday, July 17, 2023 — Forming a Team and an Idea

10 a.m.

It’s day one of Maker Week! I headed up to the 15th floor to grab a Maker Week t-shirt and to meet with Lindsey Liang, a lead product designer on the Ad Experience team.

Noon

Lindsey and I attended the Maker Week kickoff together at the 15th floor Conference Center. Our project got a mention in the hackathon kick off, which led Paul Bonneville, a staff iOS engineer on the Newsreader team, to join our project team! We then met with the rest of our teammates — Evan Kirkiles, an intern on the Engineering team, and Lord Crawford, an associate product manager for Advertising Experience. The four of us broke the ice with lunch!

2 p.m.

We all gathered in a conference room along with Denise Wang, a Software engineer on our Reader Experience team as well as Jason Thong, a product manager in Advertising. We gave brief intros, and I talked about where I got my inspiration for the project. Earlier this year, I got to attend Apple’s 2023 Worldwide Developer Conference, where I witnessed the announcement of the Vision Pro in person. As they showcased the new technology, I imagined what The New York Times reading experience could be like and I figured Maker Week would be a great time to experiment. We then brainstormed the ideal Vision Pro experience, before finally defining what our goals and practical MVPs (minimum viable products) would be.

We settled on three ideas:

1. Render a 3-D model from an Interactive article in an XR (Extended Reality) space
2. Create an immersive space for the user to see and interact with the model
3. Improve the experience for users of the New York Times app in Vision Pro.

The first concept was the easiest to implement so we agreed to start there. We quickly divided up tasks and responsibilities based on individual strengths and scheduled a regroup in the morning.

2:30 p.m.

The first idea required embedding a link of the 3-D model from this article in its code. This would make the 3-D model clickable and allow it to render in your real-world space as an AR object. We would need to add only about 2–3 lines of code, but first we had to locate the source file. We spent the rest of our day downloading the necessary software and getting access to codebases.

5 p.m.

Although we didn’t have the source code, we decided to call it a day. I decided to work a bit more at home, hoping I could find the code by inspecting the webpage in more detail. I experimented with some formatting options but everything I tried gave me an error code, so I decided to debrief with my teammates tomorrow.

Tuesday, July 18 — Getting Technical

10 a.m.

Lindsey, Lord, Evan, Jason and I regrouped so we could sync with Paul. We also got a new member — Madeline Kim, an iOS engineer — to join us! Knowing her expertise, I had been bugging Madeline to join our project for weeks. My persistence paid off!

We talked through our technical issues with Paul, and he suggested we explore what an immersive experience viewing videos in an article could look like. We all thought that was a good idea, so we pivoted!

12:45 p.m.

After lunch, Evan and I talked through the issue with the 3-D object not rendering and ultimately decided to scrap the HTML route after discovering that visionOS did not support this Apple feature. We pivoted to an idea that had initially been a stretch goal — recreating the NYT interface as its own visionOS app, with full support for immersive, XR-only features.

Evan and Paul got to work fleshing out the functionality of the visionOS NYT interface itself — wrapping our web platform in native Apple containers. We quickly realized that, while having many individual windows open at once looks great from a design perspective, it is difficult to manage in practice on the current visionOS beta. We ultimately settled on just two scoped windows: a “navigation” window with high-level pages of The Times, and an “article” window where you can cycle through articles intuitively.

The rest of us focused on the “immersives,” unique experiences linked to articles that extended reality beyond the current visionOS reader. I tried to render the 3-D object with the help of Evan’s button. Madeline and Lindsey reviewed the feasibility of the immersive scene in this time frame. They decided to play an article video on full screen with text overlay, and use the bamboo image as the immersive background. Madeline started building out this scene while we all were constantly referencing the new SwiftUI documentation for visionOS, which was scarce.

4:30 p.m.

The engineers ended early and went to the happy hour. We had made a large amount of progress, and were all feeling good!

7:30 p.m.

I wanted to help out with Madeline’s work, so I played around with adding a dummy immersive scene to her VideoView. We didn’t like how the SwiftUI warped the image, so I spent most of the evening trying to mock Lindsey’s immersive panoramic image design.

Wednesday, July 19 — Pivoting Direction

10 a.m.

While most of the team attended a breakfast, Evan and I attended a coffee tasting event hosted by one of our many Employee Resource Groups, Young Professionals Network. There were five types of coffee labeled A to E. Participants were to blind-taste test each of them and then vote on their most to least favorite. Coffee C won, which turned out to be Starbucks, but personally my favorite was B, which was coffee from a local café. Evan’s was A, which was revealed to be Dunkin’.

11 a.m.

We regrouped on final action items. Madeline achieved an immersive view for this article and was able to create a VideoView to play in the simulator. Lindsey experimented with the possibility of rendering a globe, so that users could interact with the locations featured in a simulated space. Meanwhile, Evan worked on User Interface (UI) enhancements, and refactored much of the codebase with an improved understanding about SwiftUI design patterns. This led to an issue that ran deeper than unoptimized code — and thus began our list of visionOS beta bugs.

1 p.m.

After lunch, it was crunch time. As we prepped for our presentation, which was scheduled for Friday morning, our team felt very much like a start-up.

5 p.m.

Some of us reset with additional social events, organized by the hackathon planning team, and agreed to get back on later in the evening.

11 p.m.

I ended up diverging from the original MVP design for the 3-D model and implementing something similar to one of Apple’s tutorials, in which the model appears next to the article text. I figured this would be easier because I could also copy how they made the model interactive.

Thursday, July 20 — Tackling Roadblocks

11 a.m.

We started a little late today, but we went over leftover code, possible demo recordings from the Xcode simulator and what we may need to rely on mockups for. Lord asked if there was time to render a 3-D model in a user’s physical space, so I spent the morning working on that.

2 p.m.

After lunch, we had one last sync with Paul. Overall, we felt like we were in a good place for tomorrow, and excited to be near the finish line.

3 p.m.

We reverted to the original 3-D model concept design. The article and model rendered together, but the submersible was rendering strangely under the couch! I spent a solid hour trying to figure out how to move it to the middle of the room with different commands. Evan, meanwhile, made navigation design changes that implemented our own custom containers. Madeline added the 3-D model of the globe into the VideoView. She was having some difficulties maneuvering it on the screen. With Lindsey’s guidance, we removed the 3-D globe and the title of the article to provide a focused experience of the immersive video. Lord, Lindsey and Jason continued putting the presentation slides together, adding images and placeholder content.

6:30 p.m.

We finally went home prepared to work into the night (see energy drinks). I was trying to figure out how to time the presentation perfectly; I recorded the first demo, but my computer crashed before I could record the rest.

Friday, July 21 — Adding Final Touches and Presentation!

6 a.m.

I woke up a little early so that I could record the second and third demos, as well as finish typing out my presentation notes.

8 a.m.

Commuted over to the office, still typing out notes.

9 a.m.

Let my team know that I was still typing notes, while simultaneously talking through them.

9:45 a.m.

Finally finished writing out the notes! I made my way over to The Times Center, our biggest event venue and where the presentation ceremony would take place! Meanwhile Lord, Evan and Lindsey were making finishing touches to the slides.

1 p.m.

Presenting our project!

2 p.m.

After the last presentation, Lord, Madeline, Evan and I got lunch, before finally parting ways. We were all relieved to be done, but felt that we accomplished a lot in a matter of days while also getting to know one another.

I’m very lucky and proud to have worked with such a great group of people who were willing to experiment and had such a vested interest in delivering a new way of reading our journalism. The project definitely stretched both our skills and our imaginations, giving us the space to collaborate and be creative. What an energizing week!

By John Nevins and Eric Chima

As an engineering manager, I once led a traditional Scrum team, living the mainstream Agile culture. We spent hours defining, refining, and estimating work so that we could feed it into our fine-tuned productivity machine. We were all veterans of Agile Scrum — so why did it seem like this machine wasn’t as fine-tuned as we thought?

“Ro-sham-bo, throw!” our Scrum Master would shout, as fingertip estimates darted into the air. Inevitably, a few of those numbers would match, a few wouldn’t, and then the discussion would really kick off.

“That’ll take a week,” an astute engineer would offer.

“Ah-ah! That’s a measure of time and not complexity,” I would remind the group. We would then take time to recite the ethos of complexity points, before revisiting the all-too-familiar topic of developer capabilities as they related to the estimate. Back and forth we’d go, always talking about how we described the work, rather than the actual work itself.

By the time one such session was complete, we had refined three user stories worth of work, totalling 13 points. We still needed another 10 points of work to make a full sprint at our average velocity. We agreed to meet again to be ready for Sprint Planning.

This whole process seemed very important, because these stories comprised a large body of work. It had elevated in profile enough that I needed to forecast a completion window for our stakeholders. The problem was, without knowing the estimate for each ticket, I had very little way of projecting a date without committing the team to an arbitrary deadline.

I started to look at past performance data. We were on the Fibonacci system, measuring complexity by our guts at 1, 2, 3, 5, and 8 points. 13 was too big, and 21 was unforgivable. I looked at all of the stories that crossed the finish line in the six weeks prior, and saw that the average ticket was 4 points. That’s an abomination in a Fibonacci system, I know, but the numbers don’t lie, so I went with it. I put a 4 on every ticket in the Epic, added 20% for good measure, projected a date, and handed it over.

We delivered within one week of my projection. And it wasn’t a fluke.

We continued to shape our projects that way, and delivered consistently. It got me thinking about where we spend our time in the software development lifecycle.

I really started to question the system.

I came back to the topic several years later, when my new team was struggling with velocity. It was the most senior team I had ever led, brimming with potential, but still we were stalled. We transitioned from Scrum to Kanban, trying to go lean, but the team was still spinning its wheels in meetings, debating the size of tasks we had never tried before. Refined work wasn’t making it into the “To Do” column fast enough. I couldn’t shake the sense that if we just put the time and effort we were spending on estimates into actually completing tasks, we would get a lot more done. When I suggested it to the team, I learned they felt the same way. We stopped estimating and never looked back.

By the time 2022 rolled around, I was confident that estimations were a crutch that strong teams didn’t need anymore. As long as your engineers are skilled at breaking down problems — a key muscle for any Agile team — you can keep the work flowing. If a ticket takes too long, you can learn from it and break it down better next time. And if you need to do some long-term planning, well, all you need is a little math.

Since the Agile Manifesto was first signed back in the early 2000s, software teams have built upon new norms of working. At that point, they were no longer bound to the waterfall method of …

Requirements — Design — Implementation — Test (and fix!) — Deliver

… but rather could hedge risk by fitting that entire cycle into two weeks. The struggle is real — I once worked on a waterfall-based team that operated on a one-year cycle, and that test phase really gets frantic when six months worth of work is supposed to function without issue.

Now, throughout the industry, software teams plan their calendar around the same familiar ceremonies: refine, estimate, plan, retro, and repeat. There’s a reason these processes are so commonplace: They keep teams organized and they get work done. But every hour spent in a ceremony is 2.5% of a developer’s work week gone. Gathering six people in a meeting works out to be surprisingly expensive, and most engineers would rather be doing the work than sitting on a videoconference talking about it. The cruft of ceremonies, discussions, and meetings has spawned a term — “metawork” — has made its way to Urban Dictionary.

Eliminating estimations, then, isn’t about revolutionizing Agile. It’s about simplifying it. As Agile moves deeper into its 20s, methods that seemed groundbreaking in 2001 are familiar and easy now. Many developers have been breaking down work and iterating for their entire careers. If your team has been working together for a while, they should be used to cracking chunks of their problem space into manageable tickets. And once that problem has been solved, a few key metrics can track and even predict how quickly that work is finished.

We’ll come right out and say it: Velocity is still the single most important metric to your team. Traditionally, that’s tracked in terms of story points, but it doesn’t have to be. Velocity is simply a measure of throughput, so why not leverage your issue tracking system and track the number of work items you complete each week? If your team finishes six tickets per week, and they’re breaking down tickets well, that’s your velocity right there.

“But wait,” you say. “Breaking down tickets isn’t perfect! Some of our stories are 8s and some are 3s!”

Sure they are. But in the long run, they’ll all average, so why spend the time to debate each individual ticket? What matters most is the predictable output of the team, and that’s going to be the same regardless of the number you assign to the task. If you find your velocity fluctuating too much, that just means you need to get better at breaking down problems.

Let’s make the example more concrete: If 80% of your stories are 3s and the other 20% are 8s (as an example), on a backlog of 10 tickets, your average story size is 4.

If you put a 4 on every story, or ticket, you’ve still reflected the accuracy of the backlog on average. If you complete 20 points every 2 weeks, on an average story size of 4 points, you’re going to complete 10 story points every week, regardless of the relative size of the individual task.

So, if you skip this whole exercise and put a 4 on each ticket, you’ve saved yourself the hassle. The number on each ticket is starting to feel less useful, right? Let’s just go ahead and delete it, because if your team has broken its work down efficiently, you don’t need it. Your velocity is now the number of tickets per week, or throughput.

If you’re worried that your team isn’t reducing its stories well enough, just turn to a concept from the Kanban method of Agile: Cycle Time. Track the time your stories spend between In progress and Done and you can tell how well you’re breaking down tickets. If the average cycle time of a task increases from, say, 36 hours to 7 or 10 days, use that as a catalyst for a conversation about work breakdown in those all important team retrospectives.

It’s important to note, though, that the throughput metric applies to teams. An individual does not complete a certain number of points per week. This is an important distinction, because you don’t want your developers to feel judged by the system, or to try to game it by engineering the tickets they take on. Remember, at the end of the day, it’s not about the number of story points or even the number of tickets, but rather getting real value in the hands of users. Velocity is a tool for planning projects, not a scorecard for evaluating employees.

Story estimations and velocity measurements all boil down to one thing, right? At some point in a project, someone with a manager title (product, project, engineering or otherwise) is going to ask, what is the level of effort? Or, put more plainly, how long is this going to take?

In that sense, it’s easy to see why those manager types would lean on estimations. Precision feels better when you know you’re going to be judged on it, even if it’s the faux precision of story points. But you can plan a project just as easily using the hard data of your team’s past performance.

To start, calculate your team’s velocity. Say, for example, that your team has completed six tickets per week over the last six months. If you break down your new project, and see that it has 35 tickets, you can reasonably say that it will take about six weeks to complete. Then, you’ll want to add in a margin of error for phenomena like scope growth and debugging. You can base that buffer on your team and your experience with past projects. It might feel like a bit of a guess at first, but after you complete a few projects, you’ll learn just how much inflation tends to happen. It’s data, just like anything else.

Above all, this is an iterative process, just like everything in Agile. The more you do it, the better your data will be, and the more accurate your projections. As people join or leave the team, it might take a few cycles to refine your velocity again — but that happened with estimations, too. The important thing is that you can get better at it without any extra effort from your developers — they get to focus on the work.

Meta has dubbed 2023 the Year of Efficiency, though for many of us it feels like we’ve been in the Age of Efficiency for a while now. As leaders of Agile teams, our role is to point engineers toward more of the work that matters and less of the work that doesn’t. Here at The Times, as a subscriber-based business, we have an obligation to our readers to ship meaningful value early and often. That extends from our newsrooms to our engineering teams, and we’ll embrace any approach that aids in that agenda. For some of our teams, eliminating estimations is a strategy that works.

With that in mind, we’ll admit that this method isn’t for everyone. If your team is still building the muscle for work breakdown, struggling with process discipline, or is new to Agile altogether (where have you been?), the traditional approach might still be best for you. And teams that work on very tight, week-to-week deadlines might find that they need a bit more precision than we’re suggesting.

But for most teams, we think, eliminating estimations will be an easy win. And even if your team isn’t ready quite yet, that doesn’t mean they can’t be soon. Agile is designed for iteration and experimentation. Give it a try and see how it goes, then make changes that work best for your team. We’re guessing you won’t have many engineers clamoring to get that meeting back on the calendar.

We wish you all the best in your Agile adventures. And with that, we’re going to slide this post into the Done column.

8:15 a.m.

I start off most mornings by taking my dog, Goose, to Prospect Park’s off-leash hours. It’s dog heaven!

9 a.m.

When we get back from the park, I make myself coffee and do Spelling Bee and Wordle, my morning ritual.

9:15 a.m.

I work from my apartment in Brooklyn, so I make my way over to my desk to get started for the day. I always check Slack and my emails first thing. I’m “on call” this week, so I also check our alerts Slack channel and the Loupe feedback tool channel to make sure everything is working as expected.

I have an unusually meeting-heavy day today, so I have to manage my time well so that I still have room to work on my tickets (the tasks our larger projects are broken down into!). We launched Loupe* to the newsroom at the end of 2022, so we’re now iterating it and prioritizing work in response to user feedback. So much of the journalism we publish relies on visual storytelling through photos. Before Loupe, photo editors were working with a very old system that couldn’t keep up with the needs of a modern newsroom. Loupe is a much more resilient tool and was built to be compatible with newsroom workflows.

We have a pretty active Slack channel with the photo editors who use Loupe, and it’s great to be able to communicate directly with our users. That’s one of the things I love about the Photo team. We also have bi-weekly demo meetings with our stakeholders to update them on any new features and get their feedback.

We’re also tackling two other big projects right now: replacing the system used to manage photographers and photo assignments and migrating some of our systems off an in-house inventory and provisioning tool for organizing A.W.S. (Amazon Web Services) resources.

10:15 a.m.

We have standup* every morning with the whole team — engineers, designers and program managers. We go through our list of tickets, give updates, and commiserate about how long the new operating system upgrade is taking to complete on all of our computers.

My manager, Sherry Gao, shares pictures of her new baby on Slack! There are lots of emoji reactions.

10:30 a.m.

It’s back to tickets for a while until my next meeting. (I first have to decide which Taylor Swift album I’ll listen to on repeat today. I’m going with “Midnights.”)

Today, I’m working on a couple of tickets for our photo assignment replacement tool. I’m working on fetching all of the photographer assignments from our database to display in Loupe. I’m primarily a front-end engineer, so this ticket is getting me out of my comfort zone by working with our database. Our code goes through multiple stages of peer review, from writing it to releasing it. This way, we can be confident that by the time a feature gets released, there won’t be any unexpected issues.

Of course, we can’t anticipate everything, so we have robust automated alerts so we can be notified immediately if anything in our system is going awry. First, engineers review the code. Then Jenny Hottle, our designer, will do a visual Quality Assurance (Q.A.) check, and Shonta’ Singleton does Q.A. testing. Our product manager, Kate Harman, does a final product review, and then it’s good to go. It’s now ready to get deployed to production! Kate gives me the go-ahead.

Now I have some time to review a couple of my teammates’ pull requests before my next meeting.

Noon

I head to my design system working group meeting, where we talk about the launch of version two of the system, which includes a full conversion to TypeScript*. TypeScript is being widely adopted across The Times, because it makes for a more resilient application and a better developer experience. It was one of our most requested features. (P.S.: We successfully launched the following week and were met with lots of excitement across our engineering and product teams!)

1 p.m.

I join our bi-weekly “stack chat” meeting with the Photo engineers. It’s always an open agenda, so anyone can bring in a topic they want to discuss with the group, whether it’s directly related to in-flight tickets or a more general technical concept. This week, a senior software engineer, Sherman Hewitt, led a discussion about state management in Loupe, which we’re hoping will lead to a big simplification of parts of our codebase. We aim to balance building new features with making our existing code better.

2 p.m.

It’s a marathon day of meetings. This one is another working group, where we’re redesigning the data structures and algorithms portion of our technical interview process for new hires and replacing it with a code review-style format. Today, we aligned on themes around what types of skills we want to evaluate candidates for — collaboration was a big one.

3 p.m.

It’s been a while since I’ve taken a break, so I pause for a late lunch.

5:20 p.m.

Time to wrap up for the day after working some more on my tickets for the last couple of hours. I have a little bit of time to unwind before I head to pottery class.

* What does it all mean?

• Loupe: The in-house content management system that hosts the entire archive of Times photos
• Standup: Part of agile development, a daily meeting at which the team gives status updates on in-flight work
• Typescript: A programming language and extension of JavaScript that helps developers write more reliable and understandable code for building software applications

By Akasha Archer, Angelique De Castro, Audrey Valbuena, Danielle Zhu, Joseph Kohlmann, Kika Gilbert, Mark Murray, Nate Clancy, Rachel Dixon and Véronique Brossier

The New York Times recently rolled out the long-awaited dark mode functionality for iOS and Android news apps.

For years, the case for dark mode was well-supported by both our business goals and user requests. Launching dark mode would help us achieve our long-term vision of producing a high-quality product that is valuable and worth paying for. Furthermore, our research found that dark mode was identified as a “must-have” feature for both subscribers and non-subscribers, many of whom had been requesting the feature with our Customer Care team. We knew that as a content-forward app, having accessible ways of viewing New York Times journalism was important, and we wanted to consider the high standards of our brand when implementing this feature.

There was no denying dark mode was a priority feature we wanted to make happen for our in-app readers. But as our teams discovered early on, it was easier said than done. The path to dark mode meant two major challenges:

First, creating a set of core colors to use for dark mode. And second, implementing those colors across platforms.

In late 2020, iOS engineers working on the News app team began investigating what it would take to introduce dark mode to the app and drafted a Request for Comment (RFC). They proposed an org-wide color library that would publish to independent libraries for each of the three relevant platforms: Android, iOS, and Hybrid Web.

At the same time, a Design Systems team was organizing, with an early goal of building a design system for our editorial surfaces, including the News app. We realized the design system’s color tokens could be a key to unlocking dark mode implementation across platforms and teams.

The design system’s refreshed set of core color tokens would help to achieve greater consistency, more efficient workflows, and a heightened focus on accessibility. Prior to our design system work, there was little consistency related to color usage throughout our product. Without a core color system, different designers and engineers would ultimately end up using different sets of color values. As a result, users would be exposed to inconsistent colors as they moved through the experience.

After conducting an extensive organization-wide audit, our design team worked to update our palette to a more intentional set of core colors, each with a specific target (e.g. for content, backgrounds, strokes) and a semantic name based on usage or hierarchy (e.g. primary, secondary, accent).

Aside from other requirements we had for our color system, we wanted our colors to be capable of rendering in two different themes: light and dark. Therefore, each color really has two definitions, a light value and a dark value.

When successful, attaching two different values to each semantic color means that a design system user can trust that the colors they use will convert as an entire group, and retain their intended hierarchical and contrast differences, regardless of theme.

We also solved for a number of edge cases in which the theme we wanted to use wouldn’t be solely dependent on a user’s device setting. This means that when necessary, we can modify colors such that they render sections of a screen in the dark theme, even if a user is in light mode on their device.

When we were finished defining and naming our colors, we published them in various libraries for our designers and engineers to start working with. Our color system will likely continue to evolve, but our current version enabled us to get to the next step of implementing dark mode.

The adaptation of dark mode was fairly straightforward in our iOS and Android native apps after standardizing our palette across our code base using the design system’s color tokens.

However, our apps also adopt a hybrid approach to rendering, with a native shell and webview content. This allows us to more closely match the appearance of news content in-app and on nytimes.com, while also letting us iterate on news content design and features more quickly. Thus, much of what a reader sees in the app — story articles, live blogs, interactives, etc. — is a customized version of our main website.

Thus, we encountered a paradox: To ship dark mode in our native apps, major parts of our web apps needed dark mode support.

Which parts of our web apps did we need to update for dark mode? To answer this question, the design systems team audited the colors we use in our web apps. We first looked all over the iOS and Android apps to broadly confirm which web content appears in-app. Then we built a custom source code search tool to find color values within the affected web apps. Then we identified equivalent color tokens for each color while also refining the search tool until its results were representative.

We found that migrating to dark mode compatible color tokens would ultimately span many codebases and teams. Thus, we put together a guide, which identified the existing colors and source files belonging to a given team, plus the recommended tokens to migrate to. We also beefed up our documentation, opened up Slack channels and scheduled open hours meetings to offer support and answer questions about the migration process.

With all of the above in place, different teams across the technology organization began migrating to color tokens. Beyond color token implementation, we also worked with publishing platforms teams and the newsroom to come up with solutions for making graphics and interactive assets compatible for dark mode.

Before launching publicly, we released dark mode as an internal beta to give ourselves the opportunity to test, catch and fix any bugs, as well as get others in the organization familiar with dark mode.

After launching to our users, the impact was immediate, and many readers wrote in thanking us for the feature. What started out as an initiative of one or two teams evolved into a massive effort across the technology organization and the newsroom, involving high collaboration between engineering, design, product, program management, product marketing and more.

Our design systems continue to evolve, and teams continue to build using our color tokens. We’ll also look to make improvements to the feature, ensuring teams on both the business and newsroom sides have the easiest way possible to make content dark mode compatible. Though dark mode is officially out to the public, behind the scenes, our teams continue to work towards making our NYT news apps the best possible experience for our users.

Natalia Bell is a senior analyst in the Data and Insights Group (DIG) supporting the Storytelling and Publishing mission, which aims to build a more dynamic, visual and trustworthy digital news report that transforms how our newsroom covers the news and our audience engages with us.

6:30 a.m.

Good morning, Los Angeles! ☀️

My mornings start early as a remote worker on the West Coast. I like to ease into the day with a Flat White coffee and then I wander around the block with my Goldendoodle, Arlo. I really admire how he relishes the world around him — with a big inhale of morning dew and a lingering smell of bunnies who roamed the lawn in the night!

7:30 a.m.

Log on ? I don’t need a reason to read The New York Times, but I love that it’s my job to be acquainted with the top stories. Sometimes I’ll use “The Daily” or The Morning or the home page to guide me. Or, I’ll go right to the source (an internal perk haha) with Live Admin, our internal platform where we build live news. Through the Live product, the newsroom can break stories in real time, report on them as they evolve throughout the day, or directly from reporters on the ground wherever the news takes place. Last year’s Met Gala was one of my favorite Live coverage moments.

9 a.m.

My brilliant teammate Gillian Eigo and I check in with our manager. After some very productive dog talk (I need to stay briefed on Gillian’s dog, Nena’s, activities!), we discuss the week’s priorities and project updates. Today, we discussed a presentation we will preview with our product partners in the next meeting.

9:30 a.m.

We meet with our product partners weekly to share work and make sure our priorities are in alignment with the team’s sprint goals. We have been working on a project to identify our most impactful behaviors to drive this year, and we’re ready to share our insights!

At my core, I am a storyteller. In this role, I really enjoy collecting a mix of information (like data, cross-team insights and enterprise goals) and reconciling it with a data story that inspires clarity and forward action. Hearing folks ask clarifying questions about our presentation makes me feel ? — like folks are following along and digesting our insights well. ?

10 a.m.

Now, I’m starving! I make papas con huevos and Facetime my grandmother. Working remotely has enabled me to spend more time with my family. Over the holidays, I lived and worked from my family’s home, and my grandmother frequently made me breakfast tacos (yes, I was very pampered). Especially as my grandparents and parents age, these little moments — un cafecito, un taco, a word of encouragement or sharing the same space — mean so much to me. ♥️

10:30 a.m.

A long stretch of time to work independently. During this time, I address some of the feedback received earlier from the presentation and explore a few questions raised. It’s common to receive questions without clear answers from our curious partners. So, I take a beat to consider what information and tools we have accessible to speak to them.

1 p.m.

Another meeting with our product partners! This time, they will be sharing recent trends and how experiments are performing. I enjoy this meeting because it prompts us to learn together, and I don’t need to prepare anything (haha). Instead, our partners use the tools and dashboards we built for them to explore and evaluate performance. I’m here more to listen, brainstorm ideas, and help with data interpretation.

While talking about a new feature, I’m reminded of how one of my tasks working in product analytics is the cross-functional collaboration with product managers, designers, engineers, researchers, editors and executives. So many people make the news report happen!

1:30 p.m.

Oh, hey, boss! I report to Arran Bardige, who leads data that informs newsroom and product leadership. His schedule is quite busy, but I admire how present he is to listen to what’s on my mind. Today, he suggested that I lead a workshop on presentational skills and storytelling for DIG. I block out time for that this week.

2 p.m.

I catch up on Slack threads by contributing things like data requirements, code reviews or discoveries from past projects. Naturally, I also brief myself on the latest in my favorite channels: #outdoorspeople, #dogsofnyt and #houseplants.

3:15 p.m.

Log off ⚪ Throughout the pandemic, it’s become a healthy practice for myself to consistently log off “on time,” and I’m glad my team supports that. It is also a way I try to demonstrate respect for my colleagues, many of whom are on the East Coast where it’s 6 p.m. Like many workplaces, we’re continually having conversations about working hours, when each of us works best and meeting the needs of demanding news moments. I greatly appreciate the ways my team is inclusive of me as a remote worker and the flexibility we extend to each other. I try to visit them in New York City at key moments and I look forward to seeing them soon!

But for now — I’m off to a boxing class ? and then writing my screenplay ✒️!

An in-depth look at how an animated image carousel was built to expose users to the breadth of NYT content.

By Elizabeth Levosinski

As an iOS developer at The New York Times, I’ve worked on building features using SwiftUI, Apple’s latest UI framework, in an ecosystem built with its original one, UIKit. Both of these frameworks are used to build iPhone, iPad and Apple TV applications.

Our team has turned to SwiftUI for building out new features because the framework allows a developer to build out features faster with less code. When it comes to implementing this framework, my colleagues and I are learning what it can support and what workarounds we need to build for our app’s needs.

Our team supports the Core News App, the flagship app of The New York Times, by continuing to build valuable new experiences and improve its technical quality for our users.

For this post, I chose an animated carousel I built with SwiftUI in the News app to document my thought process and technical challenges I encountered along the way.

Download this repo with a simplified version of the carousel (located under “projects/carousel”) to give more context as I explain its functionality in pieces. The demo emulates some of the same layouts as the NYT carousel with images from: https://placekitten.com/

CarouselView

CarouselView is a custom View that contains a TabView (the carousel) and initializes with the containing view’s size and paths to images. In the core news app, we use the TabView to showcase different news stories, including feature images from a story itself. The image above highlights the boundary of this feature.

Implementation of the CarouselView Inside CarouselView.swift:

The pagesState property contains models for the TabView created from the image paths. It’s set when the view first appears using the onFirstAppear view modifier. In SwiftUI a “view modifier” is a modifier you can apply to a view which will return a completely different view (or views) based on the values in that modifier.

Both @StateObject and @State property wrappers are used when a view owns the data. To keep track of the array of the TabView pages’ state I chose @StateObject instead of the @State property wrapper because the page data is an array of a more complex reference data type (Classes) vs a simpler value type like a String or a Bool.

struct CarouselView: View {.../// Models for the pages@StateObjectprivate var pagesState = PagesState()....onFirstAppear {pagesState.models = createModels(from: pagesContent)}...}

Note: Added @MainActor to any ObservableObject classes to ensure that the code will be thread safe and run on the main dispatch queue.

extension CarouselView {@MainActorfinal class PagesState: ObservableObject {@Publishedvar models = [CollagePageViewModel]()}}

TabView

The TabView builds each page from the pagesState.models array. This allows for the data to render as To make a TabView a paginating carousel add tabViewStyle(.page).

struct CarouselView: View {...TabView(selection: $selectedIndex) {ForEach(pagesState.models, id: \.pageIndex) { viewModel inCollagePage(viewModel: viewModel, totalPages: pagesContent.count)}}.tabViewStyle(.page)...}

selectedIndex property

The selectedIndex is updated whenever a user swipes through the TabView. Visually, the selectedIndex value is displayed in the pagination. It’s an important part of the carousel’s state, as it also triggers animations on each carousel page.

Subheading: Notes on using onAppear

Initially I used the onAppear method to detect when a TabView’s page was selected as I thought it would behave similarly to UIKit’s viewDidAppear method, which you can use to trigger logic when a view is added to a UITableViewController’s view hierarchy. However, after working with it and reading Apple’s documentation which stated when onAppear is called can vary:

“The exact moment that SwiftUI calls this method depends on the specific view type that you apply it to, but the action closure completes before the first rendered frame appears.”

This was problematic as sometimes onAppear would be called before the page’s view was displayed, triggering our animations too early so the user would never see them. So while I still use onAppear in other instances to trigger logic for our UI, be cautious when using it as when it’s called it can be unpredictable.

onChange(of: windowSize)

This ensures the carousel’s templates will be redrawn whenever the device’s windowSize updates (like when a user rotates an iPad) by updating the TabView view models each time.

struct CarouselView: View {.../// The device's window sizeprivate let windowSize: CGSize....onChange(of: windowSize) { newWindowSize inpagesState.models = pagesState.models.map { model inmodel.windowSize = newWindowSizereturn model}}}

Templates for Multiple Layouts

Our carousel needed to support several template layouts for iPhones and iPads in portrait mode and needed to be able to update to completely different layouts for iPads in landscape mode.

Initially I placed images using HStacks and VStacks with position offsets. HStacks are SwiftUI views that arrange their subviews horizontally and VStacks arrange their subviews vertically. However, with this approach I was fighting against layout constraints and eventually refactored the templates to place images inside a ZStack instead.

Inside CollagePage.swift:

...ZStack(alignment: .topLeading) {if !imagesLoaded.isEmpty {ForEach(Array(zip(viewModel.content.imagePaths.indices, viewModel.content.imagePaths)), id: \.0) { imageIndex, imagePath inlet imageLayout = CollageImageLayoutProvider(elementIndex: .init(rawValue: imageIndex) ?? .zero,elementCount: viewModel.content.imagePaths.count,isCompact: isCompact,isLandscape: isLandscape,windowSize: viewModel.windowSize)/// Is the current page the selected one or/// have the images already been loaded onceif (selectedIndex == pageIndex) || imagesLoaded.contains(true) {AsyncImageContainer(imagePath: imagePath, imageLoadedClosure: {imagesLoaded[imageIndex] = true}).positioned(with: imageLayout,isSelected: selectedIndex == pageIndex,isImageLoaded: imagesLoaded[imageIndex])}}}}...

To keep track of the loaded state of the AsyncImages, views for loading and displaying images asynchronously, I added a @State array object called imagesLoaded. To force the AsyncImages to not load prematurely in the TabView, I added logic to only load or display the AsyncImage if the page is currently selected or if the images have already been loaded once for that page. This solution forces the TabView to load its SwiftUI views more like the root views in UITableViewController’s viewDidAppear method, which I mentioned earlier.

Reacting to Device orientation changes

When the onAppear method is called, the isLandscape property is set. This property is used in the template logic to determine the position of images in the various template versions. I’m also updating this property when a user rotates an iPad using the onRotate view modifier.

Note: For this demo I used AppOrientationDelegate to lock the iPhone orientation to portrait because in the Core News App we don’t support landscape mode.

struct CollagePage: View {....onAppear {updateOrientationStatus(from: UIDevice.current.orientation)}.onRotate { orientation inupdateOrientationStatus(from: orientation)}/// A method that updates the 'isLandscape' state variable for determining portrait vs landscapeprivate func updateOrientationStatus(from orientation: UIDeviceOrientation) {isLandscape = orientation.isLandscape}}

Layouts Determined by Data

The template a TabView page uses is determined by the number of images passed in inside the CollageImageLayoutProvider struct.

Having the layout values change dynamically, based on the number of images, allows our code to be more reusable with a single source of truth rather than other solutions with more repeated SwiftUI view code. This also gave our designers more freedom to create many layout variations for this feature, that we could then update from a repo without having to create a new release.

struct CollageImageLayoutProvider {private enum Kind {case singlecase triplecase quad}let elementIndex: ElementIndexlet elementCount: Intlet isCompact: Boollet isLandscape: Boollet windowSize: CGSizeenum ElementIndex: Int {case zero = 0, one, two, three}private var type: Kind {switch elementCount {case 1: return .singlecase 3: return .triplecase 4: return .quaddefault:return .single}}}

Below is an example of calculating an image’s offset for a three image template. Each image’s offset is determined by:

• Template type (number of images): .single, .triple, or .quad
• isCompact: If the device is an iPhone
• isLandscape: If the device’s orientation is landscape
• elementIndex: The image’s index in the array when passed in

extension CollageImageLayoutProvider {.../// The x and y positions of each image in the templatesvar offset: CGSize {switch type {...case .triple:switch elementIndex {case .zero:if isCompact {return CGSize(width: 0,height: 0)} else if isLandscape {return CGSize(width: 0,height: (24 / iPadWidth) * windowHeight)} else {return CGSize(width: 0,height: (24 / iPadHeight) * windowHeight)}case .one:...}

The offset property’s width and height values are then used for SwiftUI’s x and y offset values inside a custom view modifier called CollageImageModifier:

.offset(x: layout.offset.width,y: isAnimatingPosition ? layout.offset.height : (layout.offset.height + layout.verticalAnimationOffset))

Image Animations

Each template image is asynchronously loaded and then animated with the CollageImageModifier’s “positioned” extension method. You can see these animations in the GIF above. They are vertically animating the images up while also animating their opacity by fading them in. I’ll explain how these animations are built below.

extension View {func positioned(with layout: CollageImageLayoutProvider,isSelected: Bool,isImageLoaded: Bool) -> some View{modifier(CollageImageModifier(layout: layout,isSelected: isSelected,isImageLoaded: isImageLoaded))}}

This view modifier gets its frame, offset, zIndex and animation values from the CollageImageLayoutProvider mentioned earlier:

private struct CollageImageModifier: ViewModifier {...func body(content: Content) -> some View {content.frame(width: layout.frame.0.width,height: layout.frame.0.height,alignment: layout.frame.1).clipped().zIndex(layout.zIndex).opacity(isAnimatingOpacity ? 1 : layout.opacityValue).offset(x: layout.offset.width,y: isAnimatingPosition ? layout.offset.height : (layout.offset.height + layout.verticalAnimationOffset)).onChange(of: isSelected) { newSelectedValue in/// Animates the page images when the page is selected or deselectedimageAnimations(isAnimating: newSelectedValue)}.onChange(of: isImageLoaded) { isLoaded inif isLoaded {imageAnimations(isAnimating: true)}}}...}

Above the isImageLoaded property triggers a fade in animation one time when the image first loads in the TabView. Watching for this change was necessary to ensure the animations didn’t fire too early.

private struct CollageImageModifier: ViewModifier {....onChange(of: isImageLoaded) { isLoaded inif isLoaded {imageAnimations(isAnimating: true)}}...}

Afterwards, every time a user swipes through the carousel, it updates the carousel’s isSelected property, which causes the onChange method to trigger a side effect for fading and vertical animations.

private struct CollageImageModifier: ViewModifier {....onChange(of: isSelected) { newSelectedValue in/// Animates the page images when the page is selected or deselectedimageAnimations(isAnimating: newSelectedValue)}...}

Below, the isAnimatingPosition and isAnimatingOpacity properties automatically animate the opacity and vertical positioning of the images because they are explicit animations. Explicit animations in SwiftUI are animations that react to specific state changes inside of the withAnimation function.

private struct CollageImageModifier: ViewModifier {.../// This property changing is what 'animates' the opacity/// of the image in the carousel@Stateprivate var isAnimatingOpacity = false/// This property changing is what 'animates' the position/// of the image in the carousel@Stateprivate var isAnimatingPosition = false....opacity(isAnimatingOpacity ? 1 : layout.opacityValue).offset(x: layout.offset.width,y: isAnimatingPosition ? layout.offset.height : (layout.offset.height + layout.verticalAnimationOffset)).../// Animations for the image's opacity and vertical positioningprivate func imageAnimations(isAnimating: Bool) {let animationDelay = layout.animationDelaywithAnimation(Animation.linear(duration: layout.opacityAnimationDuration).delay(animationDelay)) {self.isAnimatingOpacity = isAnimating}withAnimation(Animation.linear(duration: layout.verticalAnimationDuration).delay(animationDelay)) {self.isAnimatingPosition = isAnimating}}}

These are some of the main takeaways with SwiftUI that I learned as an iOS developer working on the News Core App. From building this feature in particular, I learned when it’s appropriate to use certain SwiftUI property wrappers, how to create dynamic templates which update on device rotation and how to use view modifiers to animate views each time a TabView’s state changed. I hope people can take away some useful information to use in their own work. Happy coding!

Improving visibility of our responsible disclosure program through implementation of the security.txt standard.

By Nick Leghorn

There’s really no replacement in the information security space for real people using live systems when it comes to finding vulnerabilities and bugs. No matter how many static and dynamic testing systems are used or how often we perform penetration testing on our systems, there are always going to be edge cases and interesting interactions that are only revealed when millions of live users are interacting with our sites on a daily basis. The challenge we face at The New York Times is encouraging our users to report the issues that they find.

So at the end of 2022, The New York Times Cyber Security Team deployed a single text file to the main nytimes.com website with the hopes of not only improving the ability for people to find out where to submit these kinds of vulnerabilities, but also encouraging more organizations to implement the same standard themselves.

Back when the internet was still relatively young, Congress passed the Computer Fraud and Abuse Act, a law that made it a crime to “exceed authorized access” of a computer system. It made logical sense — if someone maliciously hacks into your website, that should be a crime — but it came with an unintended consequence.

I recall experiencing this myself the hard way. One day back in my college years in the early 2000’s I was trying to pay some bills online through my bank’s online portal when I noticed an odd behavior — it wasn’t actually invalidating my session when I clicked the link to log out. I could head back to the account dashboard and see all of my accounts, interacting with them as if I were still logged in, despite having clicked the clearly marked “Log Out” button. Even worse: this persisted for a period of time, meaning that if I had just used a library computer to access my bank account, the next person to use that computer might be able to quickly pull up my account without the username or password.

It’s at this point that the unintended consequences of the Computer Fraud and Abuse Act kicked in.

As a concerned customer of this bank, my first instinct was to let them know about this issue by reporting it, but that can be a dangerous path. Back in 1995, Intel prosecuted one of its own employees in their IT division for performing some common security checks and finding weaknesses to report. One contemporary security researcher Jeffrey Kegler said at the time that “[i]f I saw someone on the Internet with a security weakness, at this point I would be reluctant to act the Good Samaritan and report it.”

That wasn’t an isolated instance — there have been a number of situations over the years where security researchers had been prosecuted for reporting security vulnerabilities, or even just the act of looking for them. That fact was front and center in my mind when I was trying to decide what to do with my newly found security vulnerability on that banking website.

So while I remain a free man after reporting the issue later that day, many other security minded users who might want to report vulnerabilities that they have found, the possibility of things going poorly and ending up facing federal charges has been and continues to be a very real specter haunting those potential reporters and preventing them from submitting their findings. (For those wondering, the bank fixed the issue quickly.)

You can’t fix unknown issues. Over the last few decades numerous scanning tools have come online from vendors, offering to try and find issues in your systems before the hackers can find them. The problem with these tools is that there is no room for improvisation or discovery — they only look for known issues that match a certain pattern. Security teams generally back this up with “penetration testing,” where they hire a team of humans to try and break into their systems. This is better, but once again these tests typically try to rush through the assessment to remain cost effective and only try a set number of exploits. The end result is a false sense of security, that the systems are fully hardened when in reality there might be something still lurking in the shadows.

The Infinite Monkey Theorem states that, purely by entropy, a sufficiently large number of monkeys and typewriters will eventually reproduce the complete works of William Shakespeare. The same kind of theory applies to finding vulnerabilities in systems: eventually, with enough people using the systems in different enough ways, some of them will find vulnerabilities, even if purely by accident.

Over the course of the last few decades the security industry has started to recognize the benefit of these kinds of stochastic good samaritan reports and has shifted to try and enable these individuals to report their findings in a safe and responsible manner. The result of that shift has been the rise of so-called “Responsible Disclosure,” which is a process where security researchers collaborate with businesses and organizations to report issues, help fix them, and in return gain reputation and credibility for the discovery.

Here at The New York Times, like many other companies with a large internet presence, the company has had a responsible disclosure program in one flavor or another for some time specifically designed to enable anyone on the internet to safely alert us to issues without fear of reprisal or prosecution. That mechanism has been used to great effect in just the time since I’ve been here, with some fantastic reports leading us to fix vulnerabilities that were so obscure and specific to our technology that finding them through other means may have been impossible. Unfortunately not every security researcher has been able to find the correct location to submit those reports, which results in more work for other customer-facing teams to handle and potential delays in assessing and remediating the issue.

The New York Times is not the only company facing this issue, with security researchers unable to find the correct location to report their discoveries or even understand if the organization in question has an appropriate program that will protect them. Within the last couple years a solution has started to emerge, and it is one that The New York Times has decided to sign on and implement: the security.txt file.

The concept borrows from another well established and well used practice of placing something known as a “robots.txt” file on the root directory of your site. This is a document that describes how search engines should proceed on your site, whether they are allowed to “crawl” the entire contents or if there are certain areas that they should not explore. These files are respected and honored by all of the major search engines, and have been around for quite some time. One of the reasons why the practice is so successful is the simplicity of configuration: the file is always in the exact same place for every site. Unlike needing to go and search for a privacy policy or similar document every time that could be in a number of strange locations, the search engine knows exactly where to find it.

With the security.txt file, the concept is the same — that it is a simple document placed in a well known location that should make it very easy for someone to find. The standard was recently proposed and documented on https://securitytxt.org/, and specifically intends to make it easy for someone (such as a security researcher trying to report a vulnerability) to identify how and where to contact the organization’s security team. It also should provide the location where they can go to learn more about the organization’s protection from prosecution or “safe harbor” that they offer for security researchers.

In September 2022 The New York Times joined the growing number of organizations who have deployed a security.txt file, which can be found on our website: https://www.nytimes.com/security.txt

We take the safety and security of The Times and our readers seriously. We make every effort to find and fix vulnerabilities, including ensuring that good samaritans wanting to report issues they find are able to do that quickly and easily. Through the implementation of this security.txt file we hope not only to make that process easier and more transparent, but we hope to encourage other organizations to follow this trend and deploy these files on their own sites.

How to cover more testing ground and discover new front end code issues by using both JSDOM and browser-based end-to-end testing.

By Katie Ishibashi

For a long time, all of the testing my team did was through React Testing Library, a popular framework that’s easy to use with the apps that my team works on. We’re a front-end team that serves as a bridge between a data team that analyzes large amounts of data about Times content and the audiences who read it and the internal stakeholders who want to see, for example, a list of the newspaper sections most popular among Gen Z readers. In broad strokes, that means we work on React apps that involve pulling in large amounts of data from an API and building a UI to display it for our users.for our users.The tests run quickly; we’re able to run hundreds of them within a few minutes. We can easily run tests with every pull request and catch bugs quickly, and having a thorough set of tests that we run all the time means there’s less opportunity for problems to sneak in.

This is where the idea to expand our testing framework came in. We added Cypress testing — not to replace React Testing Library, but to complement it. Cypress is not as quick, but it’s in many ways more thorough. Running Cypress tests means sitting and watching Cypress open a real browser, hit ouAPIs in production, and type text input into real fields. Using Cypress has made it much easier to catch entirely different sorts of issues with our apps. For instance, does a certain API take a long time to return? That’s something we’d miss with React Testing Library’s mocked APIs, but becomes obvious when we’re watching Cypress hit our real API in a browser. Using both libraries lets us cover a lot more ground, and feel a lot more confident that our code will work the way we expect it to.

How we use React Testing LibraryMuch of what our apps do is straightforward, i.e. given x input, our UI should display y results. This can make writing tests very straightforward. For example, let’s say our app is supposed to pull in a list of article topics and rank them by pageviews. To test that, we might mock a return from a topics API that looks like this:

const mockData = [{"cats":{"Pageviews": 9999999},"owls":{"Pageviews": 3},"dogs":{"Pageviews": 1234}}];

Assuming our app is pulling in this data and parsing it correctly, we’d expect its UI to look something like this:

Like many UI testing libraries, React Testing Library doesn’t actually render the UI we’re testing in a browser. Instead, it uses JSDOM, a JavaScript-based tool that parses HTML as a browser does, but without simulating layout, navigation, and other features. This is deliberately done for speed, and it works impressively well. We require all tests to pass before a branch can merge to main in our deployment process, and because JSDOM is so fast, it can run hundreds of tests in about two minutes.The tests run in a console, and since we’re using JSDOM, we don’t see anything that looks like what we’d see in a browser, just text telling us which test is being run, with a green check if the test passes and additional information if the test fails.

Since JSDOM is parsing HTML for us, most of our tests deal with checking to see what’s in that HTML. If we want to check to make sure our app is ranking our topics by pageview correctly, that would look something like this:

import { render, screen } from '@testing-library/react';describe('Table sorting', () => {it('Displays Results by pageviews', () => {// Have React Testing Library render the componentrender(<Table data={mockData} />);// Select the table rows by getting everything being rendered that has the ARIA label rowsconst rows = screen.queryAllByRole('row');// There should be 3 rows, because we've passed in 3 topicsexpect(rows.length).toBe(3);// Cats should be first because it has the largest number of pageviewsexpect(rows[0]).toHaveTextContent('Cats');// Dogs should be second, etcexpect(rows[1]).toHaveTextContent('Dogs');expect(rows[2]).toHaveTextContent('Owls');});});

Limitations of React Testing Library frameworkReact Testing Library works well as a means of running tests quickly and as a means of quickly finding and isolating errors. We know what the input is, and we know what the output ought to be. If I expect the UI to list “Cats” first, and then I make some updates to the code and suddenly “Cats” is last, it’s very likely that’s due to a bug I’ve introduced. Having these quickly running tests has been incredibly valuable, but it’s not as thorough as we wanted our test coverage to be. For one thing, 100% of our users experience our apps in a browser, but we were never actually testing them directly there. For another, the tests are not end-to-end tests — we are mocking API responses, never testing our real time API.

Enter the Cypress Framework

To more thoroughly test our apps, our team spent several sprints adding tests in Cypress, with the goal of complementing the tests we already have, rather than replacing them. It allowed us to zero in on testing angles we hadn’t previously thought of , and let us see the app in action the way our users do. Because we combined testing frameworks with different strengths, we were able to keep the benefits of both frameworks. We still had the speed of React Testing Library, and also the thoroughness of end-to-end testing.

Cypress lets us test the API from the the front end

This is especially valuable in our case, because the front end and back end of our app are developed by two separate teams, and adding tests that check how a UI made by one team interacts with an API made by another team helps ensure that the two function well together. For example, let’s say a user tells our app they want to search for topics popular among users Ages 35+ and also NOT Ages 35+.

When the user tries to submit the search, the API will (correctly) reject it, because it’s impossible for a user to be both over and under 35. Seeing this happen during testing provides us with an opportunity to optimize the front end — maybe this is something that we could handle before the user gets to the point where they’re submitting the search. We could flash an error on the front end right when a user first selects a group and also the opposite of that group, or maybe we could update the UI to make it impossible to make that selection in the first place — when a user is at the place where they’ve said they want to search for users Age 35+ and NOT [something], maybe we could remove Ages 35+ from the next drop down entirely:

Having a computer run tests in a browser is slower than having a computer run tests in JSDOM, but it’s significantly faster than having a human run those tests. Because we’re able to set up a series of Cypress tests and then quickly run them as many times as we want, it’s a great way of catching errors that occur 3% of the time. Is there a stray quotation mark that’s messing up our API call? This is a great way to catch it. Was there a change to the API so that a request that worked in the past no longer works? This is a great way to catch it.

Watching tests in a browser let us see things the way the user experiences it

Watching Cypress tests is a bit like watching a movie of your app. You actually watch a browser “select” search options from a dropdown, “click” on options, and “Fill out” a search form. Something’s always moving, and watching the tests is a great way of getting a sense of how pieces of the app move together.

This has been especially valuable in helping us get a sense of timing issues. If we’re waiting for real data to be returned from a real API, it’s much easier to get a sense of where the pain points are. Are we giving the user the option of submitting a search before the options that they need to search with have actually loaded from the API? That’s something that may not be obvious when we’re using mocked data that the app can use quickly, but it’s extremely clear when we’re watching the app wait for real data in a browser.

It’s also good for things that aren’t actually bugs, but can cause confusion. Does nothing appear to be happening on the screen after a user hits the submit button? Looks like we need to add a loading spinner.

Watching tests in a browser gives valuable context when tests aren’t passing

Many of our tests check for a certain DOM state after a user takes an action — i.e. a user submits some changes, a “Changes submitted” message should appear. When a UI test fails, it isn’t always immediately clear why, and we often want to investigate what is going on with the DOM for more information. By default, React Testing Library doesn’t actually render the DOM, but it does have a debugging feature called logTestingPlaygroundUrl which will load a snapshot of the DOM in a browser at the exact moment you invoke it. This has been invaluable for debugging, when the error occurs at that moment, but doesn’t help much when the error occurs before or after. It’s like looking at a still photo of an event, vs looking at a video. Sometimes, a still photo is what we want, but there are cases where it isn’t enough information.

For example, let’s say there’s a test where I’m trying to update a topic, and I delete its name without writing in a new one. The UI (correctly) flashes a message saying “Topics must have a name” and grays out the submit button. When I watch the entire series of events in a browser, it’s obvious why a user can’t hit the submit button and why we never get to the “Changes submitted” message.

The entire process is much more murky when we only have a screenshot of the DOM to work with. If I try the same test with React Testing Library, what is likely to happen is I see a message telling me that it cannot click on a disabled submit button, followed by a message saying that it never receives the “Changes submitted” message. If I view the changes on logTestingPlaygroundUrl, I’ll see that the Submit button is disabled and that there’s no success message, but I won’t have the context to help me see why.

Like all great things, this too has its limits

Unlike React Testing Library, Cypress tests take time to run, and aren’t well-suited to run every time before we merge a new pull request. They are powerful, in that they can test our actual APIs, but hitting our actual APIs sometimes means that they clutter the development version of our app with test data that we’ll never need again. If we have a Cypress test to make sure that our users can correctly create a new topic in our app, for example, that means we’ll eventually see our topic list cluttered with “topics” like CYPRESS-test-12349adksjhfas234987 that will have to be manually deleted from our dev server. At one point, I wrote a script to have Cypress hit the delete button to delete some of the topics it created, but our app is set up in such a way that users can only delete draft topics from the front end, not topics that are marked fully active. This means that we have to ask our back end team to periodically delete the topics Cypress creates.

The biggest obstacle to using Cypress, however, is the time that it takes to run the tests. While it is possible to open Cypress and have it run in the background while doing other things, a lot of its utility comes from watching different parts of the app interact in context. While it is a tremendous benefit, it is also a bigger time commitment.

Because Cypress and React Testing Library are so different, we’ve gained a lot by using them for different purposes. For example, we recently added a feature that would let us display the number of users in a given audience segment (i.e.there are 130 million users in Segment XYZ). Once we’ve set up the feature, React Testing Library makes it easy to test that we’ve set up the basic logic correctly to parse and display the data correctly. If our mock API says we have 130001892 users, does our UI correctly display “130M” in the correct location, rounding slightly as we’ve told it to? Great.

Watching the same portion of the app load multiple times in Cypress gives us a different kind of insight — our API is extremely slow to return the number of users in a group, but only on the first time we create an audience segment. After that, the API returns the number fairly quickly. This provides us with an opportunity to optimize what our users see — rather than making them watch a loading spinner over the entire page while we wait for the numbers API to return on segment creation, what if we load it asynchronously after a user saves a new segment? The number of users would take some time to show up, but at least they’d be able to use the rest of the page. The combination of testing approaches lets us be confident that our code is calculating the number in the way we’re expecting it to, and also flagged a non-breaking issue that might otherwise have taken a while to discover.

Our testing frameworks let us improve both code accuracy and user experience, enabling us to make changes with more confidence. React Testing Library’s speed means it’s easy for us to run tests with every pull request, so we’re able to make it a part of our everyday process. Cypress tests have become something we run periodically, whenever there’s a big change to the UI, or whenever we want to make sure everything’s ok when we make changes to the API. It’s slower than React Testing Library, but it’s able to hit multiple angles of our apps much more quickly than a human tester could. Together, these testing frameworks let us test our app much more thoroughly than either would alone.

By David E. Wheeler

When in the course of Engineering events it becomes necessary to make a decision between alternatives, how does a team go about it? Often a single engineer decides, based on their expertise and experience, and where the scope is narrow enough not to need the input of the whole team. But what about decisions that impact many teams, processes, future technology decisions, and organization-wide use-cases? Who decides then, and how?

The Identity Team at The Times, responsible for building and maintaining identity and authentication services for all of our users, has embarked on an ambitious project to build a centralized identity platform. We’re going to make a lot of decisions, such as what languages we should use, what database, how we can best protect personal information, what the API should look like, and so much more. Just thinking about the discussions and consensus-building required for a project of this scope daunts even the most experienced decision-makers among us. Fortuitously, a presentation at StaffPlus NYC by Comcast Fellow John Riviello introduced a super fascinating approach to collective decision-making, the Analytic Hierarchy Process (AHP).

AHP participants compare pairs of evaluation criteria to establish their prioritization. Then they compare each pair of alternatives for each criterion. For example, Wikipedia’s leader example compares four criteria (six pairwise comparisons) for three candidates (three pairwise comparisons for each of the four criteria).

We thought AHP might be a good tool to ensure a careful evaluation and choice we could be well-aligned on. But getting a sizable group of people together to run such a session on a slew of criteria and options will take some doing: they have to decide on the criteria and diligently evaluate each option in advance. Depending on the number of options and criteria, the geometric multiplicity can get quite large. A more detailed example for choosing a family car evaluates six models against 10 criteria for a total of 120 pairwise comparisons. It’s a lot.

The team decided to trial the process without making a huge time commitment. We just needed a more focused problem space to run through AHP. Fortunately, in designing the user schema for identity — which defines the types and structure of personal information about Times users — we have recognized the need for a new, canonical ID for New York Times readers. (Privacy is essential when handling identifiers. We won’t cover privacy here, but have written about how we think about it and how we manage it in the past.)

We decided to trial AHP by evaluating a short list of ID formats against an equally short list of pertinent criteria. Over several weeks of research and diligence, we added and removed criteria and options, and settled on the hierarchy in this AHP-style diagram:

Once we felt the options and criteria were sufficient and well-documented, we got the whole team together on a Friday to run through the AHP process. Here’s how it went. (Warning: lots of data and tables ahead; if you’re not interested in the details of how an AHP runs, skip down to Discussion for outcomes and conclusions. No judgment, this is an annoyingly long post!)

The evaluation criteria we settled on were:

• Database Support: Do database engines typically support it natively, either as an explicitly data type or via binary representation.
• Developer UX: How nice is the ID for use by developers? Is the format easy to work with, to copy-and-paste, and supported by libraries. Length is less of an issue, though shorter is still nicer than longer.
• Distributed Uniqueness: The ability to generate a unique identifier in a distributed fashion, among a number of distributed nodes, without collision.
• Ordering: The IDs are ordered and can be sorted, e.g. by time. Useful also for range partitioning but not hash partitioning.
• Randomness: Is the format sufficiently non-sequential or random to minimize the chances of ID discovery by cycling through IDs. Sequences, for example, are problematic because if a user’s ID is 93823, they can probably find that 93824 and 93825 are valid IDs, too. Even ordered but not-strictly sequential IDs are somewhat easier to discover than fully-random IDs.

For each pair of criteria, we discussed the relative importance of one over the other until we settled on a consensus score. The result is a matrix of pairs, with 1 assigned to the less-important of a pair and a number between 1 and 9 to represent the intensity of the importance of the other. This table (from Wikipedia’s leader example) defines the intensity of importance for each score:

For the criteria, our final scoring was:

Here’s how to read it:

• Developer UX is moderately more important than database support
• Distributed Uniqueness is strongly more important than database support
• Database support is slightly more important than ordering
• Randomness is slightly more important than database support

And so on. Plugging these numbers into Comcast’s AHP webapp, we get this nice diagram illustrating the relative weighting of each criterion based on the AHP calculations:

Distributed uniqueness was far and away the most important criterion. Developer UX was a distant second, with the others declining in importance from there. Here’s a sorted table of the weightings, which sum to 1:

Next, for each criterion we compared each pair of options, using the same numbering system. The options we settled on were:

• UUID
• Snowflake ID
• Nano ID
• XID

Plugging those numbers into Comcast’s AHP webapp gives us this final scoring:

UUID is the big winner here, with Nano ID in second place and Snowflake ID and XID trailing behind. Here’s a sorted table of the scores, which sum to 1:

There were a couple of surprises in the outcome of this process.

We had expected distributed uniqueness to be the most important criterion, given our plan to adopt a distributed database. We had not anticipated that developer experience would rank so highly. In discussing what “developer experience” meant, we expanded it (from the original notes about ease of double-clicking for copy and paste) to include library support across a variety of languages. This led to weighting UX higher than randomness, ordering, or database support.

This expanded interpretation of developer experience also unexpectedly allowed UUID, who’s awful default format is not double-clickable (try it on this — 3c893566-c125–4741-a68a-33e91410b7e2 — it won’t select the whole thing), to win out over Snowflake, which has an aesthetically pleasing string representation but does not have the broad industry and library support of UUID.

As for the final analysis, we had expected UUID and Snowflake to end up the top two choices, and could not have said which one would win. Finding UUID to be the clear leader was unsurprising, but Nano ID handily defeating Snowflake was a real curveball. The explanation for this result almost certainly falls entirely on Snowflake’s poorer showing in the distributed uniqueness criterion. Yes, it very much supports uniqueness in distributed systems, but the need to configure unique node IDs for each instance generating Snowflakes, together with its expiration date (Snowflake IDs cannot be created beyond a certain date, generally 69 or 174 years) led to UUID and Nano ID freezing it out: neither expire or require configuration.

Personally I’m glad Nano ID lost to UUID, because Nano has quite poor database support and is much larger than UUID (21 bytes vs 16 bytes). I was rooting for Snowflake due to its small size and efficient algorithm, and because I’m not fond of UUID’s default string format. But I take consolation in the fact that we don’t have to use that format. If we Base62 encode a UUID, rather than a typical hideous string like 3c893566-c125–4741-a68a-33e91410b7e2, we can have ifWIoI9ZU00gOqkgNrmE5B. Much more compact (22 vs. 36 chars) and double-clickable (no dashes — try it!).

One way or the other, Identity user IDs will be UUIDs. And this exercise allows that choice to be uncontroversial among members of the team.

We consider the AHP experiment to be a great success. The team enjoyed the process, as it gave us a chance to closely example the criteria and options, to better understand the features and trade-offs of each option, and to see our fine-grained analysis nicely captured by the algorithm. As we made our way through, it became pretty clear what the winner would be, and that’s a good thing, since we gave all of the options a proper airing and felt that the objective result reinforced our findings.

We plan to use AHP next to select a distributed database for the platform. We’ll likely have more criteria, options, and stakeholders, so perhaps we’ll need to take a full day to run through it (as opposed to the 90 minutes or so for this one). But the diligence it requires and consensus it builds will allow us to be confident in our conclusions, satisfied by the result, and agreed on the choice.

I’d like to thank the members of the identity team for gamely running through the AHP, including Brigitte Lamarch, Pete Saia, An Yu, Alex Detrick and Jethro Chu. Deep appreciation to Penina Kessler, Danielle Roades Ha, Shilpa Kumar, Shawn Bower, Tanya Forsheit and Robin Berjon for the close reading of drafts of this post that made it so much better.

By Jasmine Chan and Pamela Bergson

This is a story about adopting new frameworks and using new technologies in a highly collaborative culture. We know that adopting new frameworks for your team is a continual challenge. For that reason, it’s often preferred to wait until engineers are working on a greenfield project, rather than working within legacy code to implement a new framework. At The New York Times, senior Android engineers, Omar Miatello and David Morant, solved this by leveraging our team’s existing knowledge sharing session to begin using a new framework, Jetpack Compose, in production.

Jetpack Compose, a declarative UI framework introduced by Google in 2019, aims to simplify and speed up UI development time by allowing developers to define their UIs using a declarative style. This simplification caused a lot of excitement among the Android community, and the android engineers at The Times wanted to learn more about it. Given the new framework was still being worked on by Google, the implementation of the framework was fluctuating rapidly, and more capabilities were still to come. For example, animating items in lists, or transitioning between screens. Omar noted, “at the beginning there were some issues for everyone on the team. We’re used to thinking in a certain way, but with Compose it changes how you think about components because now, you have to think of building more reusable components instead of something that is for a certain case.”

Many developers learn about new technology frameworks building side projects in their spare time. Omar did so by migrating his existing side project, an app that organized tasks at home, to the Jetpack Compose framework. However, most engineers don’t have time to create substantial projects beyond their day to day work. Fellow senior engineer, David, found it challenging to get into deep learning with a side project in addition to doing his day to day role. When he did have some spare time, he would attempt to start a side project, but ultimately had his role’s work to prioritize when it came around again, and he would lose momentum on his side project.

To resolve these time constraint issues, David and Omar were thinking of ways to create a consistent space for everyone to learn Jetpack Compose. David was a leader of our Mobile Community of Practice, a forum that enables all the Android and iOS developers at The Times to share knowledge and understand what each team is focused on. Out of this meeting, David and Omar brainstormed and moved forward with a weekly Jetpack Compose Club meeting.

The club started out as a weekly space for engineers to do a show and tell on how they used the technology. We evolved it to become an introductory series, where the group would explore a pull request and go through key concepts by walking through the code and explaining what the goal was and also delving into the documentation that Google had about Jetpack Compose. The club further evolved to become a collaborative space for different side projects where they would choose a specific problem and swarm on implementing a solution with Jetpack Compose. This latest iteration of Compose Club is flexible for participants to attend any club meeting without missing out on new learning, so many of our engineers are able to come together and learn from one another. It has enabled engineers to learn how their colleagues could learn Compose concepts by applying them to a real life project they were working on, and thus, improved their own confidence in sharing their newfound knowledge of the new framework with their own teams.

Aside from the obvious benefits of having a collaborative space like this to learn from one another, it’s also just more fun, exciting and interesting to learn in a supportive group versus independently.

In addition to leading the Jetpack Compose Club, Omar and David were then able to identify opportunities to use the framework in our core New York Times mobile application and bring together mobile engineers at The Times to help in reviewing their pull requests that used the new framework, allowing for more constructive feedback from their colleagues. Omar continues to use Jetpack Compose in The New York Times mobile app, while David has transitioned to the Audio team, where he’s building a greenfield app entirely with the same framework.

We composed a community so that we can learn together and gain confidence with the new framework. We got that and so much more. The community continues to meet together weekly to learn and exchange ideas. Their most lasting contribution to the apps is likely to be not the code they’ve written, but the community they’ve helped create.

If you’d like to hear more about what our Android engineers are up to at The New York Times, David Morant and other engineering colleagues will be presenting talks at Droidcon NYC this September. Come say hi if you’ll be attending!

“Meeting …” is an ongoing series that features colleagues from different corners of The New York Times Company.

The NYT Open Team

Feb 15 · 14 min read

The technology behind The Times is driven by the contributions of different technologists across the organization. For Black History Month, we are highlighting a few of our Black colleagues who have helped shape The New York Times Company with their work, from engineering to information technology. Learn how they describe themselves in this ever-changing industry.

What are your pronouns? She/Her

What is your title and what does it mean? QA engineer on the photo team.We work on applications, most recently Loupe, that are used by the newsroom to build high-quality visual stories.

How does your identity shape the way you think about technology? How do you approach your career? How do you uplift others?
Technology plays a significant role in my personal and professional life. From my perspective, it allows me to communicate, connect and learn. When my career first began more than 15 years ago, there weren’t many people who looked like me in my profession. The industry has evolved since that time and I am optimistic about diversity continuing to improve in the future. I approach my career with the understanding that my point of view and authenticity as a Black woman is valuable. When it comes to uplifting others, I like to keep it simple by being supportive and positive. I listen. I don’t judge. I show empathy.

What is a passage or quote (from a book, movie, text) that has stuck with you? How do you apply it to your life?
“Don’t Postpone Joy” has been my motto for a long time. Since the pandemic this quote has become more meaningful to me. I wake up every morning and ask myself “How can I bring joy to my day?” Joy doesn’t have to be extravagant, it can be a trip to my local cafe to get my favorite latte, going for a run on the boardwalk or cooking one of my favorite meals.

What is the approach to mental health in your culture? What are some methods that you practice to look after your mental health and well-being?
I believe there is an awakening happening in my culture where my generation is more open to discussing mental health and exploring the methods available to them.

Dealing with mental health/well-being during a pandemic has been a journey. However, I’ve incorporated methods that help me keep peace of mind. Here are some:

• Daily ritual that starts with prayer, meditation and journaling.
• Moving my body by running, and practicing yoga.
• Learning new skills. I took tennis lessons last year.
• Watching a funny show (“Golden Girls” fan here) or movie. Laughter is a form of therapy for me.
• Spending time with family and friends.
• Helping others!

What are your pronouns? He/Him/His

What is your title and what does it mean? Senior software engineer, audio Android app. We are building a new app that brings our new New York Times audio experience to Android users (coming soon!).

What is a passage or quote (from a book, movie, text) that has stuck with you? How do you apply it to your life?
“If you’re a high visibility player, it’s on you to move the community in a positive direction” — Jonas Neubauer

Not from a movie or book, but from a former Tetris world champion (I waste a lot of time on the internet). I try my best to be welcoming and encouraging to people in any space that I feel comfortable in. If I can feel comfortable, maybe I can help make it easier for others.

How has your career shaped your understanding of the world and vice versa?
When I first started working in a large codebase I thought that the code was sacred and it must be flawless because it was here before me, and I assumed that everyone else agreed. If I didn’t understand why we were doing something, I assumed I was always wrong. But now I realize that they were just decisions that were made by other people just like me, and we can change it if it doesn’t suit us.

What are your pronouns? She/Her

What is your title? Creative technologist consultant on the growth team.

I provide technical solutions to stakeholders for monetization of our digital products. I consult within early pitch phases of a project to research and determine technical execution methods and feasibility. I collaborate with teams of developers, marketers and vendors; interfacing with production and creative departments from concept to delivery.

Are there any passion projects that you work on during your free time?
One of my passion projects is a platform and networkthat I have created whichgives you the self-evaluation tools, industry information and social access to strategize your next career move, and make an impact in the tech industry. Helping people, especially individuals from marginalized communities recognize their place in tech is a life mission of mine. I recently authored a Career Journal Planner that will help the user make a decisive move toward a new and exciting career in tech. I believe that there is a place in the tech industry for everyone and my goal is to help the user discover that place within the ecosystem.

What is a passage or quote (from a book, movie, text) that has stuck with you? How do you apply it to your life?
“Comparison is the thief of joy.” Not one path in life is the same and I apply this to my life and career as well. Speaking at various conferences in the past, I have been asked “What is your superpower?” My response, “me.” You are your own superpower. What you bring to the table is your own magic that you add to all your work. I often remind myself that leaders become great not because of their power but because of the ability to empower.

What are your pronouns?: He/Him

What is your title and what does it mean? I’m a senior software engineering manager for the User Experience Foundations (UXF) home and UXF sustainability teams. The UXF home team is responsible for the home page, and the UXF sustainability team solves unowned problems to unblock teams in the UXF group.

How does your identity shape the way you think about technology?
A friend recently pointed out that visually distinguishable minorities often feel like they constantly have to prove themselves and to continually earn their place in the room. I’ve realized that this is true of me as well, and it sits at the core of how I think, driving my choice of a major in college (economics), my choice of career (software engineering/engineering management) and the structure of my daily thoughts. This deep-seated insecurity is directly driven by my identity as a Black man in America, and for better or worse, has shaped my passion.

While I enjoy problem-solving and I take pleasure in writing elegant code and achieving strong technical craftsmanship, what I am really passionate about is creating value. I chase the most valuable problems, wherever they may be.

How has your career shaped your understanding of the world?
Being a software engineer helped me see how to organize things to make life easier and to make little chores more efficient. For example, instead of putting them in a stack, I store my T-shirts in a directly addressable “data” structure which gives me both high visibility into all my options and direct access to remove any of them while minimizing the disturbance to the others.

Becoming a manager forced me to harness empathy and become more emotionally self-aware. This initially helped me to become a better leader, but at the same time also made me a better partner and a better communicator in all aspects of my life.

And vice versa?
Years ago, my wife and I were resident fellows responsible for an ethnic theme dorm at Stanford. That experience taught me how to identify and solve hidden D.E.I. issues — we successfully campaigned to change university policy so disadvantaged students would have equal access to meals. As a manager, the same skill helps me notice and rectify hard-to-see imbalances that if left unchecked would let an unfair system propagate.

What are your pronouns? She/Her

What is your title and what does it mean? Data and insights manager on the messaging and personalization (MAPS) team.I work on interpreting trends in user behavior and results from analyses and A/B tests for our newsletter and push notification products to help inform product strategy decisions.

How does your identity shape the way you think about technology? How do you approach your career? How do you uplift others?
As a Black woman and a former policy student, I think a lot about access, and how the smallest decisions can include or exclude whole populations: from the colors we choose on websites and whether we capitalize words in hashtags to choosing to open a cashless business in a largely unbanked community. I try to mitigate how I contribute to these types of access barriers by surrounding myself with people who have wide-ranging backgrounds and experiences that I can learn from or use as sounding boards. More personally, I have an incredible amount of respect for people who have nontraditional career routes, and try to share information about technical programs and preparation tools I know of whenever helpful — it makes me happy to help other people get to where they want to be. I am also a big evangelizer of the importance of user research and understanding end users.

Do you think tech literacy is important for future generations?
I think tech literacy is important for every generation. There are too many nefarious actors who are willing to take advantage of people who don’t understand things like internet security, the importance of reading/skimming terms and conditions, or how to choose passwords that aren’t easily hacked. As more and more of our lives end up online, and more data is made available through open data initiatives and FOIA (Freedom of Information Act), it’s important to know how to protect ourselves, our identities and our rights, as well as how to utilize all the opportunities that are becoming available to us.

What are your pronouns? She/Her

What is your title and what does it mean? Senior QA engineer, engagement, I test software applications and features.

How has your career shaped your understanding of the world and vice versa?
My career has allowed me to constantly view things from another perspective. It’s one thing to find things useful and positive from your own point of view, but how useful and positive is it for everyone else? We don’t just build software and technology for the pleasure of a few, we build it for everyone to experience. When testing software, the definition of a “good” experience can be subjective to the person using the application and they’ll form their own opinions on what that actually means to them. You just have to take whatever feedback given and try to apply it in a positive way to make it better — in QA testing and in life.

What is the approach to mental health in your culture? What are some methods that you practice to look after your mental health and well-being?
Honestly, I don’t think mental health is taken seriously in my culture but this is true everywhere. We need to give people the safe space to express how they’re really feeling about themselves, their lives and what they are going through without judging. Life is hard and life is always challenging people in ways they can’t cope with. When it comes to my own mental health, I try to take things one day at a time, be thankful for the little accomplishments and positive things going on in my life. If I’m not feeling right mentally — it’s okay to NOT be okay and step away for a bit or take care of myself. Admittedly this is something I had to learn later in my life and am still working on. Sometimes I use my art as an escape — it’s not just something I enjoy doing, it’s often the thing that I go to sort my feelings either by drawing, creating things or writing stuff down. It keeps me grounded. We’re taught to suck it up and deal with issues on our own — but our well-being is important and we have to give ourselves permission to step away when we need time to heal and reflect.

What are your pronouns? He/Him

What is your title and what does it mean? Technical support specialist, end user support team, provide hands-on IT support and solutions.

What is a passage or quote (from a book, movie, text) that has stuck with you? How do you apply it to your life?
“Where your fear is, there is your task.” — Carl Jung. I often recite this quote to myself whenever I have any doubts about starting a new project, completing a task or simply sparking a conversation with someone new. Sometimes we let our fears, whether big or small, enable our procrastination, and this quote is a reminder to just get up and do it … “If I fail, I fail, it’s not that deep.”

What is the approach to mental health in your culture? What are some methods that you practice to look after your mental health and well-being?
As a first generation Haitian American, I find that mental health isn’t widely acknowledged in our culture. Having immigrant parents that have worked hard to establish themselves and provide for our family, it was often expressed that we cannot be depressed living in America. One just has to work hard and essentially “get over it.” Today, it seems that mental health from one’s personal life to work is taken into consideration. Simply going for a drive to reflect or unplugging myself from technology for a weekend are some ways I look after my mental wellbeing.

What are your pronouns? She/Her

What is your title and what does it mean? Program manager for the RISC department (technology risk, information security, and compliance). I work across all our RISC teams and manage activities and deliverables between multiple projects. I track our milestones and ensure we’re meeting our organizational goals.

Are there any passion projects that you work on during your free time
I’m passionate about classical/tribal African art and artifacts (figures, masks, sculptures and textiles). I’ve been studying pieces from cultures across the continent and I’ve fallen in love with understanding the history, intention and diversity of African art. Early on, I recognized that there are few Black dealers, gallerists and even collectors of classical African art. This was disheartening for many reasons but gave me the inspiration I needed to start sharing my interest with others like me. In 2020, I started a resource for learning about art from the continent. My goal is to expose more people of African descent to our art and artifacts with the hopes of more Black people collecting African art.

What is your ethnic background and how do you celebrate it?
I am Ethiopian-American. I was the first in my family to be born outside of Ethiopia. My parents were refugees and left Ethiopia in the early 1980s. Although they have lived outside of Ethiopia for about 40 years, they have very much kept their culture and instilled that into our family. My love for art and having a deeper understanding of history, culture and traditions comes from them.

What are your pronouns? She/Her

What is your title and what does it mean? Associate product designer, customer care. I help design experiences that allow our customer care agents to assist customers in a more efficient way.

Are there any passion projects that you work on during your free time
I’ve been working on a social enterprise project that partners up with Ghana-based artisans to create sustainable women’s clothing. It’s still very much in its infancy but the hope is that by supporting the local artisanal sector and creating businesses that operate fairly, we’ll help build up the Ghanaian economy. I was born in Ghana and spent the first 13 years of my life there so I’m very passionate about supporting the local economy there.

How does your identity shape the way you think about technology? How do you approach your career? How do you uplift others?
Being a Black woman makes me very much aware of inclusivity or the lack thereof in every space I find myself in. As such, I’m always considering how technology can be used as a vehicle to foster more inclusivity. Specifically within design, thinking about how to design experiences that are equitable for all our users with different needs. I also think about how I can contribute to the creation of a work environment that allows people to bring their whole selves to work. Practically, that looks like listening more and allowing others to shine, giving credit where it’s due, including others in the conversation.

How do you prioritize yourself and your family?
I try to be more intentional about not allowing work to creep into my personal time with family and friends. I try to work on other projects that are not about profit or advancing my career — things that I just want to do to have fun. I paint a lot, I find that very therapeutic. I’ve found that really questioning my intentions for doing the things I do allows me to be honest with myself and confront myself if I realize that I’m attaching my sense of identity to my achievements or the things I’m doing.

What are your pronouns: She/Her

What is your title? Associate software engineer on the accelerated digital growth team in care. I work on creating tools that assist customers with their online subscription experience.

How do you prioritize yourself and your family?
I prioritize myself by unplugging myself from social media and turning my phone off. It is very easy to get lost online comparing yourself to others and how much they have achieved in a short amount of time. I usually spend time with my mom and we’ll just relax together and pre-Covid, we would sit in the park and enjoy nature.

What is a passage or quote (from a book, movie, text) that has stuck with you? How do you apply it to your life?
“Stay gold, Ponyboy. Stay gold.” — “The Outsiders”

This quote stuck with me because it is so easy now to get wrapped up in a false reality due to social media. Many people become shallow and narcissistic and this is seen as the new norm. So I always remember to stay true to myself and enjoy the simpler things in life.

What is your ethnic background and how do you celebrate it?
I am Haitian, and one way I celebrate being Haitian is a big bowl of soup joumou on January 1st.

We’re hiring. Apply to work with us here.

How the New York Times Engineering team built an application catalog to keep track of our software systems.

Jul 22 · 6 min read

By Thilak Subramanian and Shesh Patel

When The New York Times first joined the world wide web in 1996, there were a handful of engineers who maintained the website and publishing software. The number of applications that fueled The Times’s online presence were limited and easy to keep track of.

Fast forward a few years and The Times’s digital portfolio has grown to include the website, mobile apps, publishing software and stand-alone products like NYT Cooking and NYT Games. The applications we use to support all of our digital services now number in the hundreds. Similarly, our technology team is now the second largest department at the company.

All of this growth means that sharing information about our applications by word of mouth is unsustainable. This is most pronounced during P1 incidents when teams need to find information about the service that is causing a problem as quickly as possible.

As The Times invests in digital products and continues to grow our engineering teams, we realized we needed a centralized source of information about all of the software systems we use. Improving the visibility of the systems our organization owns would only make our products better and the lives of our colleagues easier.

We had an idea for how we could integrate this resource with teams’ daily workflows, but we were not sure whether they would be able to keep it updated. So, we did what many engineering teams at The Times do when starting a new project: we wrote a Request for Proposal (or, RFP) and talked to engineers and product managers from different teams to understand their needs.

In our conversations, we tried to understand the pain points teams felt when looking for information about an application. Our RFP was useful in gauging whether teams were interested in our solution and whether it could actually address their problems. The responses to our RFP were overwhelmingly positive, so we began brainstorming what an application catalog might look like.

With the idea of an application catalog validated, we held interview sessions with a wider group of people across The Times to understand what details they might want from such a centralized resource. We prepared a questionnaire to guide our conversations.

At the end of these sessions, we had a list of metadata that our colleagues thought would be important. The metadata included a roster of applications we own; what an application does; whether an application is a critical system; the technology an application uses; contact information for the team that owns an application and where people can find information about it. This list informed what we decided to include in our MVP.

We chose our own team, Delivery Engineering, as the stakeholder for the MVP because we wanted a quick turnaround on feedback in order to demo the catalog for other teams; this is a process we often follow on projects within our team.

Some of the important things we considered while building the MVP included:

Adoption Friendly: Not only were we seeking to build a list of existing applications, we would be asking colleagues to change their habits while building new ones. With that in mind, it was important for us to make it easy for both engineers and non-engineers to contribute information to this resource. We decided to create a yaml file that would live in the Github repository for this project; anyone on a team could add metadata to the file.

Data Integrity: Keeping the data up-to-date was a goal of ours from the beginning of this project. We knew this had to be a shared responsibility, so we baked in some features to remind teams to update the metadata file on a periodic basis. Also, we constantly looked for ways to address this problem which resulted in the outcome that’s outlined below.

Automatic data retrieval: We constantly looked for ways to trim the metadata file that we requested teams to provide by asking ourselves if this data is something we could get on our own. If that’s something that could be achieved programmatically then we set out to do this. This resulted in a win-win situation for us.

For the technology stack, we chose to use Python, GoLang for the back-end service and MySQL for the database. These choices were fueled by The Times’s existing support of these languages through numerous communities of practice in our Engineering group. We used Python primarily because we wanted to reuse one of the existing services that our team supported. For deployment, we used Google Kubernetes Engine.

(We have since rewritten the Python service in GoLang, separated the front-end component to use ReactJS and we switched to PostGresSql to leverage its document storage and relational capabilities.)

We shipped the MVP and demoed it to a group of colleagues. The response to MVP was overwhelmingly positive and we decided on the next steps to move us forward with a general availability launch.

[If this work is interesting to you, apply to work with us. We are hiring.]

To prepare for the general availability launch, we reached out to many teams in the engineering organization and wrote user documentation. We leveraged The Times’s annual Maker Week, which allows engineering, design, data, project and product colleagues to focus on self-directed projects for a week, to talk about the application catalog and how teams might contribute to it. During Maker Week, we held workshops to help teams add their applications to the catalog — teams that participated were entered into a raffle for a gift card. Through our efforts, we onboarded 100 applications in the first 30 days.

The Application Catalog has been in use for a year, with more than 250 now listed. To understand how we might take this product to the next level, we have help feedback sessions with even more colleagues. From these discussions, we have put together a list of improvements that we will implement in the coming months, including build details and metrics; incident management for teams; the contact information for the person on call; status checks; and an inter-application dependency visualization.

We plan to enhance the user interface with a simple form that colleagues who do not use Github in their day-to-day operations can use to submit their applications.

In the future, the catalog will serve as an integral part of the day-to-day needs of developers at The Times. We envision the catalog as the foundation through which developers can create applications using application reference templates, which would seamlessly add metadata to the Application Catalog. The catalog could also be a centralized place for API documentation. This, among other things, will reduce tool fragmentation and toil on developers so they can focus their full attention on building applications.

“Meeting…” is an ongoing series that features New York Times employees from different corners of the company.

Jul 20 · 5 min read

What are your pronouns?
She/Her

What is your job title and does it mean?
My title is Managing Director, User Insights & Strategy. I’m in the Audience Insights Group, and our mission is to help The Times grow by understanding our audience. I currently lead our work for NYT Cooking. I also help teams across The Times with workshop design and meeting facilitation.

How long have you been at The Times?
Just about seven years.

Most Times employees are working remotely right now. What does working from home look like for you?
I live with my family on a relatively quiet street in Manhattan. In these work-from-home days, I’m usually sitting at a table in our living room that doubles as the family weekend craft table. I might be sharing the space with polymer clay, casting resin molds, needle felting tools, X-Acto knives or various types of glue. I love making things and I teach art workshops whenever I can.

Aside from the creative spark I get from being in the “craft zone,” the second best part of sitting at the craft table is that it’s next to a window that overlooks one of New York City’s treasured sidewalk trees. Over the last 18 months of working from home I have enjoyed watching the birds that stop by “our tree” on their way through Central Park. Other times, the scene is less bucolic: the other day, we were witness to an intense pigeon fight for space on an outside window ledge. The city can be a tough place.

How do you start your day?
My workday usually begins on paper. I am a devoted fan of the Wirecutter-endorsed Hobonichi planners. Most mornings, before I begin work at the computer, I sit with my planner, a .38mm navy blue gel pen from Muji (seriously the best pen), and a cup of tea (PG Tips, brewed strong, one sugar).

I enjoy taking this time to think about my day before Slack messages and emails start piling up.

One thing I have learned over time is that good habits are more likely to stick if you find a way to make them enjoyable. But like so many other things, this often sounds easier than it is.

What is something you’ve worked on recently?
One recent project I led was to understand — and help us build for — the first seconds of a user’s visit to one of our recipe pages. What is the very first thing people look for? What do they need to see before they commit to reading through the rest of the page? (Spoiler: it’s always about timing and ingredients.)

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
For a few years I focused on our global growth efforts, and as part of that, I was on the team that launched “NYT en Español,” a Spanish-language digital publication that was based in Mexico City. We had an amazing team and I’m so proud of the work we did together.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently?
I have worked in what we now call “digital products” for over 20 years. During that time, I have been a front-end developer, a visual designer, an information architect, a product manager and for the last decade, primarily a user researcher. At every step along the way I have been part of an emerging discipline.

There’s never been a roadmap for what a long-term career looks like in this space. We are all figuring it out as we go. It’s been fun, but also a challenge. Sometimes I look with envy at friends who have more clearly mapped out professional paths. But I don’t think I would have changed anything along the way.

What was your first job?
In high school, I worked at a frozen yogurt shop in an Atlanta strip mall. A highlight was serving a perfectly swirled frozen yogurt to Maynard Jackson who, at the time, had just finished his third term as the mayor of Atlanta.

Frozen yogurt is, in my opinion, the best fast food job because you don’t have to fry anything, which means it doesn’t smell bad and the skill of getting that perfect swirl on top of a cone will serve you well whenever you have access to a soft serve ice cream machine at another point in life.

What is something most people don’t know about you?
I think a lot of people know this, but I’m a big fan of taking on experimental side projects. For example, over six months in 2019 I wrote and distributed a monthly zine that included recipes, interviews, contributor essays, illustrations and more, and pulled them into a book that I shared with friends and contributors.

Do you have any favorite life hacks or work shortcuts?
When you’re feeling stuck, stop staring at a screen and switch to paper. Or, find someone to explain what you’re working on and talk out loud about where you’re stuck. These are simple things, but they work.

What is your superpower?
I’m a good listener. As well as being a researcher, I’m also a certified Co-Active Coach. Coaching, like user research, is 90 percent about listening.

What or who are you inspired by?
I love any documentaries or podcasts that go deep into an artist’s creative process. Song Exploder is a particular favorite.

Complete this sentence: Over time, I have realized __________.
That you have to deliberately create the right circumstances to think deeply and clearly. A mentor once told me, “If you do everything right, you just need 45 minutes of thinking a day.” The corollary is that if you don’t make time to think, you won’t have it.

What is your best advice for someone starting to work in your field?
Stay curious. Be helpful. Don’t forget the big picture.

Thea Lorentzen, product design director: Generally we think of Live as any dynamic unit of coverage, whether it’s appearing in a single format or distributed across multiple places. This could include giving readers the latest updates or providing reactions right when something happens. It can also mean using visuals to transport readers to a scene.

Chen Wu, senior product designer: We have Live Blog, Live Briefing and Live Chat. And, we have this tool called Live Admin that enables editors to publish to the Blog and to the Briefing. The Chat is what we used during the 2021 United States presidential inauguration where reporters provided live commentary and analysis. The Blog is more for news events where we have reporters on the ground and they can send minute-by-minute updates. The Briefing is small articles and not necessarily minute-by-minute updates.

Peter Rentz, principal product designer: Blogs are really great for events that are happening quickly, whereas Live Briefings are a way to synthesize the event, as opposed to giving minute-by-minute updates. Oftentimes, we use them at the same time.

Sarah: What is the design workflow?

Chen: We do both short-term and long-term work. Sometimes when there’s a short-term need, we’ll do fixes and improvements very frequently, or we’ll design short-term features that will improve the editorial workflow. The long-term work might entail a weeks-long process: brainstorming and coming up with ideas, then sketching and testing them.

We work very closely with the editors responsible for publishing content and also the people who train the editors on the Admin. For example, when I need to design a feature for the Admin tool, I work with the editors who do the training so I can understand the problems they are having. Once I have a design, I will show them and they will give me feedback. We also work with the research team on user testing and research on reader-facing products.

Peter: We sometimes try things that won’t impact editor workflow, but will hopefully contribute to a great reading experience. During the Derek Chauvin trial, we quickly designed a pinned-video experience on the blog, for example, that did not impact the editor’s workflow but provided a way to watch and read within the same screen.

Longer-term features require thinking about the reader-facing product and how editors and reporters could produce it.

Sarah: What does a very short-term design process look like when you’re reacting to breaking news?

Peter: It’s really hard to do breaking news, but it’s easier for events we expect and know how we are going to cover them. Pinning video was something we didn’t have and we thought it would be really helpful to readers who were following the Chauvin trial. We did quick sketches on what we wanted the experience to look like and then looked at how difficult it would be with the current tools. Ideally, we would spend a lot of time on the video player, but we were able to do it in a way that didn’t impact the video codebase in a major way.

In real breaking news coverage, it would be risky to introduce unvetted features because we wouldn’t want to impact editors’ workflow, or the reader experience, in those high pressure situations. But with a planned event, like an election, we can do a test run and make tweaks if we need to. Planned events are really helpful because you get to test out the full vocabulary of the Live coverage.

Sarah: Where do you pull inspiration from? Are you looking at what The Times’s Graphics or News Design desks have done in the past and how those one-off projects might be turned into products, or are you looking to other publications?

Peter: It’s both: review what the newsroom has done in the past, but like Chen mentioned, our team also has the luxury of being able to take time to focus on a problem via design sprints or longer brainstorms.

Chen: For me, it’s news publications and other things in life. News sources, like broadcast, do a great job with live reporting, and I pay attention to how other news websites handle live coverage. But we also look at social media sites like TikTok or Clubhouse and the latest apps people are using that are not news, but are live in some way.

I feel like I get inspiration from live events, in general. Sometimes if I listen to a podcast and they talk about the live events they are doing, I think, oh, we can use this.

Thea: TV is a helpful reference for showing more than one scene at the same time. This is something we want to keep working on, as there are often multiple concurrent live events. A big goal there is to make information as easy as possible to take in during urgent or stressful moments.

And as Peter was saying, the format can’t always react on the fly, but we can keep notes on scenarios we observe and use these for the next round. When we do case studies, we usually have a few buckets of event types to sketch across.

Sarah: What are the buckets of events that you look to for your case studies?

Thea: Often they are scenarios that vary in onset, as well as duration and pacing. An event might be slightly anticipated or totally unexpected when it starts, with information trickling in slowly or arriving suddenly in an announcement. Some events are time-boxed (like a sports event) while others are sprawling and covered for a long period (like the coronavirus).

Chen: When we design things, we use real news events in our designs. When we design a new feature, we always look back at the days when that news was on the Times website and we’ll ask, if we had this feature on those days, how would we have used it? We do a lot of these case studies to see how we would use the feature in the real world.

Another thing I want to add is, when we design something, we think about both the editorial workflow and the way the feature is presented to readers.

Peter: And because we have control over the tools, we can make changes. If we see something is lacking, we’re able to add features to the back-end that solves front-end problems for readers.

Sarah: As you’re working through these case studies and meeting with editors, what do those working sessions look like?

Chen: In terms of the Live Admin, we have a Slack channel with the training editors. So when I have questions, I’ll just post them in the channel. We also have meetings to quickly get in sync; I will show them designs or they will just tell us the current problems they’re seeing.

In terms of the reader-facing products, we have weekly meetings with stakeholders who are higher-level editors. We show them the latest designs and they give us some feedback or insights around editorial workflow. They will also let us know if the direction is correct or not.

Thea: There’s a lot of casual checking in with editors, gathering their perspectives both on the coverage and their workflow. Some of our most helpful discussions have been the morning after an event, talking through places where the formats might improve.

Sarah: We’ve talked about collaboration, but how does the team approach experimentation?

Chen: I just joined this team in January, but I definitely feel like there’s a great culture of not being afraid of failing, and encouraging designers to try new things — and people are very accepting of it.

Peter: We haven’t done a lot of A/B tests with live coverage only because it is hard to ask the newsroom to cover an event in two — or more — ways. A lot of the work is looking at editorial feasibility and asking whether it works. I’ve worked with other teams in the past where we felt like we had to get everything perfect because we might not revisit the feature again. With the Live team, we want things to be good, but knowing that we can iterate takes some of the pressure off.

Thea: The news is different every day and I think gives us a stream of new things to try and an eagerness to prepare for the next event.

Rebecca Lieberman, graphics/multimedia editor with News Design: [signs onto the video call]. I’m sorry I’m late to the party. I had another meeting.

Chen: Like Thea was saying, I feel like it’s very different from a traditional product design workflow. We’re not very rigid, and we don’t work on two-weeks cycles. We are quicker; we release things quickly and sketch and test things quickly. It’s very different.

Sarah: Rebecca, now that you’re here — we’ve been talking a little bit about what Live is and how the team works through case studies. Since you’re on the News Design desk, which sits in the newsroom and rapidly builds stories to meet the news cycle, I’d love to hear about your role with the Live team.

Rebecca: I think the honest answer is we’re still figuring it out, but unlike other news and product collaborations, this one feels different. Some of the work that we’ve done on Live comes from one-off projects that News Design did in a more immediate response to the news cycle. Last year for the Democratic convention, editors wanted a way to signal that we had live coverage, so we built this interactive bar that showed what was coming next in the schedule.

A lot of these little experiments before Live started out in relation to the immediate needs from the News desk. I worked on a first pass of the Live Briefing a few years ago; We wanted to think about ways that design can create constraint and shape the form of the news coverage we want.

What’s great about this group is Peter, Thea and Jordan [Vita, the Product lead on the team,] also have worked with the newsroom a lot and are very familiar with how it operates. But I do bring a smattering of one-off things that I’ve done. And, I can explain how some of these needs arise within the newsroom, which helps inform how we might systematize a lot of this stuff.

Peter: I think that’s valuable because it’s really hard for a product team to do a one-off — and those one-offs are often when you learn the most. I don’t want to say one-offs are easier to do, but there’s less pressure than when you have to consider how a feature may impact the larger tool and front-end system. If you remove those constraints, it’s just so freeing and you can think of solutions.

Rebecca: I almost see it as a spectrum: on one side is a newsroom design team that works rapidly and in a reactive way to news, and on the other side is a product design team that works in sprints, which makes the design process longer. The work we’re doing is somewhere in the middle and we work towards coverage plans and goals.

Not only are there shorter periods of time that we’re going intensely on something, but we can talk to the people that are going to be producing that coverage. The work is just as iterative, but there’s an eye toward how it can fit within the larger system, which I think is really cool.

Peter: Thea always talks about a compass: we’re iterating, but we also have a compass that we often check.

Thea: Ideally we are sketching things that aren’t possible tomorrow, yet head us in a direction we ultimately want to go in. We’re trying to set features up so that we can build quickly, keep the back-end tooling and the front-end stuff all under one team and be able to experiment continuously.

Most of the time it’s hard to know if the work is viable until it’s tested with real news. We can only get so far with mockups or prototypes and Rebecca helps us imagine broader scenarios before something is live.

Rebecca: I think Reporter Updates are a good example of how we’ve worked iteratively with editors over the course of days. Reporter Updates are short, tweet-length insights and observations that live in the stream alongside longer posts on Live blogs. Ahead of the Derek Chauvin trial, the desk was briefed on how to use the feature. But, on the first day of the trial, we realized that the reporters were writing as if readers were also watching the live video, which was confusing if readers were not following the trial live. We gave feedback to the desk and they adjusted how they were writing. I think they ended up being more successful.

Sarah: Where do you see Live going?

Thea: We’ve been talking about the reader experience of following along versus looking for an overview. Some people are tired of news and have only a short amount of time, so they should be given a quick way to find exactly what they are looking for. Other people like to feel as though they’re chasing a story as it unfolds and may want to see every update for a specific event. We’re working to serve both of these readers.

Peter: Our former design colleague Derrick Schultz used to advocate designing for behavior. That is so much easier than designing for an anonymous person or a registered person or a subscriber, because a behavior is probably exhibited by everybody. Derrick would bring up the analogy of designing for a museum: a museum exhibition designer will design for the sprinter, the stroller and the studier. The sprinter will just fly through the museum, the stroller will putter around and the studier will read every little thing next to every piece of art.

That’s what we have to do. All of our readers exhibit those behaviors. Being able to design for all of them is really helpful. I don’t think we explicitly design for every one, but we implicitly think about that kind of behavior. Takeaways are great for people that are just strolling in, whereas the blog may be great for studiers.

Rebecca: I think we think about it in those terms, I don’t know that non-designers are thinking about readers in that way. But I do agree that the blog is for somebody who wants to follow a play-by-play, because it’s a reverse-chronological feed. If you come to a live blog at the end of a day where there was a lot of news, that’s not the best way to understand what happened. We sometimes do high-level takeaways to offer a different thing to a different audience.

Sarah: What about this work has stood out to you?

Chen: Being able to follow all the news and work at the pace of news is something that’s exciting to me. I think it’s valuable work because we want to tell readers what’s going on and we want to give them a better way to understand a story.

I was a reporter before, so I know how hard it can sometimes be for reporters to enter their article and publish it because of the tooling. So being able to be on this team where we can also tackle the tools part that allows reporters to break news faster. That’s also really exciting to me.

Peter: The collaboration. I said this earlier, but the collaboration and the conversation have been really important and valuable to me. And the work shows it.

Rebecca: Over the course of my career, I’ve worked on news and a lot of planned news or big, more enterprise stories, but I do feel like breaking news is its own beast. I’ve learned so much from the key stakeholders in this project. I’ve learned a lot about how breaking news is covered. Getting a behind-the-scenes look into how that happens, basically every day nonstop at our paper is pretty incredible. It makes me pay attention to when news breaks and how it shows up in different places, whether that’s Twitter or other news outlets.

Thea: Looking across patterns of coverage and imagining how things can be better (even in small ways) is really satisfying. Since live events are often unpredictable and unavoidably messy, no format ever fits perfectly. This gives more room for change, as things are constantly happening and the only direction is forward. So it’s exciting to be observing and rethinking and learning things as a group.

Jul 13 · 4 min read

What are your pronouns?
He/Him

What is your job title and does it mean?
I am a senior engineering manager for the Data Ad Products team in the Advertising mission. I am responsible for the growth, development and wellbeing of the engineering team, and I support short- and long-term business initiatives of The New York Times from an engineering perspective.

How long have you been at The Times?
I have been at The Times for a little over a year: I joined in May, 2020.

Most Times employees are working remotely right now. What does working from home look like for you?
For a large part of last year, our four-year-old daughter was at home during the day since her day care was closed due to the pandemic. Working from home with her around was very different from how my wife and I work now that she has resumed going to the day care. We now have more time to focus on work, whereas previously we spent more time supporting her in between meetings, thinking about what to cook for her or when she would take a nap.

How do you start your day?
I usually wake up around 6 a.m. I love starting my day with morning yoga or a Peloton workout, keeping the windows slightly open to let the morning air in.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
I have recently been involved in designing the first technical onboarding program for new hires in engineering. I have been working with other engineering manager peers on one of the introductory sections called “The Essentials.” This section aims to equip the new hires with the essential tools they will need to navigate the organization, both day-to-day and in their growth beyond their immediate team. This onboarding program ensures that all engineers have clear and equal access to the organizational context necessary to grow and succeed at the Times.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently?
Some time ago, I worked in e-commerce (retail), which is a competitive business and a tough place to manage an engineering team. The senior management changed frequently and difficult decisions were made regarding things like strategy and resourcing. My biggest challenge was keeping my team focused and motivated during such uncertain times. I learned to be a more open and empathetic listener. I tried to be a safety shield for my team and absorbed a lot of unnecessary noise and distractions. This was a key lesson in management for me.

Knowing what I know now, I would have tried to be more patient and go a little easier on myself. Life happens in phases and recognizing that you are in a tough phase is key to powering through. But, cherishing the good parts is equally important.

What is something most people don’t know about you?
I love learning programming languages. I have written production code in C, C++, OCaml, Java, JavaScript, Python, Perl, PHP, Ruby, Scala and Go.

What is your superpower?
I am not easily discouraged by failure and I try to see the positive side of any given situation.

What or who are you inspired by?
I have taken inspiration from a number of people during different times in my life. Presently, I am inspired by my four-year-old daughter, who has a zeal for learning. She asks a hundred questions during the day, is not easily satisfied with our dumbed-down explanations and is willing to keep pressing until she understands. For example, here is our conversation after watching “Frozen II”:

My daughter: “Who is calling Elsa?”
Us: “Her mother.”
My daughter: “Where is her voice coming from?”
Us: “Ahtohallan.”
My daughter: “Why isn’t her mother in Ahtohallan, like in person?”
Us: “Because she is a spirit.”
My daughter: “What is a spirit?”…“Why isn’t her dad calling?”…“Does he not love her?”…

Complete this sentence: Over time, I have realized __________.
That the right opportunity is always around the corner; you need to be patient, prepared and proactive to grab it and not let it pass you by.

What is your best advice for someone starting to work in your field?
Investing time to hone your skills is the best thing you can do as a beginner. Having humility to accept shortcomings, investing time to work on them and ultimately getting better is my mantra for continuous self-improvement.

New York Times graphics editors worked with librarians and archivists to piece together a picture of Greenwood before it was burned. The data set and map files are now open source.

Jul 2 · 8 min read

By Yuliya Parshina-Kottas and Anjali Singhvi

To commemorate the centennial of the Tulsa race massacre, graphics editors at The New York Times analyzed archival maps, city directories, census data, newspaper clippings and survivor accounts to reconstruct Greenwood, a prosperous Black neighborhood in Tulsa, Okla., that perished at the hands of a violent white mob. We created a 3-D model to publish an article about what was lost in the massacre. Today we are releasing that model along with digital map files and a data set of residents and businesses to the public.

To tell this story spatially and immerse our readers in 1921 Tulsa, we set out to create a 3-D model of the Greenwood neighborhood, which served as the digital stage for our narrative.

Among the first resources we looked at were Sanborn insurance maps — large, unwieldy atlases published starting in the 1860s. They contain building footprints, heights, materials, transportation infrastructure and much more and are a wonderful resource for building-by-building details of a city. The Library of Congress has a digital collection of some of these maps, including a 1915 map of Tulsa, which helped us understand the overall geography from that time period. But we were worried about the six-year gap between the map’s creation and the massacre.

Tulsa’s population nearly quadrupled from 1910 to 1920, thanks in part to an oil boom. The city had most likely changed in the years leading up to 1921, and we didn’t want to miss anything.

We contacted libraries and archives, and we found other clues on the internet, like this blog post about a local map exhibit.

Finally, Yuliya Parshina-Kottas, one of the lead reporters and graphics editors on the project, located an original copy of the 1920 Sanborn atlas at the Tulsa Historical Society and Museum. The museum’s archivist, Luke Williams, worked with us to scan 26 pages of Greenwood and the surrounding areas.

Some details were lost to age and mold, so Anjali Singhvi, a graphics editor who led the project with Ms. Parshina-Kottas, retouched the digitized pages and stitched them together into a single image.

To create a basis for the 3-D model, Tim Wallace, senior editor for geography, and Jugal Patel, a graphics editor, cleaned and painstakingly georeferenced maps from the 1915 atlas for the areas outside Greenwood and downtown Tulsa and combined the collection with the 1920 map of Greenwood.

To turn the Sanborn maps into a three-dimensional city, Lingdong Huang, another graphics editor, trained a neural network to translate the maps into building footprints. This required some extra steps to help the neural network identify buildings on the map.

To teach the algorithm to recognize footprints in a Sanborn map, Mr. Huang generated a series of rectangles that mimicked the map style (left column) and a corresponding series of rectangles in black and white, to serve as a hint for identifying footprints (right column). The middle column is one of the algorithm’s attempts to identify and trace the footprints in the generated map.

Once the network had been trained, it correctly translated the Sanborn maps into crisp geometry that we could use to create the final 3-D map.

We studied the key and annotations on the Sanborn maps to figure out building heights. The typographic style of the annotations and age-related artifacts on the map made it difficult for the algorithm to decipher those values, so Mr. Huang created a Python tool that Matt Craig, a news assistant, used to manually enter the accurate height in feet for each building in Greenwood and downtown Tulsa.

The 100 block of Greenwood Avenue, north of Archer Street, was a bustling center of commerce and community life for Black residents. It was the perfect example of what Greenwood had to offer: cafes, theaters, physicians, lawyers, barber shops, clothing stores and more. We wanted to bring our readers down to the street level for a more intimate glimpse of the neighborhood and the people who had a hand in shaping it.

Finding visual references of pre-massacre Greenwood, however, required substantial detective work because many of the photographs taken before 1921 had been destroyed in the massacre.

Shortly after the attack, residents of Greenwood began to rebuild. Some of the structures on the 100 block were reconstructed almost brick for brick. These look-alikes made some post-1921 photographs difficult to distinguish from their pre-massacre counterparts, but a more careful analysis revealed important differences.

The graphics editors Mika Gröndahl and Guilbert Gates confirmed two pre-massacre photographs of Greenwood Avenue, offering an incomplete view of the street, as well as several closeups of buildings on the block.

Lacking sufficient references, they turned to images taken during and immediately after the massacre. Mr. Gröndahl and Mr. Gates spent hours sifting through photos of blocks that had been completely destroyed. They looked for details that might help them understand what had stood there. Barber poles, wall fragments and floor joists provided hints about what the buildings had been before they were burned.

Mr. Gröndahl, a veteran 3-D artist and a trained architect, manually crafted detailed building models in Maya based on the archival photographs. When no visual references were available, we left the buildings blank so we wouldn’t be representing anything that we hadn’t verified.

While the Sanborn maps provided a good foundation for our 3-D model of Greenwood, more references were needed to create a complete map of the remainder of Tulsa. We catalogued all the different maps we came across during our research and worked with librarians and archivists from the Tulsa Historical Society and Museum, the Library of Congress and the Tulsa City-County Library, among others, to track down undigitized content.

The Library of Congress had a copy of a 1921 street map of Tulsa in its undigitized and uncataloged collections. Julie Stoner, a reference librarian in the Geography and Maps Division, scanned and sent us a digital copy despite having limited access to the building because of Covid restrictions. This map was our main reference to create the vector street grid of Tulsa.

To create a fuller picture of the people who made up Greenwood, we analyzed and mapped census data from Ancestry.com to show where each Black resident lived and what they did for work.

One of the biggest challenges we faced in creating this map was verifying addresses and streets from 100 years ago so we could accurately map them.

Ms. Singhvi and Mr. Craig geolocated thousands of addresses from the 1920 Census to create a vivid picture of the types of occupations that were held by Greenwood residents. We are not entitled to share this data, but this is an area that others could build on.

We came across the work of Larry O’Dell, director of development and special projects at the Oklahoma Historical Society, in a 2001 state commission report on the massacre. Mr. O’Dell had manually assembled a database of Greenwood businesses and residents from multiple sources, including the Polk-Hoffhine Directory Co.’s Tulsa City Directories and the “List of Losses” published in “Events of the Tulsa Disaster” by Mary E. Jones Parrish, a massacre survivor. Mr. O’Dell graciously shared his data with us, which served as a starting point for our research and informed the vector shapefiles we created of the areas that were burned.

Inspired by Mr. O’Dell’s work, we used a process called Optical Character Recognition,or O.C.R., to create our own data set from the digitized pages of the 1921 Tulsa City Directory, available through the Tulsa City-County Library. By searching the O.C.R.-derived data for a “©” at the end of an entry — which was used by the directory to identify businesses and residents of color — we were able to create a more complete record of Black residents and businesses in the city.

For a more detailed profile of the 100 block of Greenwood, we compared businesses in the 1921 city directory with Ms. Parrish’s list of losses and advertisements in the weekly Colored Business Directory of The Tulsa Star — a Greenwood-based, Black-owned newspaper that published from 1913 to 1921.

When we came across inconsistencies in spellings or addresses, we went with the version that appeared in more than one source. In a few cases, we discovered businesses in The Tulsa Star that were not listed in the city directory. If these businesses, like the East End Doughnut Shop run by Bettie Young, appeared more than once in the newspaper between 1920 and 1921, we included them in our list of businesses on the block.

Our reconstruction and analysis of pre-massacre Greenwood is just a small glimpse into the tragic losses from the Tulsa race massacre.

By releasing the data set of Black residents and businesses, the map files and the 3-D models of Greenwood, we want others to be able to examine this devastating moment in American history in greater detail.

This is a complement to the work already being done by many others to bring this story — which was wilfully buried for decades — to light. Our hope is that researchers, historians, students and others will build on this data, add to it and propel the story further.

“Meeting…” is an ongoing series that features New York Times employees from different corners of the company.

Jun 30 · 4 min read

What are your pronouns?
He/Him

What is your job title and does it mean?
Senior Software Engineer on the UX Foundations Team. We work to ensure that The Times delivers a cohesive cross-platform user experience worthy of our journalism.

How long have you been at The Times?
Since April, 2019.

Most Times employees are working remotely right now. What does working from home look like for you?
I was working remotely from Washington, D.C. before the pandemic started, but it’s still been a big change. Before the pandemic, I would spend most of my time at coffee shops throughout the city and generally try to structure the majority of my day outside of my apartment.

Not being able to go to coffee shops meant I had to invest in my work space and get an office set up. When my partner started working from home, things got a little cramped in our one-bedroom apartment and we had to shift our furniture daily to create more space. Eventually, we moved to a two-bedroom apartment where we could have a dedicated office space, and that helped by creating a clearer boundary between work and home life.

How do you start your day?
I usually wake up and do some creative coding sketching while making coffee. Around the middle of last year, I got introduced to the creative coding library P5.js and I was instantly hooked. As a kid, I was always doodling on scrap paper, then in high school and college I did graphic design for various school clubs. I’ll usually use a platform like Observable, where I can code really quickly to try out a few different ideas until it’s time to start work. Sketching allows me to exercise some of my old graphic design skills while kick starting my brain with a little puzzle each morning.

What is something you’ve worked on recently?
Earlier this year, I updated the Live Coverage CMS to use our internal design system called Ink. The Ink project is employee-led and is really cool because it allows us to create a consistent look and feel for all of our internal tools. As an engineer, Ink streamlines my job by providing reusable code.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Much of The Times’s 2020 election coverage was powered by the tools built by the Storylines team. It was really cool to see the culmination of everything my team worked on that year in the year’s most important event.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently?
Having to deal with pay inequity as an underrepresented person in tech. In the past I’ve struggled with understanding the value that I bring to organizations and the unique view point I present as a queer Black man. It’s a constant journey, but knowing what I know now, I am able to advocate for myself and I make sure others can do the same.

What was your first job?
I sold candy in high school! Career-wise, my first job was as a back-end engineer at Legacy.com. I transitioned from an internship there right after graduation and got a lot of exposure to tools like React and GraphQL that were cutting-edge at the time, but have since become industry standard.

What is something most people don’t know about you?
I’m left-handed.

Do you have any favorite life hacks or work shortcuts?
My number one life hack is going for runs when it’s hot out. You sweat more and get to run less.

What is your superpower?
Putting my teammates’ happiness and well-being before the company’s bottom line.

What or who are you inspired by?
People who continue to show up and do the work even when it isn’t clear that progress is being made.

Complete this sentence: Over time, I have realized __________.
Most complicated things are actually pretty simple, but gatekeepers are incentivized to make them seem complicated.

What is your best advice for someone starting to work in your field?
You don’t need to be passionate about programming. There’s a trope that the field is full of genius programmers that have been writing code since they were children. I believe there is enough room in the field for those people and also those who have lacked that exposure historically and have only recently discovered programming. It’s 100 percent normal and valid to see this as a job and programming doesn’t need to be your passion.

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

Jun 15 · 6 min read

What are your pronouns?
She/Her

What is your job title and what does it mean?
Executive Director for Development & Engagement. Our team creates and fosters programs that support the professional growth of New York Times employees in the Data, Design, Engineering, Marketing, Product and Project departments (we call this group “XFun’’ for cross-functional teams). We provide platforms to broadly share our colleagues’ work, and we host events that engage and connect them to the innovation and mission of The Times. These programs and platforms include The Sponsorship Program, XFun Onboarding, Product Engineering Onboarding, Associate Engineering Bootcamp, Coaching for Managers, the NYT Open blog, Times Open Speaker Series and Maker Week.

How long have you been at The Times?
It will be seven years this June.

Most Times employees are working remotely right now. What does working from home look like for you?
This spring looks significantly different than last spring, that’s for sure! Last year, like so many of my colleagues, my wife, Martha, and I were juggling our jobs and remote schooling for our two daughters, stressed out about so many unknowns about COVID-19. We utilized every inch of our two-bedroom apartment. The extreme juggling of very different school schedules and programs, work and about a million meals a day was exhausting. It was a bleak and foreboding time.

This spring feels hopeful. We’re in a different groove. My youngest daughter is in school five days a week, and our 13-year-old daughter is on a hybrid schedule. Martha and I rotate between the room with the desk, kitchen counter and dining room table. We’re still searching for a decent office chair we can share (she’s tall; I’m not). We move between work and checking on our 13-year-old daughter to make sure she’s still awake in class throughout the day. I break at 12 p.m. for a snack or a quick bike ride now that the weather is nicer.

How do you start your day?
My day starts with being on either the wake-up team or the breakfast and lunch team. I usually walk our youngest daughter to school (she’s a speed walker, so it’s a good cardio moment for me). On the way back, I recheck my calendar for the day, the P1 (short-term) and P2 (mid- to long-term) list I put together at the end of the previous day, and I make any adjustments. Now that it’s nicer outside, I can do this in the park near our apartment. I have coffee and a small breakfast before logging on. I try to optimize my day by loading my deep-focus work and critical meetings from 9 a.m. to 2 p.m.

What is something you’ve worked on recently?
Creating and launching the XFun Onboarding and Product Engineering Onboarding programs with my colleagues Laura Prah, Pranay Prabhat and Ben Koski. The programs introduce our new joiners to the XFun and Product Engineering organization so that everyone is equally informed and we can avoid an imbalance of knowledge or unwritten rules.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
There are quite a few, but my all-time favorite has to be creating, running and continuously improving the Sponsorship Program. This was the brainchild of my Times colleague Cindy Taibi and Times alum Erin Grau, with the support from our former CTO Nick Rockwell. We started the program in the Technology department to create a more diverse pipeline for leadership positions and have since expanded to include all of XFun. We have created a program that utilizes the generosity of our C-suite, as well as a curriculum focused on business fundamentals, people and leadership skills and one-on-one coaching. It’s insanely fulfilling to watch a cohort and sponsors come together.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently?
Early in my career, I was recruited by two former colleagues to join them at an award-winning company to design and launch interactive products. While I enjoyed where I worked, I liked the products and was excited to work with the colleagues again — the job would also pay significantly more. I decided to take the leap.

Between the time I accepted the position and my start date, the company completely changed its focus, leadership and, subsequently, my role. On my first day, I knew the job wasn’t right for me. Everything felt wrong, yet I felt compelled to give the role a fair chance; I was raised to finish what you start and keep a positive attitude.

I lasted about two weeks before I called it what it was: a bad fit. I worried I would damage my relationship with my colleagues while also harming my professional reputation (the tech world on the West Coast felt tiny at that time). But, it was not the type of work I was seeking nor was it the direction I wanted to pursue in my career.

We finally had an honest and candid conversation. My colleagues shared that they had also worried about the impact of all the changes to the role. We parted ways on good terms and remain in touch 20 years later.

I learned that you make the best decision you can with the information you have. But when the information changes, you might need to make a new decision and that’s OK. My next role was much more aligned with my goals and the type of challenges I was seeking. In other words, being true to myself led me to the path I wanted.

What was your first job?
My first job was in middle school when I mowed lawns in Raleigh, North Carolina.

What is something most people don’t know about you?
I rowed against the Women’s U.S. Olympic team at the San Diego Crew Classic in a women’s eight for the Interlaken Rowing Club in Seattle. We lost, but it was a thrilling experience.

Do you have any favorite life hacks or work shortcuts?
I was a reluctant convert but it’s true: baking soda and vinegar can clean almost anything.

What is your superpower?
I listen with empathy and backbone.

What or who are you inspired by?
That’s a tough question — there are so many! My mother (she returned to school later in life to become a registered nurse, and later went on to build a successful business, then battled cancer and Alzheimer’s Disease), my family, my daughters, my friends and my father-in-law, who underwent a lung transplant and sadly passed away recently.

And of course the well-known ones: Michelle Obama, Ruth Bader Ginsberg, Maya Angelou, Beyoncé, Serena Williams and countless others who aren’t well-known. It’s wonderful to be inspired by others: you can recognize, celebrate and appreciate their achievements, as well as find inspiration and motivation for your life.

Complete this sentence: Over time, I have realized __________.
It’s OK to change your mind and to let yourself change. What may have interested you when you were 20-years-old, 30-years-old or 40-years-old may shift and change as you gain more experience or are exposed to more projects, strategies and companies. Some people may narrow their expertise and others may broaden or transition to something completely different. To let yourself change is to let yourself grow.

What is your best advice for someone starting to work in your field?
Be open, curious and generous. I have had at least 30 different jobs so far (this includes lawn mower, house painter, jewelry maker, wedding photographer, design director, program manager, product developer and executive coach), and the things that have helped me the most are an openness about others and their experience, and a curiosity about what can be learned in each opportunity. Maintaining these can create meaningful perspectives, lessons and experiences at any level or in any role. And once you learn or achieve success, be generous and give back.

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

Jun 4 · 4 min read

What are your pronouns?
He/Him

What is your job title and does it mean?
I’m currently a Principal Product Designer with a main focus on the Audience Product team. This means that I work on both internal tools as well as user-facing features, such as our new share tools on the article page and our new Site Map. I also collaborate with many other groups, including the Brand, Dataviz and Multimedia Publishing teams.

How long have you been at The Times?
September 2021 will mark nine long, amazing and fruitful years. Although, I can’t believe I don’t have more gray hairs…

Most Times employees are working remotely right now. What does working from home look like for you?
It’s much better than 2020 — one of the worst years of my life. Last year, I had to reinvent my process and develop new rituals to cope with all of the heavy stuff that came from living through a pandemic and working in news. Now, I try to stay as zen as possible throughout the workday. If I get stressed, frustrated or start feeling sad and hopeless about the news we are reporting, I drink a big glass of water and go for a walk around the block to reset.

How do you start your day?
I start the morning with stretching, breathing, some meditation and mental note-taking. Self analysis. The last several years were emotionally challenging for personal reasons, so I take my mental health and well-being very seriously. After my morning self-analysis, I have a light breakfast, plenty of water and, of course, coffee. Then I pop open my laptop and get to it.

What is something you’ve worked on recently?
Healing myself. Seriously: we are all works in progress, and for me, getting through 2020 with a focus on mental strength and self care was an epic portfolio piece.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Everything I work on here. The Times is such a historic and important institution, and I’m humbled to be a part of bringing the Paper of Record into the future. There are times when I start a project and think, “wow, so many people are going to be using this — interacting with it, sharing it, being informed by it.” It’s absolutely incredible.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently?
The designer is not always right. Remain open, admit when you’re wrong and keep it moving. Learn from your mistakes. Collaborate.

What was your first job?
I can’t pinpoint one, but several as a younger person really helped shape me. Working at a skateboard shop and a T-shirt silk screening business when I was in high school, driving a tow truck and pumping gas when I was in college. Those were so valuable in developing my work ethic and a dedication to the work I do now.

What is something most people don’t know about you?
I’m an accomplished graffiti artist. I started working in the art form in 1994, and I still carry my experiences, skills and eye for detail with me to this day. Being a part of this culture truly helped define the person I am today, for better or worse.

Do you have any favorite life hacks or work shortcuts?
Coffee, water, a decent diet, therapy and knowledge of self.

What is your superpower?
Humility.

What or who are you inspired by?
My daughter, Lila. My family and friends. My colleagues both past and present. And of course, the rock stars… Massimo Vignelli, Swoon, Stephen Powers (a.k.a. “Espo”), Paul Rand, Paula Scher, Natas Kaupas, DMX, Dondi White, Paul Sahre, KRS One, Tom Bodkin, Josef Albers, Kurt Vonnegut, Bill Cunningham, Lady Pink, Lee Quiñones, Greg Lamarche and so many more. Graphic design, typography, color theory, graffiti, hip hop, punk and hardcore music. Skateboarding. New York City. Humanity: the good, the bad and the ugly. Everything and everyone shapes me.

Complete this sentence: Over time, I have realized __________.
I’m mortal and not going to live forever. Tomorrow is not guaranteed, so I don’t take anything for granted.

What is your best advice for someone starting to work in your field?
Stay humble. We are forever students in anything we do, whether that be your day job, your hobby, your passion projects or just in everyday life. Be aware of what you do well and keep doing that. And when you make a mistake, own it, course correct and move on to the next.

Whenever I hire a data analyst for my team at The New York Times, I find myself inundated by a stack of resumes, LinkedIn messages and personal notes from colleagues recommending people they know.

It can be very difficult to sift through an often overwhelming pile of applications, but those candidates who make it through The Times’s screening process and SQL assessment can expect a 30-minute interview with yours truly. I really look forward to that conversation, as it’s my first chance for me to get to know the person behind the paperwork.

If you are one of those candidates, good news: I’m going to tell you what questions I will ask and what I’m looking for as we chat. I have come to rely on these three questions after nearly 15 years of interviewing candidates for analytics jobs at The Times, and I use them for almost every interview.

Why share these questions? One reason is because I think other hiring managers might find them thought provoking and useful, but mostly, I share them because I want you to succeed. My goal in interviews is to find great people to join our team, not to trip you up in some booby-trapped maze.

After all, I’m not just evaluating you as an employee; hopefully, you’re evaluating me as a manager, too.

The Times is known for its great storytelling, and I believe that the jobs of an analyst and a journalist are very similar. We both use critical thinking and smart investigative techniques to better understand the world around us, and it’s our job to clearly communicate the insights we uncover in compelling ways.

My first question is: “Tell me a brief story about the most surprising insight you ever developed: how you came to uncover it, how you communicated it and why it surprised you.”

Every word in this question is important. The story you share should be told well, it should be brief and it should grab my attention. Most importantly, it should be surprising.

Why surprising? Two reasons.

For something to be surprising, it must run counter to some existing belief. All analyses should start with a hypothesis — such as an assumption, an existing habit or some piece of conventional wisdom — that we can test with data. To be surprised, you must have given some thought to what you might find, and that means you probably cared enough about the outcome to develop the hypothesis in the first place.

But in addition to your hypothesis, your story must also have an element of discovery. Data analysts are at our best when we uncover hidden truths about the world we inhabit. We do not merely exist to confirm preconceptions — that would be a pretty boring career — we exist to challenge the status quo, to reveal new paths and to bring clarity to obfuscation.

Unfortunately, very few applicants are able to tell a clear and compelling story. They ramble. They try to impress me with complicated technical skills or obscure data science algorithms, which leaves big gaps and often, sadly, lacks the requirement of surprise.

Interviews can be tricky, stressful and exhausting, and I know this question is difficult. That’s exactly why I don’t want to put you on the spot. I want you to nail this question. It’s OK to prepare in advance. Just bring me a story.

My second question is: “If you take this job, what’s in it for you? What would you like to get out of it beyond a paycheck? How will this help you grow?”

Most applicants give a version of the same answer: that they are motivated by The Times’s mission and the opportunity to contribute. That’s great; it’s one of the reasons I enjoy working here. But, I’m not asking how great The Times is — I’m asking about you.

The best answers I have heard to this question showed that the candidate was curious, ambitious and sought out opportunities that would expand their knowledge. I have also heard candidates say that they want to improve their programming skills or learn more about data science. Those are fine answers, if that’s what you really want. Ultimately, I want to hear that you’re driven to learn and grow — in a way that’s specific and meaningful to you.

Why? Because curiosity is a core characteristic of a good analyst, and ambition for personal growth indicates both an openness to new challenges and a willingness to learn. Most of all, it shows that the job will be rewarding for reasons beyond the paycheck.

After all, none of us should be under any illusions that you will be doing this job forever. My hope is that the role will challenge you to not only deliver great work, but develop the skills that will help you move on to bigger and better things in your career.

It’s a chance for you to ask questions of me.

I like to leave a lot of time for this. If we can switch to your questions after 15 minutes, we’re doing great: you have told me a succinct story and you have given me a glimpse at your ambitions. It’s your turn to lead the conversation.

In many ways, this is the most important part of the interview. Asking questions is an essential skill any analyst should have.

A candidate who asks good questions reveals an interest and passion for the role that can’t be faked. It shows they care enough about the opportunity to try to understand it fully. Smart, specific questions show that they have done their research in advance; that they are truly curious about what it might be like to work at The Times. And what they ask can help me learn more about them.

I don’t think it’s a coincidence that my best hires have asked the most questions. One even came prepared with three pages of double-spaced questions, and we had to schedule a second interview so I could answer them all. (We hired her. She was great.)

Perhaps most importantly, I want you to approach this opportunity with your eyes wide open. If we’re going to work together, we need to be able to communicate with candor and respect. So, I try to answer every question honestly. Hopefully, by engaging in an open conversation from the start, I’m setting the tone for a great working relationship — one in which we’re both invested in each others’ success.

To train a 1,700-person newsroom, The New York Times added tooltips and guidance to the CMS.

May 20 · 7 min read

By Eric Athas and Taylor Poulos

In an era where storytelling is constantly evolving, how do you seamlessly train a newsroom with 1,700 journalists spread around the world?

That’s a question near and dear to the Newsroom Development & Support team (or, N.D.S.), a department responsible for driving change at The New York Times. The team has trained reporters and editors on everything from live briefings to data reporting to visual journalism and new story formats like explainers.

In 2019, N.D.S. completed one of its biggest undertakings: training the entire newsroom on an expressive and collaborative CMS, called Oak. The N.D.S. team coached every single reporter and editor one-on-one — in New York, Washington, D.C., Hong Kong, London, San Francisco and remotely for correspondents — through every part of the tool, and explained how it all worked and best practices for its powerful storytelling capabilities. And then, when journalists began using Oak, the team was there to answer questions in-person and on Slack. We also worked closely with the Oak product team to incorporate feedback and make the software more effective and easier to use.

The work was exhaustive and time intensive. And, because the team spent so much time training reporters and editors on howto use Oak, that left less time to train them to master different story formats, move swiftly during breaking news, make stories more visual and other skills needed in the new tool.

To help make Oak training more efficient, the N.D.S. and Oak teams launched a project aimed at building guidance — such as tooltips, helper text and best practices — directly into Oak. The goal: Give reporters and editors the help they need, in the places where they work, at the moments they need it most.

With that goal firmly in mind, we interviewed our colleagues within The Times and outside experts on training to understand where our documentation worked well and where it fell short. Through this research, we uncovered several major challenges.

1,700 people is a lot of people
The first challenge was the size and scope of the newsroom itself. With over 1,700 journalists who touch our CMS, there are many different roles that need to be accounted for. Our CMS users include reporters who work on one major feature story for a long period of time; editors who shepherd dozens of stories toward publication every week; and visual journalists who are focused on making our report striking and beautiful.

Varied expectations for what a CMS can do
A journalist’s previous experience with a CMS can affect their expectations of our tools. People who have joined The Times from organizations with less effective CMS teams had limited expectations about what a CMS could do. Because the first step of learning how to do a task is being aware that the task is possible, these lowered expectations made people less likely to try new things; They simply assumed that whatever they wanted to do wasn’t possible. On the other hand, some people were used to asking more of their CMS, and were comfortable pushing its limits through insightful questions and feature suggestions.

It’s hard for busy journalists to keep track of every new feature
With a dedicated product and development team for our CMS, we are constantly tweaking our tools, especially the tools our journalists use most often. This means we need to continually provide training that covers new changes. And because our tools are complex, these training sessions are sometimes over an hour and a half long. We can’t expect every person to remember every detail of the sessions.

[If this work is interesting to you, come work with us. We are hiring!]

Training isn’t one and done — tools and best practices evolve
As the world we report on changes, so do our tools and best practices. Our tools need to have the capability to surface, document and remind users about these changes. One participant in our research put it succinctly when he compared journalism to medicine, where knowledge is constantly evolving, and practitioners constantly need to stay up to date with the latest techniques.

The research was essential in learning what guidance would be most useful for the Times newsroom. It also helped us identify the challenges we would need to navigate when placing tips in the space where our journalists work.

Our next step was to make sure that the guidance would be informed by the journalists doing the work day in and day out. We made a list of every field that our reporters and editors touch in Oak — such as headline, summary, captions and URL — and asked over a dozen colleagues about each one. We wanted to know what tips we should include and how we might frame guidance in the CMS. We also wanted to know where there are sticking points in the CMS, and what is most important to flag when journalists are making decisions on deadline.

Although we already had extensive best practice documents and how-to guides for every aspect of newsroom work, space is limited in Oak. This process allowed us to narrow in on the guidance that would be most essential to surface as our colleagues write and edit.

Once the list of items was refined, then it was time to figure out where they should live in the CMS. Here’s where we landed:

For information tied to a specific field: The Right Rail

For most of the guidance we came up with, the right rail was the best place to put it. The information in the right rail shows up when a user is focused in a specific field, making it targeted to the moment users need it most. Auxiliary information, like a character count, can be shown within the same context, giving live feedback as users perform an action. And unlike tooltips, which show up only when users think to click or hover over the right icon, the guidance is shown by default; always there, but designed to be differential and not distracting.

For information tied to a element in the editor: The Detail Pane

Oak is an element-based editor, meaning that journalists build their stories out of a wide array of multimedia and text based elements, such as images, pull quotes, headings and embedded tweets. To document these elements, we expanded the element menu to include space for an image, a description and a link to learn more about the type of element, leaving plenty of room for explanations about what the element is, and how it can be used.

For when you don’t know where to start: The Help Center

In addition to this in-CMS, N.D.S. has produced a lot of documentation about our best practices for things such as SEO, visual journalism, editing processes, and story formats. However, it wasn’t clear where people could access this information within Oak. The help center is integrated into the tool itself, accessible through an icon in the lower right hand corner of the screen. It has a curated list of documentation and links, as well as the option to search our full documentation library. It also has a place for people to submit feedback about the tool directly, making it easier for people to submit feedback or questions the second they run into issues.

How we tell stories will only continue to change. And so our journalists will continually have new things to learn: new tools, updated standards and best practices to ensure they’re able to make the best journalism possible.

Just as our storytelling formats will change, so will our tools. It’s important to have a clear process for keeping all of the documentation we have placed into Oak up-to-date. Our colleague, Jin Kim, an editor with N.D.S., will lead the next chapter of this work to keep in-tool guidance relevant. Jin will coordinate between the Oak team and N.D.S. to determine when the text in Oak has become out of date and will be able to update the information on the fly. In collaboration with newsroom experts, Jin will also keep a pulse on emerging journalistic practices — such as new SEO tactics or updated standards guidance — that can be surfaced to reporters and editors. By building this capacity, we hope to continue evolving the guidance built into Oak, well into the future.

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

May 18 · 4 min read

What are your pronouns?
She/her

What is your job title and does it mean?
Software engineer. This means I write code to provide the solutions needed to drive business value. I currently work on an application called Event Tracker which enables The New York Times to collect data and metrics about how our readers use our various products.

How long have you been at The Times?
This is actually my second time working at The Times. I first worked here from 2007 to 2013 as a software engineer on the CMS team, and I rejoined in July, 2020 on the Data Collections team.

Most Times employees are working remotely right now. What does working from home look like for you?
Working from home right now is … interesting. My oldest daughter is in second grade and doing remote school. She and I share the dining area as our work space and we usually have competing video calls. My partner is also an engineer and he takes his video calls in one of the bedrooms. In between meetings, we both take turns entertaining our two-year-old. After doing this for over a year, we have settled into a routine that enables us to be surprisingly efficient and effective.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
The Times and the Data Collections team take reader privacy very seriously, and readers who have visited our site can request to have any of the data we have collected about them removed — this is called a Data Subject Request (DSR). I wrote the code that takes a DSR, finds all of the instances of that reader’s data in our database and removes them.

This was challenging because the team had to find a way to effectively search through all of the data in a fairly large database. Additionally, we needed to be able to do this on a regular basis for any reader who submits a DSR, so I wrote the code, using Apache Airflow, in a way that could be run on a schedule.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
One challenge I faced early in my career was not knowing how to advocate for myself. I approached my early career the same way that I approached school: I did the work that was assigned to me to the best of my ability. I soon realized that this was not the best way to move my career forward.

Today, I am much more deliberate in identifying the kinds of projects that excite me and I am not afraid to make suggestions or volunteer for challenging assignments. I also make sure that I frequently communicate my goals with my manager and that I am getting opportunities that keep me on track toward meeting my goals.

Do you have any favorite life hacks or work shortcuts?
I’ve recently started using TiddlyWiki to organize all of my ideas. I like it because it’s really just a self-contained HTML file. This means I don’t have to worry about installing special software to be able to view my notes.

What is your best advice for someone starting to work in your field?
Learn how to learn. Software engineering is a field that changes rapidly; You need to have curiosity and you need to understand the learning style that works best for you.

There are so many great resources out there, many available for free or with a free trial. Massive open online courses can provide a great introduction to any software topic, but it’s also important to learn by doing. There are many sites, such as Advent of Code, Exercism and Free Code Camp, that offer code challenges. Cloud platforms, such as Amazon Web Services and Google Cloud Platform, typically offer a free tier or free credits so that someone interested in learning can experiment with the platforms.

With so much information available, it can feel intimidating to get started, but the best way is to just choose a resource and stick with it. Be patient with yourself and accept that being confused is part of the job, as is the satisfaction that comes when things start to “click.”

When a reader opens The New York Times on Android, our app makes a series of calls to our core libraries. Each feature that is rendered for the reader might depend on multiple back-end calls, user input and other parallel business logic flows. To ensure that our readers are served with a fast and reliable mobile app experience, we do a lot of asynchronous work, or threading, within our app. We use our threading technology to not only offload network calls and heavy business logic, but to also chain together complex sets of processes that are updated at different times and can affect multiple observing business logic flows.

For the last few years, The Times’s Android teams have used both RxJava and Kotlin Coroutines and Flow for threading in the core libraries that power our News, Cooking and Games apps. To unify our stack, we decided to choose one technology and dug into the performance differences between RxJava and Coroutines.

While RxJava was a powerful tool, its CPU, thread usage and general complexity as a language made it an increasingly cumbersome technology for our needs. We have monitored the progress of Kotlin Coroutines and Flow and determined that it not only covered all of our feature needs, but it also had significant performance improvements.

In order to iteratively migrate our apps and libraries to Coroutines, we first needed to update our core libraries. Here are some best practices we developed during our migration.

Use Coroutines suspend for basic threading
When we migrated to Coroutines, we found that whenever we needed to use basic asynchronous code that returned a single result (or a RxJava Single), we could replace that code with an equivalent suspend function. For example:

Example RxJava vs. Kotlin Coroutines:

ControlledRunner pattern to handle multiple calls to same function
While we were migrating, we ran into multiple cases where we had several suspend functions that needed to join and wait for a single result. This was particularly important for situations when we had multiple calls in parallel for the same user-state information and we needed to avoid flooding the back-end with calls.

Instead of making duplicate calls for the same result, we used the ControlledRunner pattern. The pattern’s joinPreviousOrRun function will discard any new tasks and have the calling task wait on the result of the original task. This is useful if several areas have made the same call and expect the same result. We also used ContolledRunner’s cancelPreviousThenRun function to cancel old tasks and always run the latest task — which helped in situations where a new event supersedes the previous event.

Using Flows for reactive or observable streams
When we needed a function that could emit multiple values at different times, we used Kotlin Flows.

Flows are very much like RxJava Observables: collect{...} or collect() acts like RxJava’s subscribe(). We used mapping functions to update the result or onEach{...} to create side effects for each result. launchIn(scope) was very helpful to launch and collect the Flow in a given scope so we didn’t have to nest our Flows in a scope.launch.

We made it a best practice that our library APIs always use Flow<T> in their client-facing interfaces to prevent outside sources from interfering with our Flow business logic.

In order to avert memory leaks or using up our thread pool, we needed to prevent long-running or duplicate Flows. To do this, we made sure to cancel the Flow’s Job when the lifecycle of the object finished.

Here’s some examples of Flow patterns:

Handling Lifecycles and Scopes
When possible, we tried to use the built-in Android lifecycle-aware scopes (such as viewModelScope or lifecycleScope), which are automatically cancelled and cleaned up with the Android lifecycle. If we made our own scope within a class, we had to remember to cancel it. So, we found it was more efficient to make suspend function APIs and call them from the view model or lifecycle-aware scopes. (This also helped us with unit testing because we could test the functions directly in the unit test instead of trying to access them from the encompassing class.)

Passing Dispatchers in Class Constructors
We often had to switch between Dispatcher types: using IO Dispatchers for network calls, default Dispatchers for most business logic and Main Dispatchers for UI updates (because Android will crash if there is a UI update on a non-Main thread Dispatcher).

When we unit tested our suspend functions, we found that passing the Dispatcher types through the class constructor and providing the TestCoroutineDispatcher to our class constructor helped us easily mock our asynchronous suspend functions.

Some of our client apps contained legacy RxJava code that wasn’t quite ready to migrate to Coroutines. To support this, we used the kotlin-coroutines-rx2 library to convert between RxJava and Coroutines. The library’s rxSingle and rxObservable functions converted our Coroutines suspend functions into RxJava, and the await() or asFlow() methods converted RxJava into Coroutines.

One important note, however, is that while the library does create both threading services, it does not replace the work of manually migrating code from RxJava to Coroutines. It was simply a workaround that allowed us to iteratively migrate parts of our code to Coroutines, while leaving the remaining parts in RxJava.

One of our challenges was migrating legacy RxJava unit tests to Coroutines. When dealing with unit tests that were previously for RxJava Single results, we used runBlockingTest to make our new suspend functions run consecutively and return the results.

When working with Flows that used to be RxJava Observables, we used a TestCollector pattern (see example below) to gather and verify the values from our Flows.

When we hit a roadblock in getting our expected results from a Flow, we found that most often the issue was that our Flow was not using the TestCoroutineDispatcher in its scope. Having another Dispatcher in use caused the test to fail or be flaky (since it was not a mocked test Dispatcher but a real threaded dispatcher).

Here’s an example of a suspend method test and a Flow test:

With each migration to Coroutines, we saw performance improvements in speed, memory usage, CPU and thread usage. Using Android Studio’s Profiler, we ran debug branch builds to compare our app’s performance before and after the migration. To get reliable metrics, we averaged over six user sessions with the same steps and the same user states.

For more accurate timing data, we used coded metrics that were already built into our app. Our production metrics captured start-up time, along with feature-open and feature-load timing. Since these metrics were gathered from production, the amount of data captured allowed us to average across different devices, locations and user sessions.

Here is an example of results from one of our larger core library migrations:

Migrating from RxJava to Kotlin Coroutines opened up new opportunities and improvements for our app and development teams. Because we are now able to take advantage of the powerful threading of Jetpack libraries like Room and Lifecycle, we can set up and query databases with a couple lines of code and we no longer have to worry as much about lingering threads outside of lifecycles. We have been using Coroutines to revisit our app architecture patterns: cleaning and simplifying them, and improving our test coverage.

As each migration and iteration completes, our threading work becomes faster and more reliable, so our readers can experience Times journalism and the background work truly stays hidden in the background.

Last year, many of the Android teams at The New York Times had an ongoing debate: should we use RxJava or Kotlin Coroutines and Flow? While the question was simple, the answer and technical solution were not. We were using both in our client apps and RxJava in our core libraries, which meant we had two fragmented technologies to support.

My team, the App Platforms team, manages the core libraries and features that are used by Times mobile app teams, and part of our responsibility is to model technical best practices. We wanted to work with our Times Android colleagues to choose the best solution. Many of our colleagues believed that migrating to Kotlin Coroutines would improve performance, the readability of our code, increase developer velocity and speed up unit testing. But, some of our colleagues preferred to stick with RxJava because of its many features. We needed to first build consensus across all of the Android teams on what technology to use as a best practice.

The App Platforms team’s hypothesis was that Coroutines would be the better option for our core libraries, however, we couldn’t rush towards implementation because any technical change to our core libraries could have cross-departmental effects. In addition to building consensus, we had to articulate how the change was worth the product roadmap disruption and the shift in developer practices, and how it would positively impact our business goals.

Here’s how we did it.

Determine the level of effort
Our first step was to determine the level of effort it would take to make these changes. We looked at where the changes needed to happen: in new features, in legacy code and across several apps and teams.

At The Times, we use T-shirt sizing for the level of effort: small, medium, large and extra large. Knowing how long things will take helps establish a timeline for the change and allows teams to define the work based on priority, dependencies and difficulty. T-shirt sizing does not have to be exact — it helps to approximate how long a project will take. A good rule of thumb is if the size of a given problem is greater than medium, the problem itself needs to be split or spiked to get a more reasonable, smaller estimate.

For the Kotlin Coroutines project, the total project T-shirt size was large because we needed to migrate over four legacy core libraries and integrate them into our client News app. From there, we split each effort into the specific libraries that needed to be migrated (size: medium) and the integration (size: small).

Get everyone on board
Because we wanted to make changes that included new features, refactoring old code and future-proofing, we had to identify who needed to be convinced that this work was important. We needed advocates in Product to prioritize the work, partners in Engineering who could provide feedback and share the workload, and support from QA to identify any testing concerns. Product and Engineering were interested in Coroutine’s promise of performance improvements and increased developer velocity. QA approved the proposal to integrate regression testing as well as the improvement in unit testing.

Assess the idea and get feedback
We use a Request for Comments (RFC) process at The Times whenever an engineering team sets out on a big project. The idea behind an RFC is to document the changes we would like to make and to get feedback from our peers. This exercise includes documenting and addressing any drawbacks or alternatives to the proposed solution, and it helps prepare us for any issues that might arise.

As part of our RFC, we considered drawbacks and we proposed solutions:

• To use Kotlin Coroutines and Flow, all files would need to be migrated from Java to Kotlin. We already had an engineering decision that we wanted to move to Kotlin and any files that had to be changed should be moved to Kotlin, so this ended up being a non-issue.
• The time and effort required to do the conversion were significant, and making new features might slow developer velocity. However, the benefits from the conversion — improved performance, easier testing — outweighed the level of effort and made the project a high priority for the product team. The new technology would also help developers improve their velocity over time.
• Some teams still using RxJava needed a way to access the new Kotlin Coroutines and Flow APIs. We proposed using kotlin-coroutines-rx2 library, as well as backwards-compatible RxJava APIs (that use Coroutines behind-the-scenes) to support teams who could not migrate immediately.

By going through this RFC process for the migration, we were able to address concerns that came up around our technology change.

The App Platforms team uses tech spikes to create proofs of concept. We try to keep our tech spikes to a maximum of five working days. If our spike appears it might take longer than five days, we reevaluate the problem and think about ways it might be broken down into a more manageable bucket of work. The data generated from a spike can help strengthen the case to make a technical change on a larger scale.

For us, quantifiable results can come in many different forms:

• The change reduces the number of crashes or bugs. For example, if the app routinely crashed, we could compare the original build (with crashes) to the spike build (without crashes) and document the percent-change in results.
• The performance improves across different metrics, such as speed, CPU, memory, thread usage and network data usage.
• The change makes developers’ lives easier. There are fewer lines of code that have to be written, code test coverage percentage changes and developer velocity improves.

To provide quantifiable evidence that our migration to Coroutines would improve our app, we set up a five-day tech spike. In a small core library that integrates with The Times News app, we migrated all instances of RxJava to Coroutines, updated the unit tests in the app and integrated into a sample app and the News app on a branch. Then we used Android Studio’s Profiler to compare the average CPU, memory and thread usage of the branch with the News app. We also put in code timers to calculate the differences in timing.

With our spike results, we saw an improvement of 34 percent in CPU usage, 13 percent improvement in memory usage, 12 percent improvement in thread usage, and 21 to 63 percent improvement in speed on parts of our News and sample apps that touched the updated library.

With this strong performance improvement data, we were able to reinforce the enthusiasm of our advocates and convince those who were skeptical that migrating our core libraries to Coroutines and adopting Coroutines as best practice threading technology across Android teams was the best move.

Once we knew that we would move forward with the conversion to Kotlin Coroutines and Flow, we had to set up a timeline for the work.

For us, our plan consisted of these steps:

1. Define the timeline. A timeline makes it easier to envision and enact a plan. It also prevents dependency conflicts and keeps track of work in progress. Short timelines are best (half a year, at most. Tech fatigue is real). Our timeline started after our RFC (which took about one month to spike, write and approve), and we planned for about two months of work.
2. Incremental plans are best. We found that breaking a large plan into smaller pieces that could be developed, completed, merged and then improved upon later allowed us to achieve each goal faster and avoid a giant, unmergeable branch. Our migration plan was broken into separate core library releases and integrations. We then had several developers working on each library in parallel and created a flexible timeline with other library changes.
3. Think backwards-compatible. Some of our client app teams were not ready to move away from RxJava. So, we created RxJava-compatible library modules for them to use, and suggested using the kotlin-coroutines-rx2 library as a workaround to allow developers to transition between RxJava and Coroutines.
4. Ensure a smooth transition. Consider the level of effort for all teams involved, such as QA, and communicate widely about the changes coming up. To reduce bugs, we worked out the testing plan for feature branches with our QA team and added unit test coverage requirements.
5. Record quantifiable results. These can be used to demonstrate the value and impact of the work being done to management. As we completed each migration, we measured and published the results of the work, making sure the changes were creating performance improvements.

All core library releases included release notes as well as updates by our devs to the working team at large of any breaking API changes or library releases. Each change also included performance testing, as well as documentation. As we worked through our game plan, we demoed the performance improvements for the entire team — gaining advocates in product, engineering and management along the way.

Once we got engineering and product approval to add our proposal to the roadmap, we consolidated our working group to divide up the work. During the migration process, we kept an eye on our proposed timeline to make sure we didn’t have any scope-creep or blocked items.

The migration vastly improved different areas of our core libraries and News app integrations. Kotlin Coroutines sped up feature creation, significantly improved app and library performance, and simplified unit testing. Moving forward, we’re building all new core libraries and features with Coroutines and Flow and refining our best practices working with Coroutines. We’re also working with the Cooking app team in an advisory capacity as they migrate to Coroutines. Overall, we believe that the work we did to assess, prove and create a proposal for change is paying off in dividends.

We built a system that allows a single team to implement data privacy changes across a suite of over 70 products.

May 6 · 8 min read

By Kelsey Johnson

When I first started working on the Data Governance team at The New York Times in 2017, I would often be met by blank stares when I tried to explain my job. Over time, I perfected my elevator pitch: I work on privacy, ethics and governance at The Times, and it’s a bit like herding cats.

As the second member of The Times’s Data Governance team, I dove straight into the high-profile General Data Protection Regulation (G.D.P.R.) project, which required that our organization follow strict ground rules for handling the personal data of our users located in the European Economic Area. Our team of two had to simultaneously discover every way our business operations were processing the data of our users, all the while creating and implementing rules to honor our users’ G.D.P.R. rights.

Two people, over 70 product teams and five months to get it all in place. Don’t let anyone tell you herding cats is easy.

G.D.P.R. was just the beginning. There are now over 100 privacy laws in other countries. In the United States, there are numerous bills that have either been introduced into state legislatures or have been signed into law.

Not only do each of these laws apply to specific geographic regions, but they each tend to have their own unique requirements. This means that companies like The Times have to implement complex sets of rules to ensure their websites and apps are in compliance no matter where users are located. New regulations often go into effect with short grace periods, so companies have very little time to make changes to their technology stacks.

Because technology is constantly evolving, legislation will need to adapt as technology changes. This means companies need to be ready to modify their interpretations of the law anytime a law gets amended. That leaves companies with two options: to reactively respond any time a change is required, or to invest in privacy as part of their business strategy and dedicate resources to the task.

The Times has chosen the latter.

It was 2019 when we retired our cat-herding hats and harnessed our brain power to build PURR: Privacy, Users, Rules and Regulations.

PURR is our homegrown system that operationalizes The Times’ privacy offerings. The system centralizes our business logic and rules, and it communicates with our front-end products, instructing them on how to carry out the privacy rights of each unique user that visits us. It also connects to an internal preferences system that allows us to securely save and sync logged-in users’ privacy preferences across all of our products.

[We’re hiring. Come work with us!]

This enables us to easily and efficiently adjust our interpretations of data regulations and it simplifies the implementation of such changes across our suite of products.

Because of PURR, we have said goodbye to year-long privacy projects and roadmap disruptions whenever a new regulation is passed. Now, a single team can independently implement most changes. Purr-fect, right?

I like to think of PURR as a privacy machine. It consumes information about a reader, analyzes that information based on knowledge it has been given about our company’s interpretations of privacy regulations, and then outputs instructions on how that reader should be treated from a privacy perspective when they interact with one of our products.

Each Times product, such as News, Cooking or Games, is required to ask PURR for instructions on how to treat every single reader that visits. These instructions are called Directives.

There are several types of directives that a product needs to be given when it comes to privacy — right now, there are eight. We have broken them down into two categories: user interface directives and data handling directives.

The former tells products when they need to show a certain element on the page, such as a cookie banner, a marketing consent checkbox or an opt-out button. The latter tells products how to handle a user’s personal data. For example, a data handling directive might block certain third party tracking mechanisms from collecting data about a user.

Directives are sent via three different methods: request headers, a cookie called `nyt-purr` and JSON. These give Times products the flexibility to choose how they want to consume the information from PURR.

Let’s say you are in Europe and you visit our NYT Cooking homepage to get some inspiration for your weeknight dinner. Upon loading the page, you will be shown a banner, like this one.

This is what we call our G.D.P.R. Tracker Banner. NYT Cooking showed it to you because it received the “user-interface” directive called PURR_DataProcessingConsentUI from PURR. When you loaded the page, PURR knew you were in Europe and G.D.P.R.-eligible, so it delivered a value of show with the PURR_DataProcessingConsentUI directive, which told our products to present you with a banner that allows you to opt in or opt out. If, however, you visited NYT Cooking from New York City, the directive would have delivered a value of hide.

This is just one example of a directive. Some other examples include PURR_AcceptableTrackers, which tells products what type of trackers can be fired when a user visits a page; PURR_DeleteIPAddress, which instructs our first-party analytics tool whether to delete the IP Address of a user; and PURR_AdConfiguration, which tells products what types of ads are permitted on the page, such as non-personalized ads instead of behaviorally targeted ads.

For PURR to know what directives to issue to products, it needed to be programmed with privacy logic, which we refer to as Rules. The Data Governance team works in collaboration with the Times Legal and Engineering departments to write these rules.

Rules are written expressions that PURR must evaluate. They can become quite complex, but a rule basically says, “Hey PURR, you need to look at those inputs, and when these circumstances are met, you need to issue these specific directives.” It’s because of rules that we can provide a custom privacy experience to every user who visits our site.

Let’s go back to the NYT Cooking example. In order for PURR to know whether a user needs to be shown our G.D.P.R. Tracker Banner, it first needs to know whether the user is located in a country that is subject to the G.D.P.R. and whether they’ve already opted in or opted out.

In plain text, the rule would be written like this: IF user is in a G.D.P.R.-eligible country AND user has no set preferences with NYT, THEN show banner.

In SQL, the rule gets translated into the following expression:

PURR_DataProcessingConsentUI SQL Expression RulesCASE
WHEN ref.PURR_is_gdpr_eligible
THEN
CASE WHEN ( -- people who are gdpr eligible and have 
not opted out or in
gdpr_pref_regi = "none" AND
gdpr_pref_agent = "none"
)
THEN
"show"
ELSE -- people who are gdpr eligible and have either   
opted out or in
"hide"
END
ELSE -- people who are not gdpr eligible
"hide"
END

In order to operationalize Rules and Directives, we needed systems that could bring them to fruition. We first had to find a place where we could write and save the rules — a rules producer, of sorts.

Luckily for us, we didn’t have to build anything from scratch. In recent years, The Times had built ABRA (which is short for “Abra Basically Reports ABtests”), our homegrown testing and targeting platform. Because ABRA provides the ability to translate human-friendly rules into a widely adopted, open-source and machine-friendly format, it was a simple engineering choice to expand its capabilities to support PURR.

In addition to a system that would allow us to write and save the rules, we needed a way to run the rules. We refer to these as Rules Executors, which ingest the ABRA rules and execute on those rules in order to send directives to our products.

Because the Times has over 70 products and a diverse landscape of technical stacks, we wrote three separate Rules Executors: one for products under the nytimes.com domain that use Fastly; another for products that are under the nytimes.com domain but do not use Fastly; and one for products not on the nytimes.com domain, such as our vendor-managed sites.

It’s important to note that the PURR system does not stand on its own. It relies on several other capabilities within The Times’s data and core platforms, such as a homegrown system that keeps track of our user’s data privacy preferences. It also interacts with Samizdat, another homegrown system that our native apps leverage to integrate with PURR. And lastly, we built a customized AMP-PURR endpoint which allows us to integrate with our Google AMP products.

We have proven that PURR can adapt as new privacy laws are passed. When the United Arab Emirates passed a data privacy law in the summer of 2020, we were able to bring over 60 products into compliance within four working days and with zero roadmap disruptions to the product teams. When Brazil’s privacy law came into effect shortly after, it took us only one day to ensure our products were compliant.

Over time, we hope to implement new directives that foster privacy-forward features in our products. And as The Times explores new product offerings, such as NYT Kids, we believe PURR will be a reliable way to ensure such products are ethically built while also meeting strict legal requirements.

Although our cat herding hasn’t been reduced to zero, we have learned that implementing data governance regulations doesn’t need to be a resource-zapping, roadmap-destroying burden. It can be an opportunity to innovate and provide users with a privacy-focused digital experience.

If you’d like to learn more about PURR, check out a full talk we gave at a recent Privacy_Infra() event. And if you are seeking to expand your network of privacy peers, don’t hesitate to email us: data-governance@nytimes.com!

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

May 6 · 7 min read

What are your pronouns?
He/Him

What is your job title and does it mean?
Senior Software Engineer. It means I have significant experience and I can not only solve complicated technical problems, but I can also mentor and provide guidance to other engineers.

How long have you been at The Times?
On July 15th, it will be eight years.

Most Times employees are working remotely right now. What does working from home look like for you?
I have an office in my home. I also have a nice window to the outside world that includes a bird feeder; A look to the right and I’m virtually outside. Working from home essentially means I enter my work space, log into my laptop and get started. But before I start, I usually take the dog out for a walk and then I make coffee, so getting in the rhythm of work means grabbing my coffee and reviewing the tasks for the day. My day usually starts with a good morning announcement in our group’s development channel on Slack.

Back in the 1990s, when I got my first job where I could work from home, I learned pretty quickly that the best thing about working from home was that my bedroom was 20 feet from my office, and the worst thing about working from home was that my bedroom was 20 feet from my office. Nothing has changed since then.

How do you start your day?
I check our Slack channel to see if the clients who use our services contacted us about anything, then I check email. After that, I scan Slack to see if any of my teammates had anything to say while I was away and then I review yesterday’s work and get ready for our stand-up.

What is something you’ve worked on recently?
Our subscriptions system is part of a larger monolith known as Plato. We are in the process of building a microservice to extract from Plato the management of subscriptions that come to us from Apple through iTunes. So today, I’m working on extracting information about the subscriptions from the Plato database and figuring out how to get that data into the new microservice database.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
About two years ago, we built a microservice called the Cohort Manager, which allows us to adjust subscription packages in response to marketing, production or pricing changes. For example, when a reader purchases a New York Times subscription, they choose a particular bundle of products, such as seven-day home delivery or weekend-only delivery. Sometimes we need to update the bundle for a subset of subscribers, like when a newspaper distributor decides to stop offering Saturday deliveries to a particular region. To respond to this change, we would have to move readers subscribed to a bundle with a Saturday delivery to a bundle without a Saturday delivery.

It used to be that these changes had to be done by hand, but now the Cohort Manager moves the list of subscriptions from the original bundle to the new bundle. This was the first microservice we built, and I had an instrumental role in making it happen. It’s been working consistently for over two years with very few errors.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently?
I had a significant health problem arise a couple of years ago. It had a big impact on me and I didn’t do a good job of communicating with my team about my need for time away from the office. It caused a great deal of tension on my team at the time, and I was responsible for not explaining what was happening to my colleagues. Knowing what I know now, I would be more communicative with my team and perhaps take some time off when dealing with these types of situations.

What was your first job?
My first computer-related job was in 1979 when I worked for a small retail computer store called Mr. Computer. I was the service manager and I was responsible for both working on the floor selling computers, and for dealing with repairs and returns if customers had issues. I had this position for a little over a year and I learned a lot about how computers work and how to deal with people. The highlight of my job was a trip to King of Prussia, Pennsylvania. I drove from Poughkeepsie, New York to King of Prussia in my 1967 Triumph Spitfire. I took a three-day class on how to repair the Commodore PET, one of the very first personal computers that was a contemporary of the original Apple (which we also sold). It was my first time traveling for work.

What is something most people don’t know about you?
In April of 1984, I was working in Boca Raton, Florida on a co-op assignment through my school, the Rochester Institute of Technology. On the 6th of April that year, there was a launch of the Space Shuttle Challenger. I drove up the night before and parked on the road about 10 miles from the Kennedy Space Center. I was parked by myself that night, but when I awoke the next morning, there were cars parked all along the road. At 8:58 a.m., we all got to watch as the Challenger launched. It was probably one of the most amazing things I’ve seen in my life. Listening to hundreds of cars radios all tuned to the same broadcast was something I’ll never forget.

Do you have any favorite life hacks or work shortcuts?
Simplify and isolate. When you’re having a technical problem, or you’re trying to understand how a new framework works or how a new piece of technology is manipulated, the best thing you can do is simplify it to its essence. Extract your work from the bigger problem and isolate the solution so that it’s focused on just the one thing you need to figure out. By doing this, you make the number of variables you have to figure out as small as possible. This technique — simplify and isolate — has served me well for most of my career.

What is your superpower?
There are no superpowers; there is only observation and your own mental model of how things work. Find a path that describes the situation, take the model you have in your head — that is, your understanding of how things work — and bend it, twist it, change your viewpoint. Try to find the place where your model and the world clash, then figure out what you need to change to account for this new observation. Divide and conquer.

What or who are you inspired by?
My parents inspire me. At 85 and 88, they are still terrific people to whom I owe so much of who I am. My father spent 25 years at IBM as a programmer (what we called people who do my job back when things were complex in different ways). And my mother worked both in the home and as a computer lab manager for the local community college. I take inspiration from how they have lived their lives and aspire to be as good a person as I can be.

Complete this sentence: Over time, I have realized __________.
There’s often a push to bring new ideas to market as quickly as possible, and in my professional experience, there is often immense pressure to rush new features into production. However, a feature that has been coded for the purposes of demonstration might not stand up to the demands of production code.

One solution I have found is to implement a proof of concept in a technology stack that isn’t available for use in production. In doing so, I can quickly demonstrate the feature and potentially expose its pitfalls. And because the technology stack isn’t approved for production work, I can be reasonably certain that this lightly engineered and potentially weak proof of concept isn’t going to be rushed into production before it’s ready.

Another solution we use extensively at The Times is to limit the scope of new features to a minimum viable product, or MVP. This reduced set of capabilities is more easily engineered and can be tested in production. We often try these MVPs out with a small portion of users to see if it makes sense to continue down that path. The advantage of this technique is that the time to market is reduced and redundant work is avoided.

What is your best advice for someone starting to work in your field?
Software engineering has grown exponentially for my entire 35-year career. The number of developers doubles every five years. Keep those numbers in mind. In five years in this business, you’ll have more experience than half of the people you meet. Technology changes significantly every two to three years. So, if you want a long-term career, keep your mind focused on how you analyze problems and don’t get too wrapped up in the technology that you use to solve those problems. Chances are good that your problem-analysis techniques will last your entire career, but your technical skills will change every five to seven years.

When a team of editors, designers and strategists teamed up talk about how Times Opinion coverage is presented and packaged to readers, they thought of a dinner party.

Apr 27 · 6 min read

By Dalit Shalom

How we rebuilt a 20-year-old system that lets New York Times photo editors search through our vast store of photo outtakes.

Apr 22 · 6 min read

By Kate Brenner

On an assignment, a New York Times photographer might take hundreds of photos but only file a handful of their best shots. These best photos, often referred to as “selects,” are typically uploaded directly from the photographer’s camera to our CMS for a photo editor to access. It is a workflow that is geared towards efficiency, and to get the news out as quickly as possible.

But just because a few photos have been chosen to publish doesn’t mean that the outtakes are unusable or irrelevant. Many of these photos do in fact get published, sometimes days or even years later. Over time, The Times has collected these outtakes, which number more than 150 million, and has stored them in Amazon Web Services (AWS) storage gateways.

For Times photo editors to access the outtakes, an in-house team built a system called Scrounger, which was written as a series of Perl scripts that ran on an AWS server. Scrounger had a search UI that editors could use to find photos, but only if they knew the numerical assignment ID that is generated with each news assignment.

If an engineer needed to access the photos on the back-end, they had to connect to a remote desktop and sift through the storage gateways rendered as network drives. While this was a challenge in itself, there was another major flaw: no descriptive information was stored in a database, which meant there was no efficient way to query this massive collection of images. The only searchable information was in a photo’s file path, which included the date of the assignment, the assignment ID number and the file name, which looked something like this: 20210401/12345678A/test_photo_1.jpg.

In the past few years, the Times Photo Team has been systematically updating our photo management software. After updating the search algorithm for our main search tool for selects, we set our sights on building a system for outtakes — a Scrounger 2.0.

Some of the Photo team’s work has included consolidating our systems onto a single cloud provider — we use Google Cloud Platform (GCP). Our first step in creating Scrounger 2.0 was to transfer all of Scrounger’s photos to Google Cloud Storage (GCS) to make them easily accessible to our system.

This was more complex than we anticipated. Because the photos were stored in AWS storage gateways, they first had to be synched to an Amazon S3 bucket before they could be transferred to GCS using Google’s Storage Transfer Service.

Luckily, we were able to preserve the file path for each photo. This was a huge win because most of the information we had about these photos was in the file path.

Once the photos were in GCS, we had to determine the best way to efficiently ingest and store them. We built a database to keep relevant information about each photo, and we created a flexible schema that could be updated as needed.

We built four tables in our new system.

1. The Metastore table, which was the raw dump of all the information we had about an incoming photo and included all image metadata stored as JSON. This table was set up with a trigger to send each row to the second table.
2. The Event Queue table, in which rows were stored temporarily until they were processed and inserted into the third table.
3. The Assets table, which contained columns for metadata fields, such as description or credit, that were relevant to the photo. If we need to update the schema of the Assets table, we can “replay” ingestion and resend all rows from the Metastore table to the Assets table, without duplicating any records.
4. The Search table combined several columns from the Assets table, such as the identifier, creator, description, filename and assignment ID as a tsvector, so we could apply a full-text search on these fields. We could have our API query for assets on many different query parameters.

We built an application called Gateway as the initial point of ingestion for all past and future outtakes. Gateway listens for notifications of incoming images and runs exiftool on each image to combine its image metadata with GCS metadata and store all of that information in the Metastore table. An application called Megaphone listens for database events and publishes them to two more applications, Materializer and Enhancer. Materializer scans and externalizes specific metadata fields, such as photographer and caption, and stores them in the Assets table. Enhancer updates the Assets table with additional information from The Times’s assignment database, such as credit or photographer.

Combining the information from the original file path, externalizing several different forms of metadata, using information gathered from the assignments table, and storing all of this information in our new schema enabled us to make these photos searchable.

We wrote an API in python to interact with our database, and we created a UI for Scrounger 2.0 in React.

Once the new system was built, we ran the migration and kicked off ingestion for all 150 million photos. We created a temporary manifest from the image dump from the old system. For each photo in the GCS bucket, we extracted and parsed its original storage path to obtain the assignment ID, assignment date, file name, file size and file extension. We then put all that information into the manifest so we could perform queries and learn more about the photos in the image dump.

From there, we set up ingestion into the new system. We wrote a script that iterated through the manifest and applied each image’s metadata to the corresponding GCS object. Our GCS bucket sent notifications to the Gateway service with each metadata update.

To verify that all 150 million records had indeed been transferred over, we wrote a verifier script that iterated through the metastore table. For each Scrounger 1.0 entry, the script ensured that every row had been ingested into Scrounger 2.0 and it updated the temporary manifest with a status of “verified.”

Because the old system stored multiple versions of each image — such as high resolution, medium resolution and thumbnail — we needed to deduplicate them. Storing all resolutions from the old system would take up too much space in the Assets table, and it would create noisy search results in the UI. We didn’t want to permanently delete any photos, so we added a special metadata field that leaves the files in the Metastore table but deletes them from the Assets table. Since we still need different crops for different views in the UI, we created an API that can cut and cache renditions on the fly without taking up unnecessary storage space.

Since we launched Scrounger 2.0, our team has implemented enhancements around search, such as sorting and filtering capabilities. Now, in addition to assignment ID, Times editors can search by a variety of terms, such as photographer name or keywords that might be found in a photo’s description.

There is plenty more work to be done on this system. Not every outtake was stored in Scrounger 1.0; some outtakes were stored on DVDs in the Times newsroom. Now that our new system is up, we hope to finish transferring all of these images to help preserve the historical record these photos contain.

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

Apr 21 · 3 min read

What are your pronouns?
He/Him

What is your job title and does it mean?
Lead Software Engineer.I work with Customer Care, specifically on the Insights and Analytics team. We build platforms that gather and analyze data that help improve our readers’ experience when they contact The New York Times.

How long have you been at The Times?
It will be two years in July.

Most Times employees are working remotely right now. What does working from home look like for you?
I work from my cozy home office in central New Jersey.

How do you start your day?
My day starts with a short drive to drop my daughter at daycare. When I get home, I prepare my espresso to get my workday started.

What is something you’ve worked on recently?
We built an automated Machine learning pipeline to tag and categorize customer interactions to help our product colleagues prioritize feature work and make decisions based on the generated data.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Our team migrated our Contact Center to Amazon Connect and built an intelligent Interactive Voice Response to assist customers with their requests. It was a large project that involved a number of services including, AWS Lex and Lambda. I presented this project at AWS re:invent in December, 2019.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
Switching from accounting to software engineering wasn’t the hardest decision to make, but the challenge was in achieving the career change. The hardest part was finding the support and help to guide me along the way. The one thing that I would change is reaching out to more people in the community for their perspectives and to get more support.

What was your first job?
When I was 16, I started my technology adventure by fixing computers at an internet cafe in Alexandria, Egypt — I also learned how to plan and install wired networks. It was fun, but I learned that I love to build and architect systems more than networks.

What is something most people don’t know about you?
I am an Advanced Nitrox-certified scuba diver, and my last maximum dive depth was 102 feet. Hopefully, I will be able to travel and break that personal record soon.

What is your superpower?
Building proofs of concept is a powerful instrument that I often use to quickly verify ideas and solutions without investing too much time or energy until the idea is proven to work.

What are you inspired by?
My wife. She is very supportive, hard working and my go-to person when I need advice on complex problems.

Complete this sentence: Over time, I have realized that __________.
Learning has no limits and every day there is something new to learn.

What is your best advice for someone starting to work in your field?
Passion is the critical element to success in anything. It will guide you through your career and keep you thirsty to learn more and complete more achievements that you can be proud of every day when you look back.

How we brought site reliability engineering standards and thinking to The New York Times technology teams.

Apr 15 · 7 min read

By Tim Burke

When The New York Times launched its first website in 1996, it had already been using software to send the layout of the next day’s paper to the printing presses. The website introduced some basic web technology into The Times’s portfolio. Twenty-five years later, that portfolio has grown to include systems that deliver articles, documentaries, newsletters, games and recipes, both online and in print, to millions of readers every day.

It’s a technology landscape that is diverse and complicated. Our systems use both bleeding-edge and legacy technologies that are deployed to multiple clouds and data centers all over the world.

Because of the breadth of Times products, our engineering teams have the freedom to make their own decisions about their technology stacks and the architecture of their systems. As a result, some of The Times’s systems have been around since before Amazon Web Services had a web console, and other systems use the latest tools from Google Cloud Platform. With such a broad portfolio comes a range of operational thinking and preparedness across our engineering team.

Over the past couple of years, the Operations Engineering team (formerly Site Reliability Engineering) has worked to foster a culture of operational maturity that allows our engineers to efficiently build and maintain our technology while ensuring that we deliver a consistent experience to our readers.

Part of our work has been to define what operational maturity means in the context of The Times. For us, it is requiring teams to take a hard and objective look at their practices and ask, “how is she though?” Does a system have a disaster recovery plan? Does a system have a failover configuration in place in the case of a cloud provider outage? Does a team ineffectively translate business metrics into supply metrics and just over-provision and pray? Ultimately, we want teams to recognize that failure is a part of growth, and consistent recovery from failure comes from having mature practices and playbooks for what to do when things go wrong.

We took inspiration from the Production Maturity Assessment released by Google’s Customer Reliability Engineering team, and using their example, we developed our own Operational Maturity Assessment to provide insight into the operational maturity of The Times’s engineering teams.

The Operational Maturity Assessment is an activity that teams do together. A member of the Operations Engineering team facilitates each session, and the activity is designed to foster discussion and self-reflection within the team. (When we designed this assessment, we imagined teams gathering in person to eat pizza and complete it together. But, that was before the pandemic. Alas.)

Each session lasts from one to two hours and a member of the team drives the assessment from their screen using a tool we built. The assessment tool we built was inspired by tax preparation software and breaks six top-level categories down into manageable sections.

As the team moves through the categories — we will get into them in a minute — they answer prompts around topics that are intended to inspire conversation about whether their systems meet the maturity expectations of The Times. We want the assessment to provide a framework for thinking through parts of the system that may need to be addressed and prioritized alongside new feature development work.

The assessment goes over our Operational Maturity Model which lays out the dimensions and scales of operational maturity at The Times along six categories:

Monitoring and Metrics: Does the service have defined service-level indicators, objectives and agreements in place (things like uptime or ability to handle a certain number of requests per second)? Are data about those metrics visualized consistently and in an easy place to find by the team and downstream system teams? Do the metrics accurately reflect the experience of users?

Capacity Planning: Do the product and engineering teams work together to forecast business metrics, such as users or sales, and convert those into supply levels like servers, storage or bandwidth? Does the team acquire capacity based on forecasted business and supply metrics, and is utilization of that capacity monitored for efficiency?

Change Management: Does the team conduct releases at a predictable cadence; do they have defined practices for reviewing changes and releases; and do they automate most or all of the work of doing releases via continuous integration and deployment tools?

Emergency Response: Does the team have an organized on-call rotation? Does the team regularly analyze automated alerting to make sure engineers aren’t bombarded with alerts that don’t require human attention? Does the team conduct effective learning reviews after incidents, with metrics, lessons learned and clear steps to prevent issues in future?

Service Provision and Decommission: Does the team use infrastructure-as code-tools? Is the tooling standardized across the team, department and organization?

Reliability: Does the team design and deploy their systems to be high availability, such as running workloads and data storage across multiple data centers or availability zones? Does the team have a documented disaster recovery plan with automated backups and regular testing? Are the team’s services deployed in multiple geographic regions or with multiple cloud providers?

The questions in each section help drive discussion within the team. This discussion aims to educate teams on best practices and inspire new ideas.

[We’re hiring for our Delivery Engineering team. Come work with us.]

As the team works through each category, they score themselves on a scale of one to five against a maturity rubric; Each section includes a range from “chaotic” to “continuous improvement.” For example, from the Reliabilitycategory’s sub-topic, Multiple Region Support:

1. Chaotic: Service runs in a single region.
2. Managed: Service runs in a single region but can be spun up in a second region.
3. Defined: Services run in multiple regions with manual failover.
4. Measured: Services run in multiple regions with automated failover.
5. Continuous Improvement: Services run in multiple regions, across clouds with automated failover.

After completing the assessment, the team is presented with a report that outlines areas of strength and areas for improvement, compared to all other engineering teams at The Times.

Not only does this process encourage engineering teams to discuss their strengths and weaknesses in a structured way, it gives them real data about technical and architectural debt that needs to be prioritized alongside new feature work. This also helps teams communicate their reliability improvement needs to company leadership.

Our team, Operations Engineering, collects these team-level scores and collates them on a dashboard that allows us to see trends across the engineering organization. In aggregate, trends help us, and the leaders in engineering and product, understand where the organization as a whole needs to focus time and resources to drive improvement.

In the visualization below, each column belongs to a category, while each row belongs to a team. The columns show us cross-team trends within each metric. By using a color range of light orange to dark orange, we can see how well teams perform in a particular metric.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

What are your pronouns?
She/ Her

What is your job title and what does it mean?
Associate Software Engineer

How long have you been at The Times?
Almost a year and a half now!

Most Times employees are working remotely right now. What does working from home these days look like for you?
I wake up, make myself a latte and figure out what I’ll have for breakfast. Then, I’ll catch up on some news and emails, get ready for my first meeting and start off my workday. Around 1 p.m. is when I tend to have lunch. Thankfully, I live in a building with an amazing rooftop so I take a nice walk around to get some fresh air and then come back and continue the rest of my day. After work I try to find the willpower to work out or do something productive.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
I recently started working on an internal tool that allows us to build and deploy reader segments in a way that preserves reader privacy. I love working on data-based apps that pull together information about readers’ habits.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
Career transitioning. I used to be a technology risk consultant who would perform technical audits. I realized it wasn’t what I wanted to do and that I really wanted to build things. I decided to just quit my job and start this new chapter in my life. If there was anything I could go back and change, it would be accepting that I genuinely wanted to be an engineer and follow that career path sooner.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
Collaboration! I think we have a really great team and we are always working on trying to share knowledge with each other. Especially now that we are all working remotely, I find that collaboration is key to stay sane at home and to bond as a team.

What is a goal you hope to accomplish this year?
To get promoted!

What change do you hope to see in your community?
More diversity. I know imposter syndrome is very much a real thing and it can be discouraging and lonely to be in a field where no one looks like you. You have to work harder just to feel like you belong.

Do you have any favorite life hacks or work shortcuts?
This is not so much a life hack, but remember to always take breaks! Mental breaks are so important especially now that we’re at home and constantly staring at screens. It’s important to remember that it’s O.K. to take time to recharge.

What or who are you inspired by?
My parents. To this day I still don’t understand how they have managed to be successful and raise a family with the limited resources and lack of education they had growing up. I always remember this to motivate myself to keep going.

Fill in the blank: What is the best part of being a Latina engineer? If you had it your way, what could make it better?
The best part of being a Latina engineer is being a role model for other women and girls in this field. I wish I had a mentor who looked like me and who I could go to with questions, but the most I can do at this moment is be that person for someone else. If I had it my way, the industry would already be diverse, and equal opportunity and pay wouldn’t be issues. We are not there yet, but I’m hopeful.

Complete this sentence: Over time, I have realized __________.
Unless I’m on call for something, I’m not saving lives and the work can always wait till tomorrow.

What is your best advice for someone starting to work in your field?
Remember to still have fun! You have a ton of time to learn and things are always changing, so don’t feel like you have to rush and know it all at once. Remember, it’s all about learning.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

What are your pronouns?
She/Her

What is your job title and what does it mean?
I am a program manager who works primarily on engineering projects. I love bringing structure and order to messy and hard problems or situations. I will hustle as I need to in order to enable my teammates to perform to the best of their abilities. I hate the term “servant leadership,” but the definition suits this role.

How long have you been at The Times?
I’ve worked at The Times for two and a half years.

Most Times employees are working remotely right now. What does working from home these days look like for you?
I moved to Rockaway Beach, NY during the pandemic and I hope to stay here. In my opinion, this is one of the most beautiful places in New York City and the quality of my daily pandemic walks has greatly improved since moving here.

There are more animals than people in my household. My colleagues regularly see my cats and dog popping into my meetings. This has been one of the better parts of work-from-home life.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Though I loved my work on election readiness, the rollout of Oak, our custom article editor, to the newsroom was especially rewarding. When I arrived at The Times, a small subset of the newsroom was using Oak in limited capacity. In what now seems like a very short period of time, the Oak team fully built out the application and successfully released it to the greater newsroom. The technical challenges that were undertaken (such as track changes, collaborative editing, version history) were some of the most complex of my career. I will always be proud to have worked on this special team.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
For some time, I battled with regular bouts of imposter syndrome. Finding my voice and gaining confidence in that voice has not been easy. I have faced gatekeeping, which is not uncommon for women working in technology. Earlier in my career, I was told repeatedly, “you aren’t able to do this work” or “you can’t take on this project,” despite my track record of sound performance. I later learned that the gatekeeping stemmed from office politics happening above me.

If I could go back in time, I would have immediately removed myself from that situation. It took some time for me to prove to myself that I was capable of that project and much more. I’m grateful for those who have helped me build my confidence and I’m trying my best to pay it forward.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
Collaboration. Program managers infuse collaboration in all that we do. Our job is to reinforce the value of collaboration and ensure our teams work well together.

What is a goal you hope to accomplish this year?
I would like to learn to surf! I was too busy in the fall to take lessons and don’t have what it takes to embark on this hobby in the current temperatures. I’m looking forward to spring!

As a program manager working on a digital product, how do you approach your work with inclusivity in mind?
I believe you should approach everything, including your work, with your own reflexive statement in mind: I see the world in a certain way because paradigms, people and places have structured this viewpoint. This shapes my biases, the questions I ask and how I operate. I strive to remember that everyone around me comes from a unique place that I only see a small part of while we work together each day.

It’s non-negotiable that our work environment needs to be empathetic and supportive of all people. I (along with my teammates) strive to bring psychological safety, transparency and respect to all that we do. If we’re in a situation where any of this is challenged, we need to talk about it.

What change do you hope to see in your community?
Since I moved to the Rockaways during the pandemic, I haven’t been able to get involved in my new community as much as I would like. While many service activities have been shelved for now, I’m looking for ways I can safely contribute and meet my new neighbors!

Do you have any favorite life hacks or work shortcuts?
The basics truly keep me going. I take my own set of notes in every meeting, because writing things down helps me better process information.

I also live by and for to-do lists. I have a main to-do list that is maintained digitally. At the start of each day, I look at the list and prioritize what I want to get done. This manageable checklist goes on a Post-it. It feels great to cross things off with a pen.

What or who are you inspired by?
This has been a tough year for all of us. However, I remain in awe of our New York Times journalists. The combination of an ongoing pandemic and the 2020 election have presented a grueling and relentless news cycle. I have immense respect for their ability to keep going and provide the public with incredible journalism. Helping to serve as stage crew for these efforts has been a great honor.

Fill in the blank: What is the best part of being a New Yorker? If you had it your way, what could make it better?
The food and dining in New York! We are privileged to have endless options and standards of excellence embedded in our culture. Like many others, I’ve watched restaurants I’ve loved for years close during the pandemic. In reality, it’s been hard to operate a restaurant in NYC for way too long. If I had my way, we would look at the circumstances that have made it so difficult for these small businesses.

Complete this sentence: Over time, I have realized __________.
It is wise to pick your battles. Some goals are best tackled slowly over time. A wise friend put the phrase “Pack your patience” into my brain and I often repeat it to myself.

What is your best advice for someone starting to work in your field?
Take the time to listen. Challenge yourself to see problems from different perspectives. Always remind yourself that you have much to learn and are privileged to operate in an environment where continuous learning is possible and celebrated.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

What are your pronouns?
She/Her

What is your job title and what does it mean?
I am a Lead Software Engineer. I principally work on the iOS newsreader app. My team and I are at the end of a production line of engineers: some build tools for the newsroom to input content, others store it and still others deliver it. Our role is to fetch and present the freshest news in a multitude of structures and displays in a constantly evolving environment.

How long have been at the Times?
I started working at The New York Times right after Hurricane Sandy about eight years ago. My previous experience was in digital exhibitions for museums and interactive games.

Most Times employees are working remotely right now. What does working from home these days look like for you?
I usually start my day by listening in on the morning news meeting while having breakfast. I particularly enjoy listening to Elisabeth Bumiller, Washington Bureau Chief, and Ellen Pollock, Business Day Editor, and I see them both as role models.

I live near Union Square in Manhattan, so my break from work is to walk through the farmers’ market, check the dog run and get a baguette at Breads (best bread maker in town). I am lucky to live in a large bright apartment but I miss the workspace camaraderie. I am looking forward to going back to the office.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Developing a body of work in a medium that has become prevalent has been rewarding. The most satisfying projects are the ones where I identified a problem and figured out a solution that brought noticeable value. Last year, I developed an interface to help the newsroom streamline the process for sending notifications; I am now finishing a feature that will enhance the reading experience. During Maker Week, my favorite time of year, I created several visual and editorial experiments and tools, some of which were adopted in production.

I enjoy building things for people. Working on an app is particularly satisfying because users have a very intimate relationship with their device. I am always surprised by how quickly our readers report issues. Success for me is when the technology works seamlessly without being noticeable. My colleagues and I take a lot of pride in our craft.

What is a goal you hope to accomplish this year?
I hope to find new challenges to continue growing while contributing to The Times. It is a big place and there are always opportunities to work with new colleagues across teams. A dream project would be to collaborate with the Graphics desk on some of our most creative content and experiment using a native solution for it.

Do you have any favorite life hacks or work shortcuts?
I have been practicing yoga for several years at a studio and I now do it at home. It is my happy place. I also started sewing shortly before the beginning of the pandemic, learning by watching YouTube videos. I am now making my own clothes. I absolutely love it and am dreaming of traveling where traditional textiles are made.

What is your best advice for someone starting to work in your field?
Writing software can be both engaging and very tedious. Being curious, and a bit of a perfectionist, will help you embrace this constantly changing field and keep things exciting. Seek out people who strive for excellence. Meet people and ask questions. Be kind.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

What are your pronouns?
She/Her

What is your job title and what does it mean?
My job title is software engineer on the Search team. That means I write and maintain various software systems that my team operates. It’s not a terribly descriptive title, as I wear a lot of hats that aren’t just writing code.

I’m currently serving in another role as well: Chairperson of the Architecture Review Board, or “ARB.” The ARB is a resource for all technology teams at The Times, which helps to foster the architectural community as well as provide reviews on Requests for Comment. As Chairperson, I primarily help organize meetings and the ARB application process, promote initiatives and working groups, and I interface with management.

How long have you been at The Times?
I have been with the company for 55 months; I’m coming up quickly on the five-year mark!

Most Times employees are working remotely right now. What does working from home these days look like for you?
I’m fortunate that my job can be done from anywhere and I have been working remotely since before the pandemic. I have chronic issues that make it difficult for me to travel into the office every day. For me, working remotely means being able to show up in the morning without being exhausted from walking a half-mile to the subway, riding the train for 45 minutes and climbing a bunch of stairs to get out of the station.

Of course, I have found that others who are usually in the office recently have a newfound understanding of remote life, and that’s had a positive impact on a lot of things. Remote meetings run a lot more smoothly now that we’ve all had practice this past year!

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
When I first started out at The Times, I was fortunate to work on the schema for a publishing pipeline that was new at the time. My team collected the JSON schemas being used by various APIs around the company and combined them into a mega-schema that could be shared by everyone. We also researched which binary serialization protocol we would use (Protobuf), as well as the best way to manage constraints on the forward and backward-compatibility of the system as the schema changed over time.

Even today, my favorite part of my job is consulting with teams who want to add new asset types to the schema. We have a schema governance board that helps teams make updates, and we ensure that all of the downstream consumers of the system continue to operate smoothly through the changes. Tying everything together into a unified system makes me incredibly happy.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
This was my first job out of college and I came into work with an extremely open mind — possibly too open. I have always taken to heart commentary that could be construed as feedback and leapt to improve myself so I would “be better” the next time.

It turns out that this can come across as wishy-washy or insincere. It’s also impractical; one person’s advice might contradict another person’s advice. Now, I strive to take in all feedback and thoroughly consider it before deciding what I want to learn from it.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
I try to enable collaboration wherever possible, especially through my work with the ARB. I firmly believe that we are stronger when we work together. The ARB is well-suited to do this, as we work with all Times technology teams. We have a unique vantage point where we can highlight solutions that others have already found, and we can bring disparate groups together that are interested in solving the same outstanding problem.

What is a goal you hope to accomplish this year?
I’m working on my Masters in Library and Information Science right now, with the help of The Times’s tuition reimbursement program and dedicated 40 hours of learning time per year. It’s not as much of a stretch from library science to computer science as you might imagine. My engineering work encompasses many of the same skill sets as library science, such as indexing and finding information, managing metadata and maintaining the Times digital archive.

My biggest goal this year is to connect the dots between the things I’m learning in my classes and the real-world problems we face on the Search team. You have probably noticed that I like to make connections that others may not immediately see; This is a lot more of that same theme!

As a software engineer working on a digital product, how do you approach your work with inclusivity in mind?
I think that if we’re not a diverse and inclusive place to work, and we don’t take care of each other at work, we have no hope at all of delivering a product that puts inclusivity first.

I’m fortunate to have been involved with a grassroots group at The Times called Diversity in Digital since its inception. The group aims to drive change along the lines of recruitment; inclusivity with the aim of improving retention; and advocating for clear career advancement for employees of all backgrounds.

Additionally, I have been extremely enthusiastic about the work being done by my colleagues in creating a group focused on people with disabilities and neurodiversities. As someone who struggles with several issues myself, it’s great to find a place to push for greater understanding and acceptance of our differences.

What change do you hope to see in your community?
I want to see more people who don’t identify as male in back-end development roles. I tend to notice a pretty big difference between the gender ratios in front-end, full-stack and back-end teams. I value all of my coworkers, but I always feel more at home on a team that is not all male. I’m fortunate to have that right now — shout out to Sarah McNeil and Minnie Ko!

Do you have any favorite life hacks or work shortcuts?
My memory doesn’t always store things away as well as I would like, especially when working on such scattered topics as I encounter in my job. I started using the note-taking Zettelkasten Method about a year ago, and I just recently had to upgrade my note card storage to a filing drawer. The system organizes knowledge in a web structure that is easy to search through, so I can always find what I’m looking for.

What or who are you inspired by?
Here is where I admit that I’m a huge nerd: my biggest hero is General Leia Organa from Star Wars! She’s diplomatic, courageous and willing to put in the hard work to get things done. I have a little figurine on my desk to remind me to embody these characteristics in everything I do.

Fill in the blank: What is the best part of being ______? If you had it your way, what could make it better?
What’s the best part of being an avid reader? The book clubs! I love discussing what I read with other people and being exposed to more perspectives. Our employee resource group Times Reads is my favorite place to discover books that I never imagined existed, based on suggestions from my coworkers.

What could make it better? I honestly just need more time to read. I used to do a lot of reading on my commute, but I’m not commuting these days.

Complete this sentence: Over time, I have realized __________.
Over time, I have realized that there is always someone who thinks you can’t do it — whatever “it” is in your specific context. Sometimes, that person is going to be right, but that doesn’t mean you are worth any less as a person. Be kind to yourself.

What is your best advice for someone starting to work in your field?
Don’t pay a zillion dollars to go to a fancy university for a computer science degree. Go to a bootcamp, a community college, a state school or just watch online videos and teach yourself at home. Coding is for everyone; you don’t have to be a genius. Just pick a project and run with it, asking as many questions as you can dream up!

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

What are your pronouns?
She/Her

What is your job title and what does it mean?
Information Security Analyst. I am a member of the cybersecurity defense team.

How long have you been at The Times?
About one and a half years.

Most Times employees are working remotely right now. What does working from home these days look like for you?
I was hired as a remote employee, so the only thing that is different is that my co-workers are remote as well. But, I do miss visiting the office on occasion!

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
It would be bad “OpSec” (Operational Security) for me to disclose any specifics. However, I can generally say that I’m proud to have worked on projects that protect our human assets.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
Prior to working in Information Security, I was a librarian. When I decided that it was time for a career change about five years ago, I looked to tech and eventually realized that my natural paranoia and distrust of things made me suitable for a career in cybersecurity. (That’s a joke… sort of.)

The biggest challenge was the learning curve going from Library Science to Computer Science. Learning about computer networks and security threats required me to use a different part of my brain. I had to learn new skills and learn the ways of a completely different industry. I studied very hard every chance I got and I don’t think I would have changed a thing.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
A lot of the work as an information security analyst revolves around troubleshooting problems. Therefore, curiosity best describes my work. You can’t take things at face value and need to ask questions — the right questions — in order to solve the problem. You need curiosity to wonder what’s not being said or what’s not being seen, in addition to what has been said or is obvious. Curiosity to go beyond the surface is essential to proper problem remediation.

What is a goal you hope to accomplish this year?
A professional goal of mine for 2021 is to pass the Certified Information Systems Security Professional (or, CISSP) exam. The CISSP is an independent information security certification granted by the International Information System Security Certification Consortium, also known as (ISC)².

A personal goal of mine for 2021 is to create a non-profit for cybersecurity students and professionals in Africa. In my free time, I volunteer with a few different groups to help build their skills, help with professional networking, promote their work and assist however I can with getting more people in Africa into cybersecurity careers.

As an Information Security Analyst working on a digital product, how do you approach your work with inclusivity in mind?
Just recently, I started to type the word whitelist in context to something work-related. I stopped myself and thought, “I know there is a better word for this, but it escapes me.” So, I took a second to look it up and revised what I was writing to say allowlist instead. Words matter, even in tech.

What change do you hope to see in your community?
I dedicate a lot of my free time to helping people from underrepresented groups start information security careers. My professional community has been homogenous for too long, and my goal is to help make it more diverse. Diversity of thought solves problems, so it’s crucial for the industry to have people with different perspectives and experiences be security practitioners.

Do you have any favorite life hacks or work shortcuts?
My favorite work hack is being an observer on the Times morning news meeting! Not only is it great to connect faces and voices with names, but also to know what we are working on for publication helps the security team be aware of things that could impact the enterprise. As a news geek, it’s very exciting and it just makes me really proud to be a part of this organization.

What or who are you inspired by?
I’m inspired by people who have overcome adversity, in whatever form. I’m inspired by immigrant success stories. I’m inspired by brave LGTBQ+ folks who are living their lives in the face of hate. I’m inspired by the Black cybersecurity community who are boldly shaping their futures by not accepting the status quo in the industry. I’m inspired by anyone who, as Maxine Waters famously said, are “reclaiming my time.”

Fill in the blank: What is the best part of being ______? If you had it your way, what could make it better?
What’s the best part of being a Philadelphia-area native?

• Gritty.
• Churry Wooder Ice.
• Hoagies and Cheesesteaks. (It’s a tie.)

If I had my way, Philadelphia would have more social and financial equity for its residents. Better support and services so that the residents don’t struggle as much as they do.

(Also, more sports championships.)

Complete this sentence: Over time, I have realized __________.
It’s never too late to start something new. Whether it’s a college degree, a career change or something else. I was a non-traditional student for all three of my college degree programs. I made what looked like a strange career pivot from a librarian to a cybersecurity professional at an age when most of my peers were firmly embedded in their professional lives. I used to agonize over what I perceived as being behind everyone. It’s not that I’m behind; I’m on my own path.

What is your best advice for someone starting to work in your field?
Network with people, early and often. There is immense value in connecting with people to learn from them, to help them and to have them help you. I strongly believe that human networking is a huge factor in professional development and success.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

Mar 22 · 5 min read

What are your pronouns?
She/Her

What is your job title and what does it mean?
I’m a Software Engineer on NYT Cooking’s engineering team. That means, I get to torture myself every day by looking at amazing recipes and enticing pictures while building and maintaining features on our web application.

How long have you been at The Times?
I have been at The Times for almost three years. Wow, time really flies by!

Most Times employees are working remotely right now. What does working from home these days look like for you?
Working remotely has pretty much been the same for me since I was remote pre-pandemic. However, at the beginning of the pandemic, it was tough balancing working full-time and being a full-time teacher to my 8-year-old son. Luckily, The Times now has extremely helpful programs for employees that assists parents with child care and tutoring fees.

Tell us about a project you’ve worked on at The Times that you’re especially proud of?
Our team had upgraded all of our dependencies, but when we tried to upgrade the CSS-Loader for Webpack, all of the images that were served from the client-side would break on our website. The issue was bigger than just changing a few paths, and there was little documentation on how to upgrade from Webpack 2 to Webpack 4. Online searches returned some solutions, but nothing that fit NYT Cooking’s unique needs;I developed a fix by piecing together a few solutions that worked for others.

While working on the Webpack upgrade, I saw that it would be better if the images served from the client side were imported from only one file. Migrating the images to a single file would enable us to easily fix any image path issues we ran into in the future. I decided to incorporate a major image refactor that not only fixed the Webpack issue, but also addressed some tech debt that we had. It was one of the toughest projects I have had in my career so far, but one of the most rewarding.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
The biggest challenge I’ve faced in my career was still recovering from a traumatic brain injury while being a full-time software engineer. Feeling like I needed to catch up to my colleagues by reading and working on outside projects after work or making sure every pull request and commit was as close to perfection as it could be was very taxing.

Imposter syndrome was strong, but as I spoke with other colleagues and saw they had the same thoughts and fears that I had made me feel more confident in my abilities. Learning as much as I could outside of work was very helpful to overcoming my traumatic brain injury and imposter syndrome. But, if I could do it differently, I would not have put so much pressure on myself. No one is perfect.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
Integrity — that’s my approach in life, in general. In my work as a software engineer, shipping clean code by the deadline is extremely important to me. In tech, there are many circumstances that can affect when a project is shipped, but I try to minimize that as much as possible.

In my work as the founder and leader of The Times’s employee group, Black and Latinx in Technology, integrity is key in making real progress for our communities in and outside of The New York Times.

What is a goal you hope to accomplish this year?
Personally, I want to be able to complete a half-marathon this year. It’s a very ambitious goal since I have been pretty inactive since the beginning of the pandemic. Pray for me!

Professionally, working on increasing my technical depth is my priority. I love to learn, not only for the advancement of my career, but to be in a position where I can help others advance in their careers as well.

As a software engineer working on a digital product, how do you approach your work with inclusivity in mind?
Our team has made strides in addressing accessibility improvements where we can and continue to make it a priority.

What change do you hope to see in your community?
I identify as Black and Latinx. Our communities can only do so much to make a real change in this country. We do a lot within and for our community but I would hope to see other communities practice more equity, empathy and openness towards our communities. Action speaks louder than words.

Do you have any favorite life hacks or work shortcuts?
Learn Linux and Git command lines. With Git, create shortcuts with Git bash aliases because it will save a lot of time in the long run.

What or who are you inspired by?
People who have been counted out every step of the way but still succeed time and time again are who inspire me.

Fill in the blank: What is the best part of being ______? If you had it your way, what could make it better?
What’s the best part of being Me? I am who I am, unapologetically. As a child, I was very shy and never spoke my mind or stood up for myself. As I have grown and experienced life, I have learned to love myself first, speak up, stand strong in my beliefs, and fight for the people and things I care about.

Complete this sentence: Over time, I have realized __________.
That burnout is real and self-care should be one of your main priorities.

What is your best advice for someone starting to work in your field?
Do not get discouraged. There are people that will root for you and people who won’t, but what they cannot take from you is your knowledge, experience, and passion. If you do what you love, keep working hard at it and you will succeed.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

Mar 17 · 5 min read

What are your pronouns?
She/Her

What is your job title and what does it mean?
I’m a Senior Product Designer on a team called Workflow, which supports the editors who publish the Times homepage multiple times a day. Technically, my job is to design the end-to-end experience for a CMS called Curator — along with my design colleagues Orr Shtuhl and Tina Ye — but I prefer to think of myself as a professional listener and complexity mitigator.

How long have you been at The Times?
Three years this April!

Most Times employees are working remotely right now. What does working from home these days look like for you?
It changes month-to-month. Because we are able to be fully remote right now, I sold all my stuff in December, bought a car and started a nomad lifestyle. I don’t have an address or a lease — just a car, my dog and a few suitcases. After 10 years in NYC, it’s a huge change!

In a lot of ways it’s freeing to have control over where I am in a time where I feel like I have control of so little. If I feel uninspired or restless (or honestly, too cold) I pack up my car and I’m gone. On the flip side, safety is my highest priority so I have to take a lot of precautions — I try to move around slowly and pick places with wide open spaces. Luckily that means spending a lot of time outdoors with my dog and best friend.

Right now I have two set ups: I’m either literally working from my bed (I know, I’m terrible) or sitting outside soaking up the sun in Austin, TX.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Working with my product design “squad” to investigate the state of Diversity, Equity and Inclusion in Product Design. We have spent the last six months doing in-depth research across several themes, such as Growth, Retention and Hiring and we recently presented our findings to the rest of the Product Design team.

I am so proud to see Product Design committing to being a leader in The Times’s DEI efforts. Also, as a Black woman, it’s amazing to see my own lived experience amplified by the voices of my peers and by leadership. This is just the beginning and I’m so excited to see where this work goes.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
Making my first portfolio. I’ll be honest, before working at The Times, my confidence as a designer was absolutely trash. I spent years (yes, years) putting off making a portfolio because I was convinced that I would never meet the expectations of the companies I dreamed about working for. Working at The Times has taught me that the only expectations stopping me were my own.

I have to give a shout out to my best friend and colleague, Yuraima Estevez for: 1) putting up with me; and 2) for pushing me to apply to The Times. I can now own that I am an amazing and talented product designer.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
Curiosity. In my opinion, being curious is one of the best traits a designer can have. If you aren’t interested in the world around you, you’ll never even begin to understand how to fix it.

I’m lucky enough to spend most of my days in curiosity mode: thinking, sketching, asking questions, filling in blanks and attempting to untangle gnarly user experiences challenges. I love it.

What is a goal you hope to accomplish this year?
Do at least one pistol squat.

As a Designer working on a digital product, how do you approach your work with inclusivity in mind?
I keep universal design principles at the forefront of my design practice. Universal design principles is an approach that aims to make design as inclusive as possible with features that enhance access for everyone regardless of age, size, ability or disability. In my own practice, this looks like designing simple and intuitive internal tools that have a high tolerance for error and keep accessibility at the forefront.

What change do you hope to see in your community?
To be frank, I’d like to see more racial diversity in design, especially at The New York Times.

Do you have any favorite life hacks or work shortcuts?
If you’re feeling overwhelmed by meetings and feel like you have no time to work, download a smart assistant extension or app for your calendar. I use a smart assistant to do things like reschedule meetings so I get much-needed breaks and optimize for “focus blocks” of time that are at least two hours. Genius!

What or who are you inspired by?
I draw a lot of inspiration from my design peers, colleagues and friends. But I think I have learned the most from just talking to people who have taught me how they communicate; what they care about; and what brings them pain and pleasure. Over time I have accumulated stories that amount to a library of patterns and behaviors that I can draw upon in my everyday design practice.

Fill in the blank: What is the best part of being ______? If you had it your way, what could make it better?
What is the best part of being in Austin? Tacos and family.

What could make it better? If I could somehow be both in Austin and in New York City pre-pandemic at the same time (minus alternate-side parking).

Complete this sentence: Over time, I have realized __________.
Progress is better than perfection.

What is your best advice for someone starting to work in your field?
Don’t be so hard on yourself. Everyone has to start somewhere, so don’t be afraid to work out in the open.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

Mar 15 · 5 min read

What are your pronouns?
She/Her

What is your job title and what does it mean?
Director, Customer Care. I serve as product owner for one of Care’s cross-functional scrum teams: the Care Insights and Analytics team. Our work focuses on providing insights to the company based on what we hear directly from our customers, and maintaining efficient operations. I am also a member of the leadership team advising our Vice President of Customer Care on strategic, short-term and long-term planning decisions.

Customer Care Operations merged with our Product, Engineering and Design counterparts just over a year ago, and the Care Insights and Analytics team was an experiment to see if we could all work together in the agile product development methodology. We have been very pleased with the success of this experiment, both for our team culture and for our productivity, and we now have two fully operational scrum teams on Care.

How long have you been at The Times?
Four and a half years.

Most Times employees are working remotely right now. What does working from home these days look like for you?
I live in an area of lower Westchester, NY called the “Rivertowns” because we are right along the Hudson River — if I walk across the street I can see the Manhattan skyline in the distance. I spend my working hours at a little desk in a corner of my bedroom, facing the river and trying to keep my new rescue dog from hogging all the attention during meetings.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
In 2019, I worked with a number of my Care colleagues to completely redesign and replace our Interactive Voice Response system, a.k.a. the robo-voice you hear when you call customer service. Instead of expending a lot of effort to recreate the complex decision trees in our old system, we chose to start the phone experience simply by asking the customer why they are calling and relying on intent recognition to take it from there. Every time I encounter another company’s Interactive Voice Response or when I hear family or friends complain about the long menus they must wade through on a phone call, I’m proud of our decision to reduce that effort up front.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
My biggest challenge has been saying “yes” to too many things and being hard on myself if I don’t live up to my own internal standard for excellence. Thanks to a wonderful team (Care folks are great cheerleaders!), I’ve gotten better at prioritizing my own bandwidth, as well as asking for help when I need it — which can also be a great way to give others exposure and opportunities to grow in their own work.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
I would say collaboration best describes the nature of my work for the past few years. Through multiple roles and structures on the Care team, I have always sought out opportunities to connect people and develop a shared purpose in tackling our individual goals for the success of The Times. Everyone who works here has a deep passion for the mission of The Times; I am excited to continuously find ways to weave together various teams’ mutual commitment to upholding the mission, standards and products of the company.

What is a goal you hope to accomplish this year?
Developing connections with more colleagues outside of Care. As I’m sure many can attest, this past year of remote work has removed the ability to bump into folks in the hallways, have ad hoc catch ups in the elevator and other unscheduled socialization opportunities. I’m very proud of the culture we have developed on the Care team, and I feel that we have gotten even closer in this past year. I hope to take this a step further and deepen contact with other amazing people at The Times!

As a director working on a digital product, how do you approach your work with inclusivity in mind?
I am very proud of and humbled by numerous members of the Care team who keep important topics such as accessibility and DEI (diversity, equity and inclusion) front of mind as we develop new features and services. The team has committed to specific objectives and key results around DEI targets for ourselves and our vendor partners, to ensure we are prioritizing actions and not just conversation.

Do you have any favorite life hacks or work shortcuts?
I am a chronic multitasker. While my colleagues say I’m good at it, I believe the research that shows it is not effective and that it’s just constant, rapid context switching. I really feel the impact on my mental capacity, especially in a remote environment.

One trick that has helped me is keeping my hands busy when I’m listening to a meeting that doesn’t require as much vocal participation. If I have an embroidery needle in my hand or some Play-Doh, I’m much less likely to check email and Slack during a presentation that I would much rather give my full attention to.

What or who are you inspired by?
At risk of sounding clichéd, my mom. She has had (and continues to maintain) a high-powered career for my entire life, doing the lion’s share as a single parent when I was young and most in need of care. Her tenacity and competitive nature, coupled with boundless warmth and compassion, set the example that I could thrive in my own path without compromising my own ambition — or friendliness.

Complete this sentence: Over time, I have realized…
That it’s important to be flexible, and not take anything too personally. I like to assume folks are coming to the table with the best of intentions. Even if that isn’t the case, it starts an interaction off with a positive mindset which often carries through to a better outcome.

What is your best advice for someone starting to work in your field?
Stay curious and speak up! I love working in a customer-driven function. The more we learn about people’s experience with our products and comparable products, the better we can make our offerings. If a question or an idea seems silly, voice it anyway — you’ll never know what it could lead to if it’s never said.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

Mar 12 · 5 min read

What are your pronouns?
She/Her

What is your job title and what does it mean?
I am a Lead Software Engineer. Generally, that means I write a lot of JavaScript for our website. Currently, my team builds and promotes features that enhance the value of a New York Times subscription with the aim of retaining subscribers. Our journalists create an amazing breadth of content every day and we want to help our readers discover all we have to offer.

How long have you been at The Times?
About seven and a half years.

Most Times employees are working remotely right now. What does working from home these days look like for you?
I’ve been staying in New Jersey with my family for the last 11 months. A typical day consists of working at the desk in my childhood bedroom, taking breaks to hang out with my two cats or our family dog, and watching Jeopardy during dinner.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
I’m extremely proud of the work I’ve done on The Weekender. It started as a group project during Maker Week (an annual event where we are free to explore and build whatever we want), and it has now grown to be a weekly feature. I have gotten to work with some amazing people in the newsroom and I learned a lot about the editorial process that I wouldn’t have learned otherwise.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
A challenge I still face today is maintaining confidence in my abilities. I started at The Times right after I graduated from college. In my first year, my goal was to learn as much as possible from those around me. Achieving this was easy because my teammates were all very helpful.

As I continued my career here, I started to understand enough about our systems to be able to contribute answers rather than just ask for them. I had spent the majority of my life in school where there was always a right answer and a person to tell me what it was. It was a challenge for me to gain the confidence to embrace the ambiguity of real-life decision making. For a long time, I wouldn’t share my opinion without first asking for validation from everyone around me.

If I had trusted my gut earlier, I would have been brave enough to take chances and accept that it’s O.K. to be wrong sometimes. I think a healthy dose of skepticism in your own ideas makes you a better collaborator, but it should be balanced with some trust in yourself.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
All of these values are important to my work but I think collaboration is probably the best fit. I believe our job as developers at The Times is to create a stable infrastructure and a suite of tools that allow the newsroom to be creative and reach our readers. There’s no way to do that work without listening and learning constantly.

What is a goal you hope to accomplish this year?
I’m not much of a formal goal setter, especially at a time when it feels like everything is constantly changing around us. My informal goals this year are to learn some new things, work with great people and take things as they come.

As a lead engineer working on a digital product, how do you approach your work with inclusivity in mind?
One of my favorite parts of my job is when I get to sit in on our user research sessions. I love hearing from a wide range of readers, gaining insight into how they use our features and challenging my own assumptions. I try to bring those same concepts to my day-to-day work by asking for feedback and working to make sure those opinions come from a diverse group of colleagues.

What change do you hope to see in your community?
I think this year has taught us a lot about how necessary it is to have flexibility in our work-life. I hope we continue to give people the space they need to figure out how to achieve work-life balance in a way that makes sense for their circumstances.

Do you have any favorite life hacks or work shortcuts?
I am a very big fan of the split screen feature on Mac. It allows you to take any two apps and put them side-by-side in a single view. It’s so nice to be able to see faces in a meeting even if I’m sharing my screen.

What or who are you inspired by?
Work wise, I’m inspired by all of the complex, critical and sometimes life threatening work our journalists do every day. It’s kind of surreal that I work for the same company that they do.

Life wise, I’m inspired by those who use humor to help themselves and others process the complexity of life. Laughing is my favorite coping mechanism.

Fill in the blank: What is the best part of being ______? If you had it your way, what could make it better?
My favorite part of being an engineer, especially a web developer, is being able to rapidly prototype. It’s so satisfying to be able to quickly spin up a useful website that looks professional. I also love working with responsive design. Few people in real life spend time resizing their browser, but I love watching components seamlessly rearrange to fit the space.

Complete this sentence: Over time, I have realized __________.
Over time, I have realized that I get to define what a successful career looks like to me. I’m still working through what that means for me, but right now it includes working on a supportive team, building systems that reinforce The Times’s mission and bending the formal role of an engineer to include learning about disciplines outside of development (design, user research, product management, journalism).

What is your best advice for someone starting to work in your field?
When I start a project that will require learning something new, the long list of things I don’t yet understand can feel daunting. I take time at the end of those projects to remind myself of that feeling and to recognize that it was only temporary. This small habit has built up my confidence for when I face new unknowns and I highly recommend it.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’

What are your pronouns?
She/Her

What is your job title and what does it mean?
Senior Manager, Data & Insights (Engagement Analytics)

I manage a team responsible for data on our newsletters, push notifications and some of our personalized surfaces. We identify opportunities for our products to grow, build pipelines to power enterprise reporting, run experiments to inform business strategy and more.

How long have you been at The Times?
Just over three years.

Most Times employees are working remotely right now. What does working from home these days look like for you?
I’m currently working from home in Michigan but plan on going back to New York soon. Working from home these days looks like having coffee with my sister in the mornings and saying hi to my mail carrier in the afternoons.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
I was part of the team that first introduced the dynamic meter on our site. This involved an innovative and scrappy small core team that pushed my technical skills, as well as a highly cross-functional set of stakeholders who challenged me to communicate data insights in better ways. It was an incredibly complex project at the forefront of what the industry is doing, and it’s still powering part of our paywall today.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
Earlier in my career, I was in a pre-doctoral program with an amazing cohort of people who were all planning on getting their PhDs. I knew internally that getting a doctorate and having a career in research wasn’t for me, though I had already done a lot of preparation for it. It was difficult to overcome the expectation that was set for me in the program to pursue grad school. I started talking to folks in other industries, found allies in my cohort that also were deviating from the grad school route and got a job here at The Times! I would tell myself to trust my instincts more readily.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
Curiosity — the “open-minded inquiry” at the heart of The Times’s journalism is also the key to good data analytics.

What is a goal you hope to accomplish this year?
This year, it’s important for me to support my team and coworkers in the best way I can and to allow myself to be supported by them. Hitting the one year mark since the start of the pandemic is not easy for any of us. My goal is to make space for my team to feel like they can learn and grow in their work.

Outside of my job, I’ve been working on a manuscript for a few years that I would like to finish up this year.

As an analyst working on a digital product, how do you approach your work with inclusivity in mind?
As an analyst, it’s your job to keep an open mind and to challenge assumptions, to put yourself in the position of a reader and to approach analysis with empathy. As a people manager, inclusivity is even more important. People managers can focus more on hiring and growing people who are underrepresented in technology, to hear from different perspectives, to develop policies and expectations that are inclusive for everyone.

What change do you hope to see in your community?
I hope to see more people of color and women thrive in data, and I hope to see more people who work in the industry foster environments that support the success of people of color and women. I’m encouraged by some of The Times’s recent commitments to diversity and inclusion, and I want to push our technology and data functions to do more.

Do you have any favorite life hacks or work shortcuts?
Life hack: you can regrow green onions forever by just placing their ends in a jar of water.

Work shortcuts: write down everything you have worked on at the end of the week or at the end of a month, including small ad-hoc tasks. Whether you’re writing a year-end review or just looking back on your day-to-day, this will help you have a clear sense of all that you have accomplished.

What or who are you inspired by?
I teach a citizenship class and every week, my students inspire me to be persistent, patient and dedicated to a life of continuous learning.

Complete this sentence: Over time, I have realized __________.
Over time, I have realized the importance of having good colleagues by your side. Even with good work-life balance, you end up spending so much of your life at work. Having people to joke around with, to vent to, to cheer you on, to bounce an idea off of and to support you in times of need is so critical.

What is your best advice for someone starting to work in your field?
Stay curious! Follow your interests to explore beyond what’s expected of you. Approach your work with empathy. Advocate for yourself and find mentors, sponsors and managers who will advocate for you. And wherever you are in your career, there’s someone you can mentor and bring up behind you.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’.

Mar 8 · 3 min read

What are your pronouns?
She/Her

What is your job title and what does it mean?
Chief Information Officer. To me, it means that I lead the teams that a) empower our company and our workforce to work productively, efficiently and safely; b) run all the back office systems to keep the company doing business; and c) support all things print.

How long have you been at The Times?
40 years. I started when I was 10. Just kidding, but really some days it feels like those years have flown by, and other days it feels like it has been 10 lifetimes. I love thinking back to some of the world events I have seen unfold through the lens of working for The Times. What a long, strange trip it has been!

Most Times employees are working remotely right now. What does working from home these days look like for you?
Well, I’ll just say I don’t enjoy working from home. I would rather be around people, and I think I work better around people. For me, it’s hard to replace the value of social interaction.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
That would be our migration to the cloud, completed just about three years ago. The tech department migrated roughly 95 percent of our computing workloads to either the Amazon or Google public cloud environments, and it was all on hard deadlines having to do with ending our leases and shutting down four terrestrial data centers. Dozens of people worked on the project across the tech department and they met all of the major deadlines.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
In the early 1990s when my two children were born, I had to transform myself into a full-time working parent. Becoming a mother is a huge challenge and doing it while working was incredibly difficult. Learning to prioritize, being the best mom I could be, and for a while, managing with very little sleep! Would I do it again? YES. There’s nothing more rewarding.

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
Collaboration. For me it’s mostly about the people, and I believe that we encourage each other to do better when we do it together.

What change do you hope to see in your community?
Well, right now I would just like to see my community come out on the other side of this pandemic so that we can all be real neighbors again in a safe and healthy way.

What or who are you inspired by?
My father. He taught me so much about how to figure things out for myself, and how to advocate for myself.

What is your best advice for someone starting to work in your field?
Job satisfaction has many aspects to it, but there are three aspects which I think are fundamental. If you have work that you think is meaningful, a boss who appreciates you and you are fairly compensated, then you should have job satisfaction. You may not always be working on the most exciting project and you might not always be paid at the high end, but finding job satisfaction is important. Other aspects may wax and wane over time, but those three are the fundamentals.

This March, we are featuring colleagues from across The New York Times in a special Women’s History Month series of ‘Meeting’

Mar 4 · 6 min read

What are your pronouns?
She/Her

What is your job title and what does it mean?
I’m a Senior Software Engineer on the Messaging Pipeline team. I work on a microservice application that is mainly written in Go, but has a little JavaScript sprinkled in.

How long have you been at The Times?
Since October 2018, so a little over two years.

Most Times employees are working remotely right now. What does working from home these days look like for you?
Working from home has been a rollercoaster. I have enjoyed getting back the time I would usually spend commuting, and I have been able to spend more time with my cats. I set up a little bird feeder on the fire escape so my cats can have some entertainment on the windowsill next to my desk. Turns out, I enjoy bird watching as much as they do!

But, it’s been a challenging time, too. When the pandemic first started in New York, I really struggled to find balance between my work time and my free time. It felt oppressive and I couldn’t really disconnect the same way I was able to when I physically left the office. The idea of being able to make my lunch during lunchtime was quickly thwarted by last-minute meetings or long slack conversations, despite my attempts to reclaim my calendar.

The biggest struggle for me was getting enough movement in the day. I went from going to the gym three to five times a week and walking 20 blocks from the gym to the office in the morning to having walked only 800 steps by 6 p.m. according to my fitness watch. It was bad.

After a year of working from home, I have recalibrated my work-from-home experience and sandwiched my workday with a “commute” around my neighborhood; I have designated work areas to help define the lines of work and home; and although I haven’t unlocked the secret to maintaining a lunch hour, I have leaned on meal prep to ensure I have something good to munch on when lunch time rolls around.

I still watch the birds peck at the seeds on the fire escape while my cats look on with unflinching focus and restless tails.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
In 2019, my team worked to migrate functionality from a seven-year-old legacy system to our current messaging platform. The system was a PHP application that was responsible for sending push notifications and it had not been actively maintained in some time. My team’s goal was to shut down the application because most of the system’s original authors had left the company and the tech stack did not fit the current skill set on the team.

I had joined The Times a few months earlier with no prior experience working in Go and no professional experience working with back-end systems (other than Node applications). My role was to build functionality that took the payload scheduled for an alert and send it to the respective Android or iOS platform.

I didn’t know what the payload looked like, so I had to do a bit of reverse engineering, while learning Go and building a proof of concept, to verify our assumptions. After I built the proof of concept, I was able to complete the transformations and client setup to allow our current platform to send push notifications. I was especially proud of it because it was pretty intimidating; I never expected to be responsible for this kind of work, but I was able to build and ship it.

What is the biggest challenge you faced in your career and how did you overcome it? Knowing what you know now, would you do things differently.
Learning the language necessary to ask the right questions. It came with experience, as most things do, and being comfortable enough with my teams and my mentors along the way to ask for guidance when I needed it. Knowing what I know now, I would have stuck with my Computer Science degree instead of changing majors! I would have sought help from environments that promote “growth courses” and not “weed-out courses.”

The Times has six core values (Independence, Integrity, Curiosity, Respect, Collaboration and Excellence) by which the company operates. Is there one that you find best describes your work?
Integrity. It’s how I approach development. I come from a QA background, so I tend to think of edge cases first and then I try my best to ensure that I ship sound and stable code. It’s also how I approach the work with my team. It’s also how I approach my extracurricular work with some of The Times’s advocacy groups like Women in Tech, Black and Latinx in Tech and our Architectural Review Board. I believe that honest conversations move us forward.

What is a goal you hope to accomplish this year?
For my personal life, I would like to complete a century bike ride this year.

For my professional life, I would like to commit to putting more of my focus in increasing my technical depth and saying “no” to opportunities that don’t work towards that goal.

As a Software Engineer working on a digital product, how do you approach your work with inclusivity in mind?
I focus on end users. It doesn’t matter if you complete the work outlined in the ticket if the work that gets shipped doesn’t provide value or leaves out a group of people. This involves a lot of conversations with product and team leads, asking questions and most importantly, listening to others.

What change do you hope to see in your community?
I can’t say that there’s any major change that I would like to see specifically in my community. The community I identify with is compassionate, ambitious, resourceful, eager; but there’s only so much one side can do. I’d love to see communities outside of my own be more open, transparent, equitable and compassionate. I’d really like to see more spaces that I occupy as a professional look more like the cities that house them, the clientele they aim to serve, and the world at large.

Do you have any favorite life hacks or work shortcuts?
When you want to create a new Google Doc type “docs.new” into your browser and a new Google Doc will load; It’s the same for Sheets. It’s a pretty useful shortcut for my workflow.

What or who are you inspired by?
I’m inspired by engaging and sprawling conversations that leave me with books to read, music to listen to and perspectives to examine.

Fill in the blank: What is the best part of being ______? If you had it your way, what could make it better?
What is the best part of being in quarantine? Some days I really can’t stand it and I want to fast forward to when we can meet up in real life. But, it’s not all bad. In many ways, quarantine has forced me to be more intentional with myself and the people I care about.

I have been able to learn a lot more in my free time, ranging in subjects from algorithms to feminist theory. The pandemic has motivated me to regularly check on friends who I would not have reached out to until we hung out in person, which can be many months for friends in different parts of the country. It has taught me to listen to my body; It’s pretty wild how many mild inconveniences and discomforts a body can accumulate before you realize how garbage you feel. I have been able to try out different meal plans that hit my nutritional goals rather than grab something that’s quick and convenient and I’m feeling much more energized from it.

All of that said, I’m ready for things to get back to what they once were. I plan on taking some of the habits I learned into my post-quarantine life. If I could have it my way, I would have a full vaccine roll-out tomorrow.

Complete this sentence: Over time, I have realized __________.
That the work will still be there tomorrow.

What is your best advice for someone starting to work in your field?
Take time to explore what it is about technology that piques your interest and learn by doing; experience it the best teacher and it doesn’t always come from school or work.

We use A/B tests to make decisions about the products and features we release, but our mobile test allocation wasn’t separating users properly and we had to figure out how to fix it.

Mar 4 · 8 min read

By Julian Locke and Sameer More

In early 2020, the mobile engineering teams at The New York Times noticed an issue: our A/B tests were separating users at the wrong ratio. Tests that were supposed to allocate users into control and experimental groups were sometimes off by five percent or more. This was a big problem.

The Times uses A/B tests to make decisions about the products and features we release. Our A/B testing system works by separating users into groups that see different variants of the feature we’re testing. We then collect data on user behavior around the feature, and we draw conclusions on the effectiveness of our test.

If a test’s participation ratio is skewed — as we were seeing across many tests on our mobile platforms — it affects our confidence in feature roll-outs. When we test features, it’s important that we have control over how we allocate users into testing groups.

To figure out why this was happening, we turned to our in-house A/B testing library. Our A/B testing SDK was built for simplicity and had a synchronous API that a client could check for a test allocation. However, behind-the-scenes, the library would periodically query for results of A/B allocations and persist them across app sessions. If no result had been found and the network call had not yet returned, each test was hard-coded to fall back to a default variant, which was almost always the control.

Our hunch was that the hard-coded fallback was catching users who were browsing offline or had slow network connections. This was a problem because we needed unbiased allocation to successfully test new features. We had to figure out what was happening.

In order to make our SDK more accurate, we deployed two changes over the next few months.

First, we increased reporting and we added an analytics field to indicate the source of the allocation for a user, server or hard-coded fallback. While this didn’t make allocations more accurate, it did allow us to filter out non-random allocations during data analysis.

Second, we developed an app-side hack for reliable allocations by simply neglecting to configure the test and using native logic to assign a random variant. This was especially useful for UX tests for first-time users that had little chance of being fetched prior to the user reaching them.

Together, these changes allowed teams to continue testing, but each came with downsides. Filtering the non-random allocations from analysis shrunk our pool of testers and biased our tests to measure only users with great network connections. Our app-side hack separated all users fairly, but it limited our ability to make changes to the allocations once the apps shipped. Adding native allocation logic was a less maintainable solution because it created a second source of truth for test configurations. It also duplicated the allocation logic and the configuration validation logic that already existed on the server.

Even with the downsides, we were encouraged by the truly random synchronous allocations afforded by the native logic. We started to investigate whether we could formalize and port the logic into our shared library, while mitigating the inherent issues of the native logic.

From here it’s helpful to go a little more in depth about the constituent parts of the existing system:

1. First, tests are manually configured in a server-side GitHub repo. For each test, developers specify a name, test variants and proportions and the integration. An integration is similar to an environment, but it also specifies the application, such as `ios-newsreader-production`.
2. On each change to this repo, server-side tooling creates a new JSON rules object for each integration. It’s a distillation of the latest test configurations for that integration.
3. On launch, the mobile app sends a list of tests, an integration and targeting parameters to the allocations endpoint, which then returns a dictionary and allocations for each test.
4. The allocations endpoint computes the testing distribution using a short JavaScript allocation script. The test and targeting parameters from the app, as well as the relevant rules object, are passed into the allocation script. The script returns allocations for each test.
5. The allocations endpoint call returns the allocations to the mobile app.

We use JavaScript alongside native code in our mobile apps at The Times. Sometimes the JavaScript displays web technology-based UIs, and it occasionally executes small chunks of cross-platform logic. With that in mind, we imagined a reconfiguration of the A/B library’s components that was inspired by the JavaScript for event validation work of our colleague Krzysztof Zablocki, who wrote a simple analytics event validator that shared code with our web app. Elements of this work inspired our design here; we were able to adapt portions of it on iOS and implement analogous components on Android.

We envisioned our new configuration as a series of steps:

1. Teams would their app-side allocation tests to the server-side configurations repo, just as they did for the old system.
2. At app compile time, the tooling would download both a recent JSONLogic rules object and a recent allocation script, which it would ship to the app.
3. The tooling would use the rules object and the script to synchronously allocate all tests at app launch. This removed the network call to the allocation endpoint completely out of the picture.

This configuration would mitigate most of the issues with our native logic hack, but it would still inhibit our ability to make changes after an app release. Because our engineering teams needed the options to pull the plug on bad tests or roll them out incrementally, this was a significant roadblock.

[We are hiring! Come work with us.]

We looked back to Krzysztof’s work on JavaScript for event validation, which included a class that served as an abstraction of a remote asset. We created one instance each of the class to represent the JSONLogic rules object and the allocation script. The classes would periodically fetch the latest allocation script or asset, falling back to local versions if needed. This allowed us to update test configurations on an app already in flight.

The one caveat with this approach is that it’s not possible to apply the most up-to-date allocation rules for a user launching the app for the first time. This is because the binary may be stale and there is no time for a server call. Initial allocations rely on the rules that were live when the latest app release was cut, which may be up to two weeks old.

Finding a JavaScript engine

To implement our new A/B testing library on Android, we had to choose the right JavaScript engine to run the allocations script and we considered WebViews, J2V8 and Duktape-Android. While we had used WebViews before, they were hard to manage and couldn’t execute code synchronously. J2V8 was faster and easier to use, but it would have added about 20MB to our app’s size. We ultimately decided on Duktape-Android because it only added 2MB to our app.

Verifying Allocation Code At Runtime
We added a validation step that stops the library from trying to execute a nonfunctional script. This step reads the JavaScript and attempts to run it in Duktape-Android. However, if an exception occurs, the library will revert to the script it currently has.

Asset Management
For our A/B testing library, we relied on Store4, which brought support for coroutines. Store4 provides a framework to specify a source of truth (which in our case, are the A/B assets stored in the file system). The framework also provides a fetcher for retrieving and validating assets from the network and a reader for processing those assets.

Finding a JavaScript engine

We decided to use the Apple framework JavaScriptCore to execute the allocation script because it runs in process. While it is slightly slower than other engines we could have chosen, we opted to use it for security reasons.

Asset Management

We adapted a useful abstraction from Krzysztof’s app-side JavaScript work called the `RemoteAssetManager`. For assets like our allocation script or rules object, an instance of this manager can handle all the tricky logic around making the latest version synchronously available. The abstraction reads and materializes the asset out of the app binary, fetches new versions of the asset from the web and writes the version to disk between app runs.

The `RemoteAssetManager` is written to be flexible and capable of handling arbitrary assets. Its initializer takes a few closure arguments that allow the caller to tell it where to access assets, how to materialize assets from data, how to compare and choose the best of these assets, and how often to attempt to fetch new versions.

The core of the app-side allocation code can instantiate one of these managers and from then on synchronously access the latest and greatest possible version of the asset.

Analytics
Because an A/B test would be useless without data, our apps trigger an expose event, which reports the test name and variant at the time a feature is used. The event is our only way to know that an audience is seeing the correct test.

We added additional data including:

• Metadata for how tests are allocated.
• An error field for exceptions.
• The allocation script response.
• The input parameters to the script.
• The version of the rules object and the integration that was used.

This information is sent to the analytics service and is made available for our data team. Because of our additional analytics, we no longer need to guess why a test isn’t reaching the target audience.

In early October, 2020, we integrated this new A/B testing library into the New York Times Beta app and distributed it to a few thousand users. To test it out, our product team organized an A/A test — a 50–50 test that reported user analytics but didn’t vary the experience — and our data analysts verified the results.

Because we didn’t significantly change the SDK API or the server-side test configuration storage location, the update automatically moved all running tests over to the new system. In the beta app, we verified that users who were allocated into one group via the old SDK were allocated into the same group via the new SDK. They were.

We rolled out the release to the main Times apps. Since then, tests have been predictable, analyzable and deterministic, which means that our feature teams can deploy tests knowing that the data they get back is reliable.

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

Feb 25 · 3 min read

What are your pronouns?
He/Him

What is your job?
I’m a Senior Analyst for Audience and Coverage.

What does that mean?
I look at ways that insights from data can provide answers to questions like, ‘Which social media platform works best to promote certain pieces?’ and ‘How does reader interest in our journalism vary by region?’

My job is a combination of data analytics, data engineering and data science. I spend most of my time interpreting data, some of my time making sure we have infrastructure that supports our work and the rest of my time applying advanced techniques that help provide more insights.

How long have you been at The Times?
Since September 28th, 2020.

Most Times employees are working remotely right now. Where are you working from these days?
I’ve been working from Coral Springs, FL.

How do you start your day?
I’ll start my day at 7:30 a.m. with my eight-pound Cavachon, Kovu, asking me to walk him around the park nearby.

My wife will wake up a little later and start preparing tea and breakfast, which we’ll enjoy together on the balcony. We usually spend some time there with Kovu, just observing the birds and turtles gather in the canal behind our apartment while the sun rises over the trees.

What is something you’ve worked on recently?
At The Times, we collect a lot of complex data that can be hard to analyze, so we spend a lot of time making sure we get the insights right. I’ve been working with my team to create a way to simplify how we gather insights and share them with our colleagues.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Currently, I’m looking at the performance of Times enterprise stories, which are big journalistic projects that might take years to produce, such as the series on former President Donald J. Trump’s taxes.

What’s interesting about this project is that whenever we say ‘Enterprise Story’ we automatically assume that it will outperform other stories in every possible way, which is not necessarily true. For example, a story specific to New York City will probably perform well there, but might not be the most-read story in every state or outside of the United States. This is important because, given the amount of effort these stories take to produce, we want to get better at setting our own expectations.

What was your first job?
I was a theater usher back in Venezuela.

What is something most people don’t know about you?
I spent four years working in a libertarian think tank in Venezuela.

What is your secret to career success?
I strongly believe that if I first focus on being a good human being, success will follow — not only in my career, but in my life.

What is your superpower?
Learning and sharing knowledge.

What are you inspired by?
Most artistic expressions, and philosophy.

Name one thing you’re excited about right now.
The world after COVID.

What I’ve seen during this time is not limited to the pain and struggles of many close to me, but also their resilience, their creativity and their ability to adapt. And all those learnings make me hopeful for a future where we all go above and beyond to connect meaningfully with one another.

What is your best advice for someone starting to work in your field?
Technology is a fast-changing industry, and it can sometimes feel challenging to stay on top of the latest developments. If you can stay curious, you can turn this challenge into an opportunity to learn new things. At least, that’s how it’s worked for me.

Some of the web technologies that The New York Times has leveraged to enhance the user experience of our website and apps for all.

Feb 25 · 7 min read

By Shilpa Kumar

Last year marked the thirtieth anniversary of the Americans with Disabilities Act being signed into law, which banned discrimination against people with disabilities in all areas of public life. The A.D.A. required that accommodations be made in places like schools and businesses, and on transportation to allow equal access for people with disabilities.

The A.D.A. is often talked about in reference to physical access. There is currently a legal debate about whether the A.D.A. should apply to digital locations, too. No matter how that legal question gets resolved, as more of our modern lives are spent online, websites and apps need to be accessible, too.

Web accessibility, also known as a11y, broadly refers to web applications that can be used by all people regardless of their cognitive, physical or visual impairments, their technical proficiencies or their situational circumstances (such as poor internet connections).

Implementing web accessibility standards is not always hard. There are numerous ways designers and developers can incorporate accessibility principles in their web applications by using basic CSS and HTML.

At The New York Times, we have taken steps to improve the web accessibility of our products. Here are some of the web technologies we have leveraged in recent years to enhance the user experience of our website and apps for all.

Semantic HTML simply means HTML that is structured, organized and adds meaning to the page. Each element tag should indicate the type of content it contains, such as a heading or a button. At The Times, we use semantic HTML across all of our platforms.

Semantic HTML isn’t just a good practice in web development, it ensures that everyone can access your content.

Modern web browsers read semantic HTML tags to know what kind of content is contained within them and how they should be presented to users. Similarly, screen readers use semantic HTML tags to communicate where a user is on a web page.

ARIA is a W3C specification that consists of attributes that can be applied to any HTML element to describe the properties, roles and states of that element. The aria-label attribute can be assigned a string, such as “Main Menu” or “Main Content,” that will help provide context to someone using a screen reader.

We implemented ARIA landmarks in numerous places across our platforms.

The Times Crossword

Our Games team implemented dynamic ARIA landmarks, which change as the state of the page changes, to improve the user experience of the Crossword. The team applied the aria-label attribute to the <button> HTML tag that renders the Crossword’s play and pause button. When a user starts or pauses a game, the string in the aria-label attribute changes to reflect the state of the game. This is particularly useful for someone using a screen reader to navigate the game.

Another place where the Games implemented dynamic ARIA landmarks was in the crossword itself. The game was built using the SVG HTML tag <rect> to render each crossword square. When a user tabs to each square, the ARIA landmarks dynamically change to give more context about their location on the page. The user’s screen reader can tell them which clue they are responding to and the number of letters needed to solve the clue.

The Times paywall

If a user is not logged in to their Times account, or if they have reached their monthly free-article limit, they will be met with a paywall banner on most articles pages. The banner is located after the HTML footer of the page, which means it wouldn’t normally be read by a screen reader, so we need to provide more context for how it can be interacted with. To help users with screen readers navigate this dynamic part of the page, we added ARIA landmarks to each part of the banner.

The elements in the login and registration component, which is how users access their Times subscriptions, are visually distinct. Black buttons and text against a white background provide enough contrast so they can be viewed clearly, especially by users with visual impairments. All of the buttons and links in the component change color in their hover state to indicate that these elements are interactive.

Increasing the color contrast serves users with color blindness, as well as users who might be in bright lighting conditions (such as direct sunlight). Color contrast should be used for buttons, links, icons and images to convey the importance of elements in a web application.

Tab order

Keyboard-accessible web applications are crucial for users who are unable to use a mouse. The tabbing order of the interactive elements should be in a logical order as the user progresses through the page, from left to right and top to bottom.

Usually, the DOM position of HTML elements informs how tabbed navigation will traverse the page. However, it is possible to configure the tab order without affecting the DOM. The HTML attribute tabindex can be applied to HTML elements to indicate whether they can be accessed by tabbed navigation — and in what order.

tabindex takes any number as a value, however it is not recommended to use values greater than “1.”

• <div tabindex="0"/>: This element can be accessed by the tab key.
• <div tabindex="-1"/>: This element cannot be accessed by the tab key.
• When tabindex is set to a value greater than “1,” it will cause the element to be prioritized to the top of the tab order. If there is more than one element with a tabindex attribute greater than “1,” the tab order will start at the lowest value and work its way up.

tabindex also controls which elements get a visible border around them to indicate that they are interactive. Every element that can be accessed via the keyboard should have a focusable border around it. Most interactive elements have the focusable border by default, but it’s possible to add focus to an element by using tabindex="0".

[If this work is interesting to you, come work with us. We’re hiring!]

Our Games team specified tab order in the Crossword to improve the usability of the game. When a user pauses the game, a model appears at the center of the page with the option to resume. The tab order of the page has been set so that the user can either tab to the resume button or continue to the links farther down the page — without needing to tab through each crossword square.

The Games team also ensured that each interactive element on the page has a focusable border around it.

For users who have cognitive impairments or for those who are not technologically experienced, directional cues are very important because they indicate how to interact with an element. Directional cues can include tooltips, arrows, modals or messaging with explicit “calls to action.”

The Crossword

The Crossword’s modal explains a new autocheck feature and instructions on how to explore the page outside of the game. This prompt is a visual cue that guides users on how to navigate the game.

The skip-to-content link and the paywall

We added a skip-to-content link to the top of our site that lets users tab to an anchor link at the start of the main content of each page. The “skip” link allows users to easily bypass content that is always present at the top of our pages, such as navigation links and header images.

The “skip” link is particularly useful when a user has reached our paywall. When a paywall is present, the “skip” link takes the user directly to the prompts to log into their account or sign up for a subscription. To accomplish this, we change the tab order of the page so it skips all of the page content and goes directly to the subscriber messaging.

The app includes a ‘Popular in Your Workspace’ feature and a great read of the day selected by Times editors.

Feb 18 · 4 min read

By Anna Dubenko and Scott Sheu

Over the last few years, The New York Times Audience team began paying attention to how Times journalism was shared in private messaging spaces like text, email and other chat platforms. The team began thinking about how to deliver our journalism that fit our readers’ needs.

We knew that some readers wanted the latest breaking news, while others wanted guidance on what to cook, what to watch and what to read. Still others came to The Times to understand their industries better, and to share information and stories with their colleagues. It was this professional context that we wanted to explore.

The Audience team — which sets our audience strategy and oversees search, social and community, as well as data insights and analysis in the newsroom — connected with our Audience Product team to build a Times app for Slack that would do three things: 1) Help Slack users connect to the Times stories their colleagues were sharing and discussing; 2) Allow Slack users to save Times articles to read later; and 3) Offer a single great read of the day, selected by our editors.

We’re excited to announce that we have launched our first app for Slack.

Since Slack’s launch in 2013, it has grown as a platform for workplace communication. We thought it would be a good place for us to connect with a key audience of professionals who want to stay informed and use their time efficiently. Slack’s users are on the platform for work, but they also use it to talk about topics that touch their work, such as issues that impact working parents, remote work and more.

When we set out to build a Slack app, we didn’t want to duplicate The Times’s existing products. Our Slack app didn’t need to send users breaking news alerts or give them a list of stories they could find on our homescreen. To create an experience that fits within the Slack ecosystem, we thought a lot about features that would add to the workday.

Starting with a list of fifteen features, we ultimately winnowed them down to three:

• The Popular in Your Workspace feature allows users to easily discover the top stories their colleagues are sharing across channels. Our “home tab” within Slack also displays the three most-popular articles in a user’s workspace.
• The Recommended Read is a daily story, selected by an editor, that we hope will delight and inform users on Slack. This feature can be shared within any channel by typing “/nytimes” or can be found in the app’s home tab.
• All Times articles are accompanied by a Read Later button, which enables users to save the article to a reading list within Slack that can contain up to 50 articles. This gives users the ability to customize their own reading routines.

Using Slack’s Block Kit builder, we designed an interface based on those three features and built a working prototype that we tested with our colleagues at Wirecutter who kicked the tires and caught any bugs. Afterwards, we set up a more formal beta test with a couple hundred employees at a design consulting firm.

After receiving quantitative and qualitative feedback from our early users, we felt pretty good about launching the Slack app more widely with the features we had in place.

To prepare for public distribution in Slack’s app Directory, our teams focused on building out the app infrastructure to enhance its resiliency and its ability to handle a much larger audience. We also worked with our legal, data governance and security teams to ensure the app not only met Slack’s requirements but also fully represented The Times’s security and privacy practices.

Users are not required to have a Times subscription in order to use our Slack app, however our normal business rules apply for articles that are read on the Times website and mobile apps.

In order to install the app in Slack, users will be required to approve certain data permissions through an authorization page. While these permissions are necessary to enable features, monitor performance and troubleshoot issues, The Times will not collect users’ names, email addresses or any other contact information through the Slack app.

To help us understand app performance and where improvements can be made to the app experience, we collect aggregated data around feature usage and activity. All data collected is approved and enabled by Slack’s API, and subject to both The New York Time’s Privacy Policy and Slack’s Privacy Policy.

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

Feb 4 · 3 min read

What are your pronouns?
She/Her

What is your job?
I’m an Associate Product Manager on the Messaging team.

What does that mean?
My team is responsible for building and maintaining the back-end infrastructure for newsletters, emails and push notifications. These platforms are part of our shift to delivering tailored messages to Times readers.

How long have you been at The Times?
Almost three years as a full-time employee, although I was lucky enough to have been an intern while completing my undergraduate degree in 2016.

Most Times employees are working remotely right now. Where are you working from these days?
My studio apartment on the Upper East Side in Manhattan.

How do you start your day?
I try to get outside for a run or a walk along the East River to clear my mind and kick-start my body.

What is something you’ve worked on recently?
We built an A/B testing tool for newsletters that enables us to test different formatting options to see what best engages our readers.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
I led the migration of 51 newsletters from an old platform to a new content-creation platform. The new platform also publishes content to the Times website, so it’s a more familiar tool for editors who also work on articles. It was the first project I led from start to finish, including training editors and ironing out the inevitable bugs. I’m proud of what the team delivered and the collaborative way we worked together.

What was your first job?
This is my first full-time job. As a student I had the usual part-time jobs (tutoring, bartending, waitressing) but the most satisfying job was as a carer for an elderly woman with dementia.

What is something most people don’t know about you?
My background is firmly in the arts — I studied acting at the Royal Academy of Dramatic Arts, and English Literature and Drama at Queen Mary University of London — but I’ve spent the last year learning more about software development and I was a keynote speaker at GopherCon last year.

What is your secret to career success?
Follow your passion. And never stop looking for interesting people to learn from.

What is your superpower?
Listening. Spending time really listening to those who have more experience than I do and are happy to share it is a shortcut to learning and improvement.

What are you inspired by?
People who have a passion for what they do, are open to learn and willing to admit mistakes. Sometimes it’s hard to admit weaknesses and lack of knowledge; I most respect those who do.

Name one thing you’re excited about right now.
One of the few upsides of the pandemic has been the acceleration of digitalization and new ways of working. Innovation is working at pace fueled by new technologies such as machine learning and data analytics, helping us better engage our readers. The opportunity to ride this wave, and see where it takes us is amazing.

What is your best advice for someone starting to work in your field?
Keep learning, but not just from those in your specific field. Learn from everyone working in related disciplines: whether it’s design, engineering, marketing or project. And don’t forget to listen to the journalists and editors — they are at the heart of what we do.

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

Jan 21 · 5 min read

What is your name?
Corina Aoi (pronounced Ah-wee)

What are your pronouns?
She/Her

What is your job?
I am a technical product manager.

What does that mean?
I support platforms as a product, in particular the platforms that power The Times’s native and web apps. My focus is to provide teams and engineers with the tools, standards and systems that enable them to develop easily, work quickly, ship with quality and measure the impact of what they have built for readers.

How long have you been at The Times?
Five years. I started as an engineer then transitioned to product in 2018.

Most Times employees are working remotely right now. Where are you working from these days?
From laid-back and always sunny Southern California with my husband, my daughter and our dog, Jellybean.

How do you start your day?
Because of the time difference in relation to New York, I get up before everyone else in the house is awake and the first thing I do is take our dog out for a walk. As a parent working during the pandemic, it is rare to have a moment to myself. These walks really help to clear my head so I can start the day feeling relaxed.

What is something you’ve worked on recently?
I recently worked on decommissioning some legacy software that was used by a bunch of products and systems. It took a lot of technical detective work to identify dependent products, and I collaborated with many stakeholders and teams to strategize a plan to migrate these products off of the system.

While decommissioning the software, we also had to ensure that the security of the dependent products and systems wasn’t compromised ahead of the 2020 U.S. Elections (no pressure!). In a large organization with so many systems, removing even just one system can be complicated. But this was an important project that taught me a lot about cross-team communication, the value of project management skills for a product manager and how the process of decommissioning as a practice has a lot of room for improvement.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
I am most proud of a web accessibility sprint we did last year. It was a large project where my team partnered with 11 other teams across the organization to resolve as many issues as possible in a 90-day sprint. There is still so much work to do in this space, but I am proud that we were able to improve the experience for our readers — especially those using assistive technologies — in such a quick time frame.

What was your first job?
Fresh out of college, I worked as a game developer for Mattel, Inc., the toy company. In particular, I developed games for Barbie.com back when Flash ActionScript was the new rage. Most games amounted to interactive drag and drop where kids could decorate Barbie’s cake skirt or something like that. However, I am very proud that I engineered an underwater adventure game where I used randomized Bezier curves to create bubbles that floated up at varying speeds and had to be popped in order to save Barbie’s little sea creature friends that were trapped inside.

What is something most people don’t know about you?
Most people are surprised to hear that I am a classically trained dancer who went to a performing arts high school. I danced professionally in commercials, film, television and entertainment while I majored in Computer Science in college and I was able to continue dancing throughout my early years as an entry-level web developer.

In fact, coding was the ideal day job for a professional dancer like me because I had a lot of flexibility, as long as I got my work done. It beat working at a restaurant and it paid better too! This went on for a decade until a major injury set me back and I began to focus more on my career in tech. I was lucky to have a job in tech to fall back on!

What is your secret to career success?
Be adaptable. Not just in your career, but in life. I’ve moved so much in my adult life that I’ve learned if you are open to new situations and experiences, opportunities will follow.

What is your superpower?
Hmm, I don’t know. I guess it would be adaptability. Is that a superpower? [editor’s note: yes, it is.] The ability to adapt comes naturally to me and it has served me well in life. My friends used to say that I was a bit of a chameleon because I seemed to comfortably settle into whatever new environment I found myself in.

What are you inspired by?
Nature, 100 percent. Being out in nature reminds me of how insignificant all my worries are in the grand scheme of the planet and it inspires me to think bigger than myself.

Name one thing you’re excited about right now.
Watching my daughter grow. I don’t think there can be anything more humbling (and terrifying while simultaneously rewarding) than raising a human in this world. She is far better and smarter than I could ever be, and her presence in my life makes me a better person every single day.

What is your best advice for someone starting to work in your field?
Ask a lot of questions early on and don’t worry about how you sound to others. There will always be people in the room who were dying to ask the same questions, but don’t have the courage to. Have courage for them because everyone benefits from that.

Jan 7 · 4 min read

What is your name?
Jessie Wu

What are your pronouns?
She/Her

What is your job?
I’m a full-stack engineer with a focus in UX engineering for the Care Platforms team.

What does that mean?
The Care Platforms team works on the customer relationship management, or CRM, software that our call center agents use for customer accounts, as well as the interactive voice response system behind 1–800-NYTIMES.

As a full-stack engineer, I focus on converting designs to code, building out our design system, optimizing the workflows between design, dev and business, and I advocate for our users.

How long have you been at The Times?
A little over a year.

Most Times employees are working remotely right now. Where are you working from these days?
Crown Heights, Brooklyn, a neighborhood where people still say hi to each other in passing.

How do you start your day?
I start with a heavy breakfast and a quick meditation to mentally prepare for the day.

What is something you’ve worked on recently?
The Care team just started a diversity, equity and inclusion working group to complement the DEI efforts being done by the wider Times company. and I am working on strategies for hiring and sourcing opportunities to diversify our pipeline and career management within our team. It is at a very grassroots level and I am grateful we have the support from our team leads.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
My team and I received a Publisher’s Award last year for building the automated phone system for 1–800-NYTIMES. I learned how to build chatbots with Amazon Web Services, and I will never look at bots, voice-activated smart devices or calling customer service the same way again.

What was your first job?
I was 10 when I started working at my parents’ restaurant. I went from packing to-gos to running the restaurant over the span of 16 years. In 2018, I was able to take a two-week vacation; it was the first one I’d taken since I was a kid.

What is something most people don’t know about you?
I took wheelchair dancing classes when I worked with people with disabilities. I used to take multiple trips driving this large cargo van to transport everyone to the studio where we would learn how to dance salsa and ballroom.We learned how to take the lead and maneuver around our dancing partners in wheelchairs. I was not very good at it; I was much better at driving the cargo van.

What is your secret to career success?
Invest in yourself: take advantage of education stipends, take initiative for career development and take care of your mental well-being.

Invest in your team because when your colleagues succeed, so do you. Bring the best out of everyone and see their potential. When you treat people with respect you open up opportunities and referrals down the line. My favorite quote by Maya Angelou is, “I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.”

What is your superpower?
I am very fast at making Slack emojis and gifs of my team members.

What are you inspired by?
There have been a few conference talks that changed my perspective and shaped who I am today. Anjuan’s Lending Privilege made me realize being privileged doesn’t necessarily have to be a bad thing; use your privilege to lift others up with you. The other one is bdougie’s Swift vs. React Native talk where he compared the two languages to Taylor Swift and Kanye West. Since then, I tend to gravitate towards using analogies and gifs in my presentations.

Name one thing you’re excited about right now.
I am really into the Buy Nothing movement and similar local Facebook groups where people in your neighborhood give or lend each other things for free. Someone gave me their 100-percent silk Chinese comforter that reminded me of my childhood. I’ve seen posts looking for things like loft beds and people have actually replied saying they have one that they are happy to give away! Someone even lent me their steam mop once.

What is your best advice for someone starting to work in your field?
Learn how to ask for help efficiently: explain the task you’re trying to accomplish, the steps you took, what you think the possible solutions are and where you are stuck. By looking into possible approaches, you’re doing half of the work for the other person.

Also, when it comes to debugging CSS for layouts and styling in general, it helps to add background colors or borders around elements to better understand their relationship to each other.

The New York Times launched an advertising data program for our direct-sold ads business that uses our own data and data science techniques.

Dec 17 · 5 min read

By Pranay Prabhat

Dec 16 · 5 min read

What is your name?
Steven Nedlin

What are your pronouns?
He/Him

What is your job?
I’m an engineering director in Data Engineering, and the engineering lead of the Transformation and Records group in the Data Platforms mission.

What does that mean?
I manage the data engineers and database administrators who build and support several critical databases, data systems, data APIs and data engineering tools at The Times. I have managed the Database Reliability team for my entire Times tenure.

How long have you been at The Times?
This August marked my 10th year at The Times. A decade! I’m so proud of this milestone!

Most Times employees are working remotely right now. Where are you working from these days?
I’m working from my summer-now-turned-semi-permanent home with my husband, Patrick, and our dog, Tory, on the Jersey shore in Asbury Park.

How do you start your day?
Every day starts with a nice walk with Tory. I put in my AirPods and usually listen to news programming like “The Daily,” but lately I’ve been listening to Italian pop songs and mouthing the lyrics as I walk. People must think I am crazy!

What is something you’ve worked on recently?
Earlier this year after we all started working from home, I was chosen to be part of the Digital Operations Sounding Board, a group created to find ways to support Times employees as they work from home during the pandemic. It gave me the opportunity to offer a suggestion that the company sponsor a single day where there are no meetings so everyone can focus on work of their choosing. The very next week, we announced “No-Meeting Fridays” at the end of every month.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
I’m incredibly proud to have been a part of the first cohort of the Times Sponsorship program in 2018. It allowed me to connect with senior leaders in all corners of the company; I learned so much about myself and how much I value empathy and inclusiveness in our company’s culture. I went on to champion these values along with a group of other like-minded peers to create the Communication and Conduct Guide for Technology in 2019.

What was your first job?
My first job was as a copy editor for baseball cards for the Topps Company. They took a chance on a young writer who happened to know nothing about sports (and still doesn’t), but loved English and grammar. They quickly realized I had no talent in sports and gave me an opportunity to work on their e-commerce team to test quality assurance for their new website and it started my entire career in technology.

What is something most people don’t know about you?
I am a major Tori Amos fan! (Yes, my dog is named after her.) I have seen her play live at hundreds of concerts and when she tours, I like to travel to different cities to see her play. It is one thing I’m very much looking forward to once the pandemic ends.

What is your secret to career success?
My secret to career success is three-fold: always bring your authentic self to what you do (and tell that inner saboteur to stay quiet and let you be you); stay honest with yourself and with others; be prepared to put in some hard work and you’ll reap the rewards.

What is your superpower?
I am an autodidact. I’ve taught myself many things in life, including engineering and database administration. I’ve learned several foreign languages on my own, including most recently Italian (listening to songs in a foreign language is one of the best ways to learn language nuances!). But I’m most proud of teaching myself to play piano — I channeled all of my Tori Amos fandom into learning the craft of pop-piano performance. I play pop music, lots of Broadway and even some Chopin!

What are you inspired by?
I learned an important lesson this year in an unexpected way. In May, during the height of the pandemic, I had to travel to Florida to take care of my father who unfortunately passed away from lung cancer by the month’s end. During his home care, I assisted his hospice nurse with all of her duties, many of which required two people. In this work we bonded, perhaps a bit reluctantly, knowing full well that we would be connected for only a short time. She likely has met hundreds of families in her career and has seen it all at the end of life. But with her help, I learned that it’s not useful to fear what’s to come, and if we focus our energy instead on caring for one another, we can get through even the toughest of times.

I am inspired by those who show kindness and care deeply for one another in so many different ways. This made me realize that in this difficult year we may be the most distant from one other, in truth we are all actually as close as ever.

Name one thing you’re excited about right now.
I’m excited for 2020 to be over! Seriously, how can one year be so challenging!? I often remind myself that it’s through these tough times that we become better people. Life may seem off the rails, but we will emerge stronger for having lived through it.

What is your best advice for someone starting to work in your field?
Be kind to yourself, and stay honest with yourself and others. Everyone has their own journey and all you need to do is be open to it and it will take you where you need to go.

Dec 3 · 5 min read

What is your name?
Shay Culpepper

What are your pronouns?
She/Her

What is your job?
Software Engineer

What does that mean?
I work on tracking and analytics for our home screen, both on our website and our mobile apps. Right now, I’m building a dashboard so the newsroom can monitor click-through rates on the home screen.

How long have you been at The Times?
Almost a year and half.

Most Times employees are working remotely right now. Where are you working from these days?
I’m working from a little makeshift desk next to my bed from my apartment in Morningside Heights, a neighborhood in Manhattan. It’s one of those cheap IKEA desks. I took two of the legs off and it rests on my windowsill.

How do you start your day?
I’m a Texan, so the most important thing I do every morning is make a giant glass of iced tea. It is just Lipton tea over ice and it is delicious. The next most important thing I do is pick out a dress and loud earrings for the day.

Once I’m ready to work, I look over the paper planner that I use to keep meeting notes, a to-do list and most importantly: my daily wins. I like to check in the morning to make sure I have written down something I’m proud of from the day before. Next, I look over the to-do list items that haven’t been checked off yet. Then I check my google calendar for what meetings I have for the day. The most important step is then planning what tasks I will complete in specific blocks of time (Thank you to my colleague Carolyn Price for this tip).

What is something you’ve worked on recently?
Lately my team, Core Product Data, has been working with the data visualization team to rethink how we build dashboards for the newsroom. We have been sketching out the architecture for a completely new platform that would allow teams to quickly and easily build new dashboards. I find creating projects from the ground up fun because you end up getting to do a lot of research.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
This isn’t part of my core job responsibilities, but I’m very proud of the work I have done through Women in Tech as part of the leadership committee. Women in Tech is an internal task force at The Times that works to improve diversity and equity in recruiting, retainment and advancement.

My teammate Shelly Seroussi and I launched a buddy program for new employees in technology. We match new employees with veteran staffers to help them network across the organization and acclimate to Times culture. The buddies meet regularly for the first three months the new hire is on staff.

What was your first job?
I served fried chicken at a place called Chicken Express. I had access to so much fried okra. I love okra in all of its forms, but fried is truly the best.

What is something most people don’t know about you?
All three of my brothers are engineers, my mom is an IT administrator and my dad is a scrum master. We all got into tech by very different paths, and it is fun to share tech as a family. I like to think of us as “The Developeppers,” but that name sadly hasn’t caught on.

What is your secret to career success?
Beyond luck, privilege and working hard, I like to think I owe most of my success to radically being myself and opening space for others to be their full selves. This means being upfront and honest about my weaknesses, and it has meant being upfront about my mental health. The fact that I’m bi-polar doesn’t come up too often, but I have found that being willing to talk about it breaks down barriers. I have had several colleagues open up to me about their own journeys as a result. Being upfront also means owning mistakes and apologizing: I made some bad calls during a big project last year, which made the project stressful for my entire team. I had to own that, apologize and spend time doing the work to be better during future projects.

I also like to open up spaces for one-on-one communication with my teammates. Sometimes that’s coffee, sometimes that’s small silly Slack conversations and sometimes it is virtual hang outs after work. Being intentional about developing relationships helps you build trust and your work will be better for it.

I should also note that a lot of this isn’t possible without taking care of myself. While companies might offer support healthcare and paid time off, they can’t make me take those days off and they can’t make me go to the doctor. I have to take care of myself and my mental health. No one can do that for me.

What is your superpower?
I can whip up very silly presentations in a bind.

What are you inspired by?
Times journalism and my colleagues keep me going every day. I’m constantly pinching myself. I count myself very lucky to be part of an organization that produces such excellent journalism.

Name one thing you’re excited about right now.
I’m pursuing a master’s degree in Data Analysis and Visualization at The Graduate Center, which is part of City University of New York. Next semester, I have a data visualization course that is studio style, meaning there will be lots of critique. I’m excited because there is nothing as helpful as feedback when it comes to visualizations: you need to know if your audience understands your intended message.

What is your best advice for someone starting to work in your field?
For your first job, just try to get your foot in the door somewhere. Working at the big companies can come later; Once you have a couple of years under your belt, a lot opens up for you. Where you worked is less important than what you worked on.

What are your pronouns?
She/Her

What is your job?
Designer

What does that mean?
I’m a designer on the Commerce team, which means I work with Times Journeys, The School of The New York Times, The New York Times Store and The New York Times Wine Club on different marketing projects.

How long have you been at The Times?
I’ve been at The Times for two and a half years.

Most Times employees are working remotely right now. Where are you working from these days?
I’ve primarily been working from a house near the Jersey Shore.

How do you start your day?
Typically I wake up around 7 a.m., check the news, play with my dog, Pocco, and then I will go for a run, do yoga or go on a bike ride. I’ve been trying to get a few minutes of meditation in before I have breakfast and begin checking emails.

What is something you’ve worked on recently?
I’m currently working on a Dog Book with the Store that features a combination of Times articles, photos and original illustrations dating back to the 1800s.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
When I started on the Commerce team, I primarily worked with Times Journeys to redesign a lot of the marketing materials. The biggest redesign project was for the 100-page catalog we send out each year to showcase our trips, experts and photography. Last year, we decided to reimagine its layout and ended up with three smaller booklets that streamlined our content, incorporated multiple illustrations and a foldout of a map of the world highlighting a number of our trips. With the smaller sizes, we were not only able to decrease costs but increase mailing numbers by more than double to expand our reach.

What was your first job?
I worked in a frozen yogurt and smoothie shop, then I left to assist in a custom framing gallery.

What is something most people don’t know about you?
I’m a Phish fan!

What is your secret to career success?
I’ve always tried to stay motivated and positive even if things don’t go my way. I’m also a big believer in using the resources I’ve had and keeping my connections going — you never know who knows whom in the design world.

What is your superpower?
My attention to detail.

What are you inspired by?
I love printmaking and am especially inspired by letterpress. Anything bold or graphic makes me want to get back into the studio life.

Name one thing you’re excited about right now.
I’m going to be spending a few days in Montauk soon, and I can’t wait to bike ride around and explore!

What is your best advice for someone starting to work in your field?
Keep in touch with people from your past: a classmate, a teacher, a mentor, a former coworker or boss. They’re all great contacts to have, and you never know when you’ll need a reference for something.

Alexandra Shaheen: The purpose of this project was to assess the systems that are critical to our success on election night, to see where those systems are in terms of reliability and resilience and to improve the systems to meet the traffic levels expected on election night.

Bures: When did you start working on this project?

Shaheen: Our team was assembled two weeks prior to the democratic debate that The New York Times co-hosted with CNN in October, 2019, and support for that event truly kicked things off.

Bures: How many teams do you work with?

Shaheen: The scope of this project is really the entire Tech organization at The Times.

Bures: Vinessa Wan and Kriton Dolias wrote about some of the election preparedness work that they led in 2018. Is this essentially a continuation of the work that they started?

Suman Roy: It is a continuation, but the scope and the ambition are much greater. We are able to aim higher because we had a solid foundation laid by Vinessa, Kriton and many others.

Bures: What are some of the assessments that you did with all of the teams?

Megan Araula: We did architecture assessments for user-facing and print-facing publishing applications, e-commerce, login, registration and personalization applications. We saw a few commonalities via these assessments, particularly a lack of documentation and preparation on application architecture, monitoring and mitigation procedures.

Bures: How did you put together the criteria for the architectural reviews? Did you look at past elections to make a list of what was expected of applications?

Araula: The Delivery Engineering team created the criteria for architecture assessment, which was refined by the Election Readiness team within the past year. We helped formalize the process by creating documentation and best practices around it. We made sure that there was a guideline document that had the criteria and recommendations. We also made sure that there was a process to on-board future reviewers.

The architecture assessment is divided into eight sections; architecture and application design, application deployment, cloud configuration, security, capacity planning, logging and monitoring, failure and outages, and service maturity. These sections help us identify how well a system is prepared for a big event, how well the engineering team understands their system and how prepared they are if an issue arises. These assessments are not the final word on architectural or operational decisions, but they are recommendations intended to better prepare teams for an event.

Shaheen: In addition to the architecture reviews, we conducted operational maturity assessments to ensure that the teams had a good level of observability into their systems, and that the alerting was set up to monitor these things appropriately. We also wanted to determine whether there were enough people on the team with knowledge into all of the systems that the team owned. So we looked at, not only just the nuts and bolts, but how we as humans are monitoring these systems.

We also did stress tests, where we threw artificial loads at systems to see how things degrade. This provided insight into what users would see (like generic 500 error messaging versus graceful degradation) if our systems failed. Additionally, we wanted to see where systems could take out another system.

Between stress tests, architecture reviews and operational maturity assessments, we could see where things stood with each of these systems, which systems would benefit from additional investment, and where there were single points of failure in terms of our ability to support the critical business workflows that they map to.

Bures: How do you decide which applications are higher priority or more critical?

Shaheen: This is a conception of criticality that Vinessa pioneered, where she identified and ranked the most important workflows at the company. For example, the workflow of being able to publish content and then being able to view that content via the website and in print is our most critical workflow. And that ranks above being able to receive payment for a subscription. If all else failed, we’d want to be able to get the news out. Our revenue would fall second to that.

We then structured our resilience investment based on which systems fell into critical workflow buckets. Most of the publishing workflows related to digital or print content production are considered to be Tier Zero. The e-commerce flows are Tier One, which are still very important to us, but not as important as getting the news out. And things like targeted advertising and sending newsletters are still important, but considered to be Tier Two.

Bures: That’s really interesting. What are some of the worst-case scenarios that you prepare for?

Roy: We looked at every one of our Tier Zero workflows and every system that serves these workflows. We looked at existing business continuity plans, hardened them if necessary and devised new continuity plans to fill in the gaps.

In case of a system outage, we built a backup New York Times website that produces the homepage, interactives and articles.

Shaheen: On election night, we need to be able to communicate effectively, as this impacts the triaging of, and mobilization around, an incident. To manage incidents in an ordered, effective and repeatable way, we rely on a rotation of incident commanders to steward communication, which allows system subject matter experts to focus on resolving defects effectively.

We’ve also had to think about how our unique remote situation this year impacts us. For example, our remote work adds more traffic and strain to our VPN. Given this circumstance, how can proactively fortify the VPN so that it can handle a higher increase of users? As an organization, we’re always thinking about these things, but we need to also understand how this year presents additional nuances.

Examining silos within the organization and opportunities for knowledge sharing has also been a major theme. How many members of our teams are really versed in the steps that are needed to do a fail-over? Instilling norms around sound documentation and running practice drills have helped us share knowledge and become more confident.

We need a plan for everything, a backup plan for everything, and then an idea of what we should do if both of those plans fail.

Bures: So many plans. How do you figure out who needs practice and who doesn’t? Or do you say, “Today we’re going to do a drill and everyone’s going to participate”?

Shaheen: Sometimes things happen in production which removes the need to plan drills for practice. Looking at how teams respond to events in production gives you an inkling of how well prepared we are. Was it a really tough incident where people didn’t know what to do and it took a while to come to the right understanding of how to proceed? If so, that’s maybe an indicator that more practice is needed.

Roy: The months leading up to the election gave us ample opportunities for practice: any system that can fail, has failed. We observed how the teams fared and then we planned drills to address gaps in preparedness.

We also practiced for scenarios that are very unlikely to happen but scenarios that we must nonetheless be prepared for. As an example, the systems that support our print production are reliable and haven’t caused us to miss a print deadline in decades. Even so, our editors practiced producing the print editions without the help of Scoop, our primary CMS.

Bures: So you’re not just working with the Technology team, you’re working with editors and producers, as well.

Roy: Yes, absolutely. It is a true collaboration.

Shaheen: Honestly, we can’t take credit for most of this stuff. This has been the hard work of this entire organization. And a lot of times we’ll have a conversation with folks and they will take it upon themselves to improve their systems. It’s important to recognize that.

Shaheen: This is a highly irregular year and effort. We went from supporting the Iowa Caucus and Super Tuesday in a way that was similar to past years to confronting a general election that will be unlike any other.

Now we’re modeling our support plan on something that’s completely ambiguous. We don’t know if we are going to be supporting a one-day event or a 40-day event.

Bures: I keep thinking about how there’s so much that isn’t known about how long this election might last. We just don’t know what’s going to happen.

Shaheen: We don’t. In past years, we’ve had an election night where everyone’s in a room together and if there are problems, they can jump on them. Right now, we have two designated shifts where over a hundred engineers will sit on a video call while actively monitoring their systems.

Bures: And everyone will be there in case anything happens.

Shaheen: Exactly. Hopefully this video call is silent the entire night. There’s no need to talk unless something is happening!

Oct 29 · 3 min read

“Meeting…” is an ongoing series from NYT Open that features New York Times employees from different corners of the company.

What is your name?
Kendell Timmers

What are your pronouns?
She/Her

What is your job?
Vice President for Advertising and Growth, Data and Insights

What does that mean?
I manage the data teams responsible for advertising analytics and subscription growth analytics. It’s exciting to get to work on two intertwined but very different parts of the business at once.

How long have you been at The Times?
Two and a half years already, although I still feel new!

Most Times employees are working remotely right now. Where are you working from these days?
My home in Glen Ridge, NJ, with my husband, two kids and an absurdly loud cat.

How do you start your day?
I start the crossword, then take a spin through the Slack channels and the headlines on The Times.

What is something you’ve worked on recently?
Launching the first-party targeting product for advertising.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Launching first-party targeting made me especially proud because it was an impressive cross-functional effort — it proved that a group of brilliant Timespeople in data science, data, ad product and engineering could compete with a giant third-party data provider.

What was your first job?
First job out of college, I worked as an “inventory analyst” to tweak the overbooking system at an airline. If you ever got bumped off a flight to Florida on US Airways … that might’ve been my fault. Sorry.

What is something most people don’t know about you?
I am a voracious reader; I read more than 100 books last year. (This year… not so much.)

What is your secret to career success?
Try new things. Be curious. Fail and try again.

What is your superpower?
Curiosity. I am interested in solving problems and almost ANY problem can be interesting to me.

What are you inspired by?
My co-workers! I’ve never met such a talented, interesting and dedicated group of people as the folks I work with across The Times.

Name one thing you’re excited about right now.
There’s a gelato place that just opened within walking distance of my house. Lucky for me they are only open four days a week…

What is your best advice for someone starting to work in your field?
Never assume you can’t do it. Even if you know nothing about an industry or application today, if you have basic skills, attention to detail and willingness to learn, you can move into anything you want to do.

What are your pronouns?
She/Her

What is your job?
I am the marketing director for New York Times Games.

What does that mean?
I focus on elevating the value of game play to drive subscription growth for our business. I work cross-functionally to help shape our brand identity and voice, and I build out strategic (and delightful!) programs that support all stages of the user journey and lifecycle.

How long have you been at The Times?
About a year and a half.

Most Times employees are working remotely right now. Where are you working from these days?
I have been working from a corner of my living room in Astoria, Queens. I will occasionally switch it up and work from my “standing desk” in the kitchen (it’s really just a high bar table with cookbooks stacked on top).

How do you start your day?
I don’t wake up as early as I did before the pandemic, but I try to wake up around 7:30 a.m. (after hitting the snooze button a few times). I usually check email, Slack and my calendar in order to mentally prepare for the day ahead, but I generally aim to use my “commute” time thoughtfully. Sometimes I’ll go for a run outside or do a short workout at home, then I’ll make tea and breakfast, breathe deeply and head into all of the video meetings.

What is something you’ve worked on recently?
I’ve been building a gallery wall in my living room! It’s a great quarantine activity that spurs both nostalgia and wanderlust.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
In May, I worked alongside a large, cross-functional team to launch a new subscription landing page for Games. This was a massive undertaking that ran concurrently with a lot of other initiatives to bring our new brand to life. In addition to reflecting a new brand identity and showcasing our full games suite, the landing page displays game-specific branding based on what a user has played before. This allows us to be flexible and experimental with our marketing approach: we can lean into the existing brand equity of the Crossword and increase awareness of our other games.

What was your first job?
I worked for Parks and Recreation in my hometown. Essentially, I made arts and crafts with little kids at my neighborhood playground over the summer while wearing a bright orange, way-too-big T-shirt.

What is something most people don’t know about you?
I am weirdly good at remembering people’s birthdays.

What is your secret to career success?
My secret is probably leaning into discomfort, no matter how uncomfortable it might feel at the onset. I’ve found that the greatest rewards, both personally and professionally, come from not shying away from a challenge.

What is your superpower?
I am an active listener with a pretty good memory, which I find beneficial working at a large, cross-functional, matrixed organization.

What are you inspired by?
As a first-generation American, I continue to be inspired by my parents who instilled a work ethic in me that guides me every day.

I am also inspired by young people, especially in this moment, and those who are challenging the status quo by advocating for widespread social change.

Name one thing you’re excited about right now.
My favorite Vietnamese place opened back up after a long hiatus during New York City’s lockdown.

What is your best advice for someone starting to work in your field?
Be authentic, be open-minded, be bold and be resilient. And be patient with the process and with yourself!

Oct 8 · 8 min read

Imagine you’re a photo editor at The New York Times. It’s early evening, a few hours before the print deadline, and a story breaks involving Alexandria Ocasio-Cortez. As fast as you can, you run a search for “Alexandria Ocasio-Cortez” in our internal photo archive and begin to scan the results.

You see a handful of photos of Alexandria Ocasio-Cortez, but they’re mostly crowded out by photos of other people and, somewhat more strangely, photos like this:

Why on earth is an image of a sunset over a farm one of the top hits for “Alexandria Ocasio-Cortez”?

The answer is that when someone enters a search term, our 20-year-old search algorithm checks the caption of every photo in our archive for that term and returns anything that matches.

In this case, the caption is: “The sun sets over a farm in Central Iowa, near where Democratic Presidential hopeful Bernie Sanders held a rally today with Rep. Alexandria Ocasio-Cortez.”

The phrase “Alexandria Ocasio-Cortez” appears verbatim in that caption, so the algorithm considers the photo a perfect match.

For a photo editor, this is a frustrating experience. The names of widely covered public figures appear so often in our photo captions — whether the person is in the photo or not — that it can be impossible for photo editors to find the images they are looking for.

In the language of information retrieval, our search engine has high recall (meaning it returns nearly all of the relevant results) but low precision (meaning it also returns many results it should not).

Ideally, instead of searching against unstructured captions, we would search against structured tags of who, or what, is in each photo. However, because photos enter our archive from many different sources with different metadata conventions, those tags usually don’t exist and captions are the best data we have.

The CMS Photo team bet that if, somehow, we could accurately tag the tens of millions of photos in our archive, we could deliver a dramatically better search experience to the newsroom.

Our first thought was to use facial recognition to tag photos based on the faces detected in them, but we quickly ran into two big problems.

The first was that state-of-the-art facial recognition models continue to be bad at identifying people of color. In one of our first tests, a widely used facial recognition API told us with 95 percent confidence that a portrait of Alexandria Ocasio-Cortez was of Michelle Rodriguez. (It was wrong.) Even if the models have improved since this 2018 A.C.L.U. study — and it wasn’t clear to us that they have — this technology still has a long way to go.

The second problem was one I started calling the “Air Force One problem,” because it can be illustrated by this photo of Air Force One with President Trump aboard:

From a photo editor’s perspective, this is a photograph of the President and should be included in search results. From the perspective of a facial recognition algorithm, this is not a picture of anyone — there are no faces in it. Many of our most compelling photos capture their subjects in creative ways that facial recognition software can’t detect.

Given the limits of facial recognition, we took a second look at our captions. Earlier, I called these captions “unstructured,” but in fact they are only unstructured if you are a computer. To an English-speaker, the captions are neatly structured according to the rules of English grammar, and the grammatical subjects of each sentence can tell us immediately who is in a photo. We figured if we could just get a computer to recognize grammatical subjects in the same way a human can, we would be all set.

Making computers extract structured data from raw text is part of the study of natural language processing (N.L.P.). N.L.P. breaks down the hard problem of extracting meaning into smaller and simpler tasks. Each step in an N.L.P. sequence makes a small change to its copy of the text before passing it to the next step. The overall sequence is called a pipeline and might look something like this:

1. Input: accepts unstructured text as a String.
2. Tokenizer: splits the String into an Array of discrete word-like units called tokens.
3. Tagger: tags each token with a part of speech, like noun or verb.
4. Parser: labels the relationships between tagged tokens. A noun before a verb might be a subject, an adjective before a noun may be a modifier, and so on.

While each step in the pipeline might look mysterious, there is no magic at work here. One could write a passable implementation from scratch using just a scripting language, a dictionary and some statistics about word usage. We tried a few libraries implemented that way before landing on spaCy, an open-source Python toolkit whose more complex implementation was more accurate for our use case. (The steps in the pipeline above are the ones spaCy runs by default.)

When we ran the caption, “The sun sets over a farm in Central Iowa, near where Bernie Sanders held a rally today with Alexandria Ocasio-Cortez” through our N.L.P. pipeline, we got this:

What are your pronouns?
She/Her

What is your job?
I am the technical product manager on the Authentication team.

What does that mean?
I help make sure that New York Times readers can log-in and register seamlessly on our website and apps.

How long have you been at The Times?
A little less than a year.

Most Times employees are working remotely right now. Where are you working from these days?
I am working from Brooklyn. My office space also doubles as my living room, dining room and kitchen. We make the most of the space!

How do you start your day?
I start my day by brewing pour-over coffee while listening to “The Daily” podcast. I like to start slow and gradually take the day in before I dive into my to-do list.

What is something you’ve worked on recently?
Recently, the team worked on stress testing the Authentication system in preparation for the November election news cycle. We conduct periodic stress tests to observe how our login and registration reacts under heavy traffic and to identify any improvements we may need to make.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
Earlier this year, The Times added an option for readers to register and log-in to The Times using their Apple accounts. Launching this new feature required close coordination across many teams. However, a couple days before our scheduled launch, the entire company began working from home due to the pandemic. We found ourselves unable to collaborate in person, as planned. Despite this new challenge, none of the teams missed a beat.

What was your first job?
In my first job, I was a fellow at a community development bank on the South Side of Chicago. I’ve always been passionate about helping people.

What is something most people don’t know about you?
I used to compete in ballroom dancing and to this day I remain a big fan of the sport.

What is your secret to career success?
Always be learning!

What is your superpower?
I am extremely curious, which leads me to consistently discover new ideas.

What are you inspired by?
I am inspired by people who have changed the world for the better. For example, I marvel at the work of Dr. S. Josephine Baker who transformed the field of public health and child hygiene in New York City in the early 1900s.

What is your best advice for someone starting to work in your field?
Start by shadowing an experienced technical product manager and learn as much as you can from observing how they partner with their team.

A design system ensures that everyone working on a product has a reference point for all of the features in the project. It allows teams to build and ship features faster.

When the NYT Cooking team started preparations to build an Android app in April 2019, we didn’t have a design system. At the time, our design team was made up of two designers and a design manager, which meant we often juggled a lot of work. As the sole designer on the Android project, I knew that it was the perfect opportunity to create a design system that the NYT Cooking Android team could use. These are some of the steps I took.

Because I had created design systems before, I knew it would be a process of trial and error. It’s normal to not know whether design elements included at the beginning of the process will still be there when the system is finished. To introduce some clarity, I defined loose guidelines, which anchored my decisions down the line.

Design principles. The team wanted to ensure that the app felt native to the Android ecosystem, which meant we would utilize native Android design patterns and paradigms. We decided to use Google’s Material Design framework as a foundation for the UI elements, while making sure that we upheld the NYT Cooking brand through styling and typography.

System users and software. The design system users were members of the design and engineering teams: designers would actively contribute to the system and engineers would reference the system. We used Figma because of its powerful Team Library feature and its ease of use for engineers.

A living design system document is the place where designers actively contribute to and update the system; it’s the most frequently updated document. A design system reference is a polished document that explains the system in detail, which is most useful for non-designers. Brad Frost describes the living document as the designer’s workshop and the reference as the shopfront. To streamline the workflow for future team members, I decided that the format of the design system documentation would be simultaneously a living document and a reference.

The two most important features in Figma for a design system are Styles and Components. Styles are basic elements such as color and typography. Components are more complex elements, such as buttons or navigation bars; I like to think of components as templates.

Metaphorically, a Style is an atom and a Component is a molecule. When combined, they become templates and pages, according to the atomic design structure laid out by Brad Frost.

Atomic design is the ideal way to structure a design system because it anticipates change. If we decided to change our brand color or the font size of a headline, we would update it in one location and it would be implemented throughout the entire system. This reduces the amount of time needed to make a style or design change and ensures the change is reliably applied in every design.

I had the benefit of our existing suite of products, so I audited our iOS app and website to decide what design elements to keep.

Audit
Our recipe pages are the most visited pages in our entire app, so the NYT Cooking product manager, design director and I decided to tackle this view first. Any recipe view — even a recipe in a book — is almost all typography, so the two things I needed to define first were the spacing system and typography.

Spacing
Since I was leaning on the Material Design framework, I decided to use their recommended spacing system. We use increments of eight, and occasionally use increments of four, for spacing, as Material Design suggests.

Typography
Recipes have many types of information, so to distinguish each part of a recipe and establish a clear hierarchy, I chose a succinct number of weights and styles from NYT Cooking’s three typefaces: Cheltenham, Karnak and Franklin Gothic. Recipe views were the most challenging to design, but once the typography was set, they informed the typography on other views.

Though we had the established iOS app that we could have drawn from, I used the new Android designs as an opportunity to modernize our typography from our existing platforms.

Color
From a product perspective, we use color minimally as a way to support the content that our editors work so hard to produce. We use a variety of grays; full black is reserved for the New York Times logo. To streamline the number of grays we use across NYT Cooking, I started designing with two grays and carefully added more when they were warranted. We ended up with eight grays, ranging from a dark, almost-black gray to a light, almost-white gray.

In addition to our palette of grays, I used the NYT Cooking-brand red as our accent color.

In digital product design, red is almost always reserved for cautionary actions and messaging. Using it as an accent color is not a common choice because it can conflict with the platform-native red of things like error messages in form fields. However, we do not use many form fields in NYT Cooking. I styled our UI elements with a tasteful amount of red without conveying that users made an error. I used a brighter shade of red to indicate error states.

Iconography
Because we were building a design system while simultaneously launching our Android app, we opted to use some of the icons provided by Material Design. I didn’t see the need to reinvent the wheel at that point of the app lifecycle. The only exception we made was for our save icon, which is prominently displayed throughout our app.

Creating Components
Components are templates for elements like a button or navigation menu; The key thing about components is that they are reusable. I started creating components once I found myself reusing the same element in multiple places, like our cards.

Cards
Cards are our primary components and are used throughout our entire app. Our cards provide an abbreviated look into the content, whether it is a recipe, a collection or a guide. These three content types are fundamentally different so we made them visually distinct.

It was hard to anticipate how these cards would look across screen sizes. While there are a standard set of breakpoints for iOS and the web, Android tends to have a wider variety of screen sizes. I defined the base card size to align with the smallest screen size on Android, so that the design would adjust to subsequently larger screen sizes.

We have strict rules about how our content is displayed. The hierarchy of the content on a card is: image, title, byline, action. Our recipes can have long titles that break to multiple lines of text, and they can have long or double bylines. I tried to display as much content on our cards as possible by slightly enlarging the card containers. The logic for text to be displayed on our recipe cards is borrowed from iOS and was adapted for Android.

What are your pronouns?
He/Him

What is your job?
Lead Software Engineer

What does that mean?
Typically, I work with other engineers across teams and projects to help lead technical decisions.

How long have you been at The Times?
Time flies! It’ll be six years in December.

Most Times employees are working remotely right now. Where are you working from these days?
I’m working from my home in northern New Jersey.

How do you start your day?
Typically, it starts with my son or daughter providing a too-much-energy-for-this-hour-of-the-morning wake-up call.

What is something you’ve worked on recently?
Most recently, I’ve been working on our GraphQL strategy at The Times.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
A few years ago, we were fortunate enough to work on a green field project: the replatforming of our core site architecture. It started out with the home page and story pages, but it has grown to support multiple reader-facing products today, such as the video and audio pages.

What was your first job?
I think it was at Blockbuster, back when renting physical movies was a thing. I thought I’d love the perk of free movie rentals, but the job was super boring so I only lasted a couple months.

What is something most people don’t know about you?
I really enjoy seeing other people completely nerd out on a passion of theirs — it’s so fun seeing how much enjoyment someone gets out of something they care a lot about. What do I enjoy nerding out on? Halo!

What is your secret to career success?
I have learned a lot from others. It’s an order of magnitude better and faster than figuring things out on your own.

What is your superpower?
I love picking up a topic I know little about and digging deep into the rabbit hole to understand all the nuances that usually takes a lot of time and experience to understand.

What are you inspired by?
I’m inspired most by my dad. He’s a person of profound wisdom, but he also has deep humility and finds great joy in helping others.

Name one thing you’re excited about right now.
The NBA is back! It’s been such a slow grind these past few months without some elite-level basketball to watch.

What is your best advice for someone starting to work in your field?
Don’t be afraid to ask questions! Nobody really knows what they’re doing and we’re all just figuring it out, if someone else already has the answer, it usually just means they started earlier than you — there’s no shame in that.

What are your pronouns?
She/Her

What is your job?
Senior Software Engineer

What does that mean?
I work on all things front-end at The New York Times.

How long have you been at The Times?
It will be four years in October.

Most Times employees are working remotely right now. Where are you working from these days?
My tiny apartment in Midtown Manhattan.

How do you start your day?
I usually wake up to a loud “mama!” shouted sometime before dawn, and I make a quick coffee (or two). I then do everything I can to set up as many Montessori-style activities for my daughter that allows me to work in enough 15-minute intervals to get through a day. Basically, I feel like I accomplish a whole day’s worth of tasks before 9 a.m.

What is something you’ve worked on recently?
A book! I collaborated with incredible design and engineering leaders on a book called “Design Engineering” about modern ideas, workflows and tools. The future of design and engineering collaboration is bright.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
The year I served on The Growth Framework committee — a team tasked with updating our old software engineering career ladder. My favorite contribution was pushing for intentionally shifting language to include the variety of contributions made by engineers at The Times. For example, shifting from “having ownership” to “showing initiative” to explicitly recognize the contributions of collaborators and discourage gatekeeping.

What was your first job?
A Christmas elf! Pointy ears, stockings, candy canes. It may have also been the best job I have ever had and definitely set a high bar.

What is something most people don’t know about you?
Last year, I co-wrote a short film that was accepted into three film festivals.

What is your secret to career success?
Reconciling the potential of what something could be with the reality of what it actually is and accurately measuring the impact I can have. In other words, the ability to read a situation and know when to push through, wait out, or move on.

What is your superpower?
The simultaneous coexistence of relentless optimism and deeply pragmatic preparedness.

What are you inspired by?
People who have a clear mission and whose actions line up with their priorities. Also, toddlers: they know how to live in the moment better than anyone.

Name one thing you’re excited about right now.
The world is in a state of unprecedented disruption, but I have to believe that the potential for profoundly positive change exists. For example, the sudden shift to remote work and lack of childcare has pulled back the curtain on the realities of just how much extra work parents and caregivers do, and the fragility of the support systems we have had to cobble together in order to go to work during the pandemic. Will we, as a society, rise to the occasion and improve these systems so that we can better support people? I hope so. We have to. Teaching and childcare are both full time jobs, and they should be valued and funded.

What is your best advice for someone starting to work in your field?
Value yourself, your opinions, and your contributions. Seek out and work with people who see your value and potential, and who don’t ask you to repeatedly prove yourself to them.

There are a lot of people out there who are willing to take advantage of your self doubt. Don’t fall for it. Spend your energy on learning, growing, and creating.

What are your pronouns?
She/Her

What is your job?
Lead Software Engineer

What does that mean?
I help build products that enable the newsroom to reach our readers.

How long have you been at The Times?
Four years.

Most Times employees are working remotely right now. Where are you working from these days?
I’m working from my apartment in Astoria, Queens.

How do you start your day?
I usually try to go out for a run for 30 minutes. I’m lucky to live right next to the waterfront, which means I get to enjoy the skyline views. Also, there are very few people outside early in the day.

When I get back home, I get ready, make coffee and start working by checking email and Slack.

What is something you’ve worked on recently?
I’m working on understanding how New York Times content ranks on Google Search and collaborating with the newsroom to develop the tools that help them optimize for off-platform performance.

Tell us about a project you’ve worked on at The Times that you’re especially proud of.
That would be the work I did as the tech lead of the messaging group to scale our in-house messaging platform that sends newsroom-produced newsletters and push notifications. I led my team to redesign the platform architecture so that it can still send newsletters and push alerts to an increasingly growing audience, while also improving their delivery speed. I presented this work at O’Reilly’s Software Architecture conference in New York at the beginning of 2020.

What was your first job?
I worked as a research assistant in the Centre for Research & Technology — Hellas (CERTH) in Greece on a European-funded project for viral news detection. During this project I had the opportunity to work with journalists from different news organizations, such as the BBC and Deutsche Welle, to develop a tool that helps reporters find breaking stories from social media posts.

What is something most people don’t know about you?
I dance tango!

What is your secret to career success?
Work hard and help others as you would hope they help you.

What is your superpower?
If I’m tired, I can fall asleep anywhere regardless of what’s happening around me. I have fallen asleep in clubs, loud bars, parties, on the beach, you name it.

What are you inspired by?
My fiancé. He is brilliant and a constant inspiration for me to do more and aim higher.

Name one thing you’re excited about right now.
I have been focusing a lot on my yoga practice, so I’m excited to start reaching some new levels, such as being able to do a handstand.

What is your best advice for someone starting to work in your field?
Be intentional in your work. Know how you want to grow in your career and look out for opportunities that will help you build those skills. At the same time, be open to working on things you didn’t plan to. You might be surprised.