Uber, the global ride-sharing giant, has recently shared insights into its development team’s approach to managing the continuous deployment (CD) of microservices within large monorepos. As the company’s microservices architecture is composed of thousands of services, a reliable and efficient system is crucial for deploying updates, security patches, and new features. To ensure the safety and timeliness of this process, Uber has adopted CD and automated deployment into the production environment, which is essential for maintaining code quality and minimizing the delay in delivering changes.
Challenges of Scaling Monorepos
However, as Uber’s codebase and service demands grew, the existing deployment system faced challenges related to scale, standardization, and security, particularly in monorepos containing multiple microservices. The company recognized the need for a more streamlined and automated CD system.
In 2022, Uber embarked on a project to redesign its CD processes, aiming to create a more unified and automated solution for deploying microservices. At the time, Uber was managing approximately 4,500 microservices across three monorepos, handling 5,600 submissions and 7,000 deployments weekly. However, only 7% of these services were using CD for automated deployment, with many requiring manual intervention. This fragmented and highly flexible system led to inefficiencies and security risks, especially due to the lack of a company-wide standard among more than 100 unique deployment templates.
Introduction of Up CD
To address these challenges, Uber introduced a new system called Up CD, designed to enhance automation and security. It is tightly integrated with the company’s internal cloud platform and observability tools, ensuring that deployments follow standardized and repeatable processes by default. The new system prioritizes simplicity and transparency, especially in managing monorepos.
A key improvement is the optimization of deployments by viewing which services are affected by each commit, rather than deploying all services with every code change. This reduces unnecessary builds and allows engineers to better understand changes affecting their services.
Up CD Architecture
Up CD introduces a unified submission flow for all services, ensuring that each service goes through a series of deployment stages, each with its own security checks. These checks include time delays, deployment windows, and service alarms, ensuring that deployments are triggered only when safe. Each stage runs independently, allowing for flexibility while maintaining security. This new approach reduces manual errors and provides a more structured deployment experience.
User-Friendly Interface
A key component of Up CD’s success is its user-friendly interface, which allows engineers to intuitively view their service’s submission history and deployment status. The interface collapses unnecessary submissions, making it easier to track changes and identify potential issues. Engineers can also view real-time updates on deployment status, enabling them to quickly identify which submissions have been deployed and to which environments.
Since the launch of Up CD, Uber has seen a significant improvement in service deployment automation. The proportion of automatically deployed services has risen from less than 10% to nearly 70% within a year. Despite the increased frequency of deployments, the production incident rate per 1,000 code changes has dropped by over 50%, highlighting the system’s ability to manage more frequent deployments with enhanced security management.
Addressing New Challenges
While the deployment process has become faster, especially for changes affecting shared code in monorepos, this has introduced new risks. Uber has addressed these issues by interleaving deployments between service layers, stopping the deployment if a lower-priority service encounters problems.
Conclusion
Uber’s improved continuous deployment system has successfully achieved large-scale automation and standardization of deployments, significantly improving the security, efficiency, and transparency of deploying microservices. By closely integrating with existing tools and prioritizing automation, Up CD has enabled Uber to manage rapid growth while minimizing production incidents.
Views: 0