Lyft, the ride-hailingcompany, has shared its experience using Protocol Buffers for inter-system integration,with a focus on collaborative protocol design. The company advocates for a design approach that prioritizes knowledge sharing, consistency, and development process quality over raw efficiency optimization.
Lyft has been using Protocol Buffers for several years, having migrated from HTTP+JSON in 2020. The company cited the advantages ofProtocol Buffers as an Interface Definition Language (IDL), including its descriptive nature, open-source code generators for various languages, optimized binary format, and backward compatibility with JSON APIs.
Over time, Lyft engineers working across three language technology stacks (Python, Swift, and Kotlin) have accumulated valuable experience in protocol definition design, especially when collaborating across teams and systems. Roman Kotenko, a software engineer at Lyft, has outlined key principles for protocol design:
Clarity: Well-designed protocols should clearly define which fields are mandatory when defining messages. This prevents information from being missed during implementation. In other words, a good protocol leaves no room for ambiguity for implementers.
Extensibility: The structure of the protocol should be built with future vision and potential roadmaps in mind. This allows for anticipatedadditions and breaking changes to be considered early on.
Beyond these principles, Kotenko highlights best practices that Lyft considers essential for using Protocol Buffers as a widely used IDL for distributed systems.
Leveraging Message Validation: Lyft emphasizes the benefits of utilizing Protocol Buffers’ message validation capabilities. The protovalidate project isa recommended approach for validating messages at runtime based on user-defined validation rules. It is a successor to the protoc-gen-validate plugin (created by Lyft and still used internally) and is powered by Google’s Common Expression Language (CEL). This approach supports various validation rules for different message and field types,including unions (oneof), enums, collections (repeated), maps, and scalars, including wrapper types.
Standardizing Constant Values: The Lyft team has developed a method for unifying constant values across multiple entities within protocol definitions. Kotenko demonstrates a custom solution used to achieve this, but also cautions about potential drawbacks, includingthe requirement for complete control over protocol definition users (e.g., in internal protocol usage).
Best Practices from Official Documentation: Lyft also emphasizes following best practices outlined in the official Protocol Buffers documentation, such as using well-known data types for common use cases, including unknown enum values, and leveraging the newoptional tag introduced in Protobuf 3.15 to explicitly mark optional fields.
Language-Specific Considerations: Kotenko concludes by highlighting the importance of consulting language-specific tutorials to understand necessary setup and subtle details, such as type mapping and code generation features, as certain behaviors can differ across languages.
Lyft’s emphasis on collaborative protocol design and best practices demonstrates the company’s commitment to building robust and maintainable systems. By sharing their experience and insights, Lyft aims to encourage other organizations to adopt similar practices and reap the benefits of Protocol Buffers for inter-system communication.
Views: 0