Customize Consent Preferences

We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.

The cookies that are categorized as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... 

Always Active

Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

No cookies to display.

Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

No cookies to display.

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

No cookies to display.

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

No cookies to display.

Advertisement cookies are used to provide visitors with customized advertisements based on the pages you visited previously and to analyze the effectiveness of the ad campaigns.

No cookies to display.

上海的陆家嘴
0

By [Your Name], Senior Journalist and Editor

Pinterest, the popular visual discovery platform, hassignificantly enhanced the efficiency and resource utilization of its in-house time series database, Goku. These recent updates, detailed in a recent blog post by the Goku team,focus on optimizing storage efficiency and resource usage without compromising service quality.

Goku, developed by Pinterest, was initially created to address specific limitations of OpenTSDB. The latest advancements center around two key features: metric namespaces and top-level write-intensive metrics, both aimed at reducing the amount of data stored in Goku.

Metric Namespaces: Organizing Data for Efficiency

Metric namespaces provide a structured approach to organizing metric configurations, enabling efficient data management. This feature allows for grouping related metrics together, simplifying data access and management.

Top-Level Write-Intensive Metrics: Reducing Unnecessary Data

The Goku team has implemented a system to identify and optimize top-level write-intensive metrics. This system effectively prevents unnecessary data writes, resulting in a remarkable 37% reduction in the volume of time-series data stored.

Architectural Optimizations for Cost Reduction

Beyond these new features, Pinterest hasimplemented a series of architectural optimizations to further reduce infrastructure costs. These include:

  • Improved Metric Name Indexing: This optimization has reduced memory usage per host from 12GB to 3GB.
  • Dictionary Encoding in Goku Compactor: This technique effectively addresses memory scarcity issues, allowing for the use ofless expensive hardware.
  • Optimized Memory Allocation Strategy: Pinterest has addressed internal fragmentation and memory over-allocation issues, leading to significant memory savings. For instance, optimizations to the folly::IOBuf structure have resulted in a reduction of 8 to 11GB of memory usage per host.

Time Series Compression: A Key to Efficiency

Time series compression algorithms are crucial for efficiently storing and processing large volumes of time-stamped data. These algorithms identify patterns and redundancies in data, reducing data size and enabling faster query processing and lower storage costs.

Pinterest, like other companies such as TimescaleDBand Meta, has leveraged techniques like incremental encoding, incremental delta encoding, and XOR-based compression to achieve significant storage efficiency improvements.

Conclusion

Pinterest’s recent enhancements to Goku demonstrate a commitment to optimizing resource utilization and efficiency. These advancements, including metric namespaces, top-level write-intensive metric optimization, and architecturalimprovements, have significantly reduced data storage needs and infrastructure costs. The adoption of time series compression algorithms further contributes to efficient data management and processing. As Pinterest continues to grow, these optimizations will play a critical role in ensuring scalability and cost-effectiveness.

References:


>>> Read more <<<

Views: 0

0

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注