News

OpenAI Scales Storage to 500 PB, Read Latency Drops 12x

OpenAI has published part two of its engineering series on Habitat, the application storage platform behind its products. Habitat responds in milliseconds to more than 1 billion users across 40 geographic regions and now holds over 500 petabytes of data. The company said its team scaled the platform by putting reliability first, then performance, and finally operational efficiency at scale, working with the Azure team on a multi-tenancy approach of co-managed containers that maintains excellent read performance at massive, global scale.

Key Points

  • Habitat serves over 1 billion users across 40 geographic regions and stores more than 500 petabytes
  • The underlying Azure Cosmos DB went through three phases, from a single shared container to single-tenant isolation and back to multi-tenancy by default
  • OpenAI’s largest Cosmos DB account stores tens of petabytes and serves more than 20 million QPS
  • Global read p50 latency fell from roughly 120 ms to roughly 10 ms, a 12x improvement
  • An in-house exabyte-scale storage system combining RocksDB Cloud from Rockset with Raft-based replication is in development

From a single container to a three-phase journey

Habitat is the online storage platform OpenAI built so its products can quickly and reliably access data, running on Azure Cosmos DB underneath. The team initially relied on a single default container, leaning on the documented 10,000 RU/s limit per logical partition key for back pressure, and ran into three problems: frequent collisions on common partition keys such as user_id and workspace_id across different product use cases; logical partitions sharing a physical partition’s limits, which let workloads for niche features disrupt core login and signup flows; and shrinking effective throughput as physical partitions split, with each child receiving half the throughput capacity.

The second phase moved critical use cases onto separate containers for strong isolation. The trade-off was capacity planning for every container: product teams often shipped features to production without pre-scaling, leaving engineers in SEV rooms on launch day scaling containers at the last minute. A new use case jumping from zero to 1 million QPS could represent 1,000x more load than the baseline attributed to internal dogfooding, a pattern the team said failed to scale organizationally.

Back to multi-tenancy, co-managed with Azure

In the third phase the team shifted back to a multi-tenant solution, working closely with the Azure team on fixes for the original problems: throttling the hottest keys first so a single tenant cannot consume all of a physical partition’s throughput, tiered quality of service so offline bulk workloads are throttled before interactive traffic that serves end users, and a new improved serverless mode built for OpenAI that lets every physical partition scale up to 10,000 RU/s and introduces RU-usage-based splits in addition to storage-usage-based splits. Since moving back to a single shared container, OpenAI said it has had no provisioning or noisy neighbor incidents for months despite accelerating product development.

The article also disclosed the scale of the deployment: OpenAI’s largest Azure Cosmos DB account stores tens of petabytes of data and serves more than 20 million QPS, and the company said that to its knowledge it is the only company which has deployed Azure Cosmos DB at this scale. Two regional outages in late 2024, a failover exercise that revealed the largest accounts would take several hours to fail over, and a connection stampede that overwhelmed a gateway during failback, pushed the two teams into a co-managed model with shared tests, shared dashboards, joint ownership of failure modes and direct paging access without going through support tiers.

Read latency down to 10 milliseconds

Reads account for over 90% of Habitat requests and became a major focus in 2026. The layered strategy replicates latency-critical data such as authentication metadata to highly replicated accounts, in some cases all 40 Azure regions where OpenAI serves traffic; integrates a distributed Valkey lookaside cache whose invalidations propagate globally within seconds through change data capture, following in the footsteps of systems like Meta’s TAO; and rewrites the Habitat service in Rust to remove overhead. Together, these changes reduced global read p50 from roughly 120 ms to roughly 10 ms, a 12x improvement since the team began focusing on performance earlier this year.

Change data capture itself runs at millions of captured changes per second across thousands of containers, feeding more than a hundred Kafka topics that power secondary indexes, data-movement pipelines, offline analytics and real-time product features. Looking toward the next billion users, OpenAI is building an in-house exabyte-scale online storage system optimized for bulky Habitat data types, combining the RocksDB Cloud core from Rockset with Raft-based replication and using disaggregated storage to reduce flash capacity at the cost of a lower peak QPS.

Source: OpenAI

(Cover image is AI-generated, for illustrative purposes only)