In this article 7 sections

OpenAI moved Habitat, the online storage platform behind ChatGPT and the API, from one shared Azure Cosmos DB container to isolated containers and then back toward multi-tenancy. The return was not a retreat from isolation. It followed new controls that could limit a hot key, protect online requests from offline work, and add capacity at the physical-partition level. (OpenAI)

Habitat now serves more than one billion users across 40 geographic regions and stores more than 500 petabytes, according to OpenAI. Its largest Cosmos DB account holds tens of petabytes and processes more than 20 million queries per second. These are provider-reported operating figures, not an independent benchmark, but the sequence of failure modes is more useful than the scale claim: isolation moved operational responsibility without removing it.

Isolation solved blast radius and exported responsibility

The first Habitat design placed many product workloads in a shared container. Logical partition-key collisions let unrelated use cases compete for the same key. Physical partitions created a second interference path. When storage growth caused a partition to split, the provisioned request units were divided between the children. A workload that had not grown could suddenly have less throughput simply because its neighbor’s data had.

OpenAI responded with dedicated containers. That bounded failures, but each product team now had to predict request rate, data growth, partition keys, and launch shape. One service’s load test used 1,000 request units per second against 10,000 request units per second of provisioned capacity, then launch demand reached one million request units per second. The launch demand was 100 times the provisioned capacity.

The lesson is not that dedicated capacity is wrong. It is that isolation changes the owner of estimation error. If a central platform gives every team its own database boundary, it also needs a launch contract:

  1. A declared steady-state and burst envelope, with a confidence range rather than one forecast.
  2. A replay or load test against realistic key distributions, not only aggregate queries per second.
  3. Admission control for background work and a safe degraded mode for online traffic.
  4. Automatic evidence that partition count, request units, and hot-key concentration still match the declaration.
  5. A named response when demand exceeds the envelope: queue, shed, throttle, or request explicit emergency capacity.

Without those controls, dedicated tenancy replaces a shared noisy neighbor with many independent capacity-planning incidents.

Shared tenancy worked only after the limits moved closer to the key

OpenAI says it made multi-tenancy the default again after adding hot-key throttling, quality-of-service priority for online versus offline requests, and a serverless mode that can provide 10,000 request units per second for each physical partition. Request-unit-based splits respond to load instead of storage size alone. The company reports no provisioning or noisy-neighbor incidents for several months after the change.

That claim needs local validation, but the mechanism is portable. A shared service needs at least three nested budgets:

  • Tenant budget: one product cannot consume the whole account.
  • Key budget: one customer, conversation, or object cannot exhaust its tenant’s allocation.
  • Work-class budget: recomputation, indexing, and backfills yield before interactive reads and writes.

Apply limits where contention is visible. A global requests-per-second ceiling cannot distinguish one hot key from broad healthy traffic. A tenant limit cannot protect interactive traffic from that tenant’s own backfill. Measure throttling and latency by tenant, key percentile, work class, and physical partition, then preserve enough headroom for splits and failover.

OpenAI also avoided global database replication for most read-heavy data. It replicated only small, latency-critical datasets. Other reads go through Valkey caches whose entries are invalidated by change-data-capture events. The company reports that median read latency fell from about 120 milliseconds to about 10 milliseconds, while uncached requests can still be slow.

At the API boundary, OpenAI separately documents 30-minute and 12-hour prompt-cache lifetimes. A 12-hour write is billed at 2.5 times the uncached input-token rate, compared with 1.25 times for the default 30-minute write; cache-read pricing is the same for both. The longer setting is a minimum cache lifetime, not a deletion deadline. This does not validate Habitat’s internal design, but it creates the same decision shape for application teams: pay for longer reuse only when request gaps exceed the short window and the data policy permits the retention semantics. (OpenAI API)

That architecture makes data movement part of correctness. Habitat’s change stream handles millions of changes per second across thousands of containers, millions of container partitions, and more than 100 Kafka topics. Dead-letter queues keep a poison event from blocking later records, and checkpoint batching reduces write pressure. The operational test is not only whether change data arrives. It is whether the system can prove per-key freshness, replay from a checkpoint, switch regions, and reconcile cache state after a partial failure.

A managed database became a shared failure domain

During a regional Cosmos DB outage in late 2024, Habitat failed over, then created a connection stampede while failing back. OpenAI says this led to joint monitoring with Azure, production-sized testing on a non-production account, and closer work on SDK and service changes.

The important boundary is organizational. A managed database removes some implementation work, but it does not remove the application’s connection behavior, retry amplification, key distribution, or recovery order. The provider sees account and partition health. The application team sees which customer journeys can wait and which writes must remain ordered. Failover exercises need both views.

Write the failback plan before the incident. Cap reconnect concurrency, jitter retries, warm pools in stages, and restore high-priority paths before batch consumers. Test the procedure at realistic account size. A clean failover followed by an uncontrolled return is still a failed resilience design.

Application watch: five primitives form one authority graph

LangChain described three internal agents on October 24: a go-to-market agent, an engineering agent, and a content agent. Its recurring primitives are workplace integration, identity, scoped memory, sandboxed code execution, and scheduled or event-triggered runs. The post is a vendor account of its own platform and publishes no adoption, reliability, or outcome denominator. Its useful contribution is the interaction between those primitives. (LangChain)

An agent invoked in a direct message may act on behalf of the user. The same agent in a shared channel may need fewer tools. A scheduled review may run under a service identity because no person is present. Memory scope must follow that identity and trigger context, while sandbox egress must enforce the same policy.

Do not configure those pieces independently. Model them as an authority graph:

trigger -> actor -> credential -> memory scope -> allowed effect -> approval

Log the resolved path for every effectful action. Test the graph with negative cases, such as a shared-channel request for a personal mailbox action, a scheduled job trying to reuse an interactive token, or one user’s memory appearing in another user’s run. The product interface can live in Slack or a custom UI, but the authorization decision should remain explicit and inspectable.

Model watch: capability subtraction has a release boundary

Moonshot AI says it removed cyber-related data from Kimi K3.1 Pro’s training corpus and reinforcement-learning environments before a planned November 4 open-weight release. Reuters reports that the filtered model scored 81.0% on Moonshot’s internal vulnerability finding and exploitation benchmark, compared with 90.5% for a similar unfiltered model. The result has not been independently verified. Moonshot says general coding gains were preserved. (Reuters)

This is a more testable claim than a broad statement about safety, but it is still one provider-selected evaluation. Before adopting the weights, separate at least four questions: did offensive capability fall on unseen tasks, did defensive utility remain, did general coding regress on local work, and can fine-tuning recover the removed behavior? Open weights can be modified and redistributed after release, so a hosted-service control cannot be assumed to travel with them.

The first launch signal is operational rather than evaluative. Before today’s research cutoff, Reuters reported that Moonshot apologized for rate limiting and unnecessary quota consumption, reset subscriber quotas, and changed parameters to reduce unnecessary subagent calls and output length. That does not measure model quality. It does show why agent releases need token, delegation, and quota telemetry from the first request. (Reuters)

Stay Sharp: p50 does not survive fan-out

OpenAI’s cache result is reported at the median. Median latency matters, but a user-facing AI operation often fans out to retrieval, policy, memory, tool, and storage calls. A rare slow component becomes common at the request level.

Jeff Dean and Luiz André Barroso described this amplification in The Tail at Scale in 2013. Large services need to build predictable end-to-end latency from components whose individual latency is variable. (Google Research)

Suppose each of 100 independent subrequests has a 1% chance of exceeding a latency threshold. The probability that all 100 finish below it is:

0.99^100 = 36.6%

The probability that at least one crosses the threshold is therefore about 63.4%. The assumptions are simplified because real calls are often correlated, but the calculation exposes the design error: a good component percentile does not guarantee a good workflow percentile.

For each critical user journey, record the fan-out width and latency budget by dependency. Report end-to-end p95 and p99 alongside cache-hit rate and uncached latency. Bound fan-out, cancel work that can no longer affect the response, and use hedged requests only when duplicate side effects are impossible or idempotently fenced. A 12-fold improvement at p50 is valuable. It is not a resilience result until the tail and recovery path are measured too.

Watch next

  • Whether Habitat’s multi-tenant controls continue to prevent hot-key and provisioning incidents as account size, physical partitions, and change-stream volume grow.
  • Evidence from production-sized failover and failback exercises, including connection ramp, retry amplification, and recovery time by work class.
  • Independent evaluation of Kimi K3.1 Pro’s filtered cyber capability before and after the planned open-weight release, including whether fine-tuning restores the removed behavior.
  • Kimi launch telemetry that separates model latency from unnecessary subagent calls, output length, rate limiting, and quota accounting.
  • Company-agent implementations that publish authority-graph tests and effect-level audit data rather than only interface demonstrations.
  • DeepSeek’s Search and Fetch endpoints begin charging $0.75 per 1,000 successful calls each at 16:00 UTC on October 31, including server-side calls from supported coding harnesses. Inventory hidden tool calls before the deadline. (DeepSeek)