Bulkhead pattern

Reference Card

• Origin/Prior Art: Naval architecture, Release It! (Michael Nygard).
• Related Practices: Resource isolation, Multi-tenancy, Circuit Breakers.
• Key Concepts: Partitioning, Thread pools, Blast radius.
• Primary Failure Modes: Shared connection pools, Unbounded queues.

Introduction

In a typical application, resources like CPU, memory, and database connection pools are shared across all incoming requests. If a specific feature of the application experiences a massive surge in traffic, or if a specific downstream dependency slows down to a crawl, requests hitting that feature will consume all the shared resources.

This causes the entire application to crash, taking down unrelated features. The Bulkhead pattern prevents this by partitioning resources. By strictly separating thread pools or compute instances based on feature, tenant, or criticality, a failure in one partition is physically contained and cannot sink the whole ship.

Mental Model

Think of a submarine. If a submarine were a single, hollow tube, a single leak anywhere in the hull would fill the entire ship with water, and it would sink.

To prevent this, shipbuilders divide the submarine into separate, watertight compartments called bulkheads. If a torpedo blows a hole in the forward compartment, water floods that specific compartment. The heavy steel bulkhead doors are sealed, trapping the water. The forward compartment is destroyed, but the rest of the submarine remains dry and buoyant, allowing the ship to survive and surface.

Operational Pattern

Bulkheads can be implemented at different levels of the architecture.

1. Thread Pool Bulkheads (Application Level)

Instead of using a single global thread pool to handle all outgoing HTTP calls, the application uses isolated thread pools.

• Thread Pool A (10 threads) handles calls to the Payment API.
• Thread Pool B (10 threads) handles calls to the Recommendation API. If the Recommendation API becomes incredibly slow, Thread Pool B will quickly fill up with waiting requests and exhaust its capacity. The Recommendation feature breaks. However, Thread Pool A is completely unaffected, ensuring users can still check out and pay.

2. Infrastructure Bulkheads (Deployment Level)

Instead of running a single massive cluster that serves all customers, the infrastructure is physically partitioned.

• Cluster 1 serves Enterprise Tier customers.
• Cluster 2 serves Free Tier customers. If a Free Tier customer writes a malicious script that accidentally triggers a Denial of Service attack against the API, Cluster 2 crashes. The Enterprise Tier customers on Cluster 1 experience zero latency or downtime because they are physically isolated in a different bulkhead.

Failure Modes

Over-Partitioning (Resource Waste)

If you divide an application into 50 distinct bulkheads, each with its own tiny thread pool, you lose the statistical efficiency of shared resources. A sudden, legitimate burst of traffic to one feature will exhaust its tiny bulkhead and fail, even though the other 49 bulkheads are sitting 100% idle. Bulkheads must be sized carefully to balance isolation with resource efficiency.

Shared Databases (The Hidden Leak)

You can partition your web servers perfectly, but if all the web servers talk to the exact same shared PostgreSQL database instance, you have effectively bypassed the bulkheads. If Bulkhead A executes a terrible query that locks the database tables, Bulkhead B will crash waiting for the database. True isolation requires bulkheading down to the data tier, or implementing strict resource quotas within the database itself.

Security

Bulkheads are the primary defense against the “Noisy Neighbor” problem in multi-tenant SaaS environments. A malicious actor might realize they cannot steal data, so they simply attempt to consume all available compute resources to take the platform offline. By enforcing tenant-level bulkheads, the malicious actor can only crash their own isolated partition, leaving the rest of the customer base unaffected.

Operational Guidance

Identify your critical and non-critical paths. The checkout flow is critical; generating PDF receipts is non-critical. Place the non-critical paths into a strictly constrained bulkhead. If the PDF generation service goes haywire, it must be physically impossible for it to steal CPU cycles or database connections from the checkout flow.

Diagnostics

To verify bulkhead effectiveness, simulate a failure during load testing. Use a tool like Chaos Mesh to inject 10 seconds of network latency exclusively into the connection between the application and a non-critical downstream service. Monitor the application’s overall success rate. If the overall success rate drops to zero, your bulkheads are leaking (or non-existent). If the overall success rate remains stable, but the specific non-critical feature fails gracefully, the bulkheads are holding.

Related topics

• Circuit breaker pattern
• Rate limiting strategies

Sources & further reading

• Bulkhead Pattern - Microsoft Azure Architecture Center