On this page
Capacity planning and load forecasting
The analytical process of predicting future system resource requirements based on historical trends, business projections, and known scaling constraints.
Reference Card
- Origin/Prior Art: Datacenter operations and supply chain management.
- Related Practices: Autoscaling, Performance testing, Load shedding.
- Key Concepts: Organic growth, Step-function growth, Bottleneck analysis.
- Primary Failure Modes: Over-provisioning (wasting money) and Under-provisioning (outages).
Introduction
Autoscaling is a reactive mechanism. If traffic spikes, an autoscaler boots up new servers to handle the load. However, autoscaling takes time (often minutes), and it relies on the cloud provider actually having idle servers available to give you. If you wait until a massive event (like Black Friday or a Super Bowl commercial) to acquire capacity, the autoscaler will fail to keep up, or the cloud provider will reject your request due to regional exhaustion.
Capacity planning is the proactive discipline of ensuring your architecture can scale when required. It involves analyzing historical metrics, projecting business growth, identifying architectural bottlenecks, and reserving hard infrastructure limits before the load ever arrives.
Mental Model
Think of running a restaurant. Autoscaling is calling in an extra waiter because the dining room suddenly got busy. Capacity planning is realizing that Mother’s Day is three weeks away, calculating that you will need triple the usual staff and four extra ovens, and signing the leases and contracts today. If you wait until Mother’s Day to start looking for ovens, your restaurant will fail, regardless of how fast you can hire waiters.
Operational Pattern
Effective capacity planning involves three distinct streams of analysis:
- Organic Growth: Analyzing historical metrics over months or years. If database storage grows by 5% every month, you can mathematically forecast exactly when the 2TB disk will fill up, allowing you to schedule a maintenance window weeks in advance.
- Step-Function Growth: Planning for sudden, discrete events. This requires deep communication with the business side. If the marketing team is launching a television campaign, engineering must calculate the expected requests per second (RPS) and proactively provision the web tier to absorb the initial shock before the autoscalers engage.
- Bottleneck Identification: Hardware scales linearly, but software rarely does. You might provision 1,000 web servers, but if they all write to a single SQL database table wrapped in a mutex lock, the system will cap out at 500 RPS regardless of how many web servers exist. Capacity planning requires load testing to find where the architecture breaks first.
Failure Modes
Ignoring Hard Quotas
Every cloud provider enforces arbitrary “Service Quotas” on accounts to prevent runaway billing and protect their own capacity. A common failure mode is configuring an autoscaler to scale up to 100 EC2 instances, only to discover during an actual traffic spike that the AWS account has a hard limit of 32 instances for that specific region. Capacity planning requires auditing and requesting quota increases weeks before major events.
The “Cost vs. Risk” Vacuum
Engineers often over-provision heavily out of fear of an outage, leading to massive cloud bills. Finance departments often demand under-provisioning to save money, leading to fragile systems. Capacity planning must be a negotiated agreement based on risk. If forecasting shows a 10% chance of a traffic spike that requires $10,000 in idle servers to absorb, the business must explicitly accept either the financial cost of the servers or the reputational risk of the outage.
Security
Denial of Service (DoS) attacks actively exploit capacity limits. A well-planned architecture must include “Load Shedding” and “Rate Limiting” at the edge to protect internal capacity. If malicious traffic is allowed to trigger horizontal autoscalers endlessly, the attack transitions from an availability outage to a financial outage (Economic Denial of Sustainability), draining the organization’s bank account.
Operational Guidance
Do not rely solely on CPU and RAM for load forecasting. Often, systems fail due to hidden limits like TCP connection tracking tables, File Descriptors (inodes), or database connection pool exhaustion. Run synthetic load tests in staging environments that push the system to 200% of expected peak traffic. Document exactly which component breaks first, and establish alerts that fire when production traffic reaches 70% of that known breaking point.