SMTP Retry Scheduling & Queue Expiry

Configure outbound SMTP queue management with exponential backoff, randomized jitter, greylisting survival delays, and strict 72-to-120-hour queue retention limits.

What it is

Retry scheduling and queue expiry govern how an outbound Mail Transfer Agent (MTA) handles temporary delivery failures - such as connection timeouts, greylisting, remote server throttling, and mailbox capacity limits - before permanently abandoning delivery.

When a receiving MTA responds with a 4xx transient error code (e.g., 421 Service not available, 451 Greylisted, or 452 Too many recipients), the sending MTA places the message into a deferred queue and schedules subsequent retransmission attempts according to an automated retry policy.

Initial Attempt ──(451 Greylisted)──> Queue (Wait 15m)
Second Attempt  ──(450 Throttled)───> Queue (Wait 1h)
Third Attempt   ──(450 Throttled)───> Queue (Wait 4h)
...
After 72 Hours  ────────────────────> Permanent Expiry (Generate DSN: 5.4.7 Delivery expired)

Why it matters

Greylisting Survival: Many destination mail systems deliberately reject the first delivery attempt from an unfamiliar IP with a 451 code to block simple spam bots. Senders that automatically retry after 10–15 minutes pass greylisting filters and achieve normal inbox delivery.
Throttling & IP Ban Prevention: Aggressively hammering a remote mail server that returned 421 Too many connections triggers automated IP firewall bans and blacklisting at Microsoft, Yahoo, and Gmail.
Queue Health & Memory Protection: Unbounded message retention in deferred queues leads to disk exhaustion, memory leaks, and queue backup that delays high-priority transactional emails (like password resets).

How to implement

1. Implement Exponential Backoff with Randomized Jitter. Do not retry at constant intervals. Space consecutive retry attempts exponentially, adding randomized jitter (±10%) to prevent simultaneous retransmission spikes:

$$\text{Delay} = \min\left(\text{InitialInterval} \times 2^{(\text{attempt}-1)} + \text{jitter}, \text{MaxInterval}\right)$$

Recommended intervals:

Attempt 1 (Immediate retry): 10 to 15 minutes (ensures greylisting clearance).
Attempt 2: 30 to 45 minutes.
Attempt 3: 1 to 2 hours.
Attempt 4+: Every 4 to 6 hours up to the maximum retention limit.

2. Enforce Strict Queue Lifetime Limits (Max Age). Per RFC 5321 §4.5.4, maintain a maximum queue retention window of 72 to 120 hours (3 to 5 days). Once the maximum lifetime is reached:

Purge the message from the deferred queue.
Generate an asynchronous Delivery Status Notification (DSN) with status code 5.4.7 (Delivery time expired) to notify the sender.

In Postfix (/etc/postfix/main.cf):

# Maximum queue lifetime (default: 5 days, recommended: 3-4 days)
maximal_queue_lifetime = 3d
bounce_queue_lifetime = 3d

# Backoff intervals
minimal_backoff_time = 300s
maximal_backoff_time = 4000s
queue_run_delay = 300s

3. Distinguish Transactional from Marketing Queues. Separate outbound delivery queues by message class:

Transactional Queue (Password resets, 2FA, OTPs): Fast retries (1m, 5m, 15m) and a short total expiration window (typically 2 to 4 hours, since an authentication code is useless after 15 minutes).
Marketing / Bulk Queue (Newsletters, digests): Gradual backoff (15m, 1h, 4h) with a standard 72-hour expiration window.

Common mistakes

Instant Infinite Retries: Retrying within seconds of receiving a 421 rate-limit error, resulting in an immediate 24-hour IP ban at major mailbox providers.
Indefinite Queue Retention: Retaining deferred messages in queue for weeks without expiry, exhausting disk space and sending stale notifications to users long after they are relevant.
Applying Same Retry Policy to Time-Sensitive 2FA Codes: Leaving a one-time password code queued for 3 days before final delivery, confusing recipients who requested it days earlier.
Ignoring Destination Domain MX Throttling: Sending hundreds of concurrent parallel connections to a single destination MX (like mx.google.com) during retry bursts rather than throttling concurrent connections per destination domain.

Verification

1. Inspect deferred queue status and backoff times in Postfix:

# View active deferred queue
mailq
# Inspect detailed deferred reason for a message
postcat -q <QUEUE_ID>

2. Verify destination throttling in delivery logs: Confirm that your MTA logs deferred entries with specific backoff timestamps rather than continuous reconnection loops:

postfix/smtp[12345]: <MSG_ID>: to=<[email protected]>, relay=mx.example.com[198.51.100.1]:25, delay=912, delays=0.01/900/12/0.05, dsn=4.5.1, status=deferred (Host greylisted, retry queued in 900s)