Skip to content
Ayhan Sipahi Ayhan Sipahi

From RFC to Production: What They Don't Tell You About Implementation

Where RFC designs diverge from production reality, using notification systems as the worked example, and how to tell useful adaptation from architectural drift.

RFCs rarely survive contact with production unchanged, and that is not automatically a failure. Architecture diagrams that looked clean in review become tangled six months later, as timeline pressure, missing requirements, and operational realities force trade-offs that were never in the design.

The pattern is not bad design or bad engineers. It is the normal cost of building complex systems under business constraints. The useful skill is separating signal from drift: a gap is signal when the design met a requirement nobody could have known about at review time, and drift when the design was abandoned because following it was inconvenient. Notification systems make a good worked example, because the specification is short and the operational surface is unusually wide.

Building a Notification System from RFC to Reality

Take a hypothetical notification RFC of the kind most teams would recognise. It opens with four success criteria, and every one of them can be checked against numbers that operators have published about their own systems.

// A hypothetical RFC's success criteria, written before any code exists
interface NotificationSystemGoals {
  deliveryTime: '<100ms for in-app, <5s for email',
  throughput: '10,000+ notifications per second',
  uptime: '99.9% availability',
  timeline: '12 weeks with 2 developers'
}

Checked one line at a time against published figures, three of those four change meaning.

Uptime. The availability table in Google’s Site Reliability Engineering book converts 99.9% into 43.2 minutes of unavailability per month and 8.76 hours per year, and 99.99% into 4.32 minutes per month and 52.6 minutes per year. One deploy that takes an hour to roll back has spent the entire monthly budget at 99.9% and the entire yearly budget at 99.99%. The target buys a fixed number of minutes, and that number is smaller than most incidents.

Delivery time. LinkedIn’s engineering blog reports that its notification platform, Air Traffic Controller, processes over a billion requests per day, and that reworking the delivery path brought P90 end-to-end latency for member-to-member messaging push notifications down from about 12 seconds to about 1.5 seconds. Both ends of that range sit far from a 100ms design target, and the optimised end is the published result of a team that specialises in this.

Sub-millisecond numbers do exist, with a narrower scope. Netflix’s Pushy team reports median latency under one millisecond and 99th-percentile latency under 4ms, measured “from the incoming message arriving at Pushy to the response being sent back to the device”. That covers one hop inside infrastructure Netflix owns. A user’s phone sits several hops further out.

Throughput. The same Netflix post describes hundreds of millions of concurrent WebSocket connections and a system that regularly reaches 300,000 messages sent per second, running at an average of 200,000 connections per node with room to reach 400,000, up from 60,000 per instance previously. That is the shape of a channel the sender owns end to end. The moment a notification fans out to a channel somebody else owns, the ceiling is theirs to set:

Fan-out pathPublished limit
Slack channel postSlack’s chat.postMessage reference documents roughly 1 message per second to a specific channel, with workspace-wide limits of several hundred messages per minute and a burst allowance
SMS via a US long codeTwilio documents 1 message per second for a US long code and 100 for a short code; excess requests queue for up to 10 hours by default before surfacing as error 30001
Email via Amazon SESAWS documents sandbox accounts at 1 email per second and 200 emails per 24-hour period, with production rates that vary by use case and a hard cap of 50 recipients per message
iOS push payloadApple documents 4096 bytes for a regular remote notification and 5120 bytes for VoIP, and states that APNs refuses notifications above the maximum size

The internal target and the external ceilings are not in the same units of reality. A US long code at 1 message per second is four orders of magnitude below 10,000 per second; a short code at 100 per second is still two orders below. No amount of internal engineering moves those numbers, so the throughput line in the RFC describes buffering capacity. Sending speed belongs to whoever owns the channel.

Everything else in the RFC can be genuinely thorough: rate limiting, deduplication, preference management, quiet hours. All of that sits under your control. The ceilings in the table above do not, and being rigorous about the first set buys nothing on the second.

Implementation Challenges and Adaptations

Database Schema Evolution

The initial database schema design emphasized clean normalization with proper foreign keys and constraints:

-- Initial RFC schema design
CREATE TABLE notification_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    notification_type VARCHAR(100) NOT NULL,
    template_id UUID REFERENCES notification_templates(id),
    data JSONB DEFAULT '{}',
    status VARCHAR(20) DEFAULT 'pending',
    sent_at TIMESTAMP,
    delivered_at TIMESTAMP,
    read_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT NOW()
);

The shape that survives contact with production usually looks more like this:

-- Schema after production adaptations
CREATE TABLE notification_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID, -- Foreign key removed due to performance issues
    notification_type VARCHAR(100),
    notification_type_v2 VARCHAR(255), -- Migration in progress
    template_id UUID,
    template_id_v2 BIGINT, -- Different team used different ID type
    data JSONB DEFAULT '{}',
    data_compressed BYTEA, -- Added when JSONB got too large
    status VARCHAR(20) DEFAULT 'pending',
    status_v2 VARCHAR(50), -- More statuses than expected
    priority INTEGER DEFAULT 0, -- Not in RFC, critical for production
    retry_count INTEGER DEFAULT 0, -- Not in RFC, essential for debugging
    channel VARCHAR(50), -- Denormalized for query performance
    correlation_id UUID, -- Added for distributed tracing
    partition_key INTEGER, -- Added for sharding
    sent_at TIMESTAMP,
    delivered_at TIMESTAMP,
    read_at TIMESTAMP,
    failed_at TIMESTAMP, -- Not in RFC, very much needed
    expires_at TIMESTAMP, -- Not in RFC, prevented infinite growth
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW() -- Added once nobody could tell when a row last changed
);

-- Every new access path arrives with an index, and every index is paid for
-- on each insert and update, not just on the query it was added for
CREATE INDEX CONCURRENTLY idx_notification_events_user_created ON notification_events(user_id, created_at DESC) WHERE status != 'deleted';
CREATE INDEX CONCURRENTLY idx_notification_events_correlation ON notification_events(correlation_id) WHERE correlation_id IS NOT NULL;

Each column here answers a production question the design review had no way to ask. The indexes are the part that keeps costing. The PostgreSQL documentation states the trade plainly: once an index exists the system has to keep it synchronized with the table, which adds overhead to data manipulation operations and can prevent heap-only tuple updates, so indexes that are seldom or never used in queries should be removed. CREATE INDEX CONCURRENTLY is what makes adding one safe on a live table, since it avoids locking out writes, and PostgreSQL also documents its two costs: the build takes longer, and a build that fails leaves an invalid index behind that still carries the update overhead until somebody drops it.

WebSocket Connection Management Complexity

The RFC specified WebSocket-based delivery for optimal performance. The initial implementation approach was straightforward:

// RFC's WebSocket implementation
class NotificationWebSocketManager {
  private connections: Map<string, WebSocket> = new Map();
  
  async sendNotification(userId: string, notification: NotificationEvent) {
    const connection = this.connections.get(userId);
    if (connection && connection.readyState === WebSocket.OPEN) {
      connection.send(JSON.stringify({
        type: 'notification',
        data: notification
      }));
    }
  }
}

The transport’s own documented limits do most of the redesigning. Amazon API Gateway publishes the quotas for a WebSocket API: 32 KB per frame and 128 KB per message payload, neither of which can be raised, a 10-minute idle connection timeout, a 2-hour maximum connection duration, and 500 new connections per second per account per Region. The quota page also spells out the consequence of ignoring the first one: a message larger than 32 KB has to be split into frames of 32 KB or smaller, and a larger frame closes the connection with code 1009.

Those quotas explain most of what a second version of the class contains:

// Amazon API Gateway WebSocket quotas: 32 KB per frame, 128 KB per message
const MAX_FRAME_BYTES = 32 * 1024;
const MAX_MESSAGE_BYTES = 128 * 1024;

class NotificationWebSocketManager {
  private connections: Map<string, Set<WebSocketConnection>> = new Map();
  private connectionMetadata: Map<string, ConnectionMetadata> = new Map();
  private healthChecks: Map<string, NodeJS.Timeout> = new Map();
  private rateLimiters: Map<string, RateLimiter> = new Map();
  private deadLetterQueue: Queue<FailedNotification>;
  private circuit: CircuitBreaker;
  
  async sendNotification(userId: string, notification: NotificationEvent) {
    const connections = this.connections.get(userId);
    if (!connections || connections.size === 0) {
      await this.queueForLaterDelivery(userId, notification);
      return;
    }
    
    // Handle multiple connections per user (mobile + web + tablet)
    const results = await Promise.allSettled(
      Array.from(connections).map(async (conn) => {
        try {
          // Idle connections are dropped after 10 minutes, live ones after 2 hours
          if (!this.isConnectionHealthy(conn)) {
            await this.reconnectOrEvict(conn);
            throw new Error('Unhealthy connection');
          }
          
          // Rate limiting per connection
          const limiter = this.getRateLimiter(conn.id);
          if (!await limiter.tryAcquire()) {
            await this.backpressure(conn, notification);
            return;
          }
          
          const message = this.serializeNotification(notification);
          // Framing satisfies the 32 KB rule but not the 128 KB message cap,
          // so anything above it goes out as a reference the client fetches
          if (Buffer.byteLength(message) > MAX_MESSAGE_BYTES) {
            await this.deliverByReference(conn, notification);
            return;
          }
          
          // Circuit breaker for cascading failures
          return await this.circuit.fire(async () => {
            // An oversized frame does not fail the send, it kills the socket (1009)
            if (Buffer.byteLength(message) > MAX_FRAME_BYTES) {
              const chunks = this.chunkMessage(message, MAX_FRAME_BYTES);
              for (const chunk of chunks) {
                await this.sendChunk(conn, chunk);
              }
            } else {
              await this.sendMessage(conn, message);
            }
          });
        } catch (error) {
          await this.handleDeliveryFailure(conn, notification, error);
        }
      })
    );
    
    // Track delivery metrics
    await this.recordDeliveryMetrics(userId, notification, results);
  }
  
  // Health checks, reconnect handling and metrics round out the class
}

Nothing here is clever. The idle timeout is why health checks exist, the connection ceiling is why reconnection is a first-class path through the code, and the frame quota is why serialization and sending are separate steps. The message cap is the one quota framing cannot soften, so a payload above 128 KB leaves the socket and travels as a reference the client fetches separately. Reconnection deserves its own attention: Netflix reports that devices on Pushy reconnect roughly every 30 minutes with some staggering, which turns reconnect from an exception into steady-state traffic that has to be smoothed deliberately.

Schedule and Scope Slip in the Published Data

The phase plan in an RFC usually looks like this: core infrastructure in weeks 1 to 4, advanced features in weeks 5 to 8, integration and optimisation in weeks 9 to 12. A schedule of that kind is a prediction, and predictions of this kind have a measured distribution that nobody needs to guess at.

Flyvbjerg and Budzier analysed 1,471 IT projects for Harvard Business Review and found an average cost overrun of 27%, with one project in six behaving as an outlier: cost overrun averaging 200% and schedule overrun of almost 70%. The average is survivable and the tail is not, and the tail is one in six rather than one in a hundred. PMI’s Pulse of the Profession 2018 measures the scope side of the same problem: 52% of projects completed in the previous 12 months experienced scope creep or uncontrolled changes to scope, up from 43% five years earlier, while only 52% finished within their initially scheduled times and 57% within their initial budgets. At the population level, finishing on schedule is a coin flip.

That reframes what a growing channel list means. Adding SMS, Slack, a webhook and eventually a voice channel after launch is the modal outcome for the population any given project belongs to. Review cannot predict which channels those will be, because nobody can. It can ask which integration surfaces are likely to still exist when they arrive.

Microsoft’s retirement of Office 365 connectors is the documented version of that risk. The Microsoft 365 developer blog announced in July 2024 that new connector creation would be blocked on 15 August 2024 and that all connectors in all clouds would stop working on 1 October 2024. The shutdown date then moved four times: through December 2025, then to 31 March 2026, then to 30 April 2026, then to a rollout between 18 and 22 May 2026, with Power Automate Workflows given as the migration path. Any Teams notification built on connectors had to be rewritten, and the schedule for that rewrite came from a vendor blog post that kept being edited.

The lag runs in the other direction as well. RFC 8030, Generic Event Delivery Using HTTP Push, was published as a Proposed Standard in December 2016. WebKit announced Web Push for Home Screen web apps in the iOS and iPadOS 16.4 beta on 16 February 2023, just over six years later, and only for web apps the user had added to the Home Screen. A design that treated a published standard as an available platform capability would have been waiting the entire time.

Authentication Contexts the RFC Did Not Model

A clean API design assumes one authentication pattern. A notification system attached to an existing product inherits every pattern that product has accumulated:

// RFC assumption
interface AuthContext {
  userId: string;
  token: string;
}

// What a notification path in an older product tends to meet
type AuthContext = 
  | { type: 'jwt'; userId: string; token: string; claims: JWTClaims }
  | { type: 'oauth2'; userId: string; accessToken: string; refreshToken: string; expiresAt: Date }
  | { type: 'legacy'; sessionId: string; userId?: string; cookieData: LegacyCookie }
  | { type: 'service_account'; serviceId: string; apiKey: string }
  | { type: 'anonymous'; temporaryId: string; ipAddress: string };

// Each variant carries its own rate limiting, security validation
// and audit requirements

None of this is architecturally interesting, which is exactly why a design review prices it at zero. It is also the kind of work that expands a schedule without producing anything a demo can show.

Team Scaling and Organizational Changes

An RFC sizes the work before anyone knows what the work is. A plan written as “2 developers for 12 weeks” tends to grow along predictable lines: someone joins to absorb production support, a contractor arrives for a quick win, a database specialist is pulled in once query latency becomes the blocker, and the engineer who was only meant to consult on deployment ends up owning it.

The expensive part of those changes is context transfer. People who arrive after a decision was made will reopen it, because the reasoning lives in a review thread they never read. Reorganizations revisit architecture questions the RFC treated as settled. Sizing an RFC in calendar weeks assumes a stable team, and that is usually the first assumption to break.

Monitoring Requirements Discovery

The monitoring section of an RFC covers standard metrics: delivery rate, response time, and error rate. Operating the system produces a longer list:

// RFC monitoring plan
const plannedMetrics = [
  'delivery_rate',
  'response_time', 
  'error_rate',
  'throughput'
];

// What production operation required
const productionMetrics = [
  // Basic metrics (from RFC)
  'delivery_rate_by_channel_by_priority_by_user_segment',
  'response_time_p50_p95_p99_p999',
  'error_rate_by_type_by_service_by_retry_count',
  
  // The metrics that decide incident outcomes
  'template_render_time_by_template_by_variables_count',
  'database_connection_pool_wait_time',
  'redis_operation_time_by_operation_type',
  'webhook_retry_backoff_effectiveness',
  'notification_staleness_at_delivery',
  'user_preference_cache_hit_rate',
  'deduplication_effectiveness_by_time_window',
  'rate_limit_rejection_by_reason',
  'circuit_breaker_state_transitions',
  'message_size_distribution_by_channel',
  'websocket_reconnection_storms',
  'push_token_invalidation_rate',
  'email_bounce_classification',
  'notification_feedback_loop_latency',
  'cost_per_notification_by_channel',
  'regulatory_compliance_audit_completeness',
  
  // The unusual ones, each of which exists because something once broke
  'mobile_app_version_vs_notification_compatibility',
  'timezone_calculation_accuracy',
  'emoji_rendering_failures_by_client',
  'notification_delivery_during_database_failover',
  'memory_leak_in_template_cache',
  'thundering_herd_detection'
];

Each additional metric addresses specific operational challenges that emerged during production use, highlighting the difference between design-time and runtime observability needs.

Technical Debt Accumulation Patterns

Technical debt is absent from every RFC, because at review time there is none. Two patterns show up in almost any notification system that lives long enough to need a second version.

Template System Complexity

More than one template engine is a normal end state. Transactional messages, marketing messages and in-app copy arrive from different teams with different tooling, and each engine brings its own escaping rules, helper set and failure modes. The renderer stops being a function and becomes a dispatch table:

// A hybrid template layer, which is where multi-team requirements land
class NotificationTemplateManager {
  private engines: Map<TemplateEngineId, TemplateEngine>;
  private engineByTemplate: Map<string, TemplateEngineId>;

  async render(templateId: string, data: unknown): Promise<string> {
    const engineId = this.engineByTemplate.get(templateId);
    if (!engineId) {
      throw new UnknownTemplateError(templateId);
    }
    const engine = this.engines.get(engineId);
    if (!engine) {
      throw new UnsupportedEngineError(engineId);
    }
    // Rendering is the cheap part. The mapping above is the debt:
    // it has to stay correct for every template anyone has ever created.
    return engine.render(templateId, data);
  }
}

Rendering is also where personalisation costs surface. A template that interpolates data from three services turns one notification into three calls, and that cost is paid per recipient rather than per template, so it stays invisible until volume arrives.

Schema Migration Challenges

Running two schemas at once is the only way to reshape one without downtime, and the shape of that work has been published. Stripe describes its online migrations in four phases: dual write to the existing and the new table to keep them in sync, move every read path to the new table, move every write path to the new table, then remove the data that relies on the outdated model. The first phase exists because of a constraint: the service could not be paused for the transition and had to keep operating at full load throughout.

The same post makes the scale legible with arithmetic that anyone can repeat. At one second per object, one hundred million objects take one hundred million seconds; divided by 86,400 seconds in a day that is roughly 1,157 days, or over three years, if the work runs sequentially. Backfills are batched and parallel for exactly that reason, which is why the migration runs as a repeated small batch that pages through the rows written before dual write started:

-- One backfill batch, running alongside the dual-write path
BEGIN;
  -- Phase 1 of 4: dual write already covers everything after the cutover,
  -- so this batch walks the history in front of it, oldest rows first
  WITH batch AS (
    SELECT 
      id,
      user_id,
      notification_type,
      -- Old vocabulary mapped onto the new one, limited to the types the map knows
      CASE 
        WHEN notification_type IN ('old_type_1', 'old_type_2') THEN 'new_type_1'
        WHEN notification_type LIKE 'legacy_%' THEN REPLACE(notification_type, 'legacy_', 'classic_')
      END as notification_type_v2,
      data,
      created_at
    FROM notification_events 
    WHERE created_at < (SELECT dual_write_started_at FROM migration_status
                        WHERE migration_name = 'notification_schema_v2')
      AND status != 'migrated'
      AND NOT EXISTS (
        SELECT 1 FROM notification_events_v2 
        WHERE notification_events_v2.id = notification_events.id
      )
      AND NOT EXISTS (
        SELECT 1 FROM migration_unmapped
        WHERE migration_unmapped.id = notification_events.id
      )
    ORDER BY created_at
    LIMIT 10000 -- one page; repeat until it selects nothing
    -- Each worker claims its own page, so several can run at once
    FOR UPDATE OF notification_events SKIP LOCKED
  ),
  quarantined AS (
    -- A type the CASE does not cover would land as NULL in a nullable column,
    -- so it is parked here instead of counted as migrated
    INSERT INTO migration_unmapped (id, notification_type, seen_at)
    SELECT id, notification_type, NOW() FROM batch
    WHERE notification_type_v2 IS NULL
    RETURNING id
  ),
  inserted AS (
    INSERT INTO notification_events_v2 (id, user_id, notification_type_v2, data, created_at)
    SELECT id, user_id, notification_type_v2, data, created_at FROM batch
    WHERE notification_type_v2 IS NOT NULL
    RETURNING id
  )
  -- Progress comes from the rows this batch actually wrote
  UPDATE migration_status 
  SET last_run = NOW(), 
      records_migrated = records_migrated + (SELECT COUNT(*) FROM inserted),
      records_unmapped = records_unmapped + (SELECT COUNT(*) FROM quarantined)
  WHERE migration_name = 'notification_schema_v2';
  
  -- Conflict checks, rollback path and metrics belong in the same transaction
COMMIT;

Two clauses in that batch are load-bearing. FOR UPDATE ... SKIP LOCKED is what lets a second worker claim a different page instead of competing for the same one, which is the difference between parallel progress and retry thrashing. The quarantine branch keeps rows whose legacy type is missing from the map out of the migrated count. Without it the mapping returns NULL, the insert still succeeds, and the migration reports progress it did not make.

The Metrics the RFC Did Not Name

Every success criterion in the hypothetical RFC was technical: 99.9% uptime, sub-100ms delivery, 10,000 notifications per second. Those targets are necessary and not sufficient. The measures that decide whether a notification system is worth operating sit outside the specification entirely:

  • Opt-out rate: a high delivery rate means little if users mute the channel because the timing is wrong.
  • Integration time for other teams: an API that needs hand-holding to adopt is not a clean API, whatever the design review concluded.
  • Operational burden: automation that still requires someone on call every week has only relocated the work.
  • Feature reachability: capabilities that only the authoring team can configure are not shipped in any useful sense.

None of these appear in a throughput target, and each of them determines whether the system survives its second year.

Key Implementation Insights

Several patterns emerge consistently across notification system implementations:

1. RFCs as Starting Hypotheses

Treating RFCs as initial hypotheses rather than fixed specifications enables better adaptation. Documents should evolve with implementation learning rather than remaining static reference points.

2. Planning for Emergent Requirements

A flat percentage buffer covers the average case and misses the tail, which is where schedules actually break. The more useful planning question is which parts of the plan survive an outlier and which parts collapse with it.

3. Evolution-Ready Design

Systems inevitably require migration, versioning, and compatibility features. Building these capabilities early reduces future technical debt and operational complexity.

4. Edge Cases as Core Requirements

Scenarios discussed during design reviews typically manifest in production. Planning for these cases during initial implementation proves more efficient than reactive fixes.

5. Organizational Context Integration

Technical design success depends on organizational alignment. Team changes, restructuring, and varying stakeholder priorities affect implementation more than architectural elegance.

6. Operational Observability Focus

Effective monitoring is shaped by what an incident will demand. Business impact, user experience, and operational detail carry more debugging value than the standard metrics a design document lists.

Bridging Design and Implementation

Several strategies help minimize the RFC-to-production gap:

Progressive Feature Development

Starting with well-executed core functionality enables better iteration than comprehensive initial implementation. Perfect email notifications provide a stronger foundation than basic multi-channel support.

Designing for Change

Systems designed for graceful evolution handle changing requirements better than those optimized for predicted scenarios. Flexibility often proves more valuable than initial perfection.

Developer Experience Investment

Easy integration and operation drive adoption more effectively than raw performance. API usability often determines system success more than technical specifications.

Documentation Evolution

Maintaining documentation as living artifacts rather than historical records improves team understanding. Sections for original design, current implementation, and learned insights provide comprehensive context.

Comprehensive Feedback Integration

Feedback loops across user experience, operational metrics, and developer workflow enable rapid iteration. Quick learning cycles accelerate problem identification and resolution.

When the Gap Is Signal and When It Is Drift

Treat the RFC as a hypothesis and the gap as data. A gap is acceptable when it traces to something the review could not have known: a channel the business needed later, an authentication scheme nobody had documented, a metric that only an incident could reveal. Record the change in the document and move on. The gap becomes a problem when nobody can explain why it exists, when the current shape is defended only by “it works”, or when the RFC and the running system disagree while new engineers still read the RFC first. At that point the document is what needs fixing.

One useful next step: pick a system where the design doc and the deployment have diverged, and sort each divergence into one of those two categories.

References

Related posts