Kafka vs SQS vs EventBridge: Choosing Event-Driven Tools

Choosing an event-driven tool is less about hype and more about matching three shapes: a queue, a fan-out topic, or a router. Start with the managed primitive your cloud already gives you for that shape. Move to Kafka only when you need event replay, ordering across a partitioned key space, or throughput a managed queue quota cannot cover.

The rest of the decision comes down to delivery guarantees, message size limits, and what happens to a message nobody can process.

Message Patterns: The Foundation

Each pattern carries different tools and trade-offs:

1-to-1 (Queue Pattern)

• Message consumed by single consumer
• Use cases: Task processing, work distribution
• Tools: SQS, Azure Service Bus Queues, Cloud Tasks

1-to-Many (Topic/Fan-out Pattern)

• Message delivered to multiple subscribers
• Use cases: Event broadcasting, notifications
• Tools: SNS, Azure Service Bus Topics, Cloud Pub/Sub

Many-to-Many (Event Mesh)

• Complex routing between multiple producers/consumers
• Use cases: Microservices communication
• Tools: EventBridge, Azure Event Grid, Eventarc

The Complete Tool Landscape

Simple Queue Services

AWS SQS (Simple Queue Service)

What it excels at: Dead-simple queue operations, serverless integration, automatic scaling

Receive and DLQ config:

// Long polling keeps empty receives cheap
const params = {
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123/my-queue',
  ReceiveMessageWaitTimeSeconds: 20,  // Long polling
  MaxNumberOfMessages: 10,
  VisibilityTimeout: 30,  // Processing window
  MessageAttributeNames: ['All']
};

// The DLQ itself only needs long retention
const dlqParams = {
  QueueName: 'my-queue-dlq',
  Attributes: {
    MessageRetentionPeriod: '1209600'  // 14 days, the SQS maximum
  }
};

// The redrive policy belongs on the SOURCE queue, not on the DLQ
const sourceQueueParams = {
  QueueName: 'my-queue',
  Attributes: {
    RedrivePolicy: JSON.stringify({
      deadLetterTargetArn: dlqArn,  // ARN of my-queue-dlq
      maxReceiveCount: 3  // Deliver 3 times, then move to the DLQ
    })
  }
};

Delivery guarantees:

• Standard Queue: At-least-once (possible duplicates)
• FIFO Queue: Exactly-once processing
• Message ordering: FIFO only
• Max message size: 1MB (upgraded from 256KB in Aug 2025)

When SQS shines:

• Decoupling microservices
• Batch job processing
• Serverless architectures (Lambda triggers)
• Simple task queues

Azure Service Bus Queues

Azure’s equivalent to SQS with enterprise features:

// Service Bus with sessions and DLQ handling
var client = new ServiceBusClient(connectionString);
var processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions
{
    MaxConcurrentCalls = 10,
    AutoCompleteMessages = false,
    MaxAutoLockRenewalDuration = TimeSpan.FromMinutes(5),
    SubQueue = SubQueue.DeadLetter  // Access DLQ
});

// Message with duplicate detection
var message = new ServiceBusMessage(body)
{
    MessageId = Guid.NewGuid().ToString(),  // For deduplication
    SessionId = sessionId,  // For ordered processing
    TimeToLive = TimeSpan.FromMinutes(5)
};

Key differences from SQS:

• Built-in sessions for ordered processing
• Duplicate detection (configurable window)
• Scheduled messages
• Message size: 256KB (standard), 100MB (premium)

Google Cloud Tasks

GCP’s task queue with HTTP target integration:

import { CloudTasksClient } from '@google-cloud/tasks';

const client = new CloudTasksClient();
const parent = client.queuePath(project, location, queue);

const task = {
    httpRequest: {
        httpMethod: 'POST',
        url: 'https://example.com/process',
        headers: { 'Content-Type': 'application/json' },
        body: Buffer.from(JSON.stringify(payload))
    },
    scheduleTime: { seconds: Math.floor(timestamp / 1000) } // Delayed execution
};

const response = await client.createTask({ parent, task });

Pub/Sub Systems

AWS SNS (Simple Notification Service)

1-to-many message distribution:

// SNS with filter policies for smart routing
const publishParams = {
  TopicArn: 'arn:aws:sns:us-east-1:123:my-topic',
  Message: JSON.stringify(event),
  MessageAttributes: {
    eventType: { DataType: 'String', StringValue: 'ORDER_CREATED' },
    priority: { DataType: 'Number', StringValue: '1' }
  }
};

// Subscription with filter
const subscriptionPolicy = {
  eventType: ['ORDER_CREATED', 'ORDER_UPDATED'],
  priority: [{ numeric: ['>', 0] }]
};

SNS + SQS Pattern (Fanout):

Delivery guarantees:

• At-least-once delivery
• No message ordering
• Retry with exponential backoff
• DLQ support for failed deliveries

Azure Service Bus Topics

More sophisticated than SNS:

// Topic with multiple subscriptions and filters
var adminClient = new ServiceBusAdministrationClient(connectionString);

// Create subscription with SQL filter
await adminClient.CreateSubscriptionAsync(
    new CreateSubscriptionOptions(topicName, subscriptionName),
    new CreateRuleOptions("OrderFilter",
        new SqlRuleFilter("EventType = 'OrderCreated' AND Priority > 5"))
);

Advanced features:

• SQL-like filtering rules
• Message sessions for ordering
• Duplicate detection
• Dead-lettering with reason tracking

Google Cloud Pub/Sub

Global message distribution:

import { PubSub } from '@google-cloud/pubsub';

const pubsub = new PubSub();
// Ordering keys are ignored unless the publisher enables them
const topic = pubsub.topic(topicId, { enableMessageOrdering: true });

// Publishing with ordering key
const messageId = await topic.publishMessage({
    data: Buffer.from(data),
    orderingKey: 'user-123', // Ensures order per key
    attributes: {
        event_type: 'user_updated',
        version: '2'
    }
});

Event Routing Services

AWS EventBridge

Rule-based event routing:

// EventBridge with content-based routing
const rule = {
  Name: 'OrderProcessingRule',
  EventPattern: JSON.stringify({
    source: ['order.service'],
    'detail-type': ['Order Created'],
    detail: {
      amount: [{ numeric: ['>', 100] }],
      country: ['US', 'UK', 'DE']
    }
  }),
  Targets: [
    {
      Arn: lambdaArn,
      RetryPolicy: {
        MaximumRetryAttempts: 2,
        MaximumEventAgeInSeconds: 3600
      },
      DeadLetterConfig: {
        Arn: dlqArn
      }
    }
  ]
};

Cross-account event sharing:

// PutPermission runs on the receiving account's bus.
// Name one account, or use '*' plus the organization condition.
const eventBusPolicy = {
  EventBusName: 'default',
  StatementId: 'AllowOrgAccess',
  Action: 'events:PutEvents',
  Principal: '*',
  Condition: {
    Type: 'StringEquals',
    Key: 'aws:PrincipalOrgID',
    Value: 'o-1234567890'  // The only condition key PutPermission accepts
  }
};

// Filtering by detail-type happens in the receiving bus rules, not here

Azure Event Grid

Azure’s equivalent with powerful filtering:

{
  "filter": {
    "includedEventTypes": ["Microsoft.Storage.BlobCreated"],
    "subjectBeginsWith": "/blobServices/default/containers/images/",
    "advancedFilters": [
      {
        "operatorType": "NumberGreaterThan",
        "key": "data.contentLength",
        "value": 1048576
      }
    ]
  }
}

Google Cloud Eventarc

GCP’s unified eventing:

# Eventarc trigger configuration
apiVersion: eventarc.cnrm.cloud.google.com/v1beta1
kind: EventarcTrigger
metadata:
  name: storage-trigger
spec:
  location: us-central1
  matchingCriteria:
  - attribute: type
    value: google.cloud.storage.object.v1.finalized
  - attribute: bucket
    value: my-bucket
  destination:
    cloudRunService:
      name: process-image
      region: us-central1

Stream Processing Platforms

Apache Kafka

Open-source event streaming with configurable delivery semantics:

// Kafka Streams for real-time processing
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-processor");
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once_v2");
props.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3);

KStream<String, Order> orders = builder.stream("orders");
KTable<String, Long> orderCounts = orders
    .filter((k, v) -> v.getAmount() > 100)
    .groupByKey()
    .count(Materialized.as("order-counts-store"));

// DLQ handling with Kafka Streams
orders.foreach((key, value) -> {
    try {
        processOrder(value);
    } catch (Exception e) {
        producer.send(new ProducerRecord<>("orders-dlq", key, value));
    }
});

Kafka delivery semantics:

• At-most-once: Fire and forget (acks=0)
• At-least-once: Default (acks=1 or all)
• Exactly-once: With transactions (enable.idempotence=true)

Cloud Streaming Equivalents

AWS Kinesis Data Streams

import {
  KinesisClient,
  RegisterStreamConsumerCommand
} from '@aws-sdk/client-kinesis';

const kinesis = new KinesisClient({ region: 'us-east-1' });

// Enhanced fan-out gives each consumer its own read throughput
const consumer = await kinesis.send(new RegisterStreamConsumerCommand({
  StreamARN: streamArn,
  ConsumerName: 'low-latency-consumer'
}));

Azure Event Hubs

// Event Hubs with Kafka protocol
var config = new ConsumerConfig
{
    BootstrapServers = "namespace.servicebus.windows.net:9093",
    SecurityProtocol = SecurityProtocol.SaslSsl,
    SaslMechanism = SaslMechanism.Plain,
    GroupId = "consumer-group"
};

// Capture to Data Lake for long-term storage
var captureDescription = new CaptureDescription
{
    Enabled = true,
    IntervalInSeconds = 300,
    SizeLimitInBytes = 314572800,
    Destination = new Destination
    {
        StorageAccountResourceId = "/subscriptions/.../storageAccounts/...",
        BlobContainer = "capture"
    }
};

Google Cloud Dataflow

# Dataflow runs Apache Beam pipelines; Beam is the SDK you write against
import json
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

options = PipelineOptions(streaming=True, runner='DataflowRunner')

with beam.Pipeline(options=options) as pipeline:
    (pipeline
     | 'Read' >> beam.io.ReadFromPubSub(topic=topic)
     | 'Parse' >> beam.Map(lambda payload: json.loads(payload.decode('utf-8')))
     | 'Window' >> beam.WindowInto(beam.window.FixedWindows(60))
     | 'Filter' >> beam.Filter(lambda item: item['amount'] > 100)
     | 'Write' >> beam.io.WriteToBigQuery(table_spec))

Dead Letter Queue (DLQ) Essentials

Dead Letter Queues are critical for production resilience. They handle messages that can’t be processed successfully after retries.

Key DLQ concepts:

• Safety net for failed messages
• Prevents poison pill scenarios
• Enables error analysis and recovery
• Essential monitoring beyond queue depth

The wiring is the pair shown in the SQS section: long retention on the DLQ, maxReceiveCount on the source queue. Tune that count against your retry budget. Three deliveries absorb transient failures without keeping a poison message in the loop for minutes.

Deep Dive: For comprehensive DLQ strategies, monitoring patterns, circuit breakers, ML-based recovery, and production lessons, see our detailed guide: Dead Letter Queue Production Strategies

Edge and Hybrid Deployments

Edge Computing Considerations

Event-driven systems at the edge have unique constraints:

// Edge-optimized event processing
class EdgeEventProcessor {
  private localQueue: Queue[] = [];
  private cloudBuffer: Message[] = [];

  async processEvent(event: Event) {
    // Process locally first
    const processed = await this.localProcess(event);

    // Batch for cloud sync
    if (this.shouldSyncToCloud(processed)) {
      this.cloudBuffer.push(processed);

      if (this.cloudBuffer.length >= 100 ||
          Date.now() - this.lastSync > 60000) {
        await this.syncToCloud();
      }
    }
  }

  private async syncToCloud() {
    try {
      // Compress and batch send
      const compressed = this.compress(this.cloudBuffer);
      await this.cloudClient.sendBatch(compressed);
      this.cloudBuffer = [];
      this.lastSync = Date.now();
    } catch (error) {
      // Store locally if cloud unreachable
      await this.localStorage.store(this.cloudBuffer);
    }
  }
}

Cloudflare Workers with Queues

// Cloudflare Workers Queue Handler
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      try {
        // Process at edge
        const result = await processMessage(message.body);

        // Store in Durable Objects or KV
        await env.KV.put(
          `processed:${message.id}`,
          JSON.stringify(result),
          { expirationTtl: 3600 }
        );

        message.ack();
      } catch (error) {
        // Retry with backoff
        message.retry({ delaySeconds: 30 });
      }
    }
  }
};

AWS IoT Core for Edge Events

// Greengrass V2 component talking to IoT Core over local IPC
import * as greengrasscoreipc from 'aws-iot-device-sdk-v2/dist/greengrasscoreipc';
import * as model from 'aws-iot-device-sdk-v2/dist/greengrasscoreipc/model';

class EdgeIoTProcessor {
    private ipcClient = greengrasscoreipc.createClient();

    constructor(private deviceId: string) {}

    async connect(): Promise<void> {
        await this.ipcClient.connect();
    }

    async publishEdgeEvent(event: unknown): Promise<void> {
        // Reduce on the device first; the uplink may be offline
        const processed = this.processLocally(event);

        const request: model.PublishToIoTCoreRequest = {
            topicName: `edge/${this.deviceId}/events`,
            qos: model.QOS.AT_LEAST_ONCE,
            payload: Buffer.from(JSON.stringify(processed))
        };

        await this.ipcClient.publishToIoTCore(request);
    }

    private processLocally(event: unknown): unknown {
        // Filter, enrich, or aggregate before spending uplink bandwidth
        return event;
    }
}

Cross-Cloud Equivalents

Service Mapping Table

AWS	Azure	GCP	Use Case
SQS	Service Bus Queues	Cloud Tasks	Simple queuing
SNS	Service Bus Topics	Cloud Pub/Sub	Pub/Sub messaging
EventBridge	Event Grid	Eventarc	Event routing
Kinesis	Event Hubs	Pub/Sub + Dataflow	Stream processing
Lambda + SQS	Functions + Service Bus	Cloud Run + Pub/Sub	Serverless events
DynamoDB Streams	Cosmos DB Change Feed	Firestore Triggers	Database events
Step Functions	Logic Apps	Workflows	Event orchestration
MSK (Kafka)	Event Hubs (Kafka mode)	Confluent Cloud	Kafka-compatible

Multi-Cloud Event Bridge Pattern

// Abstract multi-cloud event interface
interface CloudEventAdapter {
  publish(event: CloudEvent): Promise<void>;
  subscribe(handler: EventHandler): Promise<void>;
}

class MultiCloudEventBridge {
  private adapters: Map<string, CloudEventAdapter> = new Map();

  constructor() {
    this.adapters.set('aws', new AWSEventBridgeAdapter());
    this.adapters.set('azure', new AzureEventGridAdapter());
    this.adapters.set('gcp', new GCPEventarcAdapter());
  }

  async publishToAll(event: CloudEvent) {
    const promises = Array.from(this.adapters.values())
      .map(adapter => adapter.publish(event));

    const results = await Promise.allSettled(promises);

    // Handle partial failures
    const failures = results.filter(r => r.status === 'rejected');
    if (failures.length > 0) {
      await this.handleFailures(failures, event);
    }
  }
}

Capability Comparison Matrix

The throughput column lists the documented quota or the axis a system scales along, not a benchmark result. Managed quotas are region-dependent and most of them can be raised on request.

Tool	Throughput ceiling	Message size	Ordering	Delivery guarantee	DLQ support
SQS Standard	Effectively unlimited	1MB	No	At-least-once	Yes
SQS FIFO	300 TPS per partition per API action, 3K/sec batched, higher in high-throughput mode	1MB	Yes	Exactly-once processing	Yes
SNS	Region-dependent publish quota	256KB	No	At-least-once	Yes
Kafka	Scales with partitions and brokers	1MB default	Per partition	Configurable	Manual
RabbitMQ	Scales with nodes and queue type	Configurable (max_message_size)	Optional	At-least-once	Yes
EventBridge	Region-dependent PutEvents quota	256KB	No	At-least-once	Yes
Kinesis	1MB/sec per shard	1MB	Per shard	At-least-once	Manual
Azure Service Bus	Scales with messaging units	256KB standard, 100MB premium	Yes	At-least-once	Yes
Cloud Pub/Sub	Region-dependent publish quota	10MB	Per ordering key	At-least-once	Yes
Redis Streams	Scales with instance size	512MB per field	Yes	At-least-once	Manual

Decision Framework

Quick Decision Tree

When to Use What

Use Simple Queues (SQS/Service Bus) when:

• Decoupling services
• Work distribution
• Simple retry requirements
• Serverless processing

Use Pub/Sub (SNS/Topics) when:

• Broadcasting events
• Fan-out patterns
• Multiple consumers
• Notification systems

Use Event Routers (EventBridge/EventGrid) when:

• Complex routing rules
• Multi-service orchestration
• SaaS integrations
• Event-driven automation

Use Streaming (Kafka/Kinesis) when:

• Real-time analytics
• Event sourcing
• Sustained volume beyond a managed queue quota
• Event replay needed

Common Pitfalls and Solutions

Pitfall 1: Message Size Limits

// Solution: Claim check pattern
class LargeMessageHandler {
  async send(largePayload: unknown) {
    const body = JSON.stringify(largePayload);

    // 256KB is the SNS and EventBridge ceiling; SQS now allows 1MB
    if (body.length > 256_000) {
      const s3Key = await this.uploadToS3(largePayload);

      // Send the reference, not the payload
      return this.queue.send({
        type: 'large_message',
        s3Key,
        size: body.length
      });
    }

    return this.queue.send(largePayload);
  }
}

Pitfall 2: Poison Messages

// Solution: Poison message detection
class PoisonMessageDetector {
  private messageAttempts = new Map<string, number>();

  async process(message: Message) {
    const messageId = message.id;
    const attempts = this.messageAttempts.get(messageId) || 0;

    if (attempts >= 3) {
      // Identified as poison message
      await this.quarantine(message);
      return;
    }

    try {
      await this.processMessage(message);
      this.messageAttempts.delete(messageId);
    } catch (error) {
      this.messageAttempts.set(messageId, attempts + 1);

      // Check if specific error pattern
      if (this.isPoisonPattern(error)) {
        await this.quarantine(message);
      } else {
        throw error; // Retry
      }
    }
  }
}

Pitfall 3: Ordering Guarantees

// Solution: Partition key strategy
class OrderedEventProcessor {
  async publishOrdered(events: Event[]) {
    // Group by entity ID for ordering
    const grouped = this.groupBy(events, e => e.entityId);

    for (const [entityId, entityEvents] of grouped) {
      // Sort by timestamp
      entityEvents.sort((a, b) => a.timestamp - b.timestamp);

      // Send with same partition key
      for (const event of entityEvents) {
        await this.kafka.send({
          topic: 'events',
          key: entityId,  // Ensures ordering
          value: event
        });
      }
    }
  }
}

Monitoring and Observability

Key Metrics to Track

// Comprehensive metrics collection
class EventMetrics {
  private metrics = {
    messagesPublished: new Counter('messages_published_total'),
    messagesConsumed: new Counter('messages_consumed_total'),
    messagesFailed: new Counter('messages_failed_total'),
    processingDuration: new Histogram('message_processing_duration_seconds'),
    queueDepth: new Gauge('queue_depth'),
    consumerLag: new Gauge('consumer_lag'),
    dlqDepth: new Gauge('dlq_depth')
  };

  async recordProcessing(message: Message, processor: Function) {
    const timer = this.metrics.processingDuration.startTimer();

    try {
      const result = await processor(message);
      this.metrics.messagesConsumed.inc();
      return result;
    } catch (error) {
      this.metrics.messagesFailed.inc({
        error_type: error.constructor.name,
        queue: message.source
      });
      throw error;
    } finally {
      timer();
    }
  }
}

Conclusion

The managed default holds for most systems: let the message pattern pick the queue, topic, or router your cloud already runs, and spend the saved effort on delivery semantics instead of brokers. Override it when you need replay of past events, ordering across a partitioned key space, or sustained volume past a managed quota, since that is where Kafka or Kinesis earns its operational cost. Whichever you land on, wire the DLQ and a consumer-lag alarm before the first production message; the failure modes arrive long before the scale ones do.

---

Related Deep Dives:

• Dead Letter Queue Production Strategies - Comprehensive DLQ patterns and monitoring

References

• Apache Kafka Introduction - Official Apache Kafka documentation covering the core concepts of distributed event streaming, topics, partitions, and consumer groups.
• Amazon SQS, Amazon SNS, or Amazon EventBridge? Decision Guide - AWS official comparison guide for choosing between the three messaging services based on use case.
• What Is Amazon EventBridge? - Official EventBridge documentation covering event buses, rules, targets, and integration patterns.
• Amazon Kinesis Data Streams: Introduction - Official Kinesis documentation on real-time data streaming, shards, and consumer applications.
• Event-Driven Architectures: Serverless Applications Lens - AWS Well-Architected guidance on designing reliable event-driven serverless systems.
• Apache Kafka Design - In-depth explanation of Kafka’s log-based architecture, delivery guarantees, and partition ordering model.
• Amazon SQS FIFO Queues - Official SQS documentation on exactly-once processing and strict message ordering guarantees.
• Amazon SQS Message Quotas - The quota table behind the throughput, message size, and retention figures used in the comparison matrix.