S3 Event Notifications: SNS, SQS, or Lambda? // DevOps World

S3 Event Notifications let a bucket announce that something happened inside it — an object was created, deleted, or restored from Glacier — to one of three destinations: SNS (Simple Notification Service), SQS (Simple Queue Service), or directly to a Lambda function. All three are valid. They are not interchangeable, and picking the wrong one shows up later as either dropped events or a tangled mess of duplicated trigger logic.

The three destinations, plainly

SNS is a pub/sub messaging service — one event can fan out to many independent subscribers (multiple Lambda functions, an SQS queue, an email address, an HTTP endpoint) without the bucket needing to know who's listening. SQS is a queue — messages sit there until something explicitly polls and processes them, which is what gives you durability and retry control. Lambda-direct means S3 invokes your function immediately, with no intermediary at all.

✉

Continue reading

Unlock the full breakdown of when each option breaks down, and the architecture from a real upload pipeline.

No spam. One confirmation email via SES. Unsubscribe anytime.

Why I didn't just use Lambda-direct

S3 → Lambda direct invocation is the simplest option on paper: one event, one function, no extra services. It also has a real limitation that's easy to miss until you hit it — a single S3 bucket+prefix+suffix combination can only have one Lambda function subscribed to a given event type. If a second consumer later needs to react to the same upload event (say, a thumbnail generator gets added alongside the original processor), Lambda-direct forces you to either cram both responsibilities into one function or restructure the whole trigger.

SNS solves exactly this problem: the bucket publishes one event to one SNS topic, and any number of subscribers — Lambda functions, SQS queues, anything else — can attach to that topic independently, without the bucket's configuration ever changing again.

The pipeline I actually built

The event-driven serverless pipeline runs like this:

1. User uploads a file to S3 bucket (PutObject)
2. S3 publishes an ObjectCreated event to an SNS topic
3. SNS fans out to a Lambda function subscribed to that topic
4. Lambda processes the event (reads object metadata,
   builds a notification payload)
5. Lambda sends a formatted alert email via SES

The SNS layer in the middle is what makes this pipeline survive future requirements without rework. When a second Lambda function was later added to log upload metadata to DynamoDB for auditing, it subscribed to the exact same SNS topic — zero changes needed to the S3 bucket notification configuration or to the first Lambda function.

Where SQS fits in instead

SQS earns its place when you need two things SNS and Lambda-direct don't give you on their own: buffering against bursts and built-in retry with a dead-letter queue.

If 500 files get uploaded in the same minute (a bulk import, for example), invoking 500 Lambda executions simultaneously can hit concurrency limits or simply overwhelm a downstream system the Lambda talks to (a database, a third-party API). Routing the event through SQS first lets a Lambda function poll the queue at a controlled rate, processing messages as fast as the downstream system can actually handle — instead of all at once.

SQS also retains a message if processing fails, retrying it automatically, and can route it to a Dead Letter Queue after a configured number of failed attempts — so a single bad file doesn't silently disappear, it lands somewhere you can inspect it.

The decision rule, condensed

One known consumer, no burst risk, no fan-out ever expected: Lambda-direct. Simplest path, fewest moving parts.
Multiple consumers now or expected later, no special need for buffering: SNS → Lambda(s). This is the fan-out problem solved cleanly.
Bursty uploads, need for retry/backoff, or a downstream system with rate limits: SQS, either standalone or behind SNS (SNS → SQS → Lambda — "fan-out with buffering" — is a common combined pattern).

Where each one actually breaks down

Lambda-direct breaks down the moment a second use case shows up for the same event — you end up either duplicating bucket notification rules in awkward ways or cramming unrelated logic into one function.

SNS alone (without SQS behind it) breaks down under burst load if your subscriber is a system with hard concurrency or rate limits — SNS will deliver every message immediately and your Lambda will scale to match, which can overwhelm whatever the Lambda calls next (an external API, a database connection pool).

SQS alone breaks down if you genuinely need true fan-out to several independent consumers reacting to the same event for different reasons — a single SQS queue is consumed by one logical set of workers, not cleanly broadcast to many unrelated downstream systems the way SNS is.

The emoji-formatted alert, for context

The final email this pipeline sends through SES is intentionally readable at a glance — formatted with status emoji and the object key, bucket, and upload timestamp, rather than a raw JSON dump. The Lambda function's only job is to take the SNS event payload, extract those three fields, and hand SES a clean subject and body. Keeping that function single-purpose is what made it trivial to add the second DynamoDB-logging consumer later without touching it at all.

The right destination isn't about which is "better" — SNS, SQS, and Lambda-direct solve three different failure modes. Pick based on whether you need fan-out, buffering, or neither, and the choice usually makes itself.