Case Study:

AWS re:invent 2023 featured a lab session on building out serverless architecture which utilized SNS, SQS, and Lambda. I found this lab particularly helpful because it helped me design a solution for a problem where data was being throttled through a single chokepoint. In this case, a large batch of data was being sent to a python script in Glue for processing. The Serverless Architecture lab I attended at re:invent inspired the idea to decouple the existing pipeline and process one datapoint at a time using SNS and Lambda. The alternative to a fan-out solution might be to convert a linearly executed python script to spark where a large dataset can be distributed and executed across a cluster. Neither solution is more correct than the other, however, converting an already working python process to use fan-out architecture removes the headache of converting all of the code to spark.

Fan out architecture is particularly useful when trying to de-throttle linear code that might be taking a long time to run. A specific use-case for this type of solution might be a scenario where we have to process a large dataset row by row and then append the processed output back together as a single dataset in S3. Specifically, the use-case might require the use of a for-loop in a data-ingesting Lambda to send an SNS message containing a row of data for processing in a subsequent Lambda function. The subsequent lambda function is subscribed to the SNS topic and is invoked each time a message is pushed to the SNS topic. Each time the processing Lambda is invoked, it will append the processed data to a single dataset in S3.

Steps:

1. Create an SNS topic. Choose Standard if processing order does not matter. Choose FIFO if ordering does matter

2. Create a Lambda function that will serve as the data-ingesting Lambda. Use boto3 to push your message to your topic. Note: DO NOT subscribe this particular Lambda to the SNS topic.

3. Create a processing Lambda function that will perform any transformations on your dataset that you require. Subscribe this Lambda function to the SNS topic that you created. Choose AWS Lambda as the Protocol and paste the Lambda ARN for the Endpoint.

4. Create an SNS EventBridge for the processing Lambda function. In the Lambda function, choose ‘Add Trigger’ and set up the trigger accordingly so the SNS topic can invoke the Lambda function.

Proposed Architecture:

CAUTION: When designing this type of architecture it is important to ingest and process the data separately. A Lambda function which is subscribed to an SNS topic and invokes itself via EventBridge will create an endless loop of Lambda invocations.

Fan-out Architecture Overview:

In modern cloud-based architectures, scalability and responsiveness are key considerations. One way to achieve these goals is by implementing a fan-out architecture, which allows you to efficiently distribute and process messages across multiple consumers. Amazon Web Services (AWS) provides a powerful combination of services – Simple Notification Service (SNS), Simple Queue Service (SQS), and AWS Lambda – to build a scalable and decoupled fan-out system. In this article, we’ll explore how to create a fan-out architecture using these AWS services.

Understanding the Components:

1. Simple Notification Service (SNS):

SNS is a fully managed pub/sub (publish/subscribe) messaging service that enables the distribution of messages to a variety of endpoints, such as AWS Lambda, and SQS. It acts as a central hub for publishing messages.

2. Simple Queue Service (SQS):

SQS is a fully managed message queuing service that allows you to decouple the components of a cloud application. Messages sent to SQS are stored in a queue and can be processed asynchronously by multiple consumers.

3. AWS Lambda:

Lambda is a serverless compute service that allows you to run code without provisioning or managing servers. It can be triggered by various AWS services, including SNS and SQS, making it an excellent choice for processing messages in a fan-out architecture.

Steps to Create a Fan-Out Architecture:

1. Set Up SNS Topic:

— Navigate to the AWS Management Console and open the SNS service.
– Create a new SNS topic and configure its settings.
– Subscribe relevant Lambda functions and SQS queues to the topic.

2. Create SQS Queues:

— In the SQS service, create multiple queues to represent different processing tasks.
– Configure the queues with appropriate settings, such as visibility timeout and message retention period.

3. Subscribe SQS Queues to SNS Topic:

— Connect each SQS queue to the SNS topic by subscribing them.
– This establishes a relationship where messages published to the SNS topic are forwarded to all subscribed SQS queues.

4. Implement Lambda Functions:

— Write Lambda functions to process messages from the SQS queues.
– Each Lambda function should be responsible for a specific task or set of tasks.

5. Configure Event Sources for Lambda:

— Set up event sources for each Lambda function, connecting them to the respective SQS queues.
– This ensures that Lambda functions are triggered when new messages arrive in the associated queues.

6. Test and Monitor:

— Publish messages to the SNS topic and observe the fan-out effect.
– Monitor the performance of the system using AWS CloudWatch and other relevant AWS monitoring tools.

Benefits of Fan-Out Architecture:

1. Scalability:

Fan-out architectures enable horizontal scaling by allowing you to add more consumers as needed. This ensures that the system can handle increased workloads efficiently.

2. Decoupling:

Components in a fan-out architecture are decoupled, meaning changes to one component do not directly impact others. This promotes flexibility and makes it easier to evolve the system over time.

3. Asynchronous Processing:

As messages are queued and processed asynchronously, the system can handle bursts of activity without affecting overall performance.

Conclusion:

Implementing a fan-out architecture using SNS, SQS, and Lambda on AWS provides a robust solution for building scalable and decoupled systems. By leveraging the strengths of these services, you can create a responsive and flexible architecture that efficiently processes messages and adapts to varying workloads. As you design and implement your fan-out architecture, keep in mind the specific requirements of your application and fine-tune the configuration to optimize performance.