EventBridge buses, as per design, are transient carriers for events, making them a black box which event-driven systems often are. This makes it difficult to track what event came through and where it went. This can be partially mitigated by using archives, wherein you can configure a rule on your event bus to automatically save all incoming events to an archive. The replay feature then lets you replay all stored events on the event bus.

LocalStack allows you to emulate and test this archive and replay workflow on your local machine. In this tutorial, you'll create a Lambda function locally using LocalStack & AWS CLI to serve as the target for an EventBridge rule. You will then create an archive, and once events are stored in the archive, you will replay them.

Prerequisites

• LocalStack CLI

• Docker

• AWS CLI with awslocal wrapper

• Node.js & zip

• LocalStack Web Application account (optional)

Start your LocalStack container

Launch the LocalStack container on your local machine using the specified command:

PROVIDER_OVERRIDE_EVENTS=v2 localstack start

The configuration variable PROVIDER_OVERRIDE_EVENTS=v2 enables you to set the EventBridge v2 provider, which enhances emulation for EventBridge features like buses, rules, patterns, and targets, and supports archive & replay.

Once initiated, you'll receive a confirmation output indicating that the LocalStack container is up and running.

     __                     _______ __             __
    / /   ____  _________ _/ / ___// /_____ ______/ /__
   / /   / __ \/ ___/ __ `/ /\__ \/ __/ __ `/ ___/ //_/
  / /___/ /_/ / /__/ /_/ / /___/ / /_/ /_/ / /__/ ,<
 /_____/\____/\___/\__,_/_//____/\__/\__,_/\___/_/|_|

 💻 LocalStack CLI 3.6.0
 👤 Profile: default

[20:19:34] starting LocalStack in    localstack.py:503
           Docker mode 🐳
...
LocalStack version: 3.6.1.dev20240725091954
LocalStack build date: 2024-07-26
LocalStack build git hash: d536652

Create a Lambda Function

Since the event bus is transient, you need a target to verify what events pass through your system. You can create a Lambda function to log the events.

To begin, create a new file named index.js. Add the following code to log events:

'use strict';

exports.handler = (event, context, callback) => {
    console.log('LogScheduledEvent');
    console.log('Received event:', JSON.stringify(event, null, 2));
    callback(null, 'Finished');
};

This code logs details of incoming events and concludes by sending a Finished message via the callback.

Next, package this file into a ZIP archive with the command:

zip function.zip index.js

Deploy the Lambda function using the following command:

awslocal lambda create-function \
    --function-name LogScheduledEvent \
    --runtime nodejs18.x \
    --role arn:aws:iam::000000000000:role/lambda-ex \
    --handler index.handler \
    --zip-file fileb://function.zip

The command outputs details about the function, including its name, pending state, and other configuration data:

{
    "FunctionName": "LogScheduledEvent",
    ...
    "State": "Pending",
    "StateReason": "The function is being created.",
    "StateReasonCode": "Creating",
    ...
    "EphemeralStorage": {
        "Size": 512
    },
    ...
    "RuntimeVersionConfig": {
        "RuntimeVersionArn": "arn:aws:lambda:us-east-1::runtime:8eeff65f6809a3ce81507fe733fe09b835899b99481ba22fd7
5b5a7338290ec1"
    }
}

Create an EventBridge Archive

You can set up an EventBridge archive to store past events. This archive can hold events indefinitely or for a specified number of days. When configuring the archive, you can either capture all events from an event bus or selectively store only targeted events. Additionally, you can apply filters or rules to store events that meet specific criteria.

To start, create a custom event bus named test-event-bus to receive events, using this command:

awslocal events create-event-bus --name test-event-bus

The output would confirm the creation of the event bus:

{
    "EventBusArn": "arn:aws:events:us-east-1:000000000000:event-bus/test-event-bus"
}

Create an EventBridge Rule

You can create a rule named ARTestRule to archive events that are sent to the event bus. EventBridge rules allow you to route specific events according to your needs. This rule filters events based on defined patterns and sends matching events to attached target services.

To create the rule, execute the following command:

awslocal events put-rule \
    --name ARTestRule \
    --event-bus-name test-event-bus \
    --event-pattern '{
      "detail-type": ["customerCreated"]
    }'

You should see the following output:

{
    "RuleArn": "arn:aws:events:us-east-1:000000000000:rule/test-event-bus/ARTestRule"
}

Create an EventBridge Target

You can then choose a target. Targets are entities that consume your events. When an event triggers a rule, all targets linked to that rule are activated.

In this case, you can use the Lambda function as the target by retrieving the ARN of the Lambda function:

awslocal lambda get-function \
    --function-name LogScheduledEvent \
    --query 'Configuration.FunctionArn' \
    --output text

The Lambda ARN output will be:

arn:aws:lambda:us-east-1:000000000000:function:LogScheduledEvent

Finally, you can associate the Lambda function as a target to the ARTestRule using this command:

awslocal events put-targets \
    --rule ARTestRule \
    --event-bus-name test-event-bus \
    --targets '[
      {
        "Id": "1",
        "Arn": "arn:aws:lambda:us-east-1:000000000000:function:LogScheduledEvent"
      }
    ]'

You should see the following output:

{
    "FailedEntryCount": 0,
    "FailedEntries": []
}

Create an EventBridge Archive

Next, establish an archive named ArchiveTest using the ARN from your event bus:

awslocal events create-archive \
    --archive-name ArchiveTest \
    --event-source-arn arn:aws:events:us-east-1:000000000000:event-bus/test-event-bus

The output would show the archive's ARN, its enabled state, and creation time:

{
    "ArchiveArn": "arn:aws:events:us-east-1:000000000000:archive/ArchiveTest",
    "State": "ENABLED",
    "CreationTime": "2024-08-03T19:03:00.917596+05:30"
}

Send a test event

With the archive and rule configured, you can send a test event to verify that the system is functioning as expected. Execute the following command to send a test event:

awslocal events put-events --entries '[
  {
    "Source": "TestEvent",
    "DetailType": "customerCreated",
    "Detail": "{}",
    "EventBusName": "test-event-bus"
  }
]'

In this command:

• The event source is identified as TestEvent.

• The event is sent to the test-event-bus.

• The event detail is empty ({}) and the type is customerCreated.

The output should confirm successful event submission:

{
    "FailedEntryCount": 0,
    "Entries": [
        {
            "EventId": "ee947775-0dcd-405c-8ba9-e100dcf244fa"
        }
    ]
}

Replay the event

Once test events are stored in the archive, you can replay them. To replay events, you need to specify both the source and the event bus they should be replayed into. Events must be replayed into the same event bus from which they were collected. You can also define a specific time window during which you want the events to be replayed into the bus.

Run the following command to start the replay:

awslocal events start-replay \
    --replay-name ReplayTest \
    --event-source-arn arn:aws:events:us-east-1:000000000000:archive/ArchiveTest \
    --event-start-time 2024-08-01 --event-end-time 2024-08-06 \
    --destination '{"Arn":"arn:aws:events:us-east-1:000000000000:event-bus/test-event-bus"}'

In the above command:

• The replay name is specified as ReplayTest.

• The event source ARN is the archive's ARN.

• The event start and end times are chosen based on the article's date but can be customized.

• The destination is the ARN for the test-event-bus.

The output should look like this:

{
    "ReplayArn": "arn:aws:events:us-east-1:000000000000:replay/ReplayTest",
    "State": "COMPLETED",
    "ReplayStartTime": "2024-08-06T12:58:01.405559+05:30"
}

To further inspect the replay details, use the command:

awslocal events describe-replay --replay-name ReplayTest

The description output will provide detailed information about the replay:

{
    "ReplayName": "ReplayTest",
    "ReplayArn": "arn:aws:events:us-east-1:000000000000:replay/ReplayTest",
    "State": "COMPLETED",
    "EventSourceArn": "arn:aws:events:us-east-1:000000000000:archive/ArchiveTest",
    "Destination": {
        "Arn": "arn:aws:events:us-east-1:000000000000:event-bus/test-event-bus"
    },
    ...
}

Replayed events will contain metadata to distinguish them from original events. You can check LocalStack logs for Lambda invocation details:

2024-08-06T07:29:24.039  INFO --- [et.reactor-1] localstack.request.aws     : AWS events.StartReplay => 200
2024-08-06T07:29:24.681  INFO --- [et.reactor-0] localstack.request.http    : POST /_localstack_lambda/58324d0bbf2060f95e80830df11f08a8/status/58324d0bbf2060f95e80830df11f08a8/ready => 202
2024-08-06T07:29:24.720  INFO --- [et.reactor-1] localstack.request.http    : POST /_localstack_lambda/58324d0bbf2060f95e80830df11f08a8/invocations/091a672b-a832-4824-a4cc-d56c75ed9893/logs => 202
2024-08-06T07:29:24.721  INFO --- [et.reactor-0] localstack.request.http    : POST /_localstack_lambda/58324d0bbf2060f95e80830df11f08a8/invocations/091a672b-a832-4824-a4cc-d56c75ed9893/response => 202

On the CloudWatch Logs Resource Browser, you can inspect the /aws/lambda/LogScheduledEvent log to view the received event:

Conclusion

With EventBridge Archive & Replay, you can capture and replay past events to troubleshoot issues or reprocess events through newly added functionalities without manually storing your events or setting up an additional infrastructure layer like Dead Letter Queues (DLQ), which can be time-consuming depending on the event consumer. LocalStack lets you test these event-driven workflows locally, simplifying the development loop and reducing development costs.

LocalStack’s EventBridge emulation supports testing event transmission across different AWS regions and accounts, integration with various targets (such as SNS, SQS, Step Functions), and Cloud Pods for state sharing & collaboration. In upcoming blog posts, we will delve into some of these use cases, showcasing how LocalStack can make the black box of event-driven systems transparent for developers.

LocalStack allows you to debug your application code in ECS tasks by setting breakpoints and improving your development and testing process without the requirement to deploy to the real cloud. This post will show you how to set up a local ECS cluster and task, open a remote debugging port on the LocalStack container, configure VS Code for debugging, and run the debugger.

Prerequisites

• localstack CLI with the LOCALSTACK_AUTH_TOKEN

• Docker

• Visual Studio Code

• AWS CDK with cdklocal

• Node.js & npm

• AWS CLI with awslocal wrapper (optional)

• curl

Node.js app on ECS with Elastic Load Balancer

The sample application uses AWS CDK to deploy a Node.js containerized app on AWS Fargate within an ECS cluster. The CDK stack:

• Creates a local VPC and an ECS Cluster.

• Builds and pushes the Docker image to a local ECR repository.

• Adds task and container Definition for the local ECS tasks.

• Runs containers on Fargate, distributing traffic using ELB.

After deploying the CDK stack, you will create a VS Code task & launch configuration and attach the debugger to the ECS task.

Start your LocalStack container

Launch the LocalStack container on your local machine using the specified command:

export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
ECS_DOCKER_FLAGS="-e NODE_OPTIONS=--inspect-brk=0.0.0.0:9229 -p 9229:9229" \
localstack start

The configuration variable ECS_DOCKER_FLAGS in the command mentioned above is used to pass additional flags to Docker when creating ECS task containers. The option is used to enable a remote debugging port for your ECS tasks. This exposes the debugger on a random port on the host machine, allowing you to remotely attach your debugger to the ECS tasks.

Alternatively, you can utilize the Docker Compose configuration provided in the repository to start the LocalStack container with the following command:

cd devops-tooling && docker compose -p ecslb up

Install the dependencies

To begin, fork the LocalStack sample repository on GitHub using this command:

git clone git@github.com:localstack-samples/sample-cdk-ecs-elb.git

After cloning the repository, navigate to the iac/awscdk directory and install all the required dependencies with the following command:

cd iac/awscdk
npm install

Deploy the application locally

To deploy the application, you should use cdklocal, which is a wrapper script for utilizing the AWS Cloud Development Kit (CDK) with local APIs provided by LocalStack.

Start by ensuring that each AWS environment you plan to deploy resources to is bootstrapped. Run the following command in the iac/awscdk directory:

cdklocal bootstrap

Next, you can deploy the CDK stack using this command in the iac/awscdk directory:

cdklocal deploy

After a successful deployment, you should see output similar to the following:

✅  RepoStack

✨  Deployment time: 20.15s

Outputs:
RepoStack.MyFargateServiceLoadBalancerDNS704F6391 = lb-bf1b158e.elb.localhost.localstack.cloud
RepoStack.MyFargateServiceServiceURL4CF8398A = http://lb-bf1b158e.elb.localhost.localstack.cloud
RepoStack.localstackserviceslb = lb-bf1b158e.elb.localhost.localstack.cloud:4566
RepoStack.serviceslb = lb-bf1b158e.elb.localhost.localstack.cloud
Stack ARN:
arn:aws:cloudformation:us-east-1:000000000000:stack/RepoStack/77d40ed8

✨  Total time: 22.55s

You can now use the URLs provided to send requests to the Node.js application running in the ECS task locally. To inspect the ELB endpoint further, if you have awslocal installed, run the following command:

awslocal elbv2 describe-load-balancers --query 'LoadBalancers[0].DNSName' --output text

The output should be:

lb-bf1b158e.elb.localhost.localstack.cloud

Configure VS Code for remote debugging

While starting LocalStack, you configured ECS_DOCKER_FLAGS to enable the required configuration for remote debugging. You can now add a VS Code task to wait for the remote debugging server. Create a tasks.json file in the .vscode directory of the project and add the following:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Wait Remote Debugger Server",
            "type": "shell",
            "command": "while [[ -z $(docker ps | grep :9229) ]]; do sleep 1; done; sleep 1;"
        }
    ]
}

Next, you can define how VS Code should connect to the remote Node.js application. Create a new launch.json file in the .vscode directory of the project and add the following:

{
    "version": "0.2.0",
    "configurations": [
        {
            "address": "127.0.0.1",
            "localRoot": "${workspaceFolder}",
            "name": "Attach to Remote Node.js",
            "port": 9229,
            "remoteRoot": "/app",
            "request": "attach",
            "type": "node",
            "preLaunchTask": "Wait Remote Debugger Server"
        }
    ]
}

Run the debugging process

For the debugging process, you can set breakpoints in the Node.js application code. Breakpoints allow you to pause the code execution on a specific line to inspect it. To run the debugger:

• Click on Run and Debug in your VS Code session.

  

• Select the Attach to Remote Node.js configuration and click Start Debugging.

  

• Navigate to src/app/server.js in your VS Code and set breakpoints (for example, on line 5).

  

• Open your terminal and execute the following command to trigger the code where you set the breakpoint:

    curl "lb-bf1b158e.elb.localhost.localstack.cloud:4566"
  

  Replace the endpoint URL with the appropriate URL in your setup.

You will now see the remote debugging in action on your VS Code:

You can now inspect the values of any identifiers within the current scope using the Variables and Watch pane in VS Code. Additionally, use the debug toolbar at the top of the editor to step through the code, continue execution, or manage breakpoints during debugging.

After the debugging process is completed, the command you ran previously will produce the following output:

Conclusion

That brings us to the end of our tour of how you can debug ECS tasks locally with LocalStack. You can mount code from the host filesystem into the ECS container for hot reloading, enabling quick debugging and testing without needing to rebuild and redeploy the Docker image each time. LocalStack not only lets you deploy and test your infrastructure but also offers various developer experience (DevEx) features, allowing you to develop your application without interacting with the actual cloud throughout your software development lifecycle.

LocalStack is a core cloud emulator that allows you to emulate various AWS services on your local machine without needing a real AWS account. LocalStack essentially operates within a Docker container, emulating or mocking different cloud services. This setup lets you connect your integrations (like Terraform, CDK, AWS CLI) to the running container for testing your application and infrastructure code without spending anything on infrastructure provisioning.

For learning, testing, and integration purposes, LocalStack supports running an emulated EC2 instance locally. This blog will walk you through launching a local EC2 instance, accessing the running instance, and deploying a basic Flask API using a user data shell script.

Docker Desktop on macOS and Windows does not make the Docker Bridge network accessible, preventing users from completing this tutorial. To ensure a smooth end-to-end experience, it is recommended to use Linux or GitHub Codespaces/GitPod.

Running Docker containers as EC2 instances

In the LocalStack community edition, you can mock EC2 APIs on your local machine. You send API requests to the mocked EC2 service running in LocalStack that mimics the real AWS service's interface and triggers a predefined behaviour. However, no actual instances (like virtual machines) are created. This setup often leads to testing errors because the mock implementation doesn't create real resources.

In LocalStack Pro, EC2 APIs utilize the Docker Engine backend to emulate EC2 instances. When you launch an EC2 instance locally, LocalStack sets up a Docker container recognized as an Amazon Machine Image (AMI). This enables users to log in to the instance, test their configurations, and conduct end-to-end integration tests on a local EC2 infrastructure.

Prerequisites

• LocalStack CLI with a LOCALSTACK_AUTH_TOKEN

• Docker

• AWS CLI with awslocal wrapper

• jq & cURL

You can sign-up for LocalStack Hobby Plan to grab a LocalStack Auth Token and use advanced AWS APIs, such as EC2 Docker backend.

Create a Flask API

To illustrate an example, set up a basic Flask API with three routes: /, /get, and /post. Create a new file called app.py and add the following code.

from flask import Flask, request

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, LocalStack!'

# GET route
@app.route('/get', methods=['GET'])
def get_example():
    return 'This is a GET request example.'

# POST route
@app.route('/post', methods=['POST'])
def post_example():
    data = request.json
    return f'This is a POST request example. Received data: {data}'

if __name__ == '__main__':
    app.run(port=8000, host="0.0.0.0")

Add this file to a fresh GitHub/GitLab repository, or use an existing template you can find at the following GitLab repository.

Start your LocalStack container

Launch the LocalStack container on your local machine using the specified command:

export LOCALSTACK_AUTH_TOKEN=...
localstack start

Once initiated, you'll receive a confirmation output indicating that the LocalStack container is up and running.

     __                     _______ __             __
    / /   ____  _________ _/ / ___// /_____ ______/ /__
   / /   / __ \/ ___/ __ `/ /\__ \/ __/ __ `/ ___/ //_/
  / /___/ /_/ / /__/ /_/ / /___/ / /_/ /_/ / /__/ ,<
 /_____/\____/\___/\__,_/_//____/\__/\__,_/\___/_/|_|

 💻 LocalStack CLI 3.5.0
 👤 Profile: default

...
─────────────────── LocalStack Runtime Log (press CTRL-C to quit) ───────────────────

LocalStack version: 3.5.1.dev20240712171301
LocalStack build date: 2024-07-14
LocalStack build git hash: 646be01
...

To confirm the startup of your LocalStack container with the Pro services, utilize the cURL command to query the /info endpoint.

curl http://localhost:4566/_localstack/info | jq
{
  "version": "3.5.1.dev20240712171301:646be01",
  "edition": "pro",
  "is_license_activated": true,
  ...
  "system": "linux",
  "is_docker": true,
  ...
}

Create an EC2 key pair

Before you make an EC2 instance, make a key pair. EC2 keeps the public key and shows the private key for you to save. To make a key pair using the awslocal CLI, use this command:

awslocal ec2 create-key-pair \
    --key-name my-key \
    --query 'KeyMaterial' \
    --output text | tee key.pem

This saves the key pair in a file named key.pem in the current directory. Apply the right permissions to the file with this command:

chmod 400 key.pem

Alternatively, you can bring in an existing key pair. If you have an SSH public key in your home directory under ~/.ssh/id_rsa.pub, run this command to import it:

awslocal ec2 import-key-pair \
    --key-name my-key \
    --public-key-material file://~/.ssh/id_rsa.pub

Add inbound roles

In LocalStack, networking features like subnets and VPCs are not emulated. LocalStack provides a default security group that manages the exposed ports for the EC2 instance. While users can create additional security groups, LocalStack focuses on the default security group.

By default, the SSH port 22 is open. To enable inbound traffic on the port 8000 for our Flask API, use this command to authorize the default security group:

awslocal ec2 authorize-security-group-ingress \
    --group-id default \
    --protocol tcp \
    --port 8000 \
    --cidr 0.0.0.0/0

The output will be:

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-84956433d958d012e",
            "GroupId": "sg-952c436416c2392b0",
            "GroupOwnerId": "000000000000",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 8000,
            "ToPort": 8000,
            "CidrIpv4": "0.0.0.0/0",
            "Description": ""
        }
    ]
}

Retrieve the security group ID with this command:

sg_id=$(awslocal ec2 describe-security-groups | jq -r '.SecurityGroups[0].GroupId')
echo $sg_id

Run the EC2 Instance locally

Now, you can start and operate the EC2 instance on your local machine. Before executing the command, create a new file named user_script.sh and include the following content:

#!/bin/bash -xeu

apt update
apt install python3 -y
apt-get -y install python3-pip curl git
git clone https://gitlab.com/HarshCasper/flask-api-example.git
cd flask-api-example
pip3 install flask
python3 app.py

Utilize the above script as a user data shell script, enabling you to send specific instructions to an instance during launch.

Modify the repository URL if you want to specify your personal GitHub/GitLab repository for deployment. Include any extra commands necessary to set up the application correctly.

Now, directly initiate the EC2 instance by executing the following command:

awslocal ec2 run-instances \
  --image-id ami-ff0fea8310f3 \
  --count 1 \
  --instance-type t3.nano --key-name my-key \
  --security-group-ids $sg_id \
  --user-data file://./user_script.sh

In the command above, the instance type is specified as t2.nano, but it has no impact since LocalStack uses a ubuntu-20.04-focal-fossa Docker image emulated as an EC2 instance. This behaviour is set by the image ID ami-ff0fea8310f3. To use an Amazon Linux AMI instead, specify the image ID as ami-024f768332f0 (requiring adjustments to the user script).

Upon successful execution, the output will be retrieved.

{
    "Groups": [
        {
            "GroupName": "default",
            "GroupId": "sg-245f6a01"
        }
    ],
    "Instances": [
        {
            "AmiLaunchIndex": 0,
            "ImageId": "ami-ff0fea8310f3",
            "InstanceId": "i-42e830289e675885f",
            "InstanceType": "t3.nano",
            ...
            "VirtualizationType": "paravirtual"
        }
    ],
    "OwnerId": "000000000000",
    "ReservationId": "r-a8f48d6e"
}

You can also confirm that LocalStack has spun an additional Docker container to emulate a locally running EC2 instance:

docker ps
CONTAINER ID   IMAGE                                                      COMMAND                  CREATED          STATUS                          PORTS                                                                                                                                    NAMES
0c00863e8fdd   localstack-ec2/ubuntu-20.04-focal-fossa:ami-ff0fea8310f3   "sleep infinity"         4 seconds ago    Up 3 seconds                    0.0.0.0:22->22/tcp, 0.0.0.0:8000->8000/tcp                                                                                               localstack-ec2.i-320e3293f4029b596
3fc719173eb2   localstack/localstack-pro                                  "docker-entrypoint.sh"   18 minutes ago   Up 18 minutes (healthy)         127.0.0.1:443->443/tcp, 127.0.0.1:4510-4560->4510-4560/tcp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, 127.0.0.1:4566->4566/tcp, 5678/tcp   localstack-main

Logging into the EC2 instance

After launching the EC2 instance, check the LocalStack logs. In these logs, you can confirm that the EC2 instance is running successfully on your local machine.

localstack logs
...
Determined main container network: bridge
Determined main container target IP: 172.17.0.2
Instance i-42e830289e675885f will be accessible 
via SSH at: 127.0.0.1:22, 172.17.0.3:22
Instance i-42e830289e675885f port mappings (container -> host): 
{'8000/tcp': 8000, '22/tcp': 22}
AWS ec2.RunInstances => 200
...

In the logs above, verify that the instance is accessible via SSH at 127.0.0.1. Depending on your setup, this configuration might change. In this example, you can use the following command to log in to the EC2 instance:

ssh -i key.pem root@127.0.0.1

You'll be prompted to establish authenticity. After verification, you can log in to the instance.

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:......
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '127.0.0.1' (ECDSA) to the list of known hosts.
root@6ef2a4c8d8ac:~#

In the local EC2 instance, you can execute various commands, similar to a real EC2 instance on the AWS cloud. You can also use cURL to confirm if your Flask API is operational:

root@6ef2a4c8d8ac:~# curl localhost:8000 
Hello, LocalStack!

Let's attempt to access the running Flask API outside of the EC2 instance.

Test the running Flask API

In the LocalStack logs, confirm that the port mappings from the container to the host are accessible on port 8000. Run the following command in a separate terminal tab to check if the Flask API is reachable:

curl localhost:8000
Hello, LocalStack!

Additionally, send GET and POST requests to test the active Flask API:

curl -X POST \
    -H "Content-Type: application/json" \
    -d '{"key": "value"}' \
    localhost:8000/post
This is a POST request example. Received data: {'key': 'value'}

curl http://localhost:8000/get
This is a GET request example.

To validate the execution of your user data shell script, use the following command in your active EC2 instance:

cat /var/log/cloud-init-output.log
...
+ python3 app.py
 * Serving Flask app 'app'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8000
 * Running on http://172.17.0.3:8000
Press CTRL+C to quit
127.0.0.1 - - [15/Feb/2024 07:51:02] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [15/Feb/2024 07:51:06] "GET / HTTP/1.1" 200 -
...

Terminate the instance

To terminate the instance, stop the LocalStack container with the command:

localstack stop

However, if you need to examine the container filesystem for debugging later on you can configure EC2_REMOVE_CONTAINERS=0 while starting LocalStack. This configuration option controls whether Docker containers created during the process are removed at instance termination or when LocalStack shuts down.

Conclusion

You can create additional EC2 instances using Terraform, CDK, or Pulumi. For a visual user interface for EC2 instance creation, utilize the LocalStack Web Application. You can also set up EBS Block Devices, Instance Metadata Service, and Elastic Load Balancers for your locally running EC2 instances. We are actively enhancing our EC2 emulation features to facilitate faster local development and testing on your machine!

In today’s tech landscape, large language models (LLMs) and AI are transforming both tech-centric and traditional businesses. AI functionality is being integrated across various platforms to enhance user experience. However, developing and testing these AI integrations often requires extensive infrastructure, leading to high costs. LocalStack enables developers to build and test AI integrations locally, which accelerates the development process and avoids extra expenses.

In this tutorial, we’ll explore building an AI-powered chatbot using Ollama, a tool that lets users interact with information through natural language. For instance, on a government website, rather than navigating complex menus, you could ask a chatbot to find a specific form. We’ll start by setting up the entire system locally with LocalStack, then move to deploying it in the cloud.

Prerequisites

• AWS free tier account

• LocalStack Pro

• Docker - for running LocalStack

• AWS CLI and AWS CLI local

• npm - for building the frontend app

Architecture Overview

To follow along with this post and get the React app, you can clone the repository for this project.

We will explore a comprehensive example of running Ollama on ECS Fargate. This example includes a backend with a VPC, a load balancer, multiple security groups, and an ECR service hosting our image. For simplicity, the frontend application will be a basic chat interface with a prompt field and an answer box.

The React application could make a direct request to our Ollama container, but in the real world, it’s not likely that you would run just one task and certainly not try to access the container using its IP address. Ideally you should have multiple tasks running to ensure high availability, in case one of them encounters an issue, you can always rely on the others to take over. In front of the tasks, you’ll need an application load balancer to handle the HTTP requests. This is how traffic is distributed across the containers. The load balancer will have a listener, which listens for client requests. The requests are routed to targets, which will be the IPs for the tasks/containers. The targets live in a target group, and that allows us to make configurations for all of them (for example, setting a routing algorithm, or healthcheck related configs). Our load balancer needs a security group that allows inbound traffic, and a second security group that only allows incoming traffic to the ECS service, which will run two tasks.

Why Ollama & Tinyllama

Ollama is an open-source platform that allows users to run large language models (LLMs) locally on their devices. In its essence, Ollama streamlines the tasks of downloading, installing, and utilizing a broad spectrum of LLMs, enabling users to discover their potential without requiring deep technical skills or dependence on cloud-based platforms. Most importantly, Ollama allows users to run their own specialized LLMs with ease.

Both tools are designed for local development, so using Ollama with LocalStack for building cloud applications offers several advantages:

• Complete local development environment: Combining Ollama and LocalStack allows developers to run complex cloud applications entirely on their local machines. Ollama can handle large language models locally, while LocalStack handles the AWS services, creating a comprehensive and integrated local development environment.

• Cost efficiency: Running applications locally avoids the costs associated with cloud resources during development and testing. This is particularly useful when working with large language models that can be resource-intensive and expensive to run in the cloud.

• Faster iteration cycles: Local development with Ollama and LocalStack allows for rapid prototyping and testing. Developers can quickly make changes and see results without the delay of deploying to the cloud. This speeds up the development cycle significantly.

• Consistent development and production environments: By using LocalStack to emulate AWS services, developers can ensure that their local development environment closely matches the production environment. This reduces the risk of environment-specific bugs and improves the application's reliability when deployed to the actual cloud.

• Improved Testing Capabilities: LocalStack provides a robust platform for testing AWS services, including ECS and Fargate. Running Ollama as a Fargate task on LocalStack allows for testing complex deployment scenarios and interactions with other AWS services, ensuring that the application behaves as expected before deploying to the cloud.

Tinyllama is a compact AI language model that stands out due to its efficient size and robust training. It occupies just 637 MB and was trained on a trillion tokens, making it not only mobile-friendly but also powerful enough to surpass similar-sized models. Designed as a smaller version of Meta’s Llama 2, it shares the same architecture and tokenizer, making it an ideal choice for development and testing, particularly with applications demanding a restricted computation and memory footprint. Depending on your needs, you can also replace it with a different, or more specialized model.

Running the application on LocalStack

Starting LocalStack

The first thing we need to do is start LocalStack by using docker compose. This permits an easy visualisation of all the necessary configs. Remember to set your LOCALSTACK_AUTH_TOKEN as an environment variable.

cd localstack
export LOCALSTACK_AUTH_TOKEN=<your_auth_token>
docker compose up

Some important configs

All the following configuration flags can be found in the docker-compose.yml file, in the localstack folder:

• DEBUG=1 - This flag enables debug mode in LocalStack. When set to 1, LocalStack provides more detailed logs, making it easier to trace issues.

• ENFORCE_IAM=1 - This configuration enables the enforcement of AWS IAM policies in LocalStack. Normally, LocalStack runs with simplified or no security checks to facilitate development.

• ECS_DOCKER_FLAGS=-e OLLAMA_ORIGINS="*" - This setting is used to pass environment variables to Docker containers spawned by the ECS service within LocalStack. Specifically, we set OLLAMA_ORIGINS="*" inside these containers to indicate that requests from any origin are allowed. This is relevant when integrating with web applications that may call APIs from various domains.

• DISABLE_CORS_CHECKS=1 - This flag disables CORS checks in LocalStack, for ease of development.

• DISABLE_CUSTOM_CORS_S3=1 - When set, this configuration disables the custom CORS handling for S3 services within LocalStack.

Building the React app

In the localstack folder, there’s a directory called frontend. To build the React application run the following commands:

cd frontend/chatbot/
npm install
npm run build

Notice the creation of the build folder. The npm run build command will create the static assets needed to run our app, and they will then be uploaded to the S3 bucket.

Creating the stack

Now we can run the bash script containing AWS CLI commands to create the necessary resources. Let’s first have a look at some of the commands in the script and identify the resources they create:

export VPC_ID=$(awslocal ec2 create-vpc --cidr-block 10.0.0.0/16 | jq -r '.Vpc.VpcId')

Creates a Virtual Private Cloud (VPC) with a specified CIDR block.

export SUBNET_ID1=$(awslocal ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.1.0/24 \
  --availability-zone us-east-1a \
  | jq -r '.Subnet.SubnetId')

export SUBNET_ID2=$(awslocal ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.2.0/24 \
  --availability-zone us-east-1b \
  | jq -r '.Subnet.SubnetId')

Creates two subnets within the VPC, each in a different availability zone.

export RT_ID=$(awslocal ec2 create-route-table --vpc-id $VPC_ID | jq -r '.RouteTable.RouteTableId')

awslocal ec2 associate-route-table \
  --route-table-id $RT_ID \
  --subnet-id $SUBNET_ID1

awslocal ec2 associate-route-table \
  --route-table-id $RT_ID \
  --subnet-id $SUBNET_ID2

awslocal ec2 create-route \
  --route-table-id $RT_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $INTERNET_GW_ID

Creates a route table, associates it with the subnets, and adds a route to the internet gateway.

export SG_ID1=$(awslocal ec2 create-security-group \
  --group-name ApplicationLoadBalancerSG \
  --description "Security Group of the Load Balancer" \
  --vpc-id $VPC_ID | jq -r '.GroupId')

awslocal ec2 authorize-security-group-ingress \
  --group-id $SG_ID1 \
  --protocol tcp \
  --port 80 \
  --cidr 0.0.0.0/0

export SG_ID2=$(awslocal ec2 create-security-group \
  --group-name ContainerFromLoadBalancerSG \
  --description "Inbound traffic from the First Load Balancer" \
  --vpc-id $VPC_ID \
  | jq -r '.GroupId')

awslocal ec2 authorize-security-group-ingress \
  --group-id $SG_ID2 \
  --protocol tcp \
  --port 0-65535 \
  --source-group $SG_ID1

Creates security groups for the load balancer and the ECS service, allowing necessary traffic.

export LB_ARN=$(awslocal elbv2 create-load-balancer \
  --name ecs-load-balancer \
  --subnets $SUBNET_ID1 $SUBNET_ID2 \
  --security-groups $SG_ID1 \
  --scheme internet-facing \
  --type application \
  | jq -r '.LoadBalancers[0].LoadBalancerArn')

export TG_ARN=$(awslocal elbv2 create-target-group \
  --name ecs-targets \
  --protocol HTTP \
  --port 11434 \
  --vpc-id $VPC_ID \
  --target-type ip \
  --health-check-protocol HTTP \
  --region us-east-1 \
  --health-check-path / \
  | jq -r '.TargetGroups[0].TargetGroupArn')

awslocal elbv2 create-listener \
  --load-balancer-arn $LB_ARN \
  --protocol HTTP \
  --port 11434 \
  --default-actions Type=forward,TargetGroupArn=$TG_ARN

Creates an internet-facing application load balancer and a target group, and sets up a listener to forward traffic.

awslocal ecr create-repository --repository-name ollama-service
export MODEL_NAME=tinyllama
docker build --build-arg MODEL_NAME=$MODEL_NAME -t ollama-service .
docker tag ollama-service:latest 000000000000.dkr.ecr.us-east-1.localhost.localstack.cloud:4510/ollama-service:latest
docker push 000000000000.dkr.ecr.us-east-1.localhost.localstack.cloud:4510/ollama-service:latest

Creates an ECR repository, builds the Docker image, and pushes it to the repository.

awslocal ecs create-cluster --cluster-name OllamaCluster

awslocal iam create-role \
  --role-name ecsTaskRole \
  --assume-role-policy-document file://ecs-task-trust-policy.json

export ECS_TASK_PARN=$(awslocal iam create-policy \
  --policy-name ecsTaskPolicy \
  --policy-document file://ecs-task-policy.json \
  | jq -r '.Policy.Arn')

awslocal iam attach-role-policy \
  --role-name ecsTaskRole \
  --policy-arn $ECS_TASK_PARN

awslocal iam update-assume-role-policy \
  --role-name ecsTaskRole \
  --policy-document file://ecs-cloudwatch-policy.json

awslocal iam create-role \
  --role-name ecsTaskExecutionRole \
  --assume-role-policy-document file://ecs-trust-policy.json

export ECS_TASK_EXEC_PARN=$(awslocal iam create-policy \
  --policy-name ecsTaskExecutionPolicy \
  --policy-document file://ecs-task-exec-policy.json | jq -r '.Policy.Arn')

awslocal iam attach-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-arn $ECS_TASK_EXEC_PARN

awslocal iam update-assume-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-document file://ecs-cloudwatch-policy.json

Creates an ECS cluster and IAM roles with necessary policies for task execution.

awslocal logs create-log-group --log-group-name ollama-service-logs
awslocal ecs register-task-definition \
  --family ollama-task \
  --cli-input-json file://task_definition.json

Creates a CloudWatch log group and registers the ECS task definition.

awslocal ecs create-service \
  --cluster OllamaCluster \
  --service-name OllamaService \
  --task-definition ollama-task \
  --desired-count 2 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID1,$SUBNET_ID2],securityGroups=[$SG_ID2],assignPublicIp=ENABLED}" \
  --load-balancers "targetGroupArn=$TG_ARN,containerName=ollama-container,containerPort=11434"

Creates an ECS service with the specified configuration, linking it to the load balancer.

awslocal s3 mb s3://frontend-bucket
awslocal s3 website s3://frontend-bucket --index-document index.html
awslocal s3api put-bucket-policy --bucket frontend-bucket --policy file://bucket-policy.json
awslocal s3 sync ./frontend/chatbot/build s3://frontend-bucket

Creates an S3 bucket, configures it as a website, sets the bucket policy, and syncs the frontend build to the bucket.

If you decide to use the AWS console to create all your resources, some of the complexity of these commands will be abstracted, and some services will be created as dependencies of other resources.

You can run the full commands.sh script and watch the LocalStack logs for updated information on the resources, as they get created. If you choose to, you can also manually run these commands, one by one, as you go through this article.

bash commands.sh

Using the app locally

Now that everything is deployed, you can go to the frontend application and try it out. In your browser, navigate to http://frontend-bucket.s3-website.us-east-1.localhost.localstack.cloud:4566/ and start typing your question. It takes a few seconds, and then the full answer appears:

If you look at the App.js, located in frontend/chatbot/src, you’ll notice the POST call payload contains a field stream: false. For simplicity purposes, we’re going to receive our answer from the LLM “in bulk” rather than streamed. This takes a few seconds to generate, and then it is fully received.

The backend call will be made to the load balancer, at http://ecs-load-balancer.elb.localhost.localstack.cloud:4566/api/generate/, so we don’t have to worry about how we access the task containers.

Running on AWS

To run this stack in the real AWS cloud, we need to make some small adjustments. The ready-to-deploy resources are in the aws folder, and they are the same as the ones for LocalStack, except:

• The AWS account number needs to be provided, so wherever you find <your_account_number>, it should be replaced with the 12-digit key (task_definition_aws.json, commands-aws.sh).

• A new bucket name needs to be set, as it needs to be unique, so the <your_bucket_name> placeholder has to be replaced with a name of your choice (commands-s3-aws.sh, bucket-policy.json).

• Since on AWS you don’t know the final DNS name of the load balancer, and we need it for the frontend component, we’ll build the app and upload the files to the S3 bucket only after creating the stack. The commands-aws.sh script will export and write the load balancer DNS name into the .env file, where the React app can pick it up from. This is generally easier using LocalStack because the DNS name of the load balancer is always as defined by the user.

The steps to getting this project on AWS are:

1. Make the aforementioned changes to your files.

2. In the aws root folder run bash commands-aws.sh.

3. Build the React app:

    cd frontend/chatbot
    npm install
    npm run build
   

4. Create the S3 bucket and prepare it to host the frontend application, by running bash commands-aws-s3.sh in the aws folder.

   After the first step, you can test the backend by running the following command:

    export LB_NAME=$(aws elbv2 describe-load-balancers --load-balancer-arns $LB_ARN | jq -r '.LoadBalancers[0].DNSName')
    curl $LB_NAME
   

If you get a message like the following, give it a few more seconds until the Fargate instances are up and running and you should see the Ollama is running response.

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>

After building the GUI part and uploading it to the S3 bucket, you’ll be able to access your chatbot at this address: http://<bucket-name>.s3-website.us-east-1.amazonaws.com/.

Conclusion

Developing and testing cloud AI-powered applications can be complex, particularly when ensuring they perform reliably in a production-like environment without incurring high costs. This is where integrating Ollama and LocalStack provides a robust solution. Ollama, which simplifies the process of downloading, installing, and interacting with various large language models (LLMs), paired with LocalStack’s ability to emulate AWS cloud services locally, allows developers to rigorously test AI functionalities in a controlled and cost-effective manner. By leveraging LocalStack, developers can validate integrations and behaviors of AI models managed with Ollama. The combination of Ollama’s straightforward LLM handling with LocalStack’s comprehensive AWS emulation offers a powerful toolkit for any developer looking to build reliable and scalable cloud AI applications.

S3 bucket remote storage

The S3 remote allows for the storage of Cloud Pod assets in an existing S3 bucket located in a real AWS account. The first action to take is to export the required AWS credentials during the terminal session. Side note: the S3 remote feature for Cloud Pods is only accessible when the localstack CLI is installed through pip, for now.

Let’s try it out:

$ export AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY_ID>
$ export AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY>

Next, we set up a new remote connection specifically for an S3 bucket. By using the command below, we create a remote called s3-storage-aws. This remote is for saving Cloud Pod items in an S3 bucket named localstack-pod-storage.

$ localstack pod remote add s3-storage-aws 's3://ls-pods-bucket-test/?access_key_id={access_key_id}&secret_access_key={secret_access_key}'

Note: When setting this up, we might encounter an error message like:

SSL validation failed forhttps://localstack-pod-storage.s3.amazonaws.com/hostname.

To fix this, we can create a list of exceptions that point to AWS instead of LocalStack by using the following configuration flag in the docker-compose file:

DNS_NAME_PATTERNS_TO_RESOLVE_UPSTREAM=.*localstack-pod-storage.s3.amazonaws.com

This setting is generally used for hybrid setups, where certain API calls target AWS, whereas other services will target LocalStack.

Now we can save the pod:

$ localstack pod save cloud-pod-product-app s3-storage-aws

Cloud Pod cloud-pod-product-app successfully created ✅
Version: 1
Remote: s3
Services: sts,s3,iam,apigateway,dynamodb,lambda

The Cloud Pod is visible in the AWS S3 dashboard:

To load the state into a new LocalStack instance, we use:

$ localstack pod load cloud-pod-product-app s3-storage-aws

Cloud Pod cloud-pod-product-app successfully loaded

ORAS remote storage

ORAS, which stands for OCI Registry As Storage, is a tool designed to help you use OCI (Open Container Initiative) registries for storing and sharing a wide range of content. While OCI registries were originally created for container images, ORAS extends their use to other types of artifacts. Essentially, ORAS allows you to push and pull any content to and from OCI-compliant registries using the same workflows you'd use for container images.

Docker Hub comes into play as a popular, OCI-compliant container registry. It's primarily known for hosting Docker container images but, thanks to the OCI specification's flexibility, it can also serve as a storage and distribution point for other types of artifacts through tools like ORAS. This makes Docker Hub not just a hub for Docker images but a versatile cloud registry for various types of application artifacts, supporting the broader ecosystem of cloud-native development and deployment practices.

Let’s illustrate how you can utilize Docker Hub to store and retrieve Cloud Pods. This is very similar to the S3 bucket storage setup:

$ export ORAS_USERNAME=your_docker_hub_id
$ export ORAS_PASSWORD=your_docker_hub_password

We can now use the CLI to create a new remote called oras-remote

$ localstack pod remote add oras-remote 'oras://{oras_username}:{oras_password}@registry.hub.docker.com/<your_docker_hub_id>'

A Cloud Pod can be stored on the newly configured remote:

$ localstack pod save cloud-pod-product-app oras-remote
Cloud Pod cloud-pod-product-app successfully created ✅
Version: 1
Remote: oras
Services: sts,s3,iam,apigateway,dynamodb,lambda

After saving the Cloud Pod, it will appear in the Docker Hub repositories dashboard:

Viewing all the remotes

By using the command localstack pod remote list, you can view all the configured remote options for saving Cloud Pods, including the AWS S3 bucket and the Docker Hub repository configuration, with the default set to the LocalStack platform.

$ localstack pod remote list
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Remote Name    ┃ URL                                                                                              ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ s3-storage-aws │ s3://localstack-pod-storage/?access_key_id={access_key_id}&secret_access_key={secret_access_key} │
│ oras-remote    │ oras://{oras_username}:{oras_password}@registry.hub.docker.com/msmuzitiger210                    │
│ default        │ platform://localstack                                                                            │
└────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘

Conclusion

We've seen how you can securely store Cloud Pod assets using S3 bucket remote storage and ORAS remote storage. S3 allows assets to be saved in an existing AWS account, while ORAS extends OCI registries for versatile artifact storage.

• Hot reloading of Lambda functions with each code change

• Ability to attach a remote debugger using an Integrated Development Environment (IDE)

• Persistence of function state using Cloud Pods

• Additional configuration like cold starts, concurrency, and more!

In the past year, we introduced the LocalStack VS Code Extension, enabling you to deploy and invoke Lambda functions directly from your code editor. This blog post provides a step-by-step guide on setting up the LocalStack VS Code Extension, utilizing a sample application deployed through the Serverless Application Model (SAM). Furthermore, we will delve into the process of viewing and managing locally created resources through the LocalStack Resource Browser.

Prerequisites

• localstack CLI with the LOCALSTACK_AUTH_TOKEN

• LocalStack Web Application

• Visual Studio Code & code CLI

• Serverless Application Model (SAM) CLI & samlocal wrapper

Recruiting Agency application with SNS, SQS, DynamoDB, Lambda and S3

This demo uses a public sample to showcase an event-driven recruiting agency application. The system utilizes SNS topics, a DynamoDB table, SQS queues, Lambda functions, and S3 buckets. The application consists of three primary services:

• An anti-corruption service that processes a change data capture (CDC) event stream.

  • This stream is converted into events and transmitted to the SNS JobEvents.fifo topic.

  • Subscribed services receive and process these events asynchronously.

• An analytics service with an SQS FIFO AnalyticsJobEvents.fifo queue linked to the SNS FIFO JobEvents.fifo topic.

  • SQS FIFO serves as an event source for Lambda functions, storing events in an S3 bucket.

• An inventory service with an SQS FIFO InventoryJobEvents.fifo queue connected to the SNS FIFO JobEvents.fifo topic.

  • It monitors JobCreated and JobDeleted events, storing data in a DynamoDB table with SNS filter policy assistance.

These resources are deployed using the Serverless Application Model (SAM). The LocalStack VS Code Extension is to be used for creating the Lambda functions and invoking them directly from the code editor. For testing the event-driven architecture, LocalStack Resource Browsers would be utilized to visualize the created application resources.

Start your LocalStack container

Launch the LocalStack container on your local machine using the specified command:

export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
DEBUG=1 localstack start

Replace <your-auth-token with your LocalStack Auth Token to start the LocalStack Pro container.

Setting DEBUG=1 activates more verbose logs, enabling you to observe the event-driven architecture in action during the invocation of Lambda functions. Once initiated, you'll receive a confirmation output indicating that the LocalStack container is up and running.

     __                     _______ __             __
    / /   ____  _________ _/ / ___// /_____ ______/ /__
   / /   / __ \/ ___/ __ `/ /\__ \/ __/ __ `/ ___/ //_/
  / /___/ /_/ / /__/ /_/ / /___/ / /_/ /_/ / /__/ ,<
 /_____/\____/\___/\__,_/_//____/\__/\__,_/\___/_/|_|

 💻 LocalStack CLI 3.1.0
 👤 Profile: default

[12:12:44] starting LocalStack in    localstack.py:494
           Docker mode 🐳
...
─── LocalStack Runtime Log (press CTRL-C to quit) ────
LocalStack supervisor: starting
LocalStack supervisor: localstack process (PID 16) starting

LocalStack version: 3.1.1.dev20240224232735
LocalStack Docker container id: 73597f01d11a
LocalStack build date: 2024-02-26
LocalStack build git hash: 323c0f8
...

Set Up the application

To set up the application on your local machine, follow these commands:

git clone https://github.com/localstack-samples/sample-sam-sns-fifo-dynamodb-lambda.git
cd sample-sam-sns-fifo-dynamodb-lambda

Next, open Visual Studio Code in the current directory using:

code .

If you encounter an issue where code is not recognized as a command, refer to the VS Code documentation to add it to the PATH. Alternatively, manually open the directory from your VS Code.

After setting up the project, you can now install the LocalStack extension to deploy and invoke Lambda functions directly from Visual Studio Code.

Install the LocalStack VS Code Extension

To install the LocalStack VS Code Extension, follow these steps:

1. Open Visual Studio Code and go to the Extensions icon in the Activity Bar on the side.

2. In the Extensions view, you'll find the most popular extensions on the VS Code Marketplace.

3. Enter LocalStack in the search bar to filter the Marketplace offerings.

   

4. Click on the Install button next to the LocalStack VS Code Extension to download and install it.

5. Once the installation is complete, you can proceed to deploy the Lambda function using the newly installed LocalStack VS Code Extension.

   

Deploy & invoke the Lambda function

To deploy and invoke the Lambda function, navigate to the anti-corruption-service directory and open the app.py file. Within the CodeLens, you'll find two options:

• Deploy Lambda function

• Invoke Lambda function

Click on Deploy Lambda function to set up local Lambdas with the running LocalStack container. The VS Code Extension supports either a CloudFormation or Serverless Application Model (SAM) deployment configuration.

In this project, use the SAM configuration available in the template.yaml file in the root directory.

You'll be asked to provide a unique name for the CloudFormation stack. Use recruiting-agency or your custom name and press Enter to confirm.

A notification will indicate the start of the Lambda function deployment, followed by another notification confirming its creation.

Return to your file and click on the Invoke Lambda function CodeLens. Choose the CloudFormation stack name, which can be either recruiting-agency or the custom name chosen during deployment.

Next, select the specific Lambda function to invoke, in this case, recruiting-agency-AntiCorruptionFunction-*.

LocalStack will initiate the invocation of the Lambda function. The invoked function logs should be displayed in the Output panel of the LocalStack VS Code Extension, which is automatically shown after invoking the Lambda function.

To check the LocalStack logs, either use localstack logs or view container logs on Docker Desktop. The logs include detailed information about the Lambda function invocation, such as duration and AWS service interactions.

localstack logs
...
2024-03-04T07:21:15.300 DEBUG --- [   asgi_gw_3] l.s.lambda_.provider       : Lambda invocation duration: 257.48ms
2024-03-04T07:21:15.308  INFO --- [   asgi_gw_3] localstack.request.aws     : AWS lambda.Invoke => 200
...
2024-03-04T07:21:15.769 DEBUG --- [   asgi_gw_3] l.services.sqs.models      : deleting message 91c9263f-1976-4cc2-9d98-11b98531f39e from queue arn:aws:sqs:us-east-1:000000000000:AnalyticsJobEvents.fifo
2024-03-04T07:21:15.814  INFO --- [   asgi_gw_3] localstack.request.aws     : AWS dynamodb.UpdateItem => 200
...

The DEBUG configuration variable allows you to access verbose logs, providing a comprehensive view of the architecture during the invocation of the Anti-Corruption Service, which serves as an event producer for the Inventory and Analytics service (event consumer).

Visualize the local resources

To view the AWS resources created locally, you can use the LocalStack Resource Browser. This browser offers an integrated interface, akin to the AWS Management Console, facilitating CRUD operations (Create-Read-Update-Delete) for managing local resources.

After invoking Lambda, navigate to the DynamoDB Resource Browser.

Select the InventoryTable resource, and then click on Items. You can now visualize job listings from the Inventory service using the InventoryJobEvents.fifo SQS queue and the recruiting-agency-InventoryFunction-* Lambda function.

You can also access the S3 Resource Browser and click on the recruiting-agency-analyticsbucket-* resource.

This bucket is utilized by the analytics service to store processed events from the recruiting-agency-AnalyticsFunction-* Lambda function, which uses the AnalyticsJobEvents.fifo SQS queue.

Similarly, you can leverage the Resource Browser for other functionalities such as querying logs in CloudWatch Logs Groups, verifying SNS subscriptions, and more!

Conclusion

The LocalStack VS Code Extension enhances the developer experience (DevX) by accelerating the development cycle for locally running Lambda functions. Using LocalStack, you can establish a developer environment that prioritises repeatability, reproducibility, and local testing — an upfront investment that pays off in the long term. Testing application code and infrastructure deployments frequently is the key to improve quality of your cloud applications, and LocalStack enables you to do that — right on your local machine!

The LocalStack VS Code Extension has a few limitations:

1. Presently, it supports only CloudFormation and Serverless Application Model.

2. Invocation of the Lambda function is restricted to the us-east-1 region.

3. It exclusively supports Python Lambdas with an empty payload invocation.

You can check out the source code for the LocalStack VS Code Extension, and share feedback/bug reports on our public issue tracker, as we continue to improve the extension experience.

The Challenge of Accurate Implementation

In a recent project, I aimed to illustrate a setup involving AWS API Gateway and Lambda functions, using Terraform for infrastructure provisioning. The goal was clear: an API Gateway integrating with two Lambda functions, one handling HTTP GET requests and the other managing POST requests. However, the simplicity of the concept did not reflect the complexity of its implementation.

Let's have a look at the snippet that started it all:

resource "aws_api_gateway_integration" "get_product_integration" {
  rest_api_id             = aws_api_gateway_rest_api.api.id
  resource_id             = aws_api_gateway_resource.product_api.id
  http_method             = aws_api_gateway_method.get_product.http_method
  type                    = "AWS_PROXY"
  integration_http_method = "GET"
  uri                     = aws_lambda_function.get_product.invoke_arn
}

The real challenge arose while configuring the integration_http_method argument, under the aws_api_gateway_integration resource. AWS documentation, dense and meticulous like a legal contract, requires careful parsing to ensure you're extracting the correct information. If you're running low on patience, it's easy to misinterpret the purpose of integration_http_method as indicating the method the Lambda function will accept.

A Twist in the Configuration

Using LocalStack to run the infrastructure makes it very obvious whether a Lambda function is invoked or not. Since every function starts its own container, it was clear that one was missing. I encountered the same behavior on AWS, so something strange was happening. This sent me on a quest through Google, StackOverflow, and AWS's documentation. Needless to say, I had to make many adjustments to the Terraform config file before finding the (not-so-obvious) answer.

Lessons lernt: configure Cloudwatch logs, but don't forget that the LocalStack logs will show you a one-liner hinting at a deeper misconfiguration.

localstack-main  | raise ApiGatewayIntegrationError("Internal server error", status_code=500)
localstack-main  | localstack.services.apigateway.helpers.ApiGatewayIntegrationError: Internal server error
localstack-main  | 2024-04-25T20:28:58.226  INFO --- [   asgi_gw_2] localstack.request.http: GET /dev/productApi => 500

It was the ApiGatewayIntegrationError that started to make things clear. The breakthrough came soon. However, LocalStack was instrumental with providing rapid feedback on my configurations and allowing me to iterate quickly without waiting on cloud deployment cycles. This fast feedback loop was critical in debunking my initial configurations and narrowing down the solution.

I guess they had a lot of questions regarding this topic, and they made it clear now:

Clarifying the Misconceptions

It turned out that the integration_http_method is not about the HTTP methods (GET, POST, DELETE, etc.) that the Lambda is supposed to handle. Rather, it is about how the API Gateway communicates with the Lambda function. It specifies the method used by the API Gateway to invoke the Lambda function, which should invariably be POST, regardless of the HTTP methods your API exposes. This is a subtle yet significant distinction that impacts how services are integrated and interact within AWS.

Concluding Thoughts

This experience underscored the importance of understanding the underlying mechanisms of cloud services and infrastructure as code tools. The ability to simulate environments with LocalStack before moving to actual AWS deployments can save countless hours of debugging and reconfiguration. For fellow developers venturing into similar territories, remember to check the details, and tools like LocalStack are invaluable in getting your IaC right.

• Region failover

• DNS failover

• Service failure simulations

All these testing scenarios can be efficiently executed within LocalStack, providing thorough coverage for critical situations in a matter of minutes rather than hours or days. To simulate service failures in LocalStack, you can use the Outages extension that enables you to start a local outage, right on your developer machine.

This allows you to quickly experiment with different failure scenarios, allowing you to perform chaos testing at an early stage by introducing errors at the infrastructure level. This is valuable as it enables you to replicate conditions that might not be feasible to mimic unless deployed to a production environment.

This blog will walk you through the process of setting up a cloud application on your local machine and leveraging the Outages extension to perform service failures in a local environment while using robust error handling to address and mitigate such issues. Furthermore, we will explore how to shift-left your chaos testing by integrating automated testing directly into your workflows.

Prerequisites

• LocalStack Docker image & LOCALSTACK_AUTH_TOKEN

• Docker Compose

• AWS CLI & awslocal wrapper

• Maven 3.8.5 & Java 17

• Python & pytest framework

• cURL

Product Management System with Lambda, API Gateway, and DynamoDB

This demo sets up an HTTP CRUD API functioning as a Product Management System. The components deployed include:

• A DynamoDB table named Products.

• Three Lambda functions:

  • add-product for product addition.

  • get-product for retrieving a product.

  • process-product-events for event processing and DynamoDB writes.

• A locally hosted REST API named quote-api-gateway.

• SNS topic named ProductEventsTopic and SQS queue named ProductEventsQueue.

• API Gateway resource named productApi with additional GET and POST methods.

Additionally, the applications set up a subscription between the SQS queue and the SNS topic, along with an event source mapping between the SQS queue and the process-product-events Lambda function.

All resources can be deployed using a LocalStack Init Hook via the init-resources.sh script in the repository. To begin, clone the repository on your local machine:

git clone https://github.com/localstack-samples/sample-outages-extension-serverless.git
cd sample-outages-extension-serverless

Let's create a Docker Compose configuration for simulating a local outage in the running Product Management System.

Set Up the Docker Compose

To start LocalStack and install the LocalStack Outages extension, create a new Docker Compose configuration. You can find the official Docker Compose file for starting the LocalStack container in our documentation.

For an extended setup, include the following in your Docker Compose file:

• Add the EXTENSION_AUTO_INSTALL=localstack-extension-outages environment variable to install the Outages extension from PyPI whenever a new container is created.

• Include the LOCALSTACK_HOST=localstack environment variable to ensure LocalStack services are accessible from other containers.

• Create the ls_network network to use LocalStack as its DNS server and enable the resolution of the domain name to the LocalStack container (also specify it via LAMBDA_DOCKER_NETWORK environment variable).

• Add a new volume attached to the LocalStack container. This volume holds the init-resources.sh file, which is copied to the LocalStack container and executed when the container is ready.

• Add another volume to copy the built Lambda functions specified as ZIP files during Lambda function creation.

• Optionally, add the LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT to wait for the runtime environment to start up, which may vary in speed based on your local machine.

The final Docker Compose configuration is as follows (also provided in the cloned repository):

version: "3.9"

services:
  localstack:
    networks:
      - ls_network
    container_name: localstack
    image: localstack/localstack-pro:latest
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
      - "127.0.0.1:443:443"
    environment:
      - DOCKER_HOST=unix:///var/run/docker.sock #unix socket to communicate with the docker daemon
      - LOCALSTACK_HOST=localstack # where services are available from other containers
      - LAMBDA_DOCKER_NETWORK=ls_network
      - LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN:?}
      - EXTENSION_AUTO_INSTALL=localstack-extension-outages
      - LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=600
    volumes:
      - "./volume:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./lambda-functions/target/product-lambda.jar:/etc/localstack/init/ready.d/target/product-lambda.jar"
      - "./init-resources.sh:/etc/localstack/init/ready.d/init-resources.sh"

networks:
  ls_network:
    name: ls_network

Deploy the local AWS infrastructure

Before deploying the demo application locally, build the Lambda functions to ensure they can be copied over during Docker Compose startup. Execute the following command:

cd lambda-functions && mvn clean package shade:shade

The built Lambda function is now available at lambda-functions/target/product-lambda.jar. Start the Docker Compose configuration, which automatically creates the local deployment using AWS CLI and the awslocal script inside the LocalStack container:

export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
docker-compose up

Check the Docker Compose logs to verify that the Outages extension is being installed, along with other local AWS resources:

localstack  | Localstack extensions installer: 
localstack  | Localstack extensions installer: Extension installation completed
localstack  | 
localstack  | LocalStack version: 3.1.1.dev20240131022456
....

localstack  | Get Product Lambda...
localstack  | 2024-02-26T05:34:18.091  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS lambda.CreateFunction => 201
...
localstack  | 2024-02-26T05:34:23.632  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS sns.CreateTopic => 200
localstack  | {
localstack  |     "TopicArn": "arn:aws:sns:us-east-1:000000000000:ProductEventsTopic"
localstack  | }
localstack  | 2024-02-26T05:34:24.229  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS sqs.CreateQueue => 200
localstack  | {
localstack  |     "QueueUrl": "http://sqs.us-east-1.localstack:4566/000000000000/ProductEventsQueue"
localstack  | }
...

After deployment, use cURL to create a product entity. Execute the following command:

curl --location 'http://12345.execute-api.localhost.localstack.cloud:4566/dev/productApi' \
--header 'Content-Type: application/json' \
--data '{
  "id": "prod-2004",
  "name": "Ultimate Gadget",
  "price": "49.99",
  "description": "The Ultimate Gadget is the perfect tool for tech enthusiasts looking for the next level in gadgetry. Compact, powerful, and loaded with features."
}'

The output should be:

Product added/updated successfully.

You can verify the successful addition by scanning the DynamoDB table:

awslocal dynamodb scan \
    --table-name Products

The output should be:

{
    "Items": [
        {
            "name": {
                "S": "Super Widget"
            },
            "description": {
                "S": "A versatile widget that can be used for a variety of purposes. Durable, reliable, and
 affordable."
            },
            "id": {
                "S": "prod-1002"
            },
            "price": {
                "N": "29.99"
            }
        }
    ],
    "Count": 1,
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

Injecting Chaos in the local infrastructure

You can now use the Outages extension for chaos testing of your locally deployed infrastructure. You can access the Outages extension through the REST API at http://outages.localhost.localstack.cloud:4566/outages, accepting standard HTTP requests.

To create an outage, taking down the DynamoDB table in the us-east-1 region, execute the following command:

curl --location --request POST 'http://outages.localhost.localstack.cloud:4566/outages' \
  --header 'Content-Type: application/json' \
  --data '
  [
    {
      "service": "dynamodb",
      "region": "us-east-1"
    }
  ]'

The output should be:

[{"service": "dynamodb", "region": "us-east-1"}]

This command creates an outage in the locally mocked us-east-1 DynamoDB tables. Verify by scanning the Products table:

awslocal dynamodb scan \
    --table-name Products

The output should be:

An error occurred (ServiceUnavailableException) when calling the Scan operation (reached max retries: 2): Service 'dynamodb' not accessible in 'us-east-1' region due to an outage

You can verify it in the LocalStack logs:

localstack  | 2024-02-26T06:12:02.196  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS dynamodb.DescribeEndpoints => 503 (ServiceUnavailableException)
localstack  | 2024-02-26T06:12:02.200  INFO --- [   asgi_gw_3] localstack.request.aws     : AWS dynamodb.PutItem => 503 (ServiceUnavailableException)

You can retrieve the current outage configuration using the following GET request:

curl --location \
    --request GET 'http://outages.localhost.localstack.cloud:4566/outages'

The output should be:

[{"service": "dynamodb", "region": "us-east-1"}]

Error handling for the outage

Now that the experiment is started, the DynamoDB table is inaccessible, resulting in the user being unable to get or post any new product. The API Gateway will return an Internal Server Error. To prevent this, include proper error handling and a mechanism to prevent data loss during a database outage.

The solution includes an SNS topic, an SQS queue, and a Lambda function that picks up queued elements and retries the PutItem operation on the DynamoDB table. If DynamoDB is still unavailable, the item will be re-queued.

Test this by executing the following command:

curl --location 'http://12345.execute-api.localhost.localstack.cloud:4566/dev/productApi' \
     --header 'Content-Type: application/json' \
     --data '{
       "id": "prod-1003",
       "name": "Super Widget",
       "price": "29.99",
       "description": "A versatile widget that can be used for a variety of purposes. Durable, reliable, and affordable."
     }'

The output should be:

A DynamoDB error occurred. Message sent to queue.

To stop the outage, send a POST request by using an empty list in the configuration. The following request will clear the current configuration:

curl --location --request POST 'http://outages.localhost.localstack.cloud:4566/outages' \
--header 'Content-Type: application/json' \
--data '[]'

Now, scan the DynamoDB table and verify that the Super Widget item has been inserted:

awslocal dynamodb scan \
    --table-name Products

The output should be:

awslocal dynamodb scan --table-name Products
{
    "Items": [
        {
            "name": {
                "S": "Super Widget"
            },
            ...
            }
        },
        {
            "name": {
                "S": "Ultimate Gadget"
            },
            ...
        }
    "Count": 2,
    "ScannedCount": 2,
    "ConsumedCapacity": null
}

Perform automated chaos testing

You can now implement a straightforward chaos test using pytest to start an outage. The test will:

• Validate the availability of Lambda functions and the DynamoDB table.

• Start a local outage and verify if DynamoDB API calls throw an error.

• Validate the ongoing outage and its appropriate cessation.

• Query the DynamoDB table for new items and assert their presence.

For integration testing, you can use the AWS SDK for Python (boto3) and the pytest framework. In a new directory named tests, create a file named test_chaos.py. Add the necessary imports and pytest fixtures:

import pytest
import time
import boto3
import requests

LOCALSTACK_ENDPOINT = "http://localhost:4566"
DYNAMODB_TABLE_NAME = "Products"
LAMBDA_FUNCTIONS = ["add-product", "get-product", "process-product-events"]

@pytest.fixture(scope="module")
def dynamodb_resource():
    return boto3.resource("dynamodb", endpoint_url=LOCALSTACK_ENDPOINT)


@pytest.fixture(scope="module")
def lambda_client():
    return boto3.client("lambda", endpoint_url=LOCALSTACK_ENDPOINT)

Add the following code to perform a simple smoke test ensuring the availability of Lambda functions and the DynamoDB table:

def test_dynamodb_table_exists(dynamodb_resource):
    tables = dynamodb_resource.tables.all()
    table_names = [table.name for table in tables]
    assert DYNAMODB_TABLE_NAME in table_names


def test_lambda_functions_exist(lambda_client):
    functions = lambda_client.list_functions()["Functions"]
    function_names = [func["FunctionName"] for func in functions]
    assert all(func_name in function_names for func_name in LAMBDA_FUNCTIONS)

Now, add the following code to chaos test the locally deployed DynamoDB table:

def test_dynamodb_outage():
    outage_payload = [{"service": "dynamodb", "region": "us-east-1"}]
    requests.post(
        "http://outages.localhost.localstack.cloud:4566/outages", json=outage_payload
    )

    # Make a request to DynamoDB and assert an error
    url = "http://12345.execute-api.localhost.localstack.cloud:4566/dev/productApi"
    headers = {"Content-Type": "application/json"}
    data = {
        "id": "prod-1002",
        "name": "Super Widget",
        "price": "29.99",
        "description": "A versatile widget that can be used for a variety of purposes. Durable, reliable, and affordable.",
    }

    response = requests.post(url, headers=headers, json=data)

    assert "error" in response.text

    # Check if outage is running
    outage_status = requests.get(
        "http://outages.localhost.localstack.cloud:4566/outages"
    ).json()
    assert outage_payload == outage_status

    # Stop the outage
    requests.post("http://outages.localhost.localstack.cloud:4566/outages", json=[])

    # Check if outage is stopped
    outage_status = requests.get(
        "http://outages.localhost.localstack.cloud:4566/outages"
    ).json()
    assert not outage_status

    # Wait for a few seconds
    time.sleep(60)

    # Query if there are items in DynamoDB table
    dynamodb = boto3.resource("dynamodb", endpoint_url=LOCALSTACK_ENDPOINT)
    table = dynamodb.Table(DYNAMODB_TABLE_NAME)
    response = table.scan()
    items = response["Items"]
    print(items)
    assert "Super Widget" in [item["name"] for item in items]

Run the test locally using the following command:

pytest

The output should be:

=========================================== test session starts ============================================
platform darwin -- Python 3.10.4, pytest-7.2.0, pluggy-1.4.0
rootdir: ...
plugins: html-3.2.0, pylint-0.19.0, json-report-1.5.0, Faker-18.4.0, cov-4.0.0, metadata-2.0.4, anyio-3.6.2, datadir-1.4.1
collected 3 items                                                                                          
collected 3 items

tests/test_outage.py ...                                                 [100%]

======================================= 3 passed in 75.86s (0:01:15) =======================================

You now have a successful outage test running on your local machine using LocalStack 🎊

You can further run the tests on a continuous integration (CI) environment, such as GitHub Actions to ensure that you can build & test your infrastructure's resilience with every commit. You can find the sample workflow on the GitHub repository.

Conclusion

Outages extension allows you to further chaos test your other resources, such as Lambda functions, S3 buckets, and more to ascertain service continuity, user experience, and the system’s resilience to the failures introduced, and how far you can go on to fix them. An ideal strategy is to design the experiments and group them in the categories of knowns and unknowns, while analyzing whatever chaos your system might end up encountering.

In the upcoming blog posts, we'll demonstrate how to perform more complex chaos testing scenarios, such as RDS & Route53 failovers, inject latency to every API call, and use AWS Resilience Testing Tools such as Fault Injection Simulator (FIS) locally. Stay tuned for more blogs on how LocalStack is enhancing your cloud development and testing experience.

You can find the code in this GitHub repository.

Continuous Integration (CI) environments are commonly used for testing CDK stacks before deploying them on the actual AWS cloud. However, configuring AWS credentials or tearing down the CDK stack post-testing requires manual setups which is often tiresome. LocalStack streamlines integration testing by allowing CDK stack deployment and testing against a cloud emulator.

This blog will guide you in creating a GitHub Action to test CDK stack deployment within a CI workflow. Additionally, we will delve into implementing a basic integration test to verify the functional aspects of the infrastructure deployed on LocalStack.

How does LocalStack work with CDK?

LocalStack runs as a Docker container either on your local machine or in an automated setting. Once started, you can utilize LocalStack alongside tools like AWS CLI or Terraform to create local AWS resources. For local deployment and testing of CDK stacks, LocalStack provides a wrapper CLI called cdklocal for utilizing the CDK library with local APIs.

To set up cdklocal, you can use the npm library with the following commands:

npm install -g aws-cdk-local aws-cdk
...
cdklocal --version
2.121.1

Internally, CDK integrates with AWS CloudFormation for infrastructure deployment and provisioning. When using cdklocal, you leverage a LocalStack-native CloudFormation engine that allows you to create the local resources, hence removing the need to deploy and test on the real cloud.

Prerequisites

• LocalStack Web Application account

• GitHub Account & gh CLI (optional)

Inventory Management System with SQS, Lambda, S3 and DynamoDB

This demo uses a public AWS example to showcase an event-driven inventory management system. The system deploys SQS, DynamoDB, Lambda, and S3, functioning as follows:

• CSV files are uploaded to an S3 bucket to centralize and secure the inventory data.

• A Lambda function reads and parses the CSV file, extracting inventory update records.

• Each record is converted into a message and sent to an SQS queue. Another Lambda function continuously checks the SQS queue for new messages.

• Upon receiving the message, it retrieves the inventory update details and updates the inventory levels in DynamoDB.

Create the GitHub Action workflow

GitHub Actions is a tool that automates workflows. It lets you make custom workflows that automatically build, test, and deploy your code when you make changes to your repository.

For this demo, you will implement a workflow that does the following:

• Checkout the repository from GitHub.

• Perform the steps to install dependencies.

• Bootstrap and deploy the CDK stack on the GitHub Action Runner.

• Run a basic integration test to verify the functionality.

To start, fork the AWS sample on GitHub. If you use GitHub's gh CLI, fork and clone the repository with this command:

gh repo fork https://github.com/aws-samples/amazon-sqs-best-practices-cdk --clone

After forking and cloning:

• Create a new directory called .github and a sub-directory called workflows.

• Create a new file called main.yml in the workflows sub-directory.

Now you're ready to create your GitHub Action workflow which will deploy your CDK stack using LocalStack's cloud emulator.

Set Up the Actions & dependencies

To achieve the goal, you can use a few prebuilt Actions:

• actions/checkout: Clone the repository for deploying the stacks.

• setup-localstack: Set up the GitHub Actions workflow with LocalStack container & localstack CLI

• setup-node: Set up the GitHub Actions workflow with NodeJS & npm.

• setup-python: Set up the GitHub Actions workflow with Python & pip.

Add the following content to the main.yml file created earlier:

name: Deploy on LocalStack 

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

This ensures that every time a pull request is raised or a new commit is pushed to the main branch, the action is triggered.

Create a new job named cdk and specify the GitHub-hosted runner executing our workflow steps, while checking out the code:

jobs:
  cdk:
    name: Setup infrastructure using CDK
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the code
        uses: actions/checkout@v4

Now, set up the step to install Python & NodeJS in the runner as part of the workflow step:

- name: Setup Node.js
  uses: actions/setup-node@v3
  with:
    node-version: 18

- name: Install Python
  uses: actions/setup-python@v4
  with:
    python-version: '3.10'

Next, set up LocalStack in your runner using the setup-localstack action:

- name: Start LocalStack
  uses: LocalStack/setup-localstack@main
  with:
    image-tag: 'latest'
    install-awslocal: 'true'
    use-pro: 'true'
    configuration: LS_LOG=trace
  env:
    LOCALSTACK_API_KEY: ${{ secrets.LOCALSTACK_API_KEY }}

This action pulls the LocalStack Pro image (localstack/localstack-pro:latest) image, installs the localstack CLI, and sets up awslocal to redirect AWS API requests to the LocalStack container. A configuration LS_LOG has been added to enable trace log level.

A repository secret LOCALSTACK_API_KEY is also specified to activate your Pro license on the GitHub Actions runner. Later in the article, you will learn the steps to configure the secret in your GitHub repository.

Finally, install other dependencies, such as CDK & cdklocal, and various Python libraries specified in requirements.txt:

- name: Install CDK
  run: |
    npm install -g aws-cdk-local aws-cdk
    cdklocal --version

- name: Install dependencies
  run: |
    pip install -r requirements.txt

Now, you are ready to deploy the CDK stack on the GitHub Action runner by specifying the appropriate CDK commands in the workflow file.

Deploy the CDK stack on LocalStack

To deploy the CDK stack, employ the cdklocal wrapper. First, ensure that each AWS environment intended for resource deployment is bootstrapped. Execute the following cdklocal bootstrap command, adjusting the AWS account ID (000000000000) and region (us-east-1) as needed:

- name: Bootstrap using CDK
  run: |
      cdklocal bootstrap aws://000000000000/us-east-1

Note that the account ID and region values can be customized for multi-account and multi-region setups in LocalStack.

Next, confirm correct stack synthesis by running cdklocal synth. If your application includes multiple stacks, specify which ones to synthesize. However, in this case with a single stack, the CDK CLI automatically detects it:

- name: Synthesize using CDK
  run: |
      cdklocal synth

Following successful synthesis, proceed to deploy the CDK stack with cdklocal deploy. To avoid manual confirmation in non-interactive environments like GitHub Actions, include --require-approval never:

- name: Deploy using CDK
  run: |
      cdklocal deploy --require-approval never

Implement integration tests against LocalStack

Now, you can implement a straightforward integration test with the following steps:

• Validate CDK outputs (cdk.out and manifest.json).

• Query the deployed S3 bucket and DynamoDB table.

• Trigger CSV processing by uploading a sample CSV file to the S3 bucket.

• Scan the DynamoDB table to confirm inventory updates.

For integration testing, you can use the AWS SDK for Python (boto3) and the pytest framework. Create a new directory called tests and create a file named test_infra.py. Add the necessary imports and pytest fixtures:

import os
import boto3
import pytest
import time


@pytest.fixture
def s3_client():
    return boto3.client(
        "s3",
        endpoint_url="http://localhost:4566",
        region_name="us-east-1",
        aws_access_key_id="test",
        aws_secret_access_key="test",
    )


@pytest.fixture
def dynamodb_client():
    return boto3.client(
        "dynamodb",
        endpoint_url="http://localhost:4566",
        region_name="us-east-1",
        aws_access_key_id="test",
        aws_secret_access_key="test",
    )

In this code, boto3 clients for interacting with the LocalStack instance are created. Two clients, s3_client and dynamodb_client, are generated, specifying the region and mock AWS Access Key ID and Secret Access Key.

Now, include the following code to execute an integration test against the deployed infrastructure:

def test_cdk(s3_client, dynamodb_client):
    # Assert CDK outputs
    assert os.path.exists("cdk.out")
    assert os.path.exists("cdk.out/manifest.json")

    # Check S3 bucket existence
    target_bucket_prefix = "sqsblogstack-inventoryupdatesbucketfe-"
    response = s3_client.list_buckets()
    target_bucket = next(
        (
            bucket["Name"]
            for bucket in response["Buckets"]
            if bucket["Name"].startswith(target_bucket_prefix)
        ),
        None,
    )
    assert target_bucket is not None

    local_file_path = "sqs_blog/sample_file.csv"
    s3_object_key = "sample_file.csv"
    s3_client.upload_file(local_file_path, target_bucket, s3_object_key)

    target_ddb_prefix = "SqsBlogStack-InventoryUpdates"
    response = dynamodb_client.list_tables()
    target_ddb = next(
        (
            table
            for table in response["TableNames"]
            if table.startswith(target_ddb_prefix)
        ),
        None,
    )
    assert target_ddb is not None
    time.sleep(10)

    # Check if there is at least one item in the DynamoDB table
    response = dynamodb_client.scan(TableName=target_ddb)
    assert response.get("Count", 0) > 0

This code uploads a sample CSV file (sqs_blog/sample_file.csv) to the local S3 bucket and checks for inserted items in the DynamoDB table. To automate running this test in a GitHub Action workflow, add the following step:

- name: Run integration tests 
  run: |
      pip3 install boto3 pytest 
      pytest

Configure a CI key for GitHub Actions

Before you trigger your workflow, set up a continuous integration (CI) key for LocalStack. LocalStack requires a CI Key for use in CI or similar automated environments.

Follow these steps to add your LocalStack CI key to your GitHub repository:

1. Go to the LocalStack Web Application and access the CI Keys page.

   

2. Switch to the Generate CI Key tab, provide a name, and click Generate CI Key.

3. In your GitHub repository secrets, set the Name as LOCALSTACK_API_KEY and the Secret as the CI Key.

   

Now, you can commit and push your workflow to your forked GitHub repository.

Run the GitHub Action workflow

With the GitHub Action Workflow in place, your CDK stack will be tested and deployed on LocalStack whenever changes are made to the main branch of your GitHub repository. 🎊

If your CDK deployment encounters issues and fails on LocalStack, you can troubleshoot by adding extra steps to generate a diagnostics report. After downloading, you can visualize logs and environment variables using a tool like diapretty:

- name: Generate a Diagnostic Report
  if: failure()
  run: |
      curl -s localhost:4566/_localstack/diagnose | gzip -cf > diagnose.json.gz

- name: Upload the Diagnostic Report
  if: failure()
  uses: actions/upload-artifact@v3
  with:
    name: diagnose.json.gz
    path: ./diagnose.json.gz

Conclusion

Testing your infrastructure code with LocalStack provides a developer-friendly experience, supporting a quick and agile test-driven development cycle. This happens without incurring any costs on the actual AWS cloud, hence no waiting around prolonged CI runs.

In the upcoming blog posts, we'll demonstrate how to inject your infrastructure state and execute application integration tests without the need for manual deployments (using CDK or Terraform). Stay tuned for more blogs on how LocalStack is enhancing your cloud development and testing experience.

You can find the GitHub Action workflow and integration test in this GitHub repository.

That said, let's skip the unnecessary headaches and look at how to correctly make requests to our S3 buckets. As always, this post is backed by an example that lives in a public GitHub repository.

TL;DR

In LocalStack, the S3 service stands out for its approach to endpoint configuration, which is distinct from all other services. Unlike the standard format used across LocalStack, the S3 service adopts a specialized format: s3.localhost.localstack.cloud.

This convention mirrors AWS S3's virtual-hosted-style of addressing behavior, facilitating a more accurate emulation of S3 interactions in a local development environment.

Path-Style vs. Virtual-Hosted-Style S3 Requests

The main difference between path-style and virtual hosting-style endpoints when accessing files in an S3 bucket lies in how the bucket name is included in the URL.

• Path-style endpoints format the URL by placing the bucket name as part of the path. The structure looks like this: http://s3.<region>.amazonaws.com/bucket-name/key-name.

• Virtual hosting-style endpoints, on the other hand, include the bucket name as a subdomain of the domain in the URL. The format is: http://bucket-name.s3.<region>.amazonaws.com/key-name. This method allows S3 to serve requests from different buckets using a single web server while making it easier to use SSL/TLS certificates tied to the bucket name as a domain. It's the preferred method for most modern applications due to its cleaner URL structure and compatibility with DNS standards.

The s3. prefix in Amazon S3 URIs serves as a component for service identification, enabling AWS to efficiently route, manage, and secure access to data stored in S3. According to the AWS documentation, this convention also supports the virtual hosting of buckets for flexible access by acting as a delimiter and allowing the service to identify the target bucket correctly.

S3 Requests in LocalStack

LocalStack, having a high parity level with AWS, also distinguishes between path-style and virtual-hosted-style requests based on the request's Host header. This means that the bucket name is part of the Host header, visible in the URL. To ensure LocalStack parses the bucket name correctly, the URL must be prefixed with s3., such as s3.localhost.localstack.cloud.

By default, most SDKs opt for virtual-hosted-style requests, automatically prefixing endpoints with the bucket name. If your endpoint doesn't start with s3., LocalStack might not process your request correctly, leading to errors. You can address this by adjusting the endpoint to use the s3. prefix or by setting your SDK to use path-style requests.

The AWS documentation also indicates that path-style requests will be discontinued in the near future. However, the SDKs currently support some method to "force path style," which needs to receive a true argument. If your endpoint does not start with s3., LocalStack treats all requests as path style by default. For consistent S3 operations, using the s3.localhost.localstack.cloud endpoint is recommended.

Example

Runs on LocalStack

Let's look at the simplest example of how to properly configure an S3 client in Java to fetch a text file from a bucket and read its content.

First, let's create an S3 bucket, give it public access, and add a text file to it.

• Create the bucket.

  aws --endpoint="http://localhost.localstack.cloud:4566" s3api create-bucket --bucket testy-mctestface-bucket

• Create the file.

  echo "Hello from the test bucket." > s3test.txt

• Add the file to the bucket.

  aws --endpoint="http://localhost.localstack.cloud:4566" s3 cp s3test.txt s3://testy-mctestface-bucket

• Programmatically getting the file and reading it.

LocalStack does not enforce IAM policies by default, so this should be enough for now.

public class S3EndpointDemo {
    public static void main(String[] args) {

        String bucketName = "testy-mctestface-bucket";
        String key = "s3test.txt";

        AwsBasicCredentials awsCreds = AwsBasicCredentials.create("test", "test");

        S3Client s3 = S3Client.builder()
                .credentialsProvider(StaticCredentialsProvider.create(awsCreds))
                .endpointOverride(URI.create("https://s3.localhost.localstack.cloud:4566"))
                .region(Region.US_EAST_1)
                .build();

        try {
            GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                    .bucket(bucketName)
                    .key(key)
                    .build();

            ResponseBytes<GetObjectResponse> objectBytes = s3.getObjectAsBytes(getObjectRequest);

            String content = new String(objectBytes.asByteArray());

            System.out.println("File content: \n" + content);
        } catch (S3Exception e) {
            System.err.println(e.awsErrorDetails().errorMessage());
            System.exit(1);
        } catch (SdkClientException | AwsServiceException e) {
            throw new RuntimeException(e);
        }

        s3.close();
    }
}

This code creates an S3 client with static credentials and a custom endpoint (https://s3.localhost.localstack.cloud:4566) to retrieve and print the content of a specific file (s3test.txt) from a bucket (testy-mctestface-bucket). In case the endpoint is misconfigured or the bucket does not exist, this will result in a The specified bucket does not exist message.

While this code runs locally and required minimal confirguration, other compute services, such as Lambda, require the same endpoint configuration.

Additionally, you can access your file content using a curl command:

• Virtual-hosted-style: curlhttp://testy-mctestface-bucket.s3.us-east-1.localhost.localstack.cloud:4566/s3test.txt

• Path-style: curlhttp://s3.us-east-1.localhost.localstack.cloud:4566/testy-mctestface-bucket/s3test.txt

Runs on AWS

• The previous commands work on AWS by removing the --endpoint flag and making the bucket public.

• Don't forget to configure your AWS CLI to use the right credentials or export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

• The S3 client will have a simpler configuration:
  S3Client s3 = S3Client.builder().region(Region.US_EAST_1).build();

LocalStack's IAM Policy Stream automates the generation of IAM policies for your AWS API requests on your local machine. This stream helps you identify the necessary permissions for your cloud application and allows you to detect logical errors, such as unexpected actions in your policies.

In this blog, we'll guide you through setting up IAM Policy Stream for a locally running AWS application. We'll use a basic example involving an SNS topic, an SQS queue, and a subscription of the queue to the SNS topic. You'll be able to generate and insert the policy without manual effort, adhering to the principle of least privilege.

Why use IAM Policy Stream?

LocalStack is a tool that lets you simulate the AWS cloud on your local machine, allowing you to run your AWS cloud and serverless applications locally. It enables you to create and enforce local IAM roles and policies using the ENFORCE_IAM feature. However, users often struggle to figure out the necessary permissions for different actions. It's important to find a balance, avoiding giving too many permissions while making sure the right ones are granted.

This challenge becomes more complex when dealing with AWS services that make requests not directly visible to users. For instance, if an SNS topic sends a message to an SQS queue and the underlying call fails, there might be no clear error message, causing confusion, especially for those less familiar with the services.

IAM Policy Stream simplifies this by automatically generating the needed policies and showing them to users. This makes it easier to integrate with resources, roles, and users, streamlining the development process. Additionally, it serves as a useful learning tool, helping users understand the permissions linked to various AWS calls and improving the onboarding experience for newcomers to AWS.

Prerequisite

• LocalStack CLI with LOCALSTACK_AUTH_TOKEN

• Docker

• AWS CLI with awslocal wrapper

• LocalStack Web Application account

• jq

Subscribing a SQS queue to a SNS topic

We've got a basic demo app featuring an SNS topic named test-topic and an SQS queue named test-queue. There's also a subscription in place. The procedure includes sending a message to SNS, which, thanks to the subscription, gets pushed into the SQS queue. With LocalStack's IAM enforcement enabled, you can thoroughly test your policy and address the IAM violations by auto-generating your policies through the IAM Policy Stream.

Start your LocalStack container

Launch the LocalStack container on your local machine using the specified command:

export LOCALSTACK_AUTH_TOKEN=...
DEBUG=1 ENFORCE_IAM=1 localstack start

Once initiated, you'll receive a confirmation output indicating that the LocalStack container is up and running.

     __                     _______ __             __
    / /   ____  _________ _/ / ___// /_____ ______/ /__
   / /   / __ \/ ___/ __ `/ /\__ \/ __/ __ `/ ___/ //_/
  / /___/ /_/ / /__/ /_/ / /___/ / /_/ /_/ / /__/ ,<
 /_____/\____/\___/\__,_/_//____/\__/\__,_/\___/_/|_|

 💻 LocalStack CLI 3.1.0
 👤 Profile: default

[11:44:39] starting LocalStack in      localstack.py:494
           Docker mode 🐳

...

──── LocalStack Runtime Log (press CTRL-C to quit) ─────
LocalStack supervisor: starting
LocalStack supervisor: localstack process (PID 18) starting

LocalStack version: 3.1.1.dev20240131022456
LocalStack Docker container id: 931fae5c27d2
LocalStack build date: 2024-02-01
LocalStack build git hash: 616ef31

Navigate to IAM Policy Stream

Access the LocalStack Web Application and go to the IAM Policy Stream dashboard. This feature enables you to directly examine the generated policies, displaying the precise permissions required for each API call.

Upon the successful launch of your LocalStack container, you'll observe the Stream active status icon, indicating that making any local AWS API request will trigger the generation of an IAM Policy. Now, let's proceed to create the SNS topic and the SQS queue.

Create the AWS resources

Create a local SNS topic with the command:

awslocal sns create-topic --name test-topic

The output will be:

{
    "TopicArn": "arn:aws:sns:us-east-1:000000000000:test-topic"
}

Go to the IAM Policy Stream dashboard, and you'll see the generated policy.

The AWS API call created an identity-based policy for the root user; If you do not set any credentials returned by IAM or STS, LocalStack will identify the request as being made by the root user. The policy is for the CreateTopic API action, allowing the specified resource.

Now, create the SQS queue with:

awslocal sqs create-queue --queue-name test-queue

The output will be:

{
    "QueueUrl": "http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test-queue"
}

On the IAM Policy Stream dashboard, you'll notice the generated policy.

Create a subscription with the topic ARN of the SNS topic, the protocol, and the notification endpoint:

awslocal sns subscribe \
    --topic-arn arn:aws:sns:us-east-1:000000000000:test-topic \
    --protocol sqs \
    --notification-endpoint arn:aws:sqs:us-east-1:000000000000:test-queue

The output will be:

{
    "SubscriptionArn": "arn:aws:sns:us-east-1:000000000000:test-topic:4283d647-18b6-4aeb-b283-19d9327a963a"
}

Testing the subscription

To test the subscription, publish a message to the SNS topic:

awslocal sns publish \
    --topic-arn arn:aws:sns:us-east-1:000000000000:test-topic \
    --message '{"some": "event"}'

The output will be:

{
    "MessageId": "63317413-c40b-41ed-982b-db722337eb5b"
}

Check if your SQS queue received the message:

awslocal sqs receive-message \
    --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test-queue

Unfortunately, there is no message due to a lack of proper IAM policy, preventing the SNS service from publishing to the SQS queue. Let's use IAM Policy Stream to resolve this issue.

Analyzing IAM Policies

Navigate to the IAM Policy Stream dashboard and observe various API calls like Publish, SendMessage, ReceiveMessage. Note that the SendMessage call was rejected due to an IAM violation.

Click on SQS.SendMessage to view the request parameters and the required resource-based policy.

LocalStack automatically suggests a resource-based policy for the arn:aws:sqs:us-east-1:000000000000:test-queue SQS queue. Copy and paste the policy into a new JSON file named policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Test432a8c7b",
      "Effect": "Allow",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:000000000000:test-queue",
      "Principal": {
        "Service": [
          "sns.amazonaws.com"
        ]
      },
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:000000000000:test-topic"
        }
      }
    }
  ]
}

Encode this policy as a string using jq, required by the AWS CLI:

jq @json < policy.json
"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"Test432a8c7b\",\"Effect\":\"Allow\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:us-east-1:000000000000:test-queue\",\"Principal\":{\"Service\":[\"sns.amazonaws.com\"]},\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"arn:aws:sns:us-east-1:000000000000:test-topic\"}}}]}"

Create a new file named sqs-queue-attributes.json and paste the generated policy:

{
    "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"Test432a8c7b\",\"Effect\":\"Allow\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:us-east-1:000000000000:test-queue\",\"Principal\":{\"Service\":[\"sns.amazonaws.com\"]},\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"arn:aws:sns:us-east-1:000000000000:test-topic\"}}}]}"
}

Set the queue attributes for the SQS queue:

awslocal sqs set-queue-attributes \
    --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test-queue \
    --attributes file://sqs-queue-attributes.json

Send another message to the SNS topic to be received by the SQS queue:

awslocal sns publish \
    --topic-arn arn:aws:sns:us-east-1:000000000000:test-topic \
    --message '{"some": "event"}'
awslocal sqs receive-message \
    --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test-queue

Verify that you have received the message:

{
    "Messages": [
        {
            "MessageId": "84a24f67-995b-45db-87d7-f6fca8891f4e",
            "ReceiptHandle": "MDUwNTAwYzQtMTJiNS00NzdiLTg2OTYtODA5MjAzMWQ3YzY1IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6dGVzdC1xdWV1ZSA4NGEyNGY2Ny05OTViLTQ1ZGItODdkNy1mNmZjYTg4OTFmNGUgMTcwNzcyMDg2My45OTEwMDQ=",
            "MD5OfBody": "a85e018e2a7d2d06866b7da00268fcc9",
            "Body": "{\"Type\": \"Notification\", \"MessageId\": \"4dbec68e-2489-4e73-bee4-90d02ffe691f\", \"TopicArn\": \"arn:aws:sns:us-east-1:000000000000:test-topic\", \"Message\": \"{\\\"some\\\": \\\"event\\\"}\", \"Timestamp\": \"2024-02-12T06:54:20.269Z\", \"UnsubscribeURL\": \"http://localhost.localstack.cloud:4566/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:000000000000:test-topic:4283d647-18b6-4aeb-b283-19d9327a963a\", \"SignatureVersion\": \"1\", \"Signature\": \"ymKIdXa+KzAy5aZ6XAA7P1TM7azzCounaFv4etpm7GL7qHawdn86aeM6q7VhDgTzCBdI3iGEjOaoAzaWCnB1RdQd9rt8Gfwckk0QtlGefEJBVdiH1DCNyGD+A48hSGUPtAk22d0Ar1AzBWtQ49DFTbfgEqfGGNlPdrri+JJmfztgg7hb0tUZoeWM3p7wfNuj7+nXGtS4JuVf5yC0f/v6ryo0IbNiGEjfsXyAU5++Lx2V3o2aZK/WfUWa5EkIqjAc6RzmnIE60IUYvn/7mVEgdl5CZLvJ0hPGrLaxFwdMd04LiYJCE4bLMfWxDyVQFysKH8GwFqRfZjrYmpdiVtI9Rw==\", \"SigningCertURL\": \"http://localhost.localstack.cloud:4566/_aws/sns/SimpleNotificationService-6c6f63616c737461636b69736e696365.pem\"}"
        }
    ]

You can now successfully verify on the IAM Policy Stream dashboard that no violation has been noticed, and your AWS API requests have successfully executed with the right IAM policies.

Generating a comprehensive policy

In scenarios where there are many AWS services, and every AWS API request generates a policy it might be cumbersome to analyze every policy. In such cases, you can generate one comprehensive policy for all your AWS resources together.

You can navigate to the Summary Policy tab on the IAM Policy Stream dashboard. This concatenates the policy per principle which the policy should be attached to. For the example above, you would be able to see the Identity Policy for the root user which has all the actions and resources inside one single policy file for the operations we performed.

On the other hand, you have the Resource Policy for the SQS queue, where you can see the permission necessary for the subscription. For larger AWS applications, you would be able to find multiple roles and multiple resource-based policies depending on your scenario.

Conclusion

IAM Policy Stream streamlines your development process by minimizing the manual creation of policies and confirming the necessity of granted permissions. However, it is advisable to manually confirm that your policy aligns with your intended actions. Your code may unintentionally make requests, and LocalStack considers all requests made during policy generation as valid.

A practical scenario is automating tests like integration or end-to-end testing against your application, allowing LocalStack to automatically generate policies with required permissions. You can then review and customize them to meet your needs, ensuring that overly permissive policies don't find their way into production environments.

We are actively working on expanding this feature and offering more advantages for developers, such as automatically analyzing deployed policies for unused permissions and more! Stay tuned for updates on our IAM feature set!

As I approach my one-year anniversary with LocalStack, it's remarkable to reflect on the transformative journey we've embarked upon. The community I joined is not the same one it is today; it's grown exponentially, becoming bigger, stronger, and more vibrant. In this blog post, I aim to shed light on the incredible strides we've made in community growth, developer advocacy, and user support. We've also evolved significantly in how we present (ourselves) to our users, marking a year of meaningful progress.

While our journey this past year has been filled with amazing achievements, it's important to acknowledge that not every moment was rosy and celebratory. We encountered our share of hurdles, with some projects proving too ambitious to launch as initially envisioned. However, these experiences are not setbacks but stepping stones. What I really appreciate at LocalStack is the academic approach to starting projects: we pitch ideas, we present the plan, we discuss, we ask questions, we execute, or go back to the whiteboard to act on the feedback. Whatever didn't work in the past will come back stronger and better.

This has been one year in the rocket ship.

How it's going

For the looks of LocalStack

Let's kick things off with a bit of a time travel exercise. Picture this: as I was digging up some gems for this article, I took a scroll through our old social media posts. Talk about a 'throwback' moment! It was almost like flipping through an old family album, seeing our early design days, seeing my colleagues before I actually knew them. It's pretty wild to see how far we've come.

So when we have new releases, we no longer look like this:

But rather like this...

Don't get me wrong, they're both amazing, but just like in the world of software (and let's not forget rockets), each upgrade is leveling up - it just keeps getting cooler and cooler. This is also our way of showing how much we value the community's support and opinion, which are like rocket fuel, pushing us to soar higher and do even better.

For the numbers

As we look back on 2023, it’s thrilling to see the leaps our social media presence has made. Our digital footprint tells the story of our community's steady engagement and growth over the year.

On Slack, we witnessed a remarkable 265% increase in active members alongside a 105% growth in our community size. This surge reflects the dynamic conversations and collaborations happening among our members. We can only go so far with helping each individual situation, so we strive to foster an environment of helping out and knowledge sharing between the people.

YouTube has been another spectacular success story. Our channel's views skyrocketed by 766% to 28.4K, while the watch time astonishingly increased by 495%. These numbers don’t just speak volumes; they shout out about the engaging and valuable content we’ve been producing. Not to mention, our subscriber count grew by 386%, and we garnered 330K impressions, clearly expanding our reach.

In the realm of Discuss, our platform has been buzzing with activity. Over 550 new posts were created by more than 150 new contributors, generating a whopping 150K consolidated page views.

As Twitter/X provides a different set of tools to do analytics, we can currently say that we've gained 356 new followers since August and 273,5K impressions in 2023.

Since our first community blog post, a little over a year ago, our project has seen tremendous growth, surpassing 50,000 GitHub stars and over 194M Docker pulls. These numbers are not just metrics; they are a reflection of the trust and confidence our users place in us. And if you really want to be up to date with it all, you should definitely read about everything that's new in LocalStack v3.

For the bond created with our users

We've seen how the LocalStack community has grown with every milestone. Every step we've taken and every initiative we've launched has been welcomed with enthusiasm by our users. Open source contributors have become colleagues and acquaintances turned into supporting friends.

Expanding Our Reach Through Webinars and Meetups

This year, we organized 8 engaging community webinars featuring esteemed guests from Wing, FABR, and XO. These events were a great success, contributing to an astounding 145% increase in the size of our Meetup group. It's been a delight to see our user base grow and engage more deeply with LocalStack. Of course, we could go on forever about how much we like it.

Making a Mark at Conferences and Summits

LocalStack's presence was felt strongly across major tech events. We had the privilege of presenting at 10 key conferences, including DockerCon, EuroPython, PyCon APAC, SpringIO, and DevFest, as well as hosting 2 webinars with LambdaTest and Lumigo. Additionally, our participation in various AWS Summits across Europe and the Middle East helped us reach out to even more cloud enthusiasts and developers. I know we always say we're local, but in this case, we're going global.

Hosting In-Person Events

We went a step further in fostering community connections by organizing three in-person "Cloud DevXchange" events in Bangalore, Chennai, and Vienna. Collaborating with Docker, MongoDB, Labyrinth Labs, and AWS, these events were a melting pot of ideas, innovation, and networking. Additionally, we had the honor of being featured speakers at several gatherings, including the Enterprise Java User Group, the Cloud Native Computing Foundation (CNCF), the Elastic User Group, and various DevOps meetups.

Driving Engagement Through Content

Our blog became a hub of knowledge sharing, with 22 new posts that attracted over 60,000 views. Highlighting significant updates like LocalStack 2.0 and 3.0 GA, and introducing new tools like our Docker Extension and Desktop App, we kept our community informed and engaged.

Collaborating with Industry Leaders

2023 was also a year of fruitful collaborations with Docker, AWS, Cloudflare, LambdaTest, AtomicJar, and Pulumi, among others. These partnerships have been instrumental in expanding our reach and enhancing the LocalStack experience through tools our users already love.

Launching the Developer Hub

Perhaps one of our proudest achievements this year is the launch of the all-new Developer Hub. Featuring 7 tutorials, 21 application samples, and the inception of LocalStack Academy with 7 comprehensive lessons, the Developer Hub has become a cornerstone for learning and development for the curious ones.

Finally, I'd like to share something special with you – a collage that captures the essence of the LocalStack team's dedication. This late-night piece of art is a snapshot of our unwavering commitment to engaging, informing, and supporting our community in every way possible. It's visual evidence of the hard work, creativity, and passion that fuels our team's efforts to connect with each and every one of you, whether in person or virtually.

How will it go

As we set our sights on 2024, we're practically buzzing with gratitude and excitement! It's like our community's growth and our shared achievements have strapped a jetpack to our backs, propelling us toward continuous innovation. We're super committed to upping our game, deepening those awesome connections, and scaling even higher peaks together. And hey, we know we can't always be everywhere for everyone, but we're giving it our all and then some. In 2024, we're rolling up our sleeves to support you even better - let's make it a year to remember!

I received this question a few times and thought it was finally time to put it to the test: Can you create X Lambdas on LocalStack?

Let’s see what happens when we want to create 50 Lambdas and invoke them sequentially…twice. It’s for science, ok?

The code and the scripts for creating and invoking the functions can be found, as always, in the Stack Bytes repository.

The setup is pretty basic: We have a Lambda function that takes an input and prints out a message using said input. To make it less boring and more unique, each function will receive its own number. We use the local AWS CLI to create all the functions using a loop, meaning this will take a while. So let’s take out our time measuring devices and see:

I was surprised it only took approximately two minutes to create 50 Lambdas. I don’t want to know how long this would take on AWS. But while we’re on the topic, a bit of a disclaimer: my setup is, by today’s standards (Aug. 2023), on the middle-upper side. There are people out there getting by with a lot less and others with a lot more, but as a developer, I’d say it’s a good place to be in:

Now think of all the things one can do with all the fancy new chips.

Let’s move on and invoke all 50 of our Lambdas:

A little discouraging and slightly disappointing that it wasn’t exactly the 5-minute mark. And if we check the logs of a randomly picked container, the message is there as expected:

The slow invocations happen because the Lambda environment provisions resources and initializes the runtime environment to execute the function code. For Java Lambda functions, this includes starting the JVM, loading classes, and performing any initialization tasks specified in your code or dependencies.

Once the Lambda function is "warmed up," meaning the runtime environment is initialized, subsequent invocations of the function are typically faster because the JVM is already running. Let’s check again with the warm starts, the second round of invocations:

Now that’s more like it. And with the dawn of the CRaC project, I’m really excited to leave the cold starts in the past.

PS: I’m looking forward to reading your stories of taking things even further ;).

localstack  | 2023-08-24T20:56:47.079 ERROR --- [Executor-1_0] l.services.lambda_.hints   : 
Failed to pull Docker image because Docker is not available in the LocalStack container but
required to run Lambda functions. Please add the Docker volume mount 
"/var/run/docker.sock:/var/run/docker.sock" to your LocalStack startup.
 https://docs.localstack.cloud/user-guide/aws/lambda/#docker-not-available

Let’s break it down a bit and see how we got here. To optimize LocalStack as much as possible, every service that is available as an image will run in its own container. You’ll be able to see this with services like Lambda, ECS, certain databases, while resources like S3, API Gateway, etc., will continue to be part of the LocalStack main container.`

There are two striking ways of achieving this behaviour, known around the Internet as Docker-in-Docker and Docker-out-of-Docker. Jérôme Petazzoni adeptly articulates what lies behind both methods in this great blog post and why you’d wanna use one over the other.

DinD involves creating a separate Docker runtime environment within a container. This means that inside the container, you're essentially running another Docker daemon, isolated from the host's Docker daemon. This approach offers isolation and encapsulation, but it comes with its own set of challenges: security vulnerabilities, risk of data corruption, and overall the complexity might outweigh the isolation benefits.

In the DooD approach, you use the Docker daemon from the host system to interact with containers. Containers themselves don't have their own Docker runtime; they communicate with the host's Docker. This offers some distinct advantages: simplicity in managing the containers and resource efficiency, as containers don't need to run their own Docker daemon.

This way the main container will have access to the Docker socket and will, therefore, be able to start containers. The only difference is that instead of starting “child” containers, it will start “sibling” containers.

So, if you’re reading this on the go, next time you set up your docker-compose file you’ll hopefully visualize these diagrams and remember to add that simple YAML line before you realize you’re missing services ;).

These are also the great advantages of LocalStack shipping as a Docker image.

Let’s say your product is a multi-functional web application that does all sorts of operations for your clients. The backend is handled by AWS-managed services, so your main focus is providing the best user experience possible. For the development phase, your setup would look something like this diagram, the center of attention being the React app, while the rest are necessary dependencies:

A new frontend engineer is joining the team and they want to get their setup running as fast as possible so that they can start making much-needed improvements to the app. Of course, they would say that talk is cheap, they want to see the code:

1. Clone the repository:

    git clone https://github.com/tinyg210/stack-bytes-apigw-lambda-s3.git
   

2. Switch to the module folder:

    cd stack-bytes-apigw-lambda-s3/frontend-client-local-machine
   

3. Run the preconfigured docker-compose file in detached mode:

    docker compose up -d
   

4. Install the React app dependencies:

    npm install
   

5. Run the web application:

    npm start
   

That’s it. In just five easy steps your new developer can start understanding and working on the app. No need for days of waiting to obtain all the necessary credentials and permissions for a new AWS account and no accidental cost spikes for learning purposes.

We can now see that our sophisticated app is running on localhost:3000. It is communicating with the API Gateway, which uses 2 separate Lambdas for creating and fetching, and an S3 bucket for storing the text files that contain the quotes that we want to remember forever. The reason why everything was so fast is a combination of features that are there to enhance the developer experience. The docker-compose file simply requests the LocalStack image, makes sure the right ports are exposed, and adds some configs regarding the network and access to the Docker socket (we’ll talk about it soon). But the more important parts lay in these lines:

volumes:
      - "../stack-bytes-lambda/target/apigw-lambda.jar:/etc/localstack/init/ready.d/target/apigw-lambda.jar"
      - "../init-resources.sh:/etc/localstack/init/ready.d/init-resources.sh"

These are the init hooks that ensure the needed resources are created at startup.

From here on, the endpoints are quickly configured for local development, and everything is set:

Notice how fast getting someone onboarded is. This combination facilitates swift resource replication across multiple environments, enabling rapid team integration and productive collaboration. 👍

This is where Testcontainers and LocalStack work beautifully together to bring you the best of integration tests and cloud services on your machine and in your CI/CD pipeline.

Today, we’re discussing the ease of setting up a workflow that will always make sure our system behaves as expected. Even visually, in a diagram, beyond the bigger picture, we can see that two key areas go hand in hand: setting up the right infrastructure and holding that in check using tests.

You can find the full working example in the Stack Bytes repository, the workflow, as is standard, will be under .github/workflow, and the test examples sit in the github-actions-testcontainers folder.

When you make changes to your application's code and push it to GitHub, GitHub Actions automatically kicks in. It takes your code and starts testing it against different scenarios, just like trying different puzzle pieces. This is especially useful for end-to-end testing, where you want to see if the entire application works well together. In this case we’re interested in testing the Lambda functions, but there are so many other apps that we can plug in.

With Testcontainers, you will set up a "sandbox" environment to put the puzzle pieces together, in our case, the AWS services we need. GitHub Actions runs your tests, simulating real user interactions. It's like a rehearsal before the big show – making sure everything runs smoothly before it's in front of your users. Another great aspect of using Testcontainrs with LocalStack is that all the steps leading up to the tests are already taken care of: Testcontainers manages the lifecycle of LocalStack, while provisioning the infrastructure is the same as in production, weather it’s initialisation hooks, Terraform, CDK, or CLI.

Here are some of the things that will make your life easier when you’re using Testcontainers in CI or on your machine:

• Use a waiter to make sure your Lambdas are ACTIVE and not just created:

    LambdaWaiter waiter = lambdaClient.waiter();
        GetFunctionRequest getFunctionRequest = GetFunctionRequest.builder()
            .functionName("create-quote")
            .build();
        WaiterResponse<GetFunctionResponse> waiterResponse = waiter.waitUntilFunctionActiveV2(
            getFunctionRequest);
        waiterResponse.matched().response().ifPresent(response -> LOGGER.info(response.toString()));
  

• Use this nifty configuration to scan the LocalStack logs and make sure your instance is in the right state before the tests are allowed to start:

    protected static LocalStackContainer localStack =
          new LocalStackContainer(DockerImageName.parse("localstack/localstack-pro:2.2.0"))
    ........
    .waitingFor(Wait.forLogMessage(".*Finished creating resources.*\\n", 1));
  

• It’s been recently discovered that some Lambda containers are not removed when the tests end. Don’t worry, a fix is on the way, but in the meantime, you can use a dedicated function to clean up at the end of the test suite (only use this on your machine):

    protected static void cleanLambdaContainers() {
        try {
          String scriptPath = "src/test/resources/delete_lambda_containers.sh";
          ProcessBuilder processBuilder = new ProcessBuilder(scriptPath);
          processBuilder.inheritIO();
          Process process = processBuilder.start();
          int exitCode = process.waitFor();
          System.out.println("Script exited with code: " + exitCode);
        } catch (IOException | InterruptedException e) {
          e.printStackTrace();
        }
      }
  

    #!/bin/bash
  
    # get a list of running container ids with the word "lambda" in their names
    container_ids=$(docker ps -q --filter name=lambda)
  
    # loop through the ids and stop and remove each container
    for id in $container_ids; do
        echo "Stopping and removing container: $id"
        docker stop $id
        docker rm $id
    done
  

Well, buckle up because what we have here is a full-blown visual Byte. We’re exploring a lesser-known instrument in the LocalStack toolbox: the web application. This isn't just your run-of-the-mill admin dashboard – oh no. On top of all the administrative things like managing your account details, subscription, etc, the web app visually interrogates your LocalStack instance on its activity and sees its every twist and turn.

As soon as you land on the dashboard on the lower side, you can find the Stack Insights:

Right within your reach lies the complete history of all those LocalStack instances you've ever conjured:

We’re currently running our Stack Bytes sample application consisting of an API Gateway, two Lambdas, and an S3 bucket. When we select the active stack, we see the comprehensive charts of API calls, service invocations, and clients used for them. You’ll also have access to the full operation-per-service list where you can see their corresponding details.

Due to constant improvements, these dashboards have been optimized to use an aggregation script that aims to save ~80-90% of the traffic by bundling and sending the information. That means operations won’t show up instantly but rather after a few seconds.

Another thing to keep in mind is that invocations executed in the context of an internal API call (the case where one API uses a boto3 client to call another API internally) will not appear in these dashboards. So, in our examples, our Lambda invocations will not be present because they happen as part of the API Gateway integration.

Above and beyond, let me tell you, the web app doesn't stop at that. It's also your real-time radar for all things service-related. Whether it's the availability overview or the running service status, this System Status has got you covered.

And the Resource Browser? It's like your user-friendly compass, guiding you through neatly categorized vistas of your AWS services.

We’re not done yet. From here, you can dive into individual services and see all the nooks and crannies of your stack. In our example, we can see that all the quotes of our beloved characters are there, stored neatly, as individual text files:

Let's talk about those Lambdas, shall we? They're not just functions anymore; they're practically an open book here, revealing all their configurations.

Hopefully, this incursion in pictures will encourage you to explore even more services and options in the LocalStack web application, and in the meantime, you can check out the docs.

In the context of infrastructure testing and system interaction checks, GitHub Actions can be used to automate and streamline essential tasks, such as infrastructure testing, integration testing, and end-to-end testing.

The infrastructure we have:

Let’s say that all of these resources are configured using Terraform. It may not look like much, but the file creating these services can get lengthy, so monitoring it and ensuring that everything is there might be hard. Not to mention, there are roles and permission policies at play that we don’t often see. Luckily, there’s a way to set up a mechanism that has repetitive actions with predictable outcomes, which will be our safety check that the services we need are always there. We achieve this through GitHub Actions.

TL;DR: GitHub Actions is a powerful CI/CD (continuous integration and continuous deployment ) platform that allows you to automate various workflows, such as building, testing, and deploying your code directly from your GitHub repository.

A workflow configuration file specifies the steps and actions to be executed whenever a certain event occurs, such as a push to the repository. GitHub Actions will automatically detect the new workflow configuration and execute the defined steps whenever a push event occurs. The workflow status and logs will be visible in the "Actions" tab of your GitHub repository.

Our entire YAML file can be found in the https://github.com/tinyg210/stack-bytes-apigw-lambda-s3 repository under the .github/workflows folder, where it will be automatically picked up from.

Whenever we build a workflow, there are key aspects that should not go unnoticed:

Name and Purpose: A clear and self-explanatory title that can answer the following: What does it do? Why is it useful? Create and Verify Infrastructure on LocalStack - a concise yet descriptive sentence.

Workflow Triggers: Indicate when and under what conditions the action is triggered. We’ll run our checks on each push against the main branch but ignore changes to the README.md file, as those are irrelevant here.

Environment: Define the environment or context in which the action runs. We’re storing our LOCALSTACK_API_KEY as a secret that GitHub will encrypt and manage.

Steps and Actions: The steps or actions composing the workflow. The steps we need to have in our infrastructure-check workflow are actually quite simple, and GitHub Actions uses YAML-based configuration files to define this process:

Dependencies: If the action depends on other services, tools, or resources, we need to bring them in for successful execution and prepare the workflow context. We start with runner (VM executing the workflow) - ubuntu-latest , and we also set up a JDK, install Python, aws-local CLI, terraform-local CLI, etc. In our diagram, we’re interested in what happens after the dotted line. Anything before that is just preparing the ground.

Error Handling: How the action handles errors or failures. In this case, It’s enough to check the logs and the messages they’re presenting, but also get a diagnose report from a LocalStack endpoint, in case that’s where the process failed.

By covering these aspects in your GitHub Action description, you'll provide your team with a clear understanding of what the action does, how to use it, and what to expect when incorporating it into other workflows. This is how you ensure that your dependencies are in place every time.

As promised, LocalStack can be a drop-in replacement for the most popular AWS services, which means there are many ways of easily configuring your clients to point to a different endpoint.

These client libraries allow developers to interact with various AWS/LocalStack services and APIs more easily and efficiently from within your preferred programming language. Each SDK typically provides a set of APIs, classes, and methods that abstract the low-level details of making HTTP requests and handling authentication, making it easier to integrate AWS services into your applications.

The TL;DR part: Here are some easy ways you can configure an S3 client in a few different other languages. Notice how similar they are and how you can always transition between AWS and LocalStack with just a few variables:

This will be a follow-along type of article :). Let’s have a look at an S3 Java client.

1. Clone the repository.

    git clone <https://github.com/tinyg210/stack-bytes-apigw-lambda-s3.git>
   

2. Export your LOCALSTACK_API_KEY as an environment variable.

    export LOCALSTACK_API_KEY=<YOUR_API_KEY>
   

   *Sidenote: make sure there’s apigw-lambda.jar in the /stack-bytes-lambda/target/ folder. If not, or if anything fails, please run mvn clean package shade:shade in the stack-bytes-lambda folder.

3. Start LocalStack (we stay in the root folder).

    docker compose up
   

   Let’s have a look at our setup’s diagram:

   

   Notice how we have a small Java app, with an S3 client (in the S3Configs class) on the right-hand side.

4. Switch to the stack-bytes-sdk folder where the new app resides.

    cd stack-bytes-sdk
   

    mvn clean package
   

   The S3 Java Client:

    private static final String ACCESS_KEY = "test";
    private static final String SECRET_KEY = "test";
   
    private static final String LOCALSTACK_ENDPOINT = "https://s3.localhost.localstack.cloud:4566";
   
    private static Region region = Region.US_EAST_1;
      protected static final String BUCKET_NAME = "quotes";
      // create an S3 client
      protected static S3Client s3Client = S3Client.builder()
          .endpointOverride(URI.create(LOCALSTACK_ENDPOINT))
          .credentialsProvider(StaticCredentialsProvider.create(
              AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY)))
          .region(region)
          .build();
   

   The Java SDK client has an easy and intuitive way of adding the necessary configurations so that everything can be replaced with real values and continue to work on the AWS platform.

5. Let’s post a file to the S3 bucket using our new client in the S3PostRequest class:

    mvn exec:java -Dexec.mainClass="s3service.S3PostRequest"
   

   This will take an existing file in the src/main/resources folder and add it to the quotes bucket.

6. Let’s read it now, using the same client:

    mvn exec:java -Dexec.mainClass="s3service.S3GetRequest"
   

    Object text: Author: Fiona
    Quote: I want what any princess wants - to live happily ever after... with the ogre I married.
   

7. Now, let’s retrieve the same object using a different client and the dedicated Lambda function:

    curl --location 'http://id12345.execute-api.localhost.localstack.cloud:4566/dev/quoteApi?author=Fiona'
   

    {"text":"Quote: I want what any princess wants - to live happily ever after... with the ogre I married."}
   

The output formats differ, as this is the processed output of the GET Lambda, but essentially, they are the same.

For this case, we used the Java client, but don’t worry, the LocalStack Developer Hub is packed with examples using different other programming languages.

Nothing can stop you from using LocalStack as a drop-in replacement for AWS now. With minimal configurations, your applications won’t know the difference.

Gone are the days of mundane infrastructure setup and data restore routines at every startup of LocalStack. Cloud Pods eliminate all those reboot worries, making it a breeze to set the stage for your top-priority work.

We’ll get to a follow-along example in a minute, but first, let’s have a look at how things work.

Instead of simply restoring a state when restarting LocalStack, Cloud Pods allow you to take snapshots of your local instance (with the save command) and inject such snapshots into a running instance (with the load command) without requiring a restart.

In addition, we provide a remote storage backend that can be used to store the state of your running application and share it with your team members.

Now, let’s explore how this is done.

1. Clone the repository.

    git clone https://github.com/tinyg210/stack-bytes-apigw-lambda-s3.git
   

2. Export your LOCALSTACK_API_KEY as an environment variable.

    export LOCALSTACK_API_KEY=<YOUR_API_KEY>
   

   *Sidenote: make sure there’s apigw-lambda.jar in the /stack-bytes-lambda/target/ folder. If not, or if anything fails, please run mvn clean package shade:shade in the stack-bytes-lambda folder.

3. Start LocalStack:

    docker compose up
   

4. Let’s add some Quotes to our S3 storage:

    curl --location 'http://id12345.execute-api.localhost.localstack.cloud:4566/dev/quoteApi' \
                                        --header 'Content-Type: application/json' \
                                        --data '{
                                        "author": "Shrek",
                                        "text": "NO! You dense, irritating, miniature beast of burden! Ogres are like onions!"
                                    }'
    curl --location 'http://id12345.execute-api.localhost.localstack.cloud:4566/dev/quoteApi' \
                                        --header 'Content-Type: application/json' \
                                        --data '{
                                        "author": "Donkey",
                                        "text": "And in the morning...I\'m making waffles!"
                                    }'
   

5. Log in to the localstack CLI tool with your registered account and password to get the full Cloud Pod experience. You can only create a snapshot on your machine if you are not logged in.

    localstack login
   

6. Now that we have a reasonable amount of data, let’s save our state:

    localstack pod save cloud-pod-quotes
   
    Cloud Pod cloud-pod-quotes successfully exported.
   

7. We can confidently shut down our containers.

    docker compose down
   

8. Verify the pod is there for your team to see.

    localstack pod list
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ local/remote ┃ Name                                 ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │    remote    │ cloud-pod-quotes                     │
   

9. Let’s start a fresh instance of LocalStack, this time using the CLI tool.

    localstack start
   

10. Load the pod into the LocalStack container.

    localstack pod load cloud-pod-quotes
    

11. Verify that the state and data are there:

    curl --location 'http://id12345.execute-api.localhost.localstack.cloud:4566/dev/quoteApi?author=Donkey'
    
    {"text":"Quote: And in the morning...I'm making waffles!"}
    

    curl --location 'http://id12345.execute-api.localhost.localstack.cloud:4566/dev/quoteApi?author=Shrek'
    
    {"text":"Quote: NO! You dense, irritating, miniature beast of burden! Ogres are like onions!"}⏎
    

Smooth sailing all the way! We did a little state and data magic and tucked them neatly into a Cloud Pod. Now, it's our secret weapon – perfect for sprinkling LocalStack goodness into those repetitive acts like CI tests. And hey, sharing is caring, right? So, we pass this Cloud Pod gem to our colleagues for a front-row seat to our LocalStack brilliance.