In my recent adventures with LocalStack, I've repeatedly been stumbling upon a question that sheds light on a critical yet frequently overlooked aspect of its configuration. This oversight can lead to puzzling issues that leave users perplexed, searching for answers that, surprisingly, are often hidden in plain sight. It's a reminder of the importance of paying attention to the finer details, especially when they hold the key to unlocking the full potential of this tool.
That said, let's skip the unnecessary headaches and look at how to correctly make requests to our S3 buckets. As always, this post is backed by an example that lives in a public GitHub repository.
TL;DR
In LocalStack, the S3 service stands out for its approach to endpoint configuration, which is distinct from all other services. Unlike the standard format used across LocalStack, the S3 service adopts a specialized format: s3.localhost.localstack.cloud.
This convention mirrors AWS S3's virtual-hosted-style of addressing behavior, facilitating a more accurate emulation of S3 interactions in a local development environment.
Path-Style vs. Virtual-Hosted-Style S3 Requests
The main difference between path-style and virtual hosting-style endpoints when accessing files in an S3 bucket lies in how the bucket name is included in the URL.
Path-style endpoints format the URL by placing the bucket name as part of the path. The structure looks like this: http://s3.<region>.amazonaws.com/bucket-name/key-name.
Virtual hosting-style endpoints, on the other hand, include the bucket name as a subdomain of the domain in the URL. The format is: http://bucket-name.s3.<region>.amazonaws.com/key-name. This method allows S3 to serve requests from different buckets using a single web server while making it easier to use SSL/TLS certificates tied to the bucket name as a domain. It's the preferred method for most modern applications due to its cleaner URL structure and compatibility with DNS standards.
The s3. prefix in Amazon S3 URIs serves as a component for service identification, enabling AWS to efficiently route, manage, and secure access to data stored in S3. According to the AWS documentation, this convention also supports the virtual hosting of buckets for flexible access by acting as a delimiter and allowing the service to identify the target bucket correctly.
S3 Requests in LocalStack
LocalStack, having a high parity level with AWS, also distinguishes between path-style and virtual-hosted-style requests based on the request's Host header. This means that the bucket name is part of the Host header, visible in the URL. To ensure LocalStack parses the bucket name correctly, the URL must be prefixed with s3., such as s3.localhost.localstack.cloud.
By default, most SDKs opt for virtual-hosted-style requests, automatically prefixing endpoints with the bucket name. If your endpoint doesn't start with s3., LocalStack might not process your request correctly, leading to errors. You can address this by adjusting the endpoint to use the s3. prefix or by setting your SDK to use path-style requests.
The AWS documentation also indicates that path-style requests will be discontinued in the near future. However, the SDKs currently support some method to "force path style," which needs to receive a true argument. If your endpoint does not start with s3., LocalStack treats all requests as path style by default. For consistent S3 operations, using the s3.localhost.localstack.cloud endpoint is recommended.
Example
Runs on LocalStack
Let's look at the simplest example of how to properly configure an S3 client in Java to fetch a text file from a bucket and read its content.
First, let's create an S3 bucket, give it public access, and add a text file to it.
LocalStack does not enforce IAM policies by default, so this should be enough for now.
public class S3EndpointDemo {
public static void main(String[] args) {
String bucketName = "testy-mctestface-bucket";
String key = "s3test.txt";
AwsBasicCredentials awsCreds = AwsBasicCredentials.create("test", "test");
S3Client s3 = S3Client.builder()
.credentialsProvider(StaticCredentialsProvider.create(awsCreds))
.endpointOverride(URI.create("https://s3.localhost.localstack.cloud:4566"))
.region(Region.US_EAST_1)
.build();
try {
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
ResponseBytes<GetObjectResponse> objectBytes = s3.getObjectAsBytes(getObjectRequest);
String content = new String(objectBytes.asByteArray());
System.out.println("File content: \n" + content);
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
} catch (SdkClientException | AwsServiceException e) {
throw new RuntimeException(e);
}
s3.close();
}
}
This code creates an S3 client with static credentials and a custom endpoint (https://s3.localhost.localstack.cloud:4566) to retrieve and print the content of a specific file (s3test.txt) from a bucket (testy-mctestface-bucket). In case the endpoint is misconfigured or the bucket does not exist, this will result in a The specified bucket does not exist message.
While this code runs locally and required minimal confirguration, other compute services, such as Lambda, require the same endpoint configuration.
Additionally, you can access your file content using a curl command:
Runs on AWS
The previous commands work on AWS by removing the --endpoint flag and making the bucket public.
Don't forget to configure your AWS CLI to use the right credentials or export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
The S3 client will have a simpler configuration:
S3Client s3 = S3Client.builder().region(Region.US_EAST_1).build();
