[AWS SDK for Java] Set a retry policy on the S3 client

The information I just wanted was not in Japanese, so I will document it. When uploading a file to S3 using the AWS SDK for Java, add some detailed settings.

table of contents

--Case outline around retry --I will implement it for the time being --Organization of related classes --Other --Reference

Case outline around retry

I want to upload a file to S3 locally using the AWS SDK for Java. Then, in anticipation of a case where communication is interrupted due to the AWS side or it cannot operate normally due to various reasons, I would like to prepare a retry policy in the AWS client. This time, we will prepare a utility class that returns a singleton of ʻAmazonS3` objects included in the AWS SDK for Java.

In this article, I will focus on "option settings when creating a client", which I was particularly troubled with. Therefore, the description related to the processing before and after the file upload is omitted.

I will implement it for the time being

Build a client packed with simple client options. As a flow, we will prepare three, a retry policy instance, a client setting instance, and a client.

First, prepare a method that returns a RetryPolicy instance that contains basic policy settings and AWS-defined strategies.

AmazonS3Utils.java


/**
 *Put the appropriate values for baseDelay and maxBackoffTime in the constructor of ExponentialBackoffStrategy.
 *Note: Here, an appropriate value is entered as a sample.
 */
private static RetryPolicy getS3BaseRetryPolicy() {
        return new RetryPolicy(
                new PredefinedRetryPolicies.SDKDefaultRetryCondition(),
                new PredefinedBackoffStrategies.ExponentialBackoffStrategy(0, 0),
                PredefinedRetryPolicies.DEFAULT_MAX_ERROR_RETRY,
                true
        );
    }

Next, call the method that returns the RetryPolicy instance defined in ↑ to define the client settings.

AmazonS3Utils.java


private static final ClientConfiguration clientConfiguration = new ClientConfiguration()
            .withRetryPolicy(getS3BaseRetryPolicy());

Then, initialize the ʻAmazonS3` object, which is the entry point of the S3 client called by other classes and methods, with the client settings.

AmazonS3Utils.java


private static final AmazonS3 amazonS3 = AmazonS3ClientBuilder
            .standard()
            .withClientConfiguration(clientConfiguration)
            .withRegion(Regions.DEFAULT_REGION)
            .build();

To summarize the settings so far,

AmazonS3Utils.java


public final class AmazonS3Utils {

    /**
     *Initialize the client settings to reflect the retry policy.
     */
    private static final ClientConfiguration clientConfiguration = new ClientConfiguration()
            .withRetryPolicy(getS3BaseRetryPolicy());

    /**
     *Initialize your Amazon S3 instance.
     */
    private static final AmazonS3 amazonS3 = AmazonS3ClientBuilder
            .standard()
            .withClientConfiguration(clientConfiguration)
            .withRegion(Regions.DEFAULT_REGION).build();

    /**
     *Returns an object that provides a file operation
     */
    public static AmazonS3 getS3() {
        return amazonS3;
    }

    /**
     *Instantiate a retry policy.
     */
    private static RetryPolicy getS3BaseRetryPolicy() {
        return new RetryPolicy(
                new PredefinedRetryPolicies.SDKDefaultRetryCondition(),
                new PredefinedBackoffStrategies.ExponentialBackoffStrategy(0, 0),
                PredefinedRetryPolicies.DEFAULT_MAX_ERROR_RETRY,
                true
        );
    }
}

Now you are ready. After that, you can call the S3 client that reflects the settings with the getS3 method.

Organize appearing objects

I would like to organize the roles of multiple classes that appear in the implementation example. I didn't feel like I could do what I wanted to do, maybe because I wasn't sure how to look it up ...

AmazonS3 (interface)

An interface that defines methods for manipulating S3 objects. The concrete implementation seems to be provided by the client builder class. AmazonS3 This object is an interface, and the tool seems to be provided by ʻAmazonS3ClientBuilder` etc.

Originally, before adding the settings around the retry, it was done with ʻAmazonS3ClientBuilder.standard (). Build ();`. I think such a minimalist client will suffice if it requires minimal functionality.

ClientConfiguration (class)

ClientConfiguration is a retry Optional configuration classes that can be added to AWS clients such as proxies and user agent strings. In this example, I passed RetryPolicy with some customization, but it seems that the default settings can be packed into the client settings by calling the static methodgetDefaultRetryPolicy ()in PredefinedRetryPolicies.

RetryPolicy (class)

RetryPolicy is A class that builds a retry policy in combination with ClientConfiguration. Initialize by packing the necessary objects and values in the constructor.

Interfaces such as RetryCondition and BackoffStrategy are specified as arguments in the constructor, but at first I was wondering what to specify for the class that matches the interface of the argument. There are several constructors, but one of them is

RetryPolicy.java


public RetryPolicy(RetryCondition retryCondition,
    BackoffStrategy backoffStrategy,
    int maxErrorRetry,
    boolean honorMaxErrorRetryInClientConfig) {
    this(retryCondition, backoffStrategy, maxErrorRetry, honorMaxErrorRetryInClientConfig, false); 
} 

After a lot of research, I found that the SDK had a class that could be packed with some pre-defined policies.

For the RetryCondition interface, we decided to follow the AWS SDK defaults for this implementation. In PredefinedRetryPolicies There are several types of implementations that can be used, so you can pass them to the RetryPolicy constructor. If you are not particular about the settings, you can specify the static property DEFAULT_RETRY_CONDITION that returns the default settings.

BackoffStrategy (https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/retry/RetryPolicy.java) interface Extended abstract class [V2CompatibleBackoffStrategy](https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/retry/V2CompatibleBackoffStrategy. java) is provided, and you can set arguments equivalent to the BackoffStrategy interface by using an adapter that extends this abstract class and various concrete classes.

More specifically, [PredefinedBackoffStrategies](https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/retry/PredefinedBackoffStrategies. extended V2CompatibleBackoffStrategyAdapter to java),

Etc. are prepared. When initializing RetryPolicy, it is possible to make detailed settings by somehow passing BackoffStrategy to the constructor.

Other

It's actually an official doc, and I feel like there's something that explains it nicely ... I couldn't find myself, so I would appreciate it if you could point out if there was something like "I'm sorry if I read this even if I can't do that."

reference

-Error retry and exponential backoff on AWS -Retry API call with Exponential Backoff in AWS SDK Java

Recommended Posts

[AWS SDK for Java] Set a retry policy on the S3 client
Set a signed cookie (for CloudFront) with a custom policy using the AWS SDK for Java
Get a list of S3 files with ListObjectsV2Request (AWS SDK for Java)
Credentials referenced by the AWS SDK for Java by default
AWS SDK for Java 1.11.x and 2.x
Encrypt data uploaded to S3 using AWS SDK for Java / SSE-KMS
Upload / download / bulk delete data to S3 using Amazon S3 Client Builder with AWS SDK for Java
Build a Maven repository on AWS S3
Check the options set for the running Java process
Until you run a Java program with the AWS SDK local to Windows
[Java] (for MacOS) How to set the classpath
I wrote a test code (Junit & mockit) for the code that calls the AWS API (Java)
A note for Initializing Fields in the Java tutorial
Get S3 object size with AWS SDK for Ruby
How to deploy a kotlin (java) app on AWS fargate
Try Spark Submit to EMR using AWS SDK for Java
A memorandum to reach the itchy place for Java Gold
Access S3 buckets using SSE-KMS encryption in an EC2 IAM Role environment (AWS SDK for Java)
Handle business logic for a set of Entity in Java class
Easily set up a Jenkins server on AWS using Bitnami's AMI
Access the in-memory data grid Apache Ignite from a Java client
[Android Studio] Set an arbitrary image for the application background [Java]
How to check for the contents of a java fixed-length string