[Java] [AWS SDK for Java] Set retry policy for S3 client

4 minute read

Just because the information I want was not in Japanese, I will document it. When uploading a file to S3 using AWS SDK for Java, add a little more detailed setting.

table of contents

  • Outline of cases around retries
  • I will implement it for the time being
  • Organize related classes
  • Other
  • Reference

Outline of cases around retries

I want to upload a file to S3 that is locally using AWS SDK for Java. Then, in anticipation of a case where communication is cut off due to AWS’s circumstances or normal operation is not possible due to various circumstances, I want to prepare a retry policy for AWS clients. This time, prepare a utility class that returns a singleton of AmazonS3 object included in AWS SDK for Java.

In this article, I focus on the option setting when creating a client, which I was especially troubled by. Therefore, the description related to the processing before and after the file upload is omitted.

Try to implement for the time being

Build a client packed with simple client options. The flow is to prepare three items, a retry policy instance, a client setting instance, and a client.

First, prepare a method that returns a RetryPolicy instance that contains basic policy settings and AWS-defined strategies.

AmazonS3Utils.java


/**
 * Put appropriate values for baseDelay and maxBackoffTime in the ExponentialBackoffStrategy constructor.
 *Note: Here, the appropriate values are entered as a sample.
 */
private static RetryPolicy getS3BaseRetryPolicy() {
        return new RetryPolicy(
                new PredefinedRetryPolicies.SDKDefaultRetryCondition(),
                new PredefinedBackoffStrategies.ExponentialBackoffStrategy(0, 0),
                PredefinedRetryPolicies.DEFAULT_MAX_ERROR_RETRY,
                true
        );
    }

Next, call the method that returns the RetryPolicy instance defined in ↑ to define the client settings.

AmazonS3Utils.java


private static final ClientConfiguration clientConfiguration = new ClientConfiguration()
            .withRetryPolicy(getS3BaseRetryPolicy());

Then, initialize the AmazonS3 object, which is the entry point of the S3 client called by other classes and methods, by filling the client settings.

AmazonS3Utils.java


private static final AmazonS3 amazonS3 = AmazonS3ClientBuilder
            .standard()
            .withClientConfiguration(clientConfiguration)
            .withRegion(Regions.DEFAULT_REGION)
            .build();

To summarize the settings so far,

AmazonS3Utils.java


public final class AmazonS3Utils {

    /**
     * Initialize client settings reflecting the retry policy.
     */
    private static final ClientConfiguration clientConfiguration = new ClientConfiguration()
            .withRetryPolicy(getS3BaseRetryPolicy());

    /**
     * Initialize Amazon S3 instance.
     */
    private static final AmazonS3 amazonS3 = AmazonS3ClientBuilder
            .standard()
            .withClientConfiguration(clientConfiguration)
            .withRegion(Regions.DEFAULT_REGION).build();

    /**
     * Returns an object that provides file operations
     */
    public static AmazonS3 getS3() {
        return amazonS3;
    }

    /**
     * Instantiate retry policy.
     */
    private static RetryPolicy getS3BaseRetryPolicy() {
        return new RetryPolicy(
                new PredefinedRetryPolicies.SDKDefaultRetryCondition(),
                new PredefinedBackoffStrategies.ExponentialBackoffStrategy(0, 0),
                PredefinedRetryPolicies.DEFAULT_MAX_ERROR_RETRY,
                true
        );
    }
}

Now you are ready. After that, you can call the S3 client that reflects the setting with the getS3 method.

Organize appearance objects

I would like to organize the roles of multiple classes that appear in the implementation example. I didn’t feel like I could do what I wanted to do, probably because I wasn’t doing my best…

AmazonS3 (interface)

Interface that defines methods for operating S3 objects. The specific implementation seems to be provided by the client builder class. AmazonS3 This object is an interface, and the tools seem to be provided by AmazonS3ClientBuilder etc.

Originally, it was done with AmazonS3ClientBuilder.standard().build(); before adding the settings around retries. I think that such a minimalist client is enough if the minimum functionality is required.

ClientConfiguration (class)

ClientConfiguration is Optional configuration class that can be added to AWS clients such as proxies and user agent strings. In this example, I passed in a slightly customized RetryPolicy, but it seems that the default settings can be packed into the client settings by calling the static method getDefaultRetryPolicy() in PredefinedRetryPolicies.

RetryPolicy (class)

RetryPolicy is A class that constructs a retry policy in combination with ClientConfiguration. Initialize by filling the necessary objects and values in the constructor.

Interfaces such as RetryCondition and BackoffStrategy are specified as arguments in the constructor, but I was wondering what to specify for the class that matches the interface of the arguments. There are several constructors, but one of them is

RetryPolicy.java


public RetryPolicy(RetryCondition retryCondition,
    BackoffStrategy backoffStrategy,
    int maxErrorRetry,
    boolean honorMaxErrorRetryInClientConfig) {
    this(retryCondition, backoffStrategy, maxErrorRetry, honorMaxErrorRetryInClientConfig, false);
}

After doing a lot of research, I found that the SDK had a class that could contain some predefined policies.

For the RetryCondition interface, we decided to follow the AWS SDK defaults in this implementation. In PredefinedRetryPolicies, There are several types of implementations that can be used, so just pass it to the RetryPolicy constructor. If you are not particular about the settings, it seems good to specify the static property DEFAULT_RETRY_CONDITION that returns the default settings.BackoffStrategyinterfaceExtendedabstractclassV2CompatibleBackoffStrategy is prepared, and an argument equivalent to the BackoffStrategy interface can be set by using an adapter that extends this abstract class and various concrete classes.

Specifically, PredefinedBackoffStrategies has extended V2CompatibleBackoffStrategyAdapter,

  • FullJitterBackoffStrategy
  • EqualJitterBackoffStrategy
  • EqualJitterBackoffStrategy

Etc. are prepared. When initializing RetryPolicy, it is possible to make detailed settings by passing BackoffStrategy to the constructor.

Other

I feel like there’s something that is really official documentation and nicely explaining it… I couldn’t find myself, so I would appreciate if you could point me out, “If you can’t do that, you can’t read this.”

Reference