Use the retry processing mode added to Boto3

Introduction

The AWS SDK implements automatic retry logic with Exponential Backoff, Boto3 offers three retry processing modes.

Checking botocore's GitHub repository, standard and adaptive are found around February 2020. Added and earlier implementations are named legacy.

Add support for new retry modes #1972

For compatibility, the default is legacy, and features added in standard / adaptive To use it, you need to explicitly change the retry processing mode.

How to change the retry processing mode

Instantiate the botocore config object and send the configuration information to the client You can pass it. The available options are max_attempts and mode. The default maximum number of retries varies depending on the retry processing mode, Can be customized by max_attempts.

import boto3
from botocore.config import Config

config = Config(
   retries = {
      'max_attempts': 10,
      'mode': 'standard'
   }
)

ec2 = boto3.client('ec2', config=config)

Overview of each retry processing mode

Legacy retry mode This is the default mode used by Boto3 clients. The factor used to calculate the backoff time is 2. The v1 retry handler (https://github.com/boto/botocore/blob/master/botocore/retryhandler.py) is used. There are a limited number of errors / exceptions for retries.

Standard retry mode Retry rules are standardized with multilingual AWS SDKs and consistent logic. It handles more throttling / limit errors and exceptions than legacy mode. The factor used to calculate the backoff time is 2, as in legacy mode, but with a maximum of 20 seconds. The v2 retry handler (https://github.com/boto/botocore/tree/master/botocore/retries) is used.

legacy standard
Default maximum number of attempts 5 3
Dealing with temporary connection errors ConnectionError
ConnectionClosedError
ReadTimeoutError
EndpointConnectionError
RequestTimeout
RequestTimeoutException
PriorRequestNotComplete
ConnectionError
HTTPClientError
throttling/Not responding to limit errors and exceptions Throttling
ThrottlingException
ThrottledException
RequestThrottledException
ProvisionedThroughputExceededException
Throttling
ThrottlingException
ThrottledException
RequestThrottledException
TooManyRequestsException
ProvisionedThroughputExceededException
TransactionInProgressException
RequestLimitExceeded
BandwidthLimitExceeded
LimitExceededException
RequestThrottled
SlowDown
EC2ThrottledException
Retry by status code 429/500/502/503/504/509 etc. 500/502/503/504
Delay time calculation formula rand(0, 1) * (2 ^ (attempts - 1)) min(rand(0, 1) * 2 ^ attempt, 20)

Adaptive retry mode In addition to the standard mode functionality, the token bucket algorithm is used on the client side. Automatic rate limiting feature and AWS service side error / exception / HTTP status code for each retry Added the ability to change rate limiting variables based on. Flexible re-execution can be performed on the client side according to the error content, ** This is an experimental mode, so its functionality and behavior may change in the future. ** **

reference

Retries - Boto3 Docs https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html

that's all. I'm glad if you can use it as a reference.

Recommended Posts

Use the retry processing mode added to Boto3
How to use the generator
How to use the decorator
Leave the troublesome processing to Python
How to use the zip function
How to use the optparse module
[Python] How to use list 3 Added
How to use the ConfigParser module
Use boto3 to mess with S3
[Linux] How to use the echo command
How to use the Linux grep command
Use numpy's .flatten () [0] to retrieve the value
How to use the IPython debugger (ipdb)
Use boto to upload / download files to s3.
3 best ways to use the less command
How to use the C library in Python
How to use MkDocs for the first time
Use the MediaWiki API to get Wiki information
Use boto3-stubs to enable code completion on boto3
Specify the Python executable to use with virtualenv
Use decorators to prevent re-execution of data processing
How to use the graph drawing library Bokeh
How to use the Google Cloud Translation API
How to use the NHK program guide API
The easiest way to use OpenCV with python
[Algorithm x Python] How to use the list
Use the Python framework "cocotb" to test Verilog.
How to use PyTorch-based image processing library "Kornia"
Use pygogo to get the log in json.
I want to use the activation function Mish
Decorator to retry
[Introduction to Python] How to use while statements (repetitive processing)
How to run the Ansible module added in Ansible Tower
How to use the Raspberry Pi relay module Python
How to use python interactive mode with git bash
I wanted to use the Python library from MATLAB
Linux user addition, how to use the useradd command
[Python] How to use the graph creation library Altair
How to use the grep command and frequent samples
How to use the exists clause in Django's queryset
Specify MinGW as the compiler to use with Python
[Introduction to Udemy Python3 + Application] 27. How to use the dictionary
[Introduction to Udemy Python3 + Application] 30. How to use the set
How to use argparse and the difference between optparse
How to use the model learned in Lobe in Python
How to use the Rubik's Cube solver library "kociemba"
(Remember quickly) How to use the LINUX command line
The programming language you want to be able to use
I want to use the R dataset in python
use something other than the default profile with boto3
Tips for Python beginners to use the Scikit-image example for themselves 8 Processing time measurement and profiler