Tested with boto3 + mock

Introduction

I wrote code that operates on AWS S3 using boto3. I decided to write a test code and decided to create it using mock and unittest.

Preparation

First, prepare the environment. Create an appropriate folder and create an environment using virtualenv. This time, it is assumed that it will be installed on AWS Lambda. After that, pip boto3 and mock.

Environmental preparation

$ mkdir s3operation
$ cd s3operation
$ virtualenv .
$ source bin/activate
$ pip install boto3
$ pip install moto

Sample class to operate on S3

$ pwd
~/s3operation/
$ vi s3access.py

s3access.py


# -*- coding:utf-8 -*-
import boto3
import os.path

class S3Access:
    s3 = None
    def __init__(self):
        self.s3 = boto3.client('s3')
    
    def upload(self, bucketname, uploadfilepath, s3folder):
        """
Upload to S3
        @param bucketname S3 bucket name
        @param uploadfilepath Path of the file to upload
        @param s3folder Folder path under the S3 bucket (filename not included)
        """
        basefilename = os.path.basename(uploadfilepath)
        s3filepath   = s3folder + '/' + basefilename
        self.s3.upload_file(uploadfilepath, bucketname, s3filepath)

Test code

This time, I made it an upload-only program. Now let's write the test code. First of all, the environment configuration.

Environmental composition

This is the file / folder structure. I wanted to create only the test code in another folder (test folder this time), so it looks like the following.

~/s3operation/
    s3access.py
    test/
        testfile.txt
        tests3access.py

Test code

The test code looks like this:

tests3access.py


# -*- coding:utf-8 -*-
import boto3
import unittest

from moto import mock_s3

import sys
import os
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + '/../')
from s3access import S3Access

class TestS3Access(unittest.TestCase):
    def setUp(self):
        pass

    def tearDown(self):
        pass

    @mock_s3
    def test_s3_upload(self):
        #Initialization
        bucketname = 'testbucket001'
        prefix     = 'firstfolder'

        ##Test preparation(Bucket, object creation)
        s3test = boto3.client('s3')
        s3test.create_bucket(Bucket=bucketname)
        s3test.put_object(Bucket=bucketname, Key=prefix)

        #Object generation to be tested
        s3 = S3Access()
        #Target method execution
        s3.upload('./test.md', bucketname, prefix)
        
        #Confirmation of execution result
        ##Read uploaded information from S3
        keyname  = prefix + '/test.md'
        response = s3test.get_object(Bucket=bucketname, Key=keyname)
        body     = response['Body'].read().decode('utf-8').encode('utf-8')
        ##Load the uploaded file
        fp       = open(filename, 'r')
        readStr  = fp.read()
        fp.close
        ##Check the result
        assert body2 == readStr

Test run

Now that we have the test code, let's actually run the test.

$ pwd
~/s3operation/test/
$ python -m unittest discover ./
.
----------------------------------------------------------------------
Ran 1 tests in 0.472s

OK

Now one test has been run and is successful.

If it fails

If there is an error in any program, it will look like this: All you have to do is analyze the error information and fix it.

python -m unittest discover ./
E
======================================================================
ERROR: test_s3_upload (tests3access.TestS3Access)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "~/s3operation/lib/python2.7/site-packages/moto/core/models.py", line 70, in wrapper
    result = func(*args, **kwargs)
  File "~/s3operation/test/tests3access.py", line 43, in test_s3_upload
    response = s3test.get_object(Bucket=bucketname, Key=keyname)
  File "~/s3operation/lib/python2.7/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "~/s3operation/lib/python2.7/site-packages/botocore/client.py", line 599, in _make_api_call
    raise error_class(parsed_response, operation_name)
NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

----------------------------------------------------------------------
Ran 6 tests in 0.519s

FAILED (errors=1)

at the end

In this example, I wrote the evaluation of only a part of S3, but it also supports the following AWS resources.

I think that I should make a habit of creating test code on a regular basis, and when the test code becomes all green, I feel that development is complete. First of all, even at the beginning.

Recommended Posts

Tested with boto3 + mock
Tested with Python
Use Mock with pytest
Region specification with boto
S3 uploader with boto
Set connection timeout with boto3
Try Google Mock with C
Change retry settings with boto3
Tested pipenv with GitHub Actions
Connect to Elastic MQ with boto
mock
Use boto3 to mess with S3
Generate S3 signed URL with boto
Use boto3 with temporary credentials with SAML authentication
Try server-side encryption on S3 with boto3
Getting started with Dynamo from Python boto
[AWS] Link Lambda and S3 with boto3
S3 server-side encryption SSE with Python boto3