Try server-side encryption on S3 with boto3

When uploading an object to S3 with a Python script I investigated how to encrypt on the server side.

S3 and object encryption

Reference: There are roughly two types.

Client-Side Encryption(CSE)

The client side encrypts the data before sending it to the server for storage. -> Data is protected during transmission and reception with the server (S3).

Server-Side Encryption(SSE)

The client sends the data as is, and the server encrypts it before storing it. -> Data is protected while stored on the server (S3).

The latter requires less effort on the client side to think about encryption. It is easy to handle because it is transparently decrypted even when downloaded. __ * Sensitive information should of course be encrypted before sending and receiving __

S3 and Server-Side Encryption

Reference: There are different types of SSE options in S3.

SSE-S3 SSE-KMS SSE-C
Encryption key Keys managed behind the scenes by S3 Keys managed by AWS KMS service User-managed keys
Feature Since everything is managed by AWS, there is little effort. However, the access authority to the key cannot be manipulated, and the key can only be used to save data in S3. You can flexibly change the access authority to the key itself by operating IAM users and roles. SSE-Although it has a smaller turn than S3, the number of setting items increases. You can choose the key to use and the storage location. It has the highest degree of freedom, but you need to ensure the secure storage of your keys yourself.

I wanted to leave the role to the AWS side as much as possible, so I tried SSE-S3 and SSE-KMS this time.

Prior confirmation in Management Console

When I try to save an object to a bucket in the Management Console Along the way, you will be given an encryption option. 0.png

"Amazon S3 master-key" stands for SSE-S3. Even if you select it, there are no particular choices.

1.png

"AWS KMS master-key" means AWS-KMS. Once selected, you will be given a choice of keys to use.

--"aws / s3" is the key that KMS tries to create for S3 services. --"test_key" is the key created by KMS (I created it in advance) --If you select Custom KMS ARN, you can specify the key in the description of ARN.

2.png

If you try to save it with SSE-S3, it will be displayed like this on the object details screen.

3.png

Do this from the script (boto3).

Script description

Because there is no description of for Python in [Document] of S3 (http://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/UsingServerSideEncryption.html) I wrote it with reference to other languages and boto3 documentation. The execution environment is Lambda's Python 3.6, but I think that the writing style and behavior do not change much anywhere.

Rewrite the role nicely so that you can access the S3 and KMS keys.

SSE-S3

SSE-S3


import boto3

def lambda_handler(event, context):
    
    file_name = 'test.txt'

    #Can be used temporarily'/tmp'Create a file in
    with open('/tmp/' + file_name, 'w') as f:
        f.write('hoge')
    
    #When uploading a file, specify the encryption method for ExtraArgs
    response = boto3.client('s3').upload_file(
        Filename='/tmp/' + file_name,
        Bucket='nanakenashi-test',
        Key=file_name,
        ExtraArgs={'ServerSideEncryption': 'AES256'})

    return True

It's very simple, with only more static arguments when uploading. At the moment (2017/09/16), it seems that the only encryption method is ʻAES256`. You can see that the actually saved object has the same shape as before.

4.png

SSE-KMS

When using the default key for S3

Rewrite only ʻExtraArgs` in the above script.

SSE-KMS (using default key for S3)


ExtraArgs={
    'ServerSideEncryption': 'aws:kms',
    }

The saved object is encrypted with the default key for S3. (Since this key was created at this timing, I think it is different from the key for SSE-S3)

5.png

However, this key is not much different from SSE-S3 because the setting cannot be changed.

When using a key created by KMS

SSE-KMS (using the key created by KMS)


#Add the ID of the key to use
ExtraArgs={
    'ServerSideEncryption': 'aws:kms',
    'SSEKMSKeyId': 'ea41458h-0c2o-496g-b92e-67441d771282'
    }

You can see that it is encrypted with the key created in advance.

6.png

Supplement

SSE-S3 and SSE-KMS /AmazonS3/latest/dev/UsingKMSEncryption.html)

With server-side encryption, only object data is encrypted. Object metadata is not encrypted.

Also

If you need server-side encryption for all objects stored in your bucket You can use bucket policies.

(It's difficult Japanese ...) In other words, it is possible to prohibit the saving of unencrypted objects. Setting this bucket policy makes it easier to maintain a secure state.

Summary

To make data storage in S3 more secure I tried several options for server-side encryption.

By the way, for SSE-C, refer to Around here. If you do, you can use it.

Recommended Posts

Try server-side encryption on S3 with boto3
S3 server-side encryption SSE with Python boto3
S3 uploader with boto
S3 operation with python boto3
Mount S3 on Ubuntu with goofys
Use boto3 to mess with S3
Generate S3 signed URL with boto
Try SVM with scikit-learn on Jupyter Notebook
[AWS] Link Lambda and S3 with boto3
Try Tensorflow with a GPU instance on AWS
View images on S3 with API Gateway + Lambda
How to get the key on Amazon S3 with Boto 3, implementation example, notes
Try working with Mongo in Python on Mac
[Python] Summary of S3 file operations with boto3
[Cloudian # 8] Try setting the bucket versioning with Python (boto3)
Install selenium on Mac and try it with python
Try debugging Python on Raspberry Pi with Visual Studio.
Manipulate S3 objects with Boto3 (high-level API and low-level API)
Issue S3 time-limited URL with boto3 (with file existence confirmation)
[Memo] Load csv of s3 into pandas with boto3
Tested with boto3 + mock
Launch Lambda on Boto3
Region specification with boto
Try SNN with BindsNET
Try FEniCS on Windows!
Try Poerty on Windows
Try NeosVR on Linux
Try regression with TensorFlow
Try deepdream on Mac
Try Amazon Simple Workflow Service (SWF) with Python and boto3
read the tag assigned to you on ec2 with boto3
Try clustering with a mixed Gaussian model on a Jupyter Notebook
Try encryption / decryption using OpenSSL key with Python3 pow function
I wanted to delete multiple objects in s3 with boto3