Copy files directly from Amazon EC2 (Amazon linux) to S3

Copy files directly from Amazon EC2 (Linux) to S3

You may want to export the data file in EC2 to another environment while verifying something in the AWS environment. (I have quite a few, but what about everyone ...) If EC2 is a Windows server OS, you can use a browser, so I think there are various ways to do it, such as uploading it to Box etc. or attaching it to Web mail, but what to do with Linux OS (´ ・ ω ・ `). Therefore, I would like to verify the method of uploading directly from EC2 to S3 using the AWS CLI and downloading to another environment from the console screen of S3.

What you want to achieve

やりたいこと.png

I want to export the data file in EC2 (Linux) to S3 so that I can download it from the S3 console screen.

procedure

  1. Deploy EC2
  2. Create directories and files in EC2
  3. Attach a role for S3 upload to EC2
  4. Create a destination S3 bucket
  5. Copy data to S3 with AWS CLI

I tried it

First, deploy EC2 and log in with SSH. Then create a directory and create a test file to copy to S3.

Switch user to #root  sudo su root  #Upload information  yum -y update Create a directory (/ test-share) in # / home  cd /home  mkdir /test-share  cd /test-share #Create a test file (test.txt)  touch test.txt #Check if the file was created  ls

Now you are ready to copy the test file to S3.

I have deployed EC2, but in the current state, I do not have the authority (role) to access S3, so I need to attach a role.

How to make a roll Access IAM from the AWS console screen and click "Role" Slide 2.PNG

Then click "Create Role" スライド3.PNG

Select "EC2" for common use cases スライド4.PNG

Select a policy to assign to a role Enter S3 in the policy filter and in the displayed policies This time, I want to access (upload, download, browse) S3, so select [AmzzonS3FullAccess].

スライド5.PNG

Tagging as needed (tags are not set this time) スライド6.PNG

Set the role name arbitrarily and click "Create" スライド7.PNG

Once you have created a role, attach it to EC2 Select the target instance on the EC2 console screen, and select "Action"-"Instance Settings"-"IAM Role Assignment / Replace". S3_Qiita用.png

Assign the target instance ID and role, and click "Apply". S3_ロール割り当て_Qiita用.png

You have now attached a role for EC2 to access S3.

Next, create an S3 bucket (test-20200624-folder) to copy the data to. Access the console screen of S3 and "Create a bucket" スライド8.PNG

Enter "test-20200624-folder" and the region (this time Asia Pacific) in the bucket name, and click "Next". スライド9.PNG

I will set the bucket, but this time there is no particular change in the setting, so "Next" as it is スライド10.PNG

Make sure that Block Public Access is checked and click "Next" スライド11.PNG

Once you've attached the role, enter the command to copy the data to S3 in the AWS CLI.

#Log in to EC2 again with SSH and switch to root  sudo su root #Move to the directory where the test file is  cd /home /test-share Copy data to S3 with #aws cli  aws s3 cp /test.txt s3://test-20200624-folder

2020-06-24 06:59:35 0 It is OK if it is displayed as test.txt

Finally, check if you can download "test.txt" from the S3 console screen. When you access the S3 console screen and check the target bucket S3_コンソールQiita用.png

I was able to confirm that test.txt was copied! After that, if you select "Download" from "Action", the file export to another environment is completed.

Finally

There are articles with the same content in various places, but there was no explanation to attach the EC2 role, and even if I followed the procedure, an error occurred, so I hope that it will be useful for memorandums and beginners.

You can use the AWS CLI to not only copy data from EC2 to S3, but also synchronize a specific directory of EC2 with a folder in the S3 bucket. It would be convenient if EC2 had a root CA function and could be used to export the created certificate or send a log file for verification. D (`・ ω ・')

Recommended Posts

Copy files directly from Amazon EC2 (Amazon linux) to S3
Copy S3 files from Python to GCS using GSUtil
[Amazon Linux] Switching from Python 2 series to Python 3 series
Copy data from Amazon S3 to Google Cloud Storage with Python (boto)
Jupyter Notebook Settings-How to use (EC2 Amazon Linux 2)
AWS EC2 2nd SSH connection to EC2 Instance (Amazon Linux2)
[AWS EC2] How to install Maven on Amazon Linux 2
[Amazon Linux 2] Change from public key authentication to password authentication
[Linux] Copy data from Linux to Windows with a shell script
Recursively copy files from the directory directly under the directory using Python
Linux script to convert Markdown files from JupyterLab format to Qiita format
[AWS EC2] Settings you want to do on Amazon Linux 2
[Scp] Copy files locally from Linux server with Tera Term ssh scp
Easy copy to clipboard on Linux
How to install wkhtmltopdf (Amazon Linux2)
Add users with SSH access to your Amazon EC2 Linux instance
Install pyenv on EC2 (Amazon Linux)
Introducing Amplify to Amazon Linux 2 AMI
Data integration from Python app on Linux to Amazon Redshift with ODBC
How to display Django static files correctly under Amazon Linux2 + Apache environment
Change AWS EC2 instance from t2 to t3
I'll install Ruby on EC2 (Amazon Linux2) 2020
[Linux] How to subdivide files and folders
Save files using EC2 storage without using S3
[Linux] Flow from power-on to PC startup
PUT gzip directly to S3 in Python
How to update php on Amazon linux 2
[linux] Split files to the specified size
Upgraded mysql on Cloud9 (Amazon Linux) (5.5 to 5,7)
How to install Anisble on Amazon Linux 2
How to operate Linux from the console
Introducing Docker Engine to Linux From Scratch
How to find large files on Linux
Use boto to upload / download files to s3.
Install Python 3.8, Pip 3.8 on EC2 (Amazon Linux 2)
Easy script migration from CentOS to Oracle Linux
EC2 (Amazon Linux) certbot-auto renew error /pkg_resources.DistributionNotFound: The'virtualenv == 15.1.0'
How to operate Linux from the outside Procedure
How to handle Linux commands well from Python
Creating an AWS EC2 EC2 Instance (Amazon Linux AMI) 2
Added firewalld to Amazon Linux 2 (IP address restrictions)
(Translation) From Arrow to pandas at 10GB / s
Download files directly to Google Drive (using Google Colaboratory)
Back up from QNAP to Linux with rsync
[Linux] When you want to search for a specific character string from multiple files
[AWS EC2] How to install only MySQL client on Amazon Linux 2 and connect to RDS