Install the latest version of the AWS CLI
pip3 install awscli --upgrade --user
It seems that it was installed, but
aws --version
Then
zsh: command not found: aws
Will come out ...
Add the AWS CLI version 1 executable to the macOS command line path (https://docs.aws.amazon.com/ja_jp/cli/latest/userguide/install-macos.html#awscli-install-osx- path)
Based on the above article, "Add the aws program to the PATH environment variable of the operating system".
→ But after all
zsh: command not found: aws
Will come out ...?
which python
It comes up as / Users / user name / anaconda3 / bin / python
.
Install AWS CLI According to the article above If you use --user during the first installation, it will be installed in .local. I need to put my PATH in ~ / .local / bin, but now
ʻExport PATH =" / Users / username / anaconda3 / bin: $ PATH "` I wonder if it is because it has become ...
Is the method different when using anaconda? how to install AWSCLI on a Anaconda python distribution aws codecommit aws: command not found
conda install -c conda-forge awscli
After running, I was able to use AWS commands brilliantly.
Reference site
aws configure
↓
AWS Access Key ID [None]: 〜〜 AWS Secret Access Key [None]: 〜〜 Default region name [None]: ap-northeast-1 Default output format [None]: json
AWS CLI Command List Confirm that you can see the contents of s3 with ʻaws s3 ls`
ʻAws s3 cp s3: // {bucket name} / {path} {local path} `
You can copy the path of the file on s3 by clicking "Click File-> Click Copy Path". (The following is when downloading to the download folder) ʻAws s3 cp s3: // ~ ~ ~ / Users / user name / Downloads `
Reference: You can read compressed files with pandas.read_csv. Very convenient!
python
import pandas as pd
df = pd.read_csv('file name.csv.gz')
#Delete unused columns
df=df.drop(columns=['A','B','C'],axis=1)
#Delete any missing values in column a
df=df.dropna(subset=['a'])
df.to_csv('./renamed_file/File name after compression.csv.gz', index=False, compression='gzip')
[Introduction to Python] Let's execute commands using subprocess!
python
import subprocess
subprocess.call(["aws","s3","ls"])
→ Success
python
import subprocess
path_list=[List of PATH files on S3 you want to download]
for s in range(len(path_list)):
cmd="aws s3 cp"+" "+path_list[s]+" "+"/Users/username/Downloads/point_data"
subprocess.call(cmd.split())
df = pd.read_csv(filename_list[s])
#Delete unused columns
df=df.drop(columns=df.columns[[1,2,3,4,5]], axis=1)#Numbers are column numbers Left is an example
#Delete row without column name A
df=df.dropna(subset=['A'])
#Save
file_name='/Users/username/Downloads/renamed_file/'+file name
df.to_csv(file_name, index=False, compression='gzip')
cmd2="aws s3 cp"+" "+"/Users/username/Downloads/renamed_file/"+file name+" "+ 'PATH of location on S3 you want to upload'
subprocess.call(cmd2.split())
Recommended Posts