[Memo] Load csv of s3 into pandas with boto3

import pandas as pd
from io import StringIO
import boto3
ACCESS_KEY = 'your access key'
SECRET_KEY = 'your secret key'
client = boto3.client('s3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
paginator = client.get_paginator('list_objects')
pageresponse = paginator.paginate(Bucket="your bucket")
for pageobject in pageresponse:
    for file in pageobject["Contents"]:
        obj = client.get_object(Bucket="your bucket", Key=file["Key"])
        content = obj['Body'].read().decode('utf-8')
        reader = pd.read_csv(StringIO(content))
        print(reader)

Recommended Posts

[Memo] Load csv of s3 into pandas with boto3
Load csv with duplicate columns in pandas
[Python] Summary of S3 file operations with boto3
S3 uploader with boto
How to output CSV of multi-line header with pandas
[Python] A memo to write CSV vertically with Pandas
Read csv with python pandas
Load nested json with pandas
S3 operation with python boto3
[Python] Operation memo of pandas DataFrame
Format the CSV file of "National Holiday" of the Cabinet Office with pandas
[Memo] Small story of pandas, numpy
Use boto3 to mess with S3
Generate S3 signed URL with boto
How to deal with SSL error when connecting to S3 with boto of Python
Use Bucket (). Objects.filter instead of list_objects_v2 when outputting S3 list with Boto3
[BigQuery] Load a part of BQ data into pandas at high speed
Import of japandas with pandas 1.0 and above
Try server-side encryption on S3 with boto3
[AWS] Link Lambda and S3 with boto3
S3 server-side encryption SSE with Python boto3
Read CSV and analyze with Pandas and Seaborn
Example of efficient data processing with PANDAS
Pandas memo
Automatic operation of Chrome with Python + Selenium + pandas
Read Python csv data with Pandas ⇒ Graph with Matplotlib
Get a list of IAM users with Boto3
CSV output of pulse data with Raspberry Pi (CSV output)
Aggregate VIP values of Smash Bros. with Pandas
Make holiday data into a data frame with pandas
Example of reading and writing CSV with Python
Combine multiple csv files into one csv file with python (assuming only one line of header)
When reading a csv file with read_csv of pandas, the first column becomes index