How to use MinIO with the same function as S3 Use docker-compose

Overview

I tried to make it usable with docker for development even in the situation where AWS services can not be used due to various circumstances (contract problems, budget, etc.). The title is MinIO, but DynamoDB and MySQL are also available.

Advance preparation

• Install docker first. For windows Home, use ** Docker Quickstart Terminal **.
• Since the app uses Python, install python (3.7 at the time of article creation).
• It is also possible with Jupyter Notebook etc.
• boto3

docker-compose

• This time, create yml to start with docker-compose.

• Create by referring to Official.

• As it is, the error of Invalid command line arguments: use path style endpoint for FS setup. has occurred, so fix it by referring to here. Did.

• Only one is used for development, but it seems to use 4 nodes originally.

• In addition to MinIO, MySQL and DynamoDB are also available.

• The final file is as follows.

• docker-compose.yml

version: '3.7'
services:
  dynamodb-local:
    build: ./dynamodb
    volumes:
      - dynamodb_data:/home/dynamodblocal/data
    ports:
      - 8000:8000
    command: -jar DynamoDBLocal.jar -dbPath ./data
  mysql:
    image: mysql:5.7
    command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci
    volumes:
      - db_data:/var/lib/mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 'password'
    ports:
      - 3306:3306
  minio1:
    image: minio/minio:RELEASE.2020-08-27T05-16-20Z
    volumes:
      - data1-1:/data1
      - data1-2:/data2
    ports:
      - "9000:9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server data1
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
volumes:
  dynamodb_data:
  db_data: {}
  data1-1:
  data1-2:

• DynamoDB Docker file

FROM amazon/dynamodb-local

WORKDIR /home/dynamodblocal

# UID=1000 → DynamoDB Local execution user
RUN mkdir data && chown -R 1000 data

• When you start it, the endpoint is displayed as shown below, so make a note of it.
• ** For Docker Quickstart Terminal **, replace the IP address with the IP displayed when docker is started.

minio1_1          | Endpoint:  http://172.20.0.4:9000  http://127.0.0.1:9000
minio1_1          |
minio1_1          | Browser Access:
minio1_1          |    http://172.20.0.4:9000  http://127.0.0.1:9000

• By accessing http://192.168.99.100:9000/minio/ (IP address is different because of Docker Quickstart Terminal), you can check and create the inside of the bucket on the browser. I will.

Implementation of S3 function

• The source from bucket creation to file upload is as follows.
• If you create the directory structure when uploading to S3 with os.path.join, it will be connected with\\, so it is converted with replace.

import boto3
import os
from datetime import datetime

endpoint_url = os.getenv('S3_ENDPOINT', 'http://192.168.99.100:9000/')
aws_access_key_id = os.getenv('S3_ACCESS_KEY_ID', 'minio')
aws_secret_access_key_id = os.getenv('S3_SECRET_ACCESS_KEY_ID', 'minio123')

s3 = boto3.resource(
    's3',
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key_id,
    endpoint_url=endpoint_url
)

bucket=s3.Bucket('testbukcet')
bucket.create()
bucketPath=os.path.join('dir',datetime.now().strftime('%Y%m%d%H%M%S'),'out','test.csv')
filePath=bucketPath.replace('\\','/')
bucket.upload_file('test.csv',filePath)

• If you check with a browser, you can check as follows.