Cloudian is an object storage device with AWS S3 compatible API.
Since it is an S3 compatible API, it can be operated using the AWS SDK, but since there are many articles that assume connecting to S3, I will write a method to connect to an object storage other than S3.
This time, we will check that you can get the bucket list (bucket list) of object storage using AWS SDK for Python (boto3).
In order to work with S3 files, you need to import the AWS SDK for Python (boto3) in Python. boto3 can be easily installed with the pip command
$ pip install boto3
Set the credential information (access key and secret key) to use boto3 from Python. Easy to set up by installing the AWS CLI
Install with pip command
$ pip install awscli
Run the aws configure command --Enter Cloudian credential information (access key and secret key) --Default region name is an empty enter --Default output format is json
$ aws configure
AWS Access Key ID: xxxxxxxxxxxxxxxxx
AWS Secret Access Key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
Default region name:
Default output format: json
This completes the setting of credential information. You can check the settings in "~/.aws/credentials" and "~/.aws/config".
$ cat ~/.aws/credentials
[default]
aws_access_key_id = xxxxxxxxxxxxxxxxx
aws_secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
$ cat ~/.aws/config
[default]
output=json
In order to work with S3 files, you need to import boto3 in Python.
import boto3
client = boto3.client(
's3',
endpoint_url='https://xxx.yyy.com'
)
Set the following as arguments of boto3.client.
--Set the AWS service type in the first argument. Set's3'here.
--In the second argument, set "endopoint_url ='Object storage S3 endpoint'" to specify the object storage S3 endpoint instead of the AWS endpoint.
client = boto3.client('s3', endpoint_url='https://xxx.yyy.com')
I have set ‘s3’ as the AWS service type and endpoint_url ='http://xxx.yyy.com' as the URL of the S3 endpoint.
The endpoint_url setting overrides the AWS S3 endpoint, which is the default reference, with the object storage S3 endpoint.
Reference information) If you want to set the credential information in the Python program (if you do not set it in the AWS CLI/Configure settings) You can also connect by passing the credential information (access key and secret key) to the boto3.client parameter.
client = boto3.client( 's3', endpoint_url='https://xxx.yyy.com', # Hard coded strings as credentials, not recommended. aws_access_key_id='4dfba0a142971dbde38b', aws_secret_access_key='FdgZNJ7udliVpbc9HE5M8sgAz5nvzn1y6uCHl7YQ' )Note) You can try it immediately without AWS CLI/Configure settings, but it is not recommended to write the credentials programmatically (for verification purposes).
_
Reference information) About creating S3 objects The AWS SDKs, including boto3, provide a 1: 1 "low-level API" for the S3 API and an object-oriented "high-level API" for more advanced operations.
・ Client API (low level API): S3Client (boto3.client) It has a one-to-one correspondence with AWS REST API.
・ Resource API (high level API): S3Resource (boto3.resource) AWS resources can be handled in an object-oriented manner.
This sample uses the low-level API S3Client (boto3.client) object.
test.py
import boto3
client = boto3.client(
's3',
endpoint_url='https://xxx.yyy.com'
)
#Get a list of buckets
client.list_buckets()
You can execute the above code to get a list of buckets set in object storage/Cloudian.
$ python test1.py
{'ResponseMetadata': {'RequestId': '9dad38b5-0e30-1dbc-a754-06bdfcde1d5e',
'HostId': '',
'HTTPStatusCode': 200,
'HTTPHeaders': {'date': 'Sun, 13 Dec 2020 21:57:43 GMT',
'x-amz-request-id': '9dad38b5-0e30-1dbc-a754-06bdfcde1d5e',
'content-type': 'application/xml;charset=UTF-8',
'content-length': '519',
'server': 'CloudianS3'},
'RetryAttempts': 0},
'Buckets': [{'Name': 'bucket1',
'CreationDate': datetime.datetime(2020, 12, 1, 3, 2, 25, 876000, tzinfo=tzutc())},
{'Name': 'pythonbucket2',
'CreationDate': datetime.datetime(2020, 12, 13, 19, 51, 20, 267000, tzinfo=tzutc())},
{'Name': 'pythonbucket3',
'CreationDate': datetime.datetime(2020, 12, 13, 21, 41, 8, 495000, tzinfo=tzutc())}],
'Owner': {'DisplayName': '', 'ID': '27b8e84694ca0b529d5379049564ebe1'}}
The boto3 S3Client object created by the Python program has many methods that can perform various operations (create/delete buckets, upload/download data, etc.) to the object storage/Cloudian.
Detailed information on the methods available in S3Client (boto3.client) https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
Next time wants to use this S3Client object to operate Cloudian, which is an object storage, in various ways.
Recommended Posts