Try to make your own AWS-SDK with bash

Overview

I made AWS-SDK with bash. AWS-AWS will be able to handle even PCs that do not have an SDK or programming environment.

When making it, I try to observe the following conditions.

Articles can be opened and closed for each service, so please read only the services you need.

Diagram

draw_1.png

The finished product

Since it is not possible to check the operation of all the functions of the SDK, we will only check the operation below in this article.

Before use: Prepare profile

When using it, please create profile information in ~ / .aws / credentials. The format is text and no extension is required.

~/.aws/credentials


[default]
aws_access_key_id = AKIA**************************
aws_secret_access_key = *****************************

Operation check: Operate DynamoDB data

Steps to operate DynamoDB (click to open)

Get data from DynamoDB


./aws-sdk-bash.sh "default" "ap-northeast-1" "dynamodb" "/" "" \
"content-type:application/x-amz-json-1.0;host:@;x-amz-date:@;x-amz-target:DynamoDB_20120810.GetItem" \
"{\"TableName\": \"target_table\", \"Key\": {\"id\": {\"S\": \"key\"}}}"

{"Item":{"entity":{"S":"string_data"},"id":{"S":"key"}}}

** When the same request is written in AWS-SDK (boto3) **

boto3


from boto3 import Session
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "dynamodb"
).get_item(
    TableName = "target_table",
    Key = {"id" : {"S" : "key"}}
)

** Confirm that you can register and scan data by changing the argument **

Other operations of DynamoDB


#Enter data
./aws-sdk-bash.sh "default" "ap-northeast-1" "dynamodb" "/" "" \
"content-type:application/x-amz-json-1.0;host:@;x-amz-date:@;x-amz-target:DynamoDB_20120810.PutItem" \
"{\"TableName\": \"target_table\", \"Item\": {\"id\": {\"S\": \"newData\"}, \"entity\":{\"S\":\"new_string_data\"}}}"

{}

#After entering the data, check that the number of cases is increasing with Scan
./aws-sdk-bash.sh "default" "ap-northeast-1" "dynamodb" "/" "" \
"content-type:application/x-amz-json-1.0;host:@;x-amz-date:@;x-amz-target:DynamoDB_20120810.Scan" \
"{\"TableName\": \"target_table\"}"

{"Count":2,"Items":[{"entity":{"S":"new_string_data"},"id":{"S":"newData"}},{"entity":{"S":"string_data"},"id":{"S":"key"}}],"ScannedCount":2}

** Request parameter details **

Parameter name Contents of the above request example Remarks
Profile name default Authentication name
region ap-northeast-1 region情報
Service name dynamodb Service name。DynamoDB
URI path / Endpoint path, root path in DynamoDB
URL query Not used in DynamoDB
HTTP header content-type:application/x-amz-json-1.0;
host:@;
x-amz-date:@;
x-amz-target:DynamoDB_20120810.GetItem
@Is given on the script side
x-amz-You can specify the process to be executed by target.
Get (GetItem)
Add (PutItem)
Scan… etc
POST payload {"TableName":"target_table", "Key": {"id" : "S" : "key"}}

Operation check: Run Lambda directly

Procedure for operating Lambda (click to open)

Run Lambda


./aws-sdk-bash.sh "default" "ap-northeast-1" "lambda" "/2015-03-31/functions/sample_lambda/invocations" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-invocation-type:RequestResponse;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"{\"Message\":\"Hello\"}"

{"statusCode": 200, "body": "\"Hello from Lambda! I am SAMPLE\""}

** When the same request is written in AWS-SDK (boto3) **

boto3


from boto3 import Session
import json
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "lambda"
).invoke(
    FunctionName = "sample_lambda",
    InvocationType = "RequestResponse",
    Payload = json.dumps({"Message" : "Hello"})
)

** Request parameters **

Parameter name Contents of the above request example Remarks
Profile default Authentication name
region ap-northeast-1 region情報
Service name lambda Service name。Lambda
URI path /2015-03-31/functions/sample_lambda/invocations Operation to be performed
Lambda name "sample"_Issue an "invoke" command to "lambda"
URL query Not used with Lambda
HTTP header host:@;
x-amz-content-sha256:@;
x-amz-date:@;
x-amz-invocation-type:RequestResponse;
x-amz-user-agent:aws-sdk-js/2.668.0 callback
@Is given on the script side
Specify RequestResponse for synchronous processing with response
UA borrows JavaScript version of UA
POST payload {"Message":"Hello"}

Operation check: Manipulate SQS data

Steps to operate SQS (click to open)

Run SQS


#Send message
./aws-sdk-bash.sh "default" "ap-northeast-1" "sqs" "/" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"Action=SendMessage&MessageBody=%7B%22id%22%3A%22NewMessage%22%7D&QueueUrl=https%3A%2F%2Fsqs.ap-northeast-1.amazonaws.com%2F***********************%2Fsqs-send-request-test-0424&Version=2012-11-05"

<?xml version="1.0"?><SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><SendMessageResult><MessageId>013a849d-f344-4d90-a6be-e37302c6b029</MessageId><MD5OfMessageBody>abc832604cb20908715bca6f197c8945</MD5OfMessageBody></SendMessageResult><ResponseMetadata><RequestId>01e07779-a593-5162-b698-2050d59e1524</RequestId></ResponseMetadata></SendMessageResponse>

#Receive the message you sent
./aws-sdk-bash.sh "default" "ap-northeast-1" "sqs" "/" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"Action=ReceiveMessage&QueueUrl=https%3A%2F%2Fsqs.ap-northeast-1.amazonaws.com%2F*****************%2Fsqs-send-request-test-0424&Version=2012-11-05"

<?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult><Message><MessageId>6335c486-d1b6-4478-84c4-40b92f5d655b</MessageId><ReceiptHandle>***********</ReceiptHandle><MD5OfBody>abc832604cb20908715bca6f197c8945</MD5OfBody><Body>{&quot;id&quot;:&quot;NewMessage&quot;}</Body></Message></ReceiveMessageResult><ResponseMetadata><RequestId>9e05f85b-4a27-5d6f-97c4-af723bb68a70</RequestId></ResponseMetadata></ReceiveMessageResponse>

** When sending a message with boto3 **

from boto3 import Session
import json
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "sqs"
).send_message(
    QueueUrl = "https://sqs.ap-northeast-1.amazonaws.com/**************/sqs-send-request-test-0424",
    MessageBody = json.dumps({"id": "NewMessage"})
)

** Request parameters **

Parameter name Contents of the above request example Remarks
Profile default Authentication name
region ap-northeast-1 region情報
Service name sqs Service name。SQS
URI path / Not used in SQS
URL query Not used in SQS
HTTP header host:@;
x-amz-content-sha256:@;
x-amz-date:@;
x-amz-user-agent:aws-sdk-js/2.668.0 callback
@Is given on the script side
UA borrows JavaScript version of UA
POST payload Action=SendMessage&MessageBody=%7B%22id%22%3A%22NewMessage%22%7D&QueueUrl=https%3A%2F%2Fsqs.ap-northeast-1.amazonaws.com%2F***********************%2Fsqs-send-request-test-0424&Version=2012-11-05 SQS is submitted in form data format.
Use Action to switch operations such as sending and receiving.

Operation check: Get own profile information from STS

Steps to operate STS (click to open)

Run STS


#Get your own information
./aws-sdk-bash.sh "default" "ap-northeast-1" "sts" "/" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"Action=GetCallerIdentity&Version=2011-06-15"

<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <GetCallerIdentityResult>
    <Arn>arn:aws:iam::*****************:user/**************</Arn>
    <UserId>*****************</UserId>
    <Account>*****************</Account>
  </GetCallerIdentityResult>
  <ResponseMetadata>
    <RequestId>1703851c-5b55-49b5-8bcf-130ab4dbb6f2</RequestId>
  </ResponseMetadata>
</GetCallerIdentityResponse>

** When the same process is written in boto3 **

from boto3 import Session
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "sts"
).get_caller_identity()

** Request parameters **

Parameter name Contents of the above request example Remarks
Profile default Authentication name
region ap-northeast-1 region情報
Service name sts Service name。STS
URI path / Not used in SQS
URL query Not used in SQS
HTTP header host:@;
x-amz-content-sha256:@;
x-amz-date:@;
x-amz-user-agent:aws-sdk-js/2.668.0 callback
@Is given on the script side
UA borrows JavaScript version of UA
POST payload Action=GetCallerIdentity&Version=2011-06-15 STS is submitted in form data format.
Switch operations with Action.

Source code

If you want to run it, save it in any location with the file name aws-sdk-bash.sh. For the command to be executed, refer to "Operation check".

aws-sdk-bash.sh


#!/bin/bash

#Profile path is fixed
CREDENTIALS_FILE=~/.aws/credentials
#Algorithm is signature v4, hmac SHA256
ALGORITHM='AWS4-HMAC-SHA256'

#Script arguments
_INPUT_PROFILE_NAME=$1
_INPUT_REGION=$2
_INPUT_SERVICE=$3
_INPUT_CANONICAL_URI=$4
_INPUT_CANONICAL_QUERY_STRING=$5
_INPUT_OPTIONAL_HEADERS=$6
_INPUT_PAYLOAD=$7

#Creating an endpoint
METHOD=POST
PROTOCOL=https
HOST_NAME=${_INPUT_SERVICE}.${_INPUT_REGION}.amazonaws.com

# .Get profile information from aws file
#Input profile path, profile name, profile key name, number of lines to read file
get_credentials () {
    _CREDENTIALS_FILE=$1; _PROFILE_NAME=$2; _KEY_NAME=$3; _READ_LENGTH=$4;

    # PROFILE_Get the line number of NAME
    PROFILE_IDX=`nl $_CREDENTIALS_FILE | grep $_PROFILE_NAME | head -n 1 | awk '{print $1}'`
    PROFILE_IDX_END=`expr $PROFILE_IDX + $_READ_LENGTH`

    #Get access key ID
    RESULT=`cat $CREDENTIALS_FILE | sed -n "${PROFILE_IDX},${PROFILE_IDX_END}p" | grep "=" | grep ${_KEY_NAME} | \
    tr -d " " | sed "s/=/ /g" | awk '{print $2}' | \
    head -n 1`

    echo -n $RESULT
}

#Create a hash with SHA256
#Input message: file, key: none
#Output hash: hex format
create_digest_from_file () {
    cat $1 | openssl dgst -sha256 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: text
#Output hash: hex format
sign_from_string () {
    echo -n $2 | openssl dgst -sha256 -hmac $1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: hex format
#Output hash: hex format
sign_from_string_with_hex_key () {
    echo -n $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: file, key: hex format
#Output hash: hex format
sign_from_file_with_hex_key () {
    cat $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

#HMAC the basic information of signature v4 (access key ID, date and time of transmission, region, service name)-Hash with SHA256
get_signature_key () {
    TEMP_DATE=`sign_from_string AWS4$1 $2`
    TEMP_REGION=`sign_from_string_with_hex_key $TEMP_DATE $3`
    TEMP_SERVICE=`sign_from_string_with_hex_key $TEMP_REGION $4`
    sign_from_string_with_hex_key $TEMP_SERVICE 'aws4_request'
}

#Get access key ID
ACCESS_KEY_ID=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_access_key_id 2`

#Get a secret access key
SECRET_ACCESS_KEY=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_secret_access_key 2`

#Get UTC date and time (Format example: 2020/12/31T12:34:At 50 AMZ_DATE:20201231T123450Z DATE_STAMP:20201231)
UTC_DATE=`date -Iseconds -u | sed "s/+/ /g" | awk '{print $1 "Z"}'`
AMZ_DATE=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/://g"`
DATE_STAMP=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/T/ /g" | awk '{print $1}'`

#Create temporary files, delete temporary files when finished
TEMP_HEADERS=`mktemp`
TEMP_CANONICAL_REQUEST=`mktemp`
TEMP_STRING_TO_SIGN=`mktemp`
TEMP_PAYLOAD=`mktemp`
trap "rm -f $TEMP_HEADERS; rm -f $TEMP_CANONICAL_REQUEST; rm -f $TEMP_STRING_TO_SIGN; rm -f $TEMP_PAYLOAD" EXIT

#Copy the payload to a temporary file
echo -n ${_INPUT_PAYLOAD} > $TEMP_PAYLOAD

#Create a SHA256 hash (without key) from the BODY data to be sent
PAYLOAD_HASH=`create_digest_from_file $TEMP_PAYLOAD`

#Set the send header
echo -n "${_INPUT_OPTIONAL_HEADERS}" | sed "s#x-amz-content-sha256:@#x-amz-content-sha256:${PAYLOAD_HASH}#" | sed "s#host:@#host:${HOST_NAME}#" | sed "s#x-amz-date:@#x-amz-date:${AMZ_DATE}#" | xargs -d ";" -r -I @ echo @ >> $TEMP_HEADERS
SIGNED_HEADERS=`echo -n "${_INPUT_OPTIONAL_HEADERS}" | xargs -d ";" -r -I @ echo ";@" | sed 's/:.*//'`
SIGNED_HEADERS=`echo -n $SIGNED_HEADERS | sed 's/ //g' | sed 's/^;//'`

#Make a canonical request
#Standardize request information
#Reference information: Task 1:Make a legitimate request for signature version 4
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-create-canonical-request.html
echo $METHOD > $TEMP_CANONICAL_REQUEST
echo $_INPUT_CANONICAL_URI >> $TEMP_CANONICAL_REQUEST
echo $_INPUT_CANONICAL_QUERY_STRING >> $TEMP_CANONICAL_REQUEST
cat $TEMP_HEADERS >> $TEMP_CANONICAL_REQUEST
echo "" >> $TEMP_CANONICAL_REQUEST
echo $SIGNED_HEADERS >> $TEMP_CANONICAL_REQUEST
echo -n $PAYLOAD_HASH >> $TEMP_CANONICAL_REQUEST

#Create a signature string
#Reference information: Task 2:Create a signature version 4 signature string
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-create-string-to-sign.html
CANONICAL_REQUEST_HASH=`create_digest_from_file $TEMP_CANONICAL_REQUEST`
CREDENTIAL_SCOPE=${DATE_STAMP}/${_INPUT_REGION}/${_INPUT_SERVICE}/"aws4_request"

#Set signature algorithm and signature scope
echo ${ALGORITHM} > $TEMP_STRING_TO_SIGN
echo ${AMZ_DATE} >> $TEMP_STRING_TO_SIGN
echo ${CREDENTIAL_SCOPE} >> $TEMP_STRING_TO_SIGN
echo -n ${CANONICAL_REQUEST_HASH} >> $TEMP_STRING_TO_SIGN

#Calculate Signature for signature
#Reference information: Task 3:Calculate AWS Signature Version 4 Signatures
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-calculate-signature.html
SIGNING_KEY=`get_signature_key ${SECRET_ACCESS_KEY} ${DATE_STAMP} ${_INPUT_REGION} ${_INPUT_SERVICE}`
SIGNATURE=`sign_from_file_with_hex_key ${SIGNING_KEY} ${TEMP_STRING_TO_SIGN}`

#Set the signature in the header of the HTTP request
#Reference information: Task 4:Add signature to HTTP request
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-add-signature-to-request.html
AUTHORIZATION_HEADER="${ALGORITHM} Credential=${ACCESS_KEY_ID}/${CREDENTIAL_SCOPE}, SignedHeaders=${SIGNED_HEADERS}, Signature=${SIGNATURE}"

#Send an HTTP request.
QUERY_STRING=${_INPUT_CANONICAL_QUERY_STRING}
if [ ! $QUERY_STRING = "" ]; then
    QUERY_STRING="?${QUERY_STRING}"
fi
curl -s -X POST ${PROTOCOL}://${HOST_NAME}${_INPUT_CANONICAL_URI}${QUERY_STRING} -d @$TEMP_PAYLOAD -H @$TEMP_HEADERS -H "Authorization: ${AUTHORIZATION_HEADER}"

Explanation of source code

The processing flow is as follows. There are different SDKs for different languages, but if you implement these four you can do the same thing as the AWS-SDK.

\def\of{\unicode[serif]{x306E}}
\bbox[8px, border: 2px solid gray]{\rlap{\tt 1.\quad profile information\Get}\hspace{80mm}}
\triangledown
\def\of{\unicode[serif]{x306E}}
\bbox[8px, border: 2px solid gray]{\rlap{\tt 2.\quad parameter\Get and create}\hspace{80mm}}
\triangledown
\bbox[8px, border: 2px solid gray]{\rlap{\tt 3.\Hash quad parameters with SHA256}\hspace{80mm}}
\triangledown
\bbox[8px, border: 2px solid gray]{\rlap{\tt 4.\Send HTTP request with quad curl}\hspace{80mm}}

For version 4 signing, see the official AWS "Completely Signed Version 4 Signing Process Example (Python)" -request-examples.html) Please refer to.

Hereafter, each process will be explained.

1. Acquisition of profile information

This is the process to get profile information from ~ / .aws / credentials.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
command&Purpose\\
\hline 
{\tt nl} &line number+File\Displays the contents of.\\
&line number\It is a cat that comes out.\\
\hdashline 
{\tt sed\;\text{-n}} &Specified line\Get the text of.\\
\hline 
\end{array}

Use "nl + grep profile name" to get the number of lines of the profile name to get. "Sed -n" will take the line after the profile name, so take the access key ID and secret access key from there.

aws-sdk-bash.sh_Acquisition of signature information


# .Get profile information from aws file
#Input profile path, profile name, profile key name, number of lines to read profile
get_credentials () {
    _CREDENTIALS_FILE=$1; _PROFILE_NAME=$2; _KEY_NAME=$3; _READ_LENGTH=$4;

    # PROFILE_Get the line number of NAME
    PROFILE_IDX=`nl $_CREDENTIALS_FILE | grep $_PROFILE_NAME | head -n 1 | awk '{print $1}'`
    PROFILE_IDX_END=`expr $PROFILE_IDX + $_READ_LENGTH`

    #Get access key ID
    RESULT=`cat $CREDENTIALS_FILE | sed -n "${PROFILE_IDX},${PROFILE_IDX_END}p" | grep "=" | grep ${_KEY_NAME} | \
    tr -d " " | sed "s/=/ /g" | awk '{print $2}' | \
    head -n 1`

    echo -n $RESULT
}

aws-sdk-bash.sh_Caller


#Get access key ID
ACCESS_KEY_ID=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_access_key_id 2`

#Get a secret access key
SECRET_ACCESS_KEY=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_secret_access_key 2`

2. Get and create parameters

** Get date and time **

Get the date and time in UTC. Create a string in a format that includes hours, minutes, and seconds, and a format that does not include hours, minutes, and seconds.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
value&format\\
\hline 
{\tt \text{AMZ_DATE}} & {\tt YYYYMMDDTHHMMSS}\Specify in the format of.\\
&Behind{\tt UTC}Point to{\tt Z}I will put on.\\
&Colon, hyphen\Do not include symbols such as.\\
\hdashline 
{\tt \text{DATE_STAMP}} & {\tt YYYYMMDD}\Specify in the format of.\\
&Specify only the date.\\
&Colon, hyphen\Do not include symbols such as.\\
\hline 
\end{array}

aws-sdk-bash.sh_Get UTC date and time


#Get UTC date and time (Format example: 2020/12/31T12:34:At 50 AMZ_DATE:20201231T123450Z DATE_STAMP:20201231)
UTC_DATE=`date -Iseconds -u | sed "s/+/ /g" | awk '{print $1 "Z"}'`
AMZ_DATE=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/://g"`
DATE_STAMP=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/T/ /g" | awk '{print $1}'`

** Handle text **

The character string is written to a temporary file and handled. Temporary files are deleted when the process ends.

You can handle it with bash variables, but it is easier to implement if you handle strings in a file. For signature information, blank lines, line breaks at the end, and the order of appearance of data are strictly determined, and if even one shifts, the signature will not pass.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
command&Purpose\\
\hline 
{\tt mktemp} & /Create a temporary file in tmp.\\
&File names are unique and\\
&Access rights are also set appropriately.\\
\hdashline 
{\tt trap} &specific\The command is executed at the timing of.\\
&Deletes temporary files at the end of the process.\\
\hline 
\end{array}

aws-sdk-bash.sh_Creating a temporary file


#Create temporary files, delete temporary files when finished
TEMP_HEADERS=`mktemp`
TEMP_CANONICAL_REQUEST=`mktemp`
TEMP_STRING_TO_SIGN=`mktemp`
TEMP_PAYLOAD=`mktemp`
trap "rm -f $TEMP_HEADERS; rm -f $TEMP_CANONICAL_REQUEST; rm -f $TEMP_STRING_TO_SIGN; rm -f $TEMP_PAYLOAD" EXIT

** Parameter replacement **

There are some parameters that the user is not aware of when making a request. Such parameters are set on the script side by omitting the value with @.

\begin{array}{ll}
User-set parameters& {\tt \text{x-amz-date:@}} \\
Parameters actually sent& {\tt \text{x-amz-date:20200504T145432Z}}\\
& \\
\end{array}
\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
header\key of&Value to be set\\
\hline 
{\tt \text{x-amz-content-sha256}} &payload\Hash value of\\
\hdashline 
{\tt host} & {\tt AWS endpoint\URL}\\
\hdashline 
{\tt \text{x-amz-date}}  &Date and time of request\\
\hline 
\end{array}

aws-sdk-bash.sh_Set header


#Create a SHA256 hash (without key) from the BODY data to be sent
PAYLOAD_HASH=`create_digest_from_file $TEMP_PAYLOAD`

#Set the send header
echo -n "${_INPUT_OPTIONAL_HEADERS}" | sed "s#x-amz-content-sha256:@#x-amz-content-sha256:${PAYLOAD_HASH}#" | sed "s#host:@#host:${HOST_NAME}#" | sed "s#x-amz-date:@#x-amz-date:${AMZ_DATE}#" | xargs -d ";" -r -I @ echo @ >> $TEMP_HEADERS
SIGNED_HEADERS=`echo -n "${_INPUT_OPTIONAL_HEADERS}" | xargs -d ";" -r -I @ echo ";@" | sed 's/:.*//'`
SIGNED_HEADERS=`echo -n $SIGNED_HEADERS | sed 's/ //g' | sed 's/^;//'`

3. Hashing

To get HMAC-SHA256, write as follows in Python.

HMAC-Get SHA256


def sign(key, msg):
    return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(("AWS4" + key).encode("utf-8"), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, "aws4_request")
    return kSigning

To do the same with bash, use openssl.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
command&Purpose\\
\hline 
{\tt openssl\;dgst\;\text{-}sha256} &Text{\tt SHA256}Hash with.\\
&The result will be a hexadecimal number.\\
\hdashline 
{\tt openssl\;dgst\;\text{-}sha256} &Text{\tt SHA256}Hash with.\\
\qquad{\tt \text{-}hmac\;(key)} &Plain text\Specify the key of.\\
&The result will be a hexadecimal number.\\
\hdashline 
{\tt openssl\;dgst\;\text{-}sha256} &Text{\tt SHA256}Hash with.\\
\qquad{\tt \text{-}mac\;hmac} &Hexadecimal\Specify the key of.\\
\qquad{\tt \text{-}macopt\;hexkey:(key)}&The result will be a hexadecimal number.\\
\hline 
\end{array}

aws-sdk-bash.sh_Hashing


#Create a hash with SHA256
#Input message: file, key: none
#Output hash: hex format
create_digest_from_file () {
    cat $1 | openssl dgst -sha256 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: text
#Output hash: hex format
sign_from_string () {
    echo -n $2 | openssl dgst -sha256 -hmac $1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: hex format
#Output hash: hex format
sign_from_string_with_hex_key () {
    echo -n $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: file, key: hex format
#Output hash: hex format
sign_from_file_with_hex_key () {
    cat $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

#HMAC the basic information of signature v4 (access key ID, date and time of transmission, region, service name)-Hash with SHA256
#Same process as getSignatureKey in Python
get_signature_key () {
    TEMP_DATE=`sign_from_string AWS4$1 $2`
    TEMP_REGION=`sign_from_string_with_hex_key $TEMP_DATE $3`
    TEMP_SERVICE=`sign_from_string_with_hex_key $TEMP_REGION $4`
    sign_from_string_with_hex_key $TEMP_SERVICE 'aws4_request'
}

How to check AWS-SDK request

The format of the payload and header thrown by the AWS-SDK varies from service to service. There is no official documentation, so you have to look it up yourself.

service How to specify the method payload
Lambda URL path JSON
DynamoDB header JSON
SQS payload Form format
STS payload Form format

How to check with boto3 + Wireshark

If you communicate as it is, it will be encrypted, so I will add a little work.

Turn off encryption and use HTTP communication


import boto3
client = boto3.client("dynamodb", use_ssl = False)

print(client.get_item(TableName = "target_table", Key = {"id" : {"S":"key"}}))

If use_ssl is set to False, it will be sent to port 80. Since it communicates in plain text, it can be read by Wireshark.

capture_1.png

If you look at Wireshark, you can see that the data we are sending is as follows.

text


POST / HTTP/1.1
Host: dynamodb.ap-northeast-1.amazonaws.com
Accept-Encoding: identity
X-Amz-Target: DynamoDB_20120810.GetItem
Content-Type: application/x-amz-json-1.0
User-Agent: Boto3/1.12.43 Python/3.8.2 Windows/10 Botocore/1.15.43
X-Amz-Date: 20200501T213154Z
Authorization: AWS4-HMAC-SHA256 Credential=AKIA**********/20200501/ap-northeast-1/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=***********************************************************
Content-Length: 58

{"TableName": "target_table", "Key": {"id": {"S": "key"}}}

It is OK if you set the data so that the same request will fly.

Note that use_ssl cannot be used with some services. For example, Lambda and MQTT. When I try to connect to port 80 with a service that only supports HTTPS, it times out without a response.

How to check with the verification function of the browser

If you want to see the payload on a service that requires SSL, use the javascript version of the SDK.

test.html


<script src="https://sdk.amazonaws.com/js/aws-sdk-2.668.0.min.js"></script>
<script type="text/javascript">
    AWS.config.update({
        accessKeyId : 'AKIA******************',
        secretAccessKey : '**********************************'
    });
    AWS.config.region = 'ap-northeast-1';
    
    let lambda = new AWS.Lambda();
    let params = {
        FunctionName : 'sample_lambda',
        InvocationType : 'RequestResponse',
        Payload : JSON.stringify({
            "Message" : "Hello"
        })
    };
    
    lambda.invoke(params, (err, data) => console.log(JSON.parse(data.Payload)));
</script>

After writing the process you want to check in the HTML file, open it in a browser. With Chrome, you can check network communication data by right-clicking and opening "Verify" in your browser.

lambda.png

There are differences such as browser-specific data included and CORS may be required, but we have all the necessary data. With reference to this, it is OK if you set the same request to fly.

Recommended Posts

Try to make your own AWS-SDK with bash
Try HeloWorld in your own language (with How to & code)
To import your own module with jupyter
How to make your own domain site with heroku (free plan)
Introduction to Deep Learning (2) --Try your own nonlinear regression with Chainer-
Try to make a "cryptanalysis" cipher with Python
Make your own module quickly with setuptools (python)
Try to make a dihedral group with Python
Try to make client FTP fastest with Pythonista
Make your own music player with Bottle0.13 + jPlayer2.5!
Steps to install your own library with pip
Memo to create your own Box with Pepper's Python
Try to make a command standby tool with python
Try to improve your own intro quiz in Python
Try to make RESTful API with MVC using Flask 1.0.2
Try to put LED in your own PC (slightly)
Try to factorial with recursion
Easy to make with syntax
Try sorting your own objects with priority queue in Python
Try to make Qiita's Word Cloud from your browser history
Try to make a blackjack strategy by reinforcement learning (③ Reinforcement learning in your own OpenAI Gym environment)
[Python] Make your own LINE bot
Make your own manual. [Linux] [man]
Try to operate Facebook with Python
Solve your own maze with Q-learning
How to create your own Transform
Try implementing k-NN on your own
Try to profile with ONNX Runtime
Bridge ROS to your own protocol
Train UGATIT with your own dataset
Try to output audio with M5STACK
Solve your own maze with DQN
Make the theme of Pythonista 3 like Monokai (how to make your own theme)
Make your own VPC with a Single Public Subnet Only with boto
Try to make a web service-like guy with 3D markup language
[Introduction to StyleGAN] Unique learning of anime with your own machine ♬
Try to make capture software with as high accuracy as possible with python (1)
Try adding a wall to your IFC file with IfcOpenShell python
[TCP / IP] After studying, try to make an HTTP client-like with Python
Try to reproduce color film with Python
Add your own content view to mitmproxy
Your own Twitter client made with Django
Try logging in to qiita with Python
[Reinforcement learning] DQN with your own library
Try to make a kernel of Jupyter
Migrate your own CMS data to WordPress
Create your own DNS server with Twisted
Try to make a capture software with as high accuracy as possible with python (2)
Create your own Composite Value with SQLAlchemy
Make your own PC for deep learning
Fractal to make and play with Python
Try to make foldl and foldr with Python: lambda. Also time measurement
Let's make an image recognition model with your own data and play!
Try to predict cherry blossoms with xgboost
Try converting to tidy data with pandas
Quickly try to visualize datasets with pandas
How to install your own (root) CA
First YDK to try with Cisco IOS-XE
Publish your own Python library with Homebrew
Try to generate an image with aliasing
Try to make something like C # LINQ