Send a request from AWS Lambda to Amazon Elasticsearch Service

Amazon Elasticsearch Service is Elasticsearch. You can use the endpoint as Elasticsearch normally, and choose the access policy from the following.

By the way, I thought about a request from Lambda based on IAM to lightly parse the file placed in S3 for a little convenience and thrust it into Elasticsearch, but at present it seems that I make my own signature request and throw it in.

result. I tried it, but now it seems easier to stream it to the CloudWatch Logs stream and then use the preset functions. If boto3 resource is implemented for ES, it seems that there will be no trouble.

It worked for the time being, so make a note of the code.

Elasticsearch domain side policy

--Allow roles applied to Lambda in IAM template --ʻEs: * , ʻes: ESHttpGet, etc.

Lambda function

Aside from EventSource, just send a GET to the ES / for the time being. The following materials were helpful.

-Complete Version 4 Signing Process Example (Python)-Amazon Web Services --Bringed a function to create an Authorization header. --Javascript for LogsToElasticsearch that can be created with CloudWatch Logs --Used as a reference for headers and environment variables other than Authorization. ――It's almost like transplanting this.

Mostly the code to make the header. .. ..

function.py


import urllib2, datetime, os, sys
import hashlib, hmac

endpoint = 'search-sandbox01-xxxxxxxxxxxxxxxxxxxx.ap-northeast-1.es.amazonaws.com'
url = "https://" + endpoint
region = 'ap-northeast-1'
service = 'es'
method = 'GET'

access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
if access_key is None or secret_key is None:
    print 'No access key is available.'
    sys.exit()

def sign(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, 'aws4_request')
    return kSigning


def lambda_handler(event, context):
    t = datetime.datetime.utcnow()
    amzdate = t.strftime('%Y%m%dT%H%M%SZ')
    datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope
    
    canonical_uri = '/'
    canonical_querystring = ""
    canonical_headers = 'host:' + endpoint + '\n' + 'x-amz-date:' + amzdate + '\n'
    signed_headers = 'host;x-amz-date'
    payload_hash = hashlib.sha256('').hexdigest()
    canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash
    
    algorithm = 'AWS4-HMAC-SHA256'
    credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
    string_to_sign = algorithm + '\n' +  amzdate + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request).hexdigest()
    
    signing_key = getSignatureKey(secret_key, datestamp, region, service)
    signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()
    
    authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' +  'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature
    
    headers = {
          'Content-Type': 'application/json',
                'Host': endpoint,
                'X-Amz-Security-Token': os.environ.get('AWS_SESSION_TOKEN'),
                'X-Amz-Date': amzdate,
                'Authorization': authorization_header
    }
    
    req = urllib2.Request(url, None, headers)
    
    try:
        print urllib2.urlopen(req).read()
    except urllib2.URLError, e:
        print e.read()

Is it possible to make this header quickly by using the botocore module? While thinking.

Addendum: Click here for an example of "Make it quickly using the botocore module". [Small story] Use only the signing process of botocore's AWS API request | Developers.IO

You Know, for Search

When I try to cURL to ES from my hand, it is blocked properly.

$ curl search-sandbox01-xxxxxxxxxx.ap-northeast-1.es.amazonaws.com
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:ap-northeast-1:xxxxxxxxx:domain/sandbox01/"}

When I opened Lambda and Save and Test the above code, I got a reply from ES safely.

START RequestId: b6ce7399-85eb-11e5-b2c3-99a74c2c1765 Version: $LATEST
{
  "status" : 200,
  "name" : "Chief Examiner",
  "cluster_name" : "xxxxxxxxxxx:sandbox01",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

END RequestId: b6ce7399-85eb-11e5-b2c3-99a74c2c1765
REPORT RequestId: b6ce7399-85eb-11e5-b2c3-99a74c2c1765	Duration: 64.87 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 16 MB

Recommended Posts

Send a request from AWS Lambda to Amazon Elasticsearch Service
Terraform configured to launch AWS Lambda from Amazon SQS
A quick explanation from creating AWS Lambda Layers to linking
A story that I was addicted to calling Lambda from AWS Lambda.
Send a message from Python to Slack
Send a message from Slack to a Python server
Bulk deployment with CFn to take a manual snapshot of Elasticsearch Service with Lambda
Tweet from AWS Lambda
Amazon SNS → AWS Lambda → Slack → AWS Chatbot to execute AWS commands
Input Zaim data to Amazon Elasticsearch Service with Logstash
How to launch AWS Batch from a python client app
Send a signal to subprocess
I want to send a message from Python to LINE Bot
Created a package to support AWS Lambda development in Go language
Send images taken with ESP32-WROOM-32 to AWS (API Gateway → Lambda → S3)
Make a request from Device Farm (appium python) to API Gateway
Run Systems Manager from Lambda to get a backup of EC2
How to send a request to the DMM (FANZA) API with python
How to create a serverless machine learning API with AWS Lambda
From Elasticsearch installation to data entry
[Lambda] [Python] Post to Twitter from Lambda!
Send commands from Atom to Maya
Tweet in Chama Slack Bot ~ How to make a Slack Bot using AWS Lambda ~
Until you get a snapshot of Amazon Elasticsearch service and restore it
How to get a value from a parameter store in lambda (using python)
I tried to send a registration completion email from Gmail with django.
Upload what you got in request to S3 with AWS Lambda Python
I tried to make a url shortening service serverless with AWS CDK
Change AWS EC2 instance from t2 to t3
How to create a clone from Github
Execute a script from Jupyter to process
How to access RDS from Lambda (python)
How to create a repository from media
Connect to s3 with AWS Lambda Python
[Amazon Linux] Switching from Python 2 series to Python 3 series
Summary of how to write AWS Lambda
[LINE Messaging API] I want to send a message from the program to everyone's LINE
I made a library to operate AWS CloudFormation stack from CUI (Python Fabric)
[AWS; Introduction to Lambda] 2nd; Extract sentences from json file and save S3 ♬
[Free] Hit the Clash Royale API from lambda and send it to LINE