Create Cognito user list in S3 with SQS Deploy queue function and API to Lambda with SAM

The title is inconvenient in Japanese, but I will set it in order.

Build a function on Lambda that automatically creates a CSV file of the user list managed by AWS Cognito on S3. Set SQS as the trigger to activate that Lambda, and implement the API to add the message queue to SQS in another Lambda so that HTTP requests can be received via API Gateway.

The configuration diagram is as follows.

image.png

AWS-CLI installation and IAM configuration

$ brew install awscli
$ aws configure
AWS Access Key ID [None]: ************
AWS Secret Access Key [None]: ************************
Default region name [None]: ap-northeast-1
Default output format [None]: json

SAM installation

$ brew tap aws/tap
$ brew install aws-sam-cli
$ sam --version
SAM CLI, version 1.0.0

Initialize SAM project

This time, we will implement it in Python 3.8 based on the Hello World template.

% sam init
Which template source would you like to use?
	1 - AWS Quick Start Templates
	2 - Custom Template Location
Choice: 1

Which runtime would you like to use?
	1 - nodejs12.x
	2 - python3.8
	3 - ruby2.7
・ ・ ・
Runtime: 2

Project name [sam-app]: sample-app

Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

AWS quick start application templates:
	1 - Hello World Example
	2 - EventBridge Hello World
・ ・ ・
Template selection: 1

-----------------------
Generating application:
-----------------------
Name: sample-app
Runtime: python3.8
Dependency Manager: pip
Application Template: hello-world
Output Directory: .

Next steps can be found in the README file at ./sample-app/README.md

Creating a Cognito user pool

This time, I created it with the default settings for the time being, and manually added one suitable user. Use the created user pool ID when creating the file.

image.png

image.png

Implementation

I implemented the main files as follows.

app.py


import json
import requests
import boto3
from datetime import datetime
import pandas as pd


def lambda_handler(event, context):

    try:
        sqs_client = boto3.client("sqs")
        #Once you deploy SQS, a queue will be automatically created in SQS, so set it.
        queue_url = "https://sqs.ap-northeast-1.amazonaws.com/********/**********"
        print(queue_url)
        now = datetime.now()
        date_str = now.strftime('%Y/%m/%d-%H:%M:%S')
        sqs_response = sqs_client.send_message(
            QueueUrl=queue_url,
            MessageBody=json.dumps(date_str)
        )
    except requests.RequestException as e:
        # Send some context about this error to Lambda Logs
        print(e)

        raise e

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "hello world"
        }),
    }

def csv_create_handler(event, context):
    for record in event['Records']:
        payload=record["body"]

        date_str = str(payload)
        s3 = boto3.resource('s3')
        bucket = s3.Bucket('arkth-user-list-files')

        cognito_client = boto3.client('cognito-idp')

        response = cognito_client.list_users(
            UserPoolId = 'ap-northeast-***********',
            AttributesToGet = ['email','sub']
        )

        data = []
        for user in response["Users"]:
            data.append([user['Username'], user['Enabled']])

        Coulum = ['Username', 'Enabled']
        df = pd.DataFrame(data, columns=Coulum)

        df_csv = df.to_csv(index=None)
        objkey = 'user-list.csv'
        print(objkey)
        putobj = bucket.Object(objkey)
        putobj.put(Body=df_csv)

def support_datetime_default(obj):
    if isinstance(obj, datetime):
        return obj.isoformat()
    raise TypeError(repr(obj) + " is not JSON serializable")

template.yaml


AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sample-app

  Sample SAM Template for sample-app

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3
    Runtime: python3.8

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

  UsersCsv:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 60

  CsvCreateFunction:
    Properties:
      CodeUri: hello_world/
      Handler: app.csv_create_handler
      Events:
        MySQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt UsersCsv.Arn
            BatchSize: 10
    Type: AWS::Serverless::Function

requirement.txt


requests
boto3
pandas

SAM build

$ sam build
Building function 'HelloWorldFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Building function 'CsvCreateFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Launch SAM locally

$ sam local start-api
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-09-05 22:11:22  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)

An error will occur because the SQS endpoint is dummy, but access http://127.0.0.1:3000/hello from a browser and check if there is a Syntax Error etc.

Deploy SAM

The first time you deploy with --guided, samconfig.toml is created interactively.

$ sam deploy --guided

Configuring SAM deploy
======================

	Looking for samconfig.toml :  Not found

	Setting default arguments for 'sam deploy'
	=========================================
	Stack Name [sam-app]: sample-app
	AWS Region [us-east-1]: ap-northeast-1
	#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
	Confirm changes before deploy [y/N]: y
	#SAM needs permission to be able to create roles to connect to the resources in your template
	Allow SAM CLI IAM role creation [Y/n]: Y
	HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
	Save arguments to samconfig.toml [Y/n]: Y
・ ・ ・ ・
・ ・ ・ ・
・ ・ ・ ・

It will take some time, but IAM and Api Gateway will be set up and deployed appropriately.

Environmental setting

image.png

Then build & deploy again.

$ sam build
$ sam deploy

Operation check

Access the API Gateway endpoint set in the Lambda function of Hello World with a browser.

After that, when you access the S3 Bucket, the following CSV file is created.

image.png

Recommended Posts

Create Cognito user list in S3 with SQS Deploy queue function and API to Lambda with SAM
Challenge to create time axis list report with Toggl API and Python
Create API with Python, lambda, API Gateway quickly using AWS SAM
How to create dataframes and mess with elements in pandas
Send images taken with ESP32-WROOM-32 to AWS (API Gateway → Lambda → S3)
Crawling with Python and Twitter API 2-Implementation of user search function
How to create a serverless machine learning API with AWS Lambda
Create user authentication function in Airflow
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python
Set up a Lambda function and let it work with S3 events!
Upload what you got in request to S3 with AWS Lambda Python
[AWS] Link Lambda and S3 with boto3
Connect to s3 with AWS Lambda Python
Create and deploy Flask apps in PTVS
Function to extract the maximum and minimum values ​​in a slice with Go
Create an API to convert PDF files to TIF images with FastAPI and Docker