Use AppSync on the front and back ends

Use AppSync on the front and back ends

This article is a hands-on article for reviewing and fixing the knowledge gained by developing Serverless Web App Mosaic. It is one of w2or3w / items / 87b57dfdbcf218de91e2).

It would be nice to read this article after looking at the following.

Introduction

An API called AppSync is used for data management of uploaded images and processed images, and for data transfer with the client side. DynamoDB is used as the data source for AppSync. AppSync can also be set up with the Amplify CLI, but I didn't use the Amplify CLI because it seemed that I couldn't specify the partition key or sort key for DynanoDB. Build DynamoDB and AppSync in the AWS console. Request from Vue on the front end using Amplify, and from Lambda (Python) on the back end by HTTP.

content

AppSync setup

I can set up AppSync with the Amplify CLI, but it seems that I can't specify the partition key or sort key of DynamoDB. I may be able to do it, but I didn't know how to do it. If anyone knows, please let me know. So, it's a little more troublesome than the command line, but let's make it with the AWS console.

Creating a DynamoDB table

Create DynamoDB as the data source for AppSync first. AWS Console> DynamoDB> Create Table Create a table with the following settings. Table name: sample_appsync_table Partition key: group (string) Sort key: path (string) Screenshot 2020-01-01 at 15.25.19.png

Creating the AppSync API

Once DynamoDB is created, it's time to create AppSync. AWS Console> AppSync> Create API

Step1: Getting started Select "Import DynamoDB table" and press the "Start" button. ![Screenshot 2020-01-02 at 23.36.05.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/394775/0d34eb39-937e-1b41-215e- 5216cc8228dc.png) Step2: Create a model Select the table you created earlier (sample_appsync_table). Select "New role" in "Create or use an existing role". Press the "Import" button. Screenshot 2020-01-02 at 23.36.23.png Just press the "Create" button. Screenshot 2020-01-02 at 23.40.08.png Step3: Create resource Set the API name and press the "Create" button. Screenshot 2020-01-02 at 23.42.14.png

Get schema.json

AWS Console> AppSync> sample_appsync_table> From the Schema menu Download schema.json. This file is used in the web application. Screenshot 2020-01-01 at 16.48.25.png

Obtaining credentials

AWS Console> AppSync> sample_appsync_table> From the Settings menu Check the information of ʻAPI URL and ʻAPI KEY in API Detail. This information will be used in the app. ink.png

Leave the authentication mode as "API Key". The API key is valid for 7 days by default, but you can extend it up to 365 days by editing it. I will write another article about how to use the authentication mode as a Cognito user like Storage.

Subscription on the front end (Vue web app)

After setting up AppSync, we will continue to update the web application.

Add graphql definition file

Create a src / graphql folder and add 3 files.

src/graphql/schema.json Add the file you downloaded earlier to the project as it is.

src/graphql/queries.js


export const listSampleAppsyncTables = `query listSampleAppsyncTables($group: String) {
  listSampleAppsyncTables(
    limit: 1000000
    filter: {
      group: {eq:$group}
    }
  )
  {
    items 
    {
      group
      path
    }
  }
}
`;

It is a query to get the record list by specifying the group of the partition key. This is a mystery, but if you don't specify the limit, you won't be able to get the data. I think it's an AppSync specification, not graphql, but what about it? I've specified a reasonably large number of 1000000, but honestly it's too subtle. If anyone knows a better way to write, please let me know.

src/graphql/subscriptions.js


export const onCreateSampleAppsyncTable = `subscription OnCreateSampleAppsyncTable($group: String) {
    onCreateSampleAppsyncTable(group : $group) {
        group
        path
    }
}
`;

This is a subscription for specifying the partition key group and notifying you with the information when a record is inserted. I wrote "when a record is inserted", but inserting a record directly into DynamoDB doesn't work and it must be inserted by AppSync create.

Added AppSync information to aws-exports.js

Add the information required to access AppSync to src / aws-exports.js.

src/aws-exports.js


const awsmobile = {
    "aws_project_region": "ap-northeast-1",
    "aws_cognito_identity_pool_id": "ap-northeast-1:********-****-****-****-************",
    "aws_cognito_region": "ap-northeast-1",
    "aws_user_pools_id": "ap-northeast-1_*********",
    "aws_user_pools_web_client_id": "**************************",
    "oauth": {},
    "aws_user_files_s3_bucket": "sample-vue-project-bucket-work",
    "aws_user_files_s3_bucket_region": "ap-northeast-1", 
    "aws_appsync_graphqlEndpoint": "https://**************************.appsync-api.ap-northeast-1.amazonaws.com/graphql",
    "aws_appsync_region": "ap-northeast-1",
    "aws_appsync_authenticationType": "API_KEY",
    "aws_appsync_apiKey": "da2-**************************"
};
export default awsmobile;

The information contained in this file is important, so please handle it with care so as not to leak it.

Web application implementation

After selecting an image in Home and uploading it, implement it to list the information of the image uploaded or monochrome processed image at the page transition destination.

Add a page called List. Router settings for that.

src/router.js


import Vue from 'vue'; 
import Router from 'vue-router'; 
import Home from './views/Home.vue'; 
import About from './views/About.vue'; 
import List from './views/List.vue'; 
 
Vue.use(Router); 
 
export default new Router({ 
  routes: [ 
    { 
      path: '/', 
      name: 'home', 
      component: Home,
    }, 
    { 
      path: '/about', 
      name: 'about', 
      component: About, 
    }, 
    { 
      path: '/list', 
      name: 'list', 
      component: List, 
    }, 
  ] 
});

List page view

src/views/List.vue


<template> 
  <List /> 
</template> 
 
<script> 
  import List from '../components/List' 
  export default { 
    components: { 
      List 
    } 
  } 
</script> 

List page components

src/components/List.vue


<template>
  <v-container>
    <p>list</p>
    <router-link to="/" >link to Home</router-link>
    <hr>
    
    <v-list>
      <v-list-item v-for="data in this.dataList" :key="data.path">
        <v-list-item-content>
          <a :href="data.image" target=”_blank”>
            <v-list-item-title v-text="data.path"></v-list-item-title>
          </a>
        </v-list-item-content>
        <v-list-item-avatar>
          <v-img :src="data.image"></v-img>
        </v-list-item-avatar>
      </v-list-item>
    </v-list>

  </v-container>
</template>

<script>
import { API, graphqlOperation, Storage } from 'aws-amplify';
import { listSampleAppsyncTables } from "../graphql/queries";
import { onCreateSampleAppsyncTable } from "../graphql/subscriptions";

const dataExpireSeconds = (30 * 60);
export default {
  name: 'List',
  data: () => ({
    group: null, 
    dataList: [], 
  }), 
  mounted: async function() {
    this.getList();
  }, 
  methods:{
    async getList() {
      this.group = this.$route.query.group;
      console.log("group : " + this.group);
      if(!this.group){
          return;
      }
      
      let apiResult = await API.graphql(graphqlOperation(listSampleAppsyncTables, { group : this.group }));
      let listAll = apiResult.data.listSampleAppsyncTables.items;
      for(let data of listAll) {
        let tmp = { path : data.path, image : "" };
        let list = [...this.dataList, tmp];
        this.dataList = list;
        console.log("path : " + data.path);
        Storage.get(data.path.replace('public/', ''), { expires: dataExpireSeconds }).then(result => {
          tmp.image = result;
          console.log("image : " + result);
        }).catch(err => console.log(err));
      }
      
      API.graphql(
          graphqlOperation(onCreateSampleAppsyncTable, { group : this.group } )
      ).subscribe({
          next: (eventData) => {
            let data = eventData.value.data.onCreateSampleAppsyncTable;
            let tmp = { path : data.path, image : "" };
            let list = [...this.dataList, tmp];
            this.dataList = list;
            console.log("path : " + data.path);
            Storage.get(data.path.replace('public/', ''), { expires: dataExpireSeconds }).then(result => {
              tmp.image = result;
              console.log("image : " + result);
            }).catch(err => console.log(err));
          }
      });
    }, 
  }
}
</script>

Get the group with the query parameter. With mounted before the screen is displayed, record data is acquired by specifying a group, or record data is acquired when an insert event is received. The acquired record data is held in a member variable array called dataList and displayed side by side in v-list on the screen. In v-list, the path and image of record data are displayed. The image is accessed by getting the address with the expiration date (30 minutes) with Strage.

src/components/Home.vue


<template>
  <v-container>
    <p>home</p>
    <router-link to="about" >link to About</router-link>
    <hr>
    <v-btn @click="selectFile">
      SELECT A FILE !!
    </v-btn>
    <input style="display: none" 
      ref="input" type="file" 
      @change="uploadSelectedFile()">
  </v-container>
</template>

<script>
import Vue from 'vue'
import { Auth, Storage } from 'aws-amplify';

export default {
  name: 'Home',
  data: () => ({
    loginid: "sample-vue-project-user", 
    loginpw: "sample-vue-project-user", 
  }), 
  mounted: async function() {
    this.login();
  }, 
  methods:{
    login() {
      console.log("login.");
      Auth.signIn(this.loginid, this.loginpw)
        .then((data) => {
          if(data.challengeName == "NEW_PASSWORD_REQUIRED"){
            console.log("new password required.");
            data.completeNewPasswordChallenge(this.loginpw, {}, 
              {
                onSuccess(result) {
                    console.log("onSuccess");
                    console.log(result);
                },
                onFailure(err) {
                    console.log("onFailure");
                    console.log(err);
                }
              }
            );
          }
          console.log("login successfully.");
        }).catch((err) => {
          console.log("login failed.");
          console.log(err);
        });
    },
    
    selectFile() {
      if(this.$refs.input != undefined){
        this.$refs.input.click();
      }
    }, 
    
    uploadSelectedFile() {
      let file = this.$refs.input.files[0];
      if(file == undefined){
        return;
      }
      console.log(file);

      let dt = new Date();
      let dirName = this.getDirString(dt);
      let filePath = dirName + "/" + file.name;      
      Storage.put(filePath, file).then(result => {
        console.log(result);
      }).catch(err => console.log(err));

      this.$router.push({ path: 'list', query: { group: dirName }});      
    }, 
    
    getDirString(date){
      let random = date.getTime() + Math.floor(100000 * Math.random());
      random = Math.random() * random;
      random = Math.floor(random).toString(16);
      return "" + 
        ("00" + date.getUTCFullYear()).slice(-2) + 
        ("00" + (date.getMonth() + 1)).slice(-2) + 
        ("00" + date.getUTCDate()).slice(-2) + 
        ("00" + date.getUTCHours()).slice(-2) + 
        ("00" + date.getUTCMinutes()).slice(-2) + 
        ("00" + date.getUTCSeconds()).slice(-2) + 
        "-" + random;
    }, 
  }
}
</script>

Use uploadSelectedFile to move to the List page after uploading the file. At that time, a query parameter called group is attached.

This completes the repair of the front end (web application), but the operation check is done after the back end side is completed.

Hit from the backend (Lambda Python)

We will implement a record insertion via AppSync for the file uploaded from the web application and the path (S3 Key) of the monochrome image generated and uploaded by Lambda.

Install gql.

pip install gql -t .

Update lambda_function.py as follows:

lambda_function.py


# coding: UTF-8
import boto3
import os
from urllib.parse import unquote_plus
import numpy as np
import cv2
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
s3 = boto3.client("s3")

from gql import gql, Client
from gql.transport.requests import RequestsHTTPTransport
ENDPOINT = "https://**************************.appsync-api.ap-northeast-1.amazonaws.com/graphql"
API_KEY = "da2-**************************"
_headers = {
    "Content-Type": "application/graphql",
    "x-api-key": API_KEY,
}
_transport = RequestsHTTPTransport(
    headers = _headers,
    url = ENDPOINT,
    use_json = True,
)
_client = Client(
    transport = _transport,
    fetch_schema_from_transport = True,
)

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    logger.info("Function Start (deploy from S3) : Bucket={0}, Key={1}" .format(bucket, key))

    fileName = os.path.basename(key)
    dirPath = os.path.dirname(key)
    dirName = os.path.basename(dirPath)
    
    orgFilePath = u'/tmp/' + fileName
    processedFilePath = u'/tmp/processed-' + fileName

    if (key.startswith("public/processed/")):
        logger.info("not start with public")
        return
    
    apiCreateTable(dirName, key)

    keyOut = key.replace("public", "public/processed", 1)
    logger.info("Output local path = {0}".format(processedFilePath))

    try:
        s3.download_file(Bucket=bucket, Key=key, Filename=orgFilePath)

        orgImage = cv2.imread(orgFilePath)
        grayImage = cv2.cvtColor(orgImage, cv2.COLOR_RGB2GRAY)
        cv2.imwrite(processedFilePath, grayImage)

        s3.upload_file(Filename=processedFilePath, Bucket=bucket, Key=keyOut)
        apiCreateTable(dirName, keyOut)

        logger.info("Function Completed : processed key = {0}".format(keyOut))

    except Exception as e:
        print(e)
        raise e
        
    finally:
        if os.path.exists(orgFilePath):
            os.remove(orgFilePath)
        if os.path.exists(processedFilePath):
            os.remove(processedFilePath)

def apiCreateTable(group, path):
    logger.info("group={0}, path={1}".format(group, path))
    try:
        query = gql("""
            mutation create {{
                createSampleAppsyncTable(input:{{
                group: \"{0}\"
                path: \"{1}\"
              }}){{
                group path
              }}
            }}
            """.format(group, path))
        _client.execute(query)
    except Exception as e:
        print(e)

For ʻENDPOINT and ʻAPI_KEY, refer to the API settings you created earlier in AppSync. Zip it up, upload it to S3 and deploy it to Lambda.

Operation check

When you run the web app and upload an image, Lambda hits AppSync, detects it and lists it on the web app side. Even if I hit the URL with the query parameter directly, I get the list from AppSync and list it. Screenshot 2020-01-03 at 09.19.47.pngScreenshot2020-01-03at09.20.39.png

The web application (Vue) project is as follows. https://github.com/ww2or3ww/sample_vue_project/tree/work5

The Lambda project is below. https://github.com/ww2or3ww/sample_lambda_py_project/tree/work3

Afterword

This was the first topic I asked aloud after participating in JAWS UG Hamamatsu. Isn't it possible to specify the DynamoDB partition key or sort key in the Amplify API? You don't often use DynamoDB without key settings, right? I don't know WebSocket in detail, but is it something like long polling? I remember speaking with excitement.

By the way, networks are difficult, aren't they? I respect those who can call themselves network engineers. To be honest, I don't really understand even if it is called MQTT over WebSocket. Please tell me in an easy-to-understand manner.

AppSync samples are often talked about as a set with the Amplify CLI, so they are only used from the front end. Chat app, TODO app. DynamoDB is also a full scan. I think that DynamoDB tends to increase the number of records, or it tends to be used for that purpose, and in that sense, scanning all records is not good.

Recommended Posts

Use AppSync on the front and back ends
Launch and use IPython notebook on the network
Sakura Use Python on the Internet
How to use Jupyter on the front end of supercomputer ITO
Use the Grove sensor on the Raspberry Pi
Looking back on 2016 in the Crystal language
Deploy and use the prediction model created in Python on SQL Server
Add lines and text on the image
Use python on Raspberry Pi 3 and turn on the LED when it gets dark!
Looking back on the data M-1 Grand Prix 2020
Use the latest version of PyCharm on Ubuntu
I tried to understand how to use Pandas and multicollinearity based on the Affairs dataset.
Run the flask app on Cloud9 and Apache Httpd
Render and synthesize objects from front, back, left and right
How to use the grep command and frequent samples
Looking back on the transition of the Qiita Advent calendar
Ubuntu 20.04 on raspberry pi 4 with OpenCV and use with python
Execute the command on the web server and display the result
How to use argparse and the difference between optparse
Django-Overview the tutorial app on Qiita and add features (1)
Install django on python + anaconda and start the server
Use Python to monitor Windows and Mac and collect information on the apps you are working on