How to use Serverless Framework & Python environment variables and manage stages

This article is a port of my personal blog Practical Cloud Python --SERVERLESS FRAMEWORK & PYTHON How to use environment variables to Qiita. Thank you for visiting Practical Cloud Python.

Abstract

Explains how to manage environment variables that can be enjoyed by stage management and coding.

Source

The source of the example is released, so please see here for the implementation example.

https://github.com/hassaku63/sls-env-example

It has a configuration of Lambda (scheduled)-> SQS-> Lambda (Queue consumer).

Premise

The following packages and plug-ins are used.

Serverless Framework

Python

Environment variables declared in .env. and serverless.yml

The example source is code that is supposed to be posted to Slack. Since Token and channel name correspond to confidential information, they are described in .env.

The Serverless Framework uses the Serverless Dotenv Plugin to include the definition of the .env file in provider.environment during deployment packaging.

The Dotenv plugin automatically loads an .env file suffixed with the specified stage name. For example, in the dev stage, load .env.dev as the definition of the environment variable (see the section on Automatic Env file name resolution). This function is used for settings that change API Token etc. for each stage.

It is also declared in the environment variable to manage the resource name of SQS and DynamoDB. This is because the queue and table resource names are needed when using the boto3 client from within the code.

https://gist.github.com/hassaku63/0e4e61db60bd48b5d86459ceabb9dd34

provider:
  name: aws
  runtime: python3.8
  profile: ${opt:profile, default}
  stage: ${opt:stage, dev}
  region: ${opt:region, us-east-1}
  timeout: 30
  environment:
    # To tell lambda function to AWS AccountId by use pseudo-parameters plugin.
    # AccountId is used to construct SQS Queue URL
    AWS_ACCOUNT_ID: '#{AWS::AccountId}'
    # MY_QUEUE_NAME is used to construct Queue URL in lambda.
    # And also convenient to reference resource ARN in IAM Policy statement
    MY_QUEUE_NAME: ${self:service}-my-queue-${self:provider.stage}

The part marked $ {self: service} -my-queue- $ {self: provider.stage} is the name of the queue used in this source. The service and stage names are included to ensure the uniqueness of the resource names when using multiple deployment stages.

When enqueue from Python to SQS, you need to specify QueueUrl as the target resource identifier. To assemble QueueUrl, you need three things: AWS account ID, AWS region, and QueueName. I want to be able to refer to them in Lambda, so I declare it with an environment variable (Note that only the region needs to be defined by yourself because Lambda automatically sets it as an environment variable by default at runtime. there is not).

There is talk of whether environment variables should be written in .env or serverless.yml, but I think you should judge whether this is information you want to keep secret. If the number of environment variables becomes too large and it becomes redundant, it is better to pick up only those that are not really environment variables and declare them as environment variables. If there are parts that are not essential (such as substrings that make up part of the resource name), I think it's a good idea to define them in the custom section.

Define environment variables in Python in settings.py

If you want to handle environment variables from Python, you should use os.environ. However, this way of writing doesn't work with IDE auto-completion, so there's no way to notice the variable name Typo when coding. It's not a good development experience to notice mistyping of environment variables only after deploying. Also, it feels like it's a violation of DRY to use os.environ each time the number of Lambda handlers increases.

To prevent this, load environment variables and create settings.py to define them as Python constants. All Python code should read environment variables from this module. This will allow you to take advantage of the IDE's auto-completion while preserving DRY and reducing the possibility of mistyping.

https://gist.github.com/hassaku63/0e971fb0823aea561f33db880d0269e4

"""
Declare all environment variables defined in .env.<stage-name> and serverless.yml as python variables.
If you would like to use environment variables in your code, you import this module then reference as constants of python variable.
By use this technique, you are able to use IDE auto-completion effectively when you writing python code.
It is useful to reduce miss-typing of environment variables.
"""
import os
import pathlib
import json
import dotenv

project_root_dir = pathlib.Path(__file__).parent / '../'

dotenv.load_dotenv()
MYAPP_STAGE_NAME = os.environ.get('MYAPP_STAGE_NAME')

stage_env_path = project_root_dir / f'.env.{MYAPP_STAGE_NAME}'
dotenv.load_dotenv(stage_env_path.resolve())

# Slack
SLACK_BOT_TOKEN = os.environ.get('SLACK_BOT_TOKEN')
SLACK_CHANNEL = os.environ.get('SLACK_CHANNEL')

# AWS
AWS_ACCOUNT_ID = os.environ.get('AWS_ACCOUNT_ID')
# Notice: You don't need to define AWS_REGION in .env and serverless.yml
# because of lambda runtime automatically set AWS_REGION env var when execute.
AWS_REGION = os.environ.get('AWS_REGION') 

# SQS
MY_QUEUE_NAME = os.environ.get('MY_QUEUE_NAME', '')

At this time, python-dotenv is used to load .env according to the stage. I expect .env to be in the project root. Find the relative directory for settings in Pathlib, use the MYAPP_STAGE_NAME environment variable to determine the stage name, and use that name as the .env suffix.

Summary

Both Serverless Framework and Python have additional packages for working with .env. Let's use these to make good use of environment variables.

There may be references to resource names in your code, such as DynamoDB or SQS resource identifiers. Environment variables can also be used in such cases. By defining it as an environment variable, the variable can be reused even when referring to the resource name in the template.

Let's use os.environ to create a module for grouping environment variables into Python constants. You can also use IDE auto-completion, and since the description is centralized in one place, mistyping is less likely to occur and it is easy.

Recommended Posts

How to use Serverless Framework & Python environment variables and manage stages
How to access environment variables in Python
How to install and use pandas_datareader [Python]
python: How to use locals () and globals ()
How to use Python zip and enumerate
How to use is and == in Python
[Python] How to use hash function and tuple.
Use os.getenv to get environment variables in Python
[Python] [Django] How to use ChoiceField and how to add options
python3: How to use bottle (2)
[Python] How to use list 1
How to use Python argparse
Python: How to use pydub
[Python] How to use checkio
Julia Quick Note [01] How to use variables and constants
[Python] How to use input ()
How to use Python lambda
[Python] How to use virtualenv
python3: How to use bottle (3)
python3: How to use bottle
How to use Python bytes
[Python] How to play with class variables with decorator and metaclass
[Introduction to Udemy Python 3 + Application] 36. How to use In and Not
[Python] Summary of how to use split and join functions
Comparison of how to use higher-order functions in Python 2 and 3
Overview of Python virtual environment and how to create it
Python: How to use async with
How to install and use Tesseract-OCR
[Python] How to use Pandas Series
How to use Requests (Python Library)
How to use SQLite in Python
How to use .bash_profile and .bashrc
How to install and use Graphviz
[Python] How to use list 3 Added
How to use Mysql in python
How to use OpenPose's Python API
How to use ChemSpider in Python
How to use FTP with Python
Python: How to use pydub (playback)
How to use PubChem in Python
How to use python zip function
[Python] How to use Typetalk API
How to use functions in separate files Perl and Python versions
Install pip in Serverless Framework and AWS Lambda with Python environment
How to use Python with Jw_cad (Part 2 Command explanation and operation)
[Introduction to Python] How to use the Boolean operator (and ・ or ・ not)
How to build Python and Jupyter execution environment with VS Code
[Python] Summary of how to use pandas
How to package and distribute Python scripts
[Python] Use and and or when creating variables
How to dynamically define variables in Python
[python] How to use __command__, function explanation
How to prepare Python development environment [Mac]
[Python] How to use import sys sys.argv
[Python] Organizing how to use for statements
Memorandum on how to use gremlin python
[Python2.7] Summary of how to use unittest
To reference environment variables in Python in Blender
How to use __slots__ in Python class
How to use tensorflow under docker environment
How to use "deque" for Python data