[Python/Django] Summary of frequently used commands (4) -Part 2- <Production operation: Amazon EC2 (Amazon Linux 2)>

background

Here are some frequently used commands when developing with Python/Django.

I would like to take this opportunity to thank you for solving the problem by borrowing the wisdom of our predecessors, and I am very sorry to say that I will summarize it here as my own memo.

environment

(Production environment)

(Development environment)

Premise

Build a WEB server, AP server, and DB server on Amazon EC2 (Elastic Computing Cloud): Amazon Linux 2.

(Diagram) 構成図_基本形.png

--Execute the following commands with the SSH connection to the EC2 instance.

Please also refer to [Python/Django] Summary of frequently used commands (4) -Part 1- <Production operation: Amazon EC2 (Amazon Linux 2)>.

9. Set environment variables.

9-1. Open the file for setting environment variables.

Terminal


$ vi ~/.bash_profile

9-2. Set environment variables.

You can edit it by pressing the "i" key. After editing, press the "esc" key and enter ": wq" to save the contents and finish editing. (Enter ": q!" To exit without saving.)

~/.bash_profile


(...Abbreviation...)

export DJANGO_SECRET_KEY='<SECRET given when creating a Djnago project_KEY(50 digits)>'
export DJANGO_SETTINGS_MODULE=<Project name>.settings
export DB_USER=<Arbitrary user name>
export DB_PASSWORD=<Arbitrary password>
export ALLOWED_HOSTS=<Elastic IP> (or <unique domain name>) *
export AWS_SES_ACCESS_KEY_ID=<Access key ID obtained when setting up Amazon SES usage>
export AWS_SES_SECRET_ACCESS_KEY=<Secret Access key obtained when setting up Amazon SES usage>

9-3. To make the edited contents valid, execute the following.

Terminal


$ source ~/.bash_profile

9-4. Check the contents of the environment variables.

Terminal


$ env

(...Abbreviation...)
XDG_SESSION_ID=514
HOSTNAME=ip-XXX-XXX-XXXX-XXX.ap-northeast-1.compute.internal
AWS_SES_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=XXX.XXX.XXX.XXX 57002 22
DJANGO_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SSH_TTY=/dev/pts/0
USER=<Arbitrary user name>
AWS_SES_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
(...Omission...)
ALLOWED_HOSTS=<Original domain name>
MAIL=/var/spool/mail/<Arbitrary user name>
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/<Arbitrary user name>/.local/bin:/home/<Arbitrary user name>/bin
PWD=/home/<Arbitrary user name>
DJANGO_SETTINGS_MODULE=<Project name>.settings
DB_PASSWORD=<Arbitrary password>
LANG=ja_JP.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/<Arbitrary user name>
LOGNAME=<Arbitrary user name>
DB_USER=<Arbitrary user name>
SSH_CONNECTION=XXX.XXX.XXX.XXX 57002 XXX.XXX.XXX.XXX 22
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/1001
_=/usr/bin/env

10. Place the static file in the distribution directory.

To reduce the load on the AP server, static files are returned from the WEB server (Nginx).

10-1. Create a static file placement directory and change the owner to .

Terminal


$ sudo mkdir -p /usr/share/nginx/html/static For static files
$sudo chown <any user>/usr/share/nginx/html/static

$ sudo mkdir /usr/share/nginx/html/media for media files
$sudo chown <any user>/usr/share/nginx/html/media

10-2. Enter the Python virtual environment and execute the following in the directory where manage.py exists.

(Premise):

settings.py


STATIC_ROOT = 'usr/share/nginx/html/static'

By executing the following, static files such as css and jpg will be placed in the / usr/share/nginx/static directory.

Terminal


(venv_<Project name>)$ python manage.py collectstatic

236 static files copied to '/usr/share/nginx/html/static'.

11. Perform DB migration.

Enter the Python virtual environment and execute the following in the directory where manage.py exists.

Terminal


(venv_<Project name>)$ python manage.py migrate

12. Install Nignx and configure the settings.

12-1. Installing Nginx

Terminal


$ amazon-linux-extras list | grep nginx

 38  nginx1=latest            enabled      [ =stable ]

We have found that nginx1 is the highest version that can be installed. Execute the following to install.

Terminal


$ sugo amazon-linux-extras install nginx1

12-2. Nginx settings

Edit the configuration file for Nginx and configure the environment settings.

You can edit it by pressing the "i" key. After editing, press the "esc" key and enter ": wq" to save the contents and finish editing. (Enter ": q!" To exit without saving.)

Terminal


$ sudo vi /etc/nginx/nginx.conf

nginx.conf


(...Abbreviation...)

    include /etc/nginx/conf.d/*.conf;

    server {
        listen      80;
        listen      [::]:80;
        server_name <Elastic IP or your own domain name>;
        return 301 https://$host/$request_uri;
    }

    # Setting for a TLS enabled server.    
    server {
        listen      443;
        listen      [::]:443;
        server_name <Elastic IP or your own domain name>;
        root        /usr/share/nginx/html;

        ssl_certificate "/etc/letsencrypt/live/<Own domain>/fullchain.pem";
        ssl_certificate_key "/etc/letsencrypt/live/<Own domain>/privkey.pem";

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location /static {
          alias /usr/share/nginx/html/static;
        }

        location /media {
          alias /usr/share/nginx/html/media;
        }

        location / {
          proxy_set_header Host $http_host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-proto $scheme;

          proxy_pass http://127.0.0.1:8000;
        }

        location /.well-known/acme-challenge {
          root /usr/share/nginx/html;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

(...Abbreviation...)

12-3. Nginx autostart settings

Terminal


$ sudo systemctl enable nginx.service

12-4. Start Nginx

Terminal


$ sudo systemctl start nginx.service

$ systemctl status nginx.service

● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
   Active: active (running)since tree 2020-12-10 08:20:46 JST; 2 days ago
  Process: 13919 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
  Process: 13916 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
  Process: 13915 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
 Main PID: 13922 (nginx)
   CGroup: /system.slice/nginx.service
           ├─13922 nginx: master process /usr/sbin/nginx
           └─13923 nginx: worker process

12-5. Restart Nginx

If you want to restart Nginx, do the following:

Terminal


$ sudo systemctl reload nginx.service

12-6. Stop Nginx

If you want to stop Nginx, do the following:

Terminal


$ sudo systemctl stop nginx.service

13. Operate Gunicorn.

Enter the Python virtual environment and execute the following in the directory where manage.py exists.

13-1. Start Gunicorn

Terminal


(venv_Project name)$ gunicorn --bind 127.0.0.1:8000 <Project name>.wsgi -D

(venv_Project name)$ ps ax | grep gunicorn

  671 pts/0    S+     0:00 grep --color=auto gunicorn
13937 ?        S      0:26 /home/<Arbitrary user name>/venv_<Project name>/bin/python3 /home/<Arbitrary user name>/venv_<Project name>/bin/gunicorn --bind 127.0.0.1:8000 <Project name>.wsgi -D
13940 ?        S      0:03 /home/<Arbitrary user name>/venv_<Project name>/bin/python3 /home/<Arbitrary user name>/venv_<Project name>/bin/gunicorn --bind 127.0.0.1:8000 <Project name>.wsgi -D

13-2. Stop Gunicorn

If you want to stop Gunicorn, do the following:

Terminal


(venv_Project name)$ pkill gunicorn

14. Make HTTPS

Please refer to [Nginx] What to do if you cannot get the SSL certificate from Let ’s Encrypt with certbot-auto.

15. Set up scheduled batch execution.

15-1. (Example) Back up the database.

backup_db.py


import csv
import datetime
import os

from django.conf import settings
from django.core.management.base import BaseCommand

from ...models import Post


class Command(BaseCommand):
    help = "Backup Database"

    def handle(self, *args, **options):
        date = datetime.date.today().strftime("%Y%m%d")
        file_path = settings.BACKUP_PATH + 'database_' + date + '.csv'
        os.makedirs(settings.BACKUP_PATH, exist_ok=True)

        with open(file_path, 'w') as file:
            writer = csv.writer(file)
            header = [field.name for field in Post._meta.fields]
            writer.writerow(header)
            blogs = Post.objects.all()
            for customer in customers:
                writer.writerow([str(customer.id,
                                 customer.name,
                                 customer.address,
                                 str(customer.phone_number),
                                 str(customer.created_at),
                                 str(customer.updated_at)])

        files = os.listdir(settings.BACKUP_PATH)
        if len(files) >= settings.NUM_SAVED_BACKUP:
            files.sort()
            os.remove(settings.BACKUP_PATH + files[0])

settings.py


BACKUP_PATH ="backup/"
NUM_SAVED_BACKUP = 30

15-2. Use the cron function to set periodic execution.

Cron is a program that automatically executes registered commands on a regular basis. A feature built into Amazon Linux 2 from the beginning.

You can edit it by pressing the "i" key. After editing, press the "esc" key and enter ": wq" to save the contents and finish editing. (Enter ": q!" To exit without saving.)

Terminal


$ crontab -e

Contents added by the crontab command


 0 4 * * * source ~/venv_<Project name>/bin/activate; cd ~/venv_<Project name>/<Project name>; python manage.py backup_db > ~/cron.log 2>&1

** Basic cron format ** <minute> <hour> <day> <month> <day of the week> <command> The above example is an example of executing a command at 4:00 AM (UTC) every day.


Notes
The specification in cron format cannot be changed because the time zone is UTC only. To set in Japan Standard Time (JST), specify a value of "-9 hours" (minus 9 hours) for UTC. Therefore, the above example is an example of executing the command at 1 pm (13:00) (JST) every day.

16. Perform a Django security check.

It will be done after HTTPS conversion is completed. Enter the Python virtual environment and execute the following in the directory where manage.py exists.

settings.py


# security.W004
SECURE_HSTS_SECONDS = 31536000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
# security.W006
SECURE_CONTENT_TYPE_NOSNIFF = True
# security.W007
SECURE_BROWSER_XSS_FILTER = True
# security.W008
SECURE_SSL_REDIRECT = True
# security.W012
SESSION_COOKIE_SECURE = True
# security.W016
CSRF_COOKIE_SECURE = True
# security.W019
X_FRAME_OPTIONS = 'DENY'
# security.W021
SECURE_HSTS_PRELOAD = True

Terminal


(venv_<Project name>)$ python manage.py check --deploy

If the WARNING message is displayed, it means that some security measures are insufficient. Normally, WARNING is not displayed in the above settings in the project settings file (settings.py).

Reference: Security in Django


(Relation)

[Python/Django] Summary of frequently used commands (1) <Creating virtual environment, project, application> [Python/Django] Summary of frequently used commands (2) [Python/Django] Summary of frequently used commands (3) [Python/Django] Summary of frequently used commands (4) -Part 1- <Production operation: Amazon EC2 (Amazon Linux 2)>


(Editor's note)

You can use EC2 as a development environment as well as during production operation, so I would like to consider using Docker as well.

Recommended Posts

[Python/Django] Summary of frequently used commands (4) -Part 2- <Production operation: Amazon EC2 (Amazon Linux 2)>
[Python/Django] Summary of frequently used commands (4) -Part 1- <Production operation: Amazon EC2 (Amazon Linux 2)>
[Python/Django] Summary of frequently used commands (2) <Installing packages>
[Linux] Frequently used Linux commands (file operation)
List of frequently used Linux commands
[Anaconda3] Summary of frequently used commands
[Linux] Frequently used Linux commands (folder operation)
[Linux] Review of frequently used basic commands 2
Summary of frequently used commands of django (beginner)
[Linux] Review of frequently used basic commands
Frequently used Linux commands
Frequently used Linux commands
Frequently used linux commands
[Linux command] A memorandum of frequently used commands
Summary of frequently used commands (with petit commentary)
Selenium webdriver Summary of frequently used operation methods
Linux Frequently Used Commands [Personal Memo]
Frequently used Linux commands (for beginners)
Summary of installing PHP7.2 on EC2 (Amazon Linux 2) and setting php.ini
Python + Selenium Frequently used operation method summary
[Linux] Summary of middleware version confirmation commands
[Linux] List of Linux commands used in practice
Summary of petit techniques for Linux commands
Summary of frequently used Python arrays (for myself)
Summary of Linux (UNIX) commands that appeared in Progate
A collection of commands frequently used in server management
Display a list of frequently used commands on Zsh
pyenv Frequently used commands
Frequently used tmux commands
New Linux commands! !! Part 2
OpenVPN Summary + Amazon Linux2
Frequently used pip commands
Addition of amazon linux swap
Summary of Linux distribution types
Frequently used subpackages of SciPy
Frequently used commands in virtualenv
A brief summary of Linux