Do Django with CodeStar (Python3.6.8, Django2.2.9)

Create a Django app with CodeStar and CI / CD with CodePipline.

Development environment

Local machine

• git for windows
• anaconda 2019
• python 3.6.9
• Django 1.11.18 / 2.2.9

AWS EC2

• python 3.6.8
• Django 1.11.18 / 2.2.9

Creating a project

1. 1. Log in to the aws console and open CodeStar (maybe IAM users needed AWS CloudFormationFullAccess before)
2. Click Create New Project
3. 3. Select Python (Django), Amazon EC2 Four. Enter the project name (eg helloworld) and click Next Five. After confirming the details of the project, click Create Project
4. Open the Amazon EC2 Management Console in a new tab
5. 7. Create a key pair

8. 8. Save helloworld.pem

9. Select the key pair you created and click Create Project Ten. Skip (AWS Cloud9 was in the Oregon region)

11. CodeStar dashboard is displayed

12. CodePipline Source-> Build-> Wait until Deploy is completed

• An error occurred in the Oregon region

Your requested instance type (t2.micro) is not supported in your requested Availability Zone (us-west-2d). Please retry your request by not specifying an Availability Zone or choosing us-west-2a, us-west-2b, us-west-2c. (Service: AmazonEC2; Status Code: 400; Error Code: Unsupported; Request ID: 7ee63da9-fc85-4e2a-999c-629fe3997dbe)

13. After Deploy is complete, open the application endpoint

14. OK if you can see the page

• If you stop / start EC2, the page will disappear, so you need to set the script so that the Django app will also start when you start it. See below.

Editing the project (Practice of CodeCommit)

1. 1. Click the code from the CodeStar dashboard to move to CodeCommit

2. Copy URL from URL clone

3. 3. Install git for Windows

Four. Open anaconda prompt

Five. Create a virtual environment for django1 and django2

conda create -n django1 python=3.6
conda create -n django2 python=3.6

6. IAM CodeCommit settings

The permissions of the IAM group look like the following.

AmazonEC2FullAccess
AWSCodeCommitFullAccess
AWSLambdaFullAccess
AmazonS3FullAccess
AmazonDynamoDBFullAccess
AmazonSageMakerFullAccess
AmazonSageMaker-ExecutionPolicy-20191208Txxxxxx
AWSCloudFormationFullAccess

Generate credentials from IAM user credentials

• Make a note of this user name and password as they are required for git clone.
• After changing the password, How to reset the authentication information using git for Windows

7. 7. Activate the django1 environment and clone the project. You will be asked for the user name and password you have set. The rest is git configure.

   (base) $ conda activate django1
(django1) $ git config --global user.email "[email protected]"
(django1) $ git config --global user.name "s-fujimoto"
(django1) $ git clone https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/helloworld
(django1) $ cd helloworld

8. 8. Cut the develop branch * Do not commit directly to master. Cut the develop branch as soon as you create the project.

(django1) $ git branch develop
(django1) $ git checkout develop
Switched to branch 'develop'

9. Cut the feature branch * Do not commit directly to develop. Break the feature branch every time you add or change new features.

(django1) $ git branch feature
(django1) $ git checkout feature
Switched to branch 'feature'

Ten. Install the required libraries for your project

(django1) $ pip install -r requirements/dev.txt
Collecting Django==1.11.18
  Using cached https://files.pythonhosted.org/packages/e0/eb/6dc122c6d0a82263bd26bebae3cdbafeb99a7281aa1dae57ca1f645a9872/Django-1.11.18-py2.py3-none-any.whl
Collecting pytz
  Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Installing collected packages: pytz, Django
Successfully installed Django-1.11.18 pytz-2019.3

Django 1.11.18 is installed

11. Launch Django locally

(django1) $ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
December 23, 2019 - 18:15:46
Django version 1.11.18, using settings 'ec2django.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[23/Dec/2019 18:15:54] "GET / HTTP/1.1" 200 6652
[23/Dec/2019 18:15:54] "GET /static/helloworld/css/styles.css HTTP/1.1" 200 2690
[23/Dec/2019 18:15:54] "GET /static/helloworld/js/set-background.js HTTP/1.1" 200 137
[23/Dec/2019 18:15:54] "GET /static/helloworld/css/gradients.css HTTP/1.1" 200 2133
[23/Dec/2019 18:15:54] "GET /static/helloworld/img/tweet.svg HTTP/1.1" 200 1418
[23/Dec/2019 18:15:54] "GET /favicon.ico HTTP/1.1" 404 77

12. If you access http://127.0.0.1:8000/ from your browser and see the same page, it's OK.

13. See program changes in git status

(django1) $ git status
On branch feature
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        db.sqlite3
        ec2django/__pycache__/
        helloworld/__pycache__/
        helloworld/migrations/__pycache__/

nothing added to commit but untracked files present (use "git add" to track)

14. If pycache exists, CodePipline deployment will fail, so it is unnecessary, add the following .gitignore file (db.sqlite3 is optional)

 Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

 C extensions
*.so

 Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

 PyInstaller
  Usually these files are written by a python script from a template
  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

 Installer logs
pip-log.txt
pip-delete-this-directory.txt

 Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/

 Translations
*.mo
*.pot

 Django stuff:
*.log
local_settings.py

 Flask stuff:
instance/
.webassets-cache

 Scrapy stuff:
.scrapy

 Sphinx documentation
docs/_build/

 PyBuilder
target/

 Jupyter Notebook
.ipynb_checkpoints

 pyenv
.python-version

 celery beat schedule file
celerybeat-schedule

 SageMath parsed files
*.sage.py

 dotenv
.env

 virtualenv
.venv
venv/
ENV/

 Spyder project settings
.spyderproject
.spyproject

 Rope project settings
.ropeproject

 mkdocs documentation
/site

 mypy
.mypy_cache/


.idea/
db.sqlite3
migrations/

15. 15. Commit to feature

(django1) $ git add .
(django1) $ git commit -m "add .gitignore"
[feature 6d9a594] add .gitignore
 1 file changed, 106 insertions(+)
 create mode 100644 .gitignore
(django1) $ git push origin feature
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 939 bytes | 939.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/helloworld
 * [new branch]      feature -> feature

16. If you check CodeCommit, a feature branch has been created

17. create a develop branch

18. Creating a pull request

19. Merge from feature to develop

20. After merging, delete the feature. (But when you add or change new features, cut the feature branch. The feature branch remains locally, so use it as it is.)

twenty one. Successful merge

twenty two. Similarly merge from develop to master

twenty three. Do not delete the develop branch

twenty four. CodePipline starts when merged into master

twenty five. Make sure the page is visible

Edit project (update from Django1 to Django2)

1. 1. Exit the django1 environment and activate to the django2 environment

(django1) $ conda deactivate
   (base) $ conda activate django2
(django2) $ 

2. Edit requirements / common.txt Change before

 dependencies common to all environments
Django==1.11.18

After change

 dependencies common to all environments
Django==2.2.9

2. Library installation

(django2) $ pip install -r requirements/dev.txt
Collecting Django==2.2.9
  Using cached https://files.pythonhosted.org/packages/cb/c9/ef1e25bdd092749dae74c95c2707dff892fde36e4053c4a2354b2303be10/Django-2.2.9-py3-none-any.whl
Collecting pytz
  Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Collecting sqlparse
  Using cached https://files.pythonhosted.org/packages/ef/53/900f7d2a54557c6a37886585a91336520e5539e3ae2423ff1102daf4f3a7/sqlparse-0.3.0-py2.py3-none-any.whl
Installing collected packages: pytz, sqlparse, Django
Successfully installed Django-2.2.9 pytz-2019.3 sqlparse-0.3.0

Django == 2.2.9 is installed

3. 3. Edit ec2django / setting.py Change before

MIDDLEWARE_CLASSES = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True

After change

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

LANGUAGE_CODE = 'ja'

TIME_ZONE = 'Asia/Tokyo'

USE_I18N = True

USE_L10N = True

USE_TZ = True

Four. Edit ec2django / urls.py

Change before

from django.conf import settings
from django.conf.urls.static import static
from django.conf.urls import include, url
from django.contrib import admin
from django.contrib.staticfiles.urls import staticfiles_urlpatterns

urlpatterns = [
   url(r'^admin/', admin.site.urls),
   url(r'^', include('helloworld.urls')),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

After change

from django.conf import settings
from django.conf.urls.static import static
from django.conf.urls import include, url
from django.contrib import admin
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('helloworld.urls')),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

Five. Move helloworld / templates / index.html to helloworld / templates / helloworld / index.html

6．helloworld/urls.py

Change before

 helloworld/urls.py
from django.conf.urls import url
from django.conf.urls.static import static
from helloworld import views

urlpatterns = [
    url(r'^$', views.HomePageView.as_view()),
]

After change

 helloworld/urls.py
from django.urls import path
from django.conf.urls import url
from django.conf.urls.static import static
from helloworld import views

urlpatterns = [
    path('', views.HomePageView.as_view(), name='index'),
]

7．helloworld/views.py

Change before

 helloworld/views.py
from django.shortcuts import render
from django.views.generic import TemplateView


 Create your views here.
class HomePageView(TemplateView):
    def get(self, request, **kwargs):
        return render(request, 'index.html', context=None)

After change

 helloworld/views.py
from django.shortcuts import render
from django.views.generic import TemplateView
from django.views import generic

 Create your views here.
class HomePageView(generic.TemplateView):
    template_name = 'helloworld/index.html'

8. 8. Start Django localhost, access http://127.0.0.1:8000/ from your browser and see the same page, it's OK

(django2) $ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
December 23, 2019 - 19:27:46
Django version 2.2.9, using settings 'ec2django.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[23/Dec/2019 19:27:49] "GET / HTTP/1.1" 200 6652
[23/Dec/2019 19:27:49] "GET /static/helloworld/js/set-background.js HTTP/1.1" 200 137
[23/Dec/2019 19:27:49] "GET /static/helloworld/css/gradients.css HTTP/1.1" 200 2133
[23/Dec/2019 19:27:49] "GET /static/helloworld/css/styles.css HTTP/1.1" 200 2690
[23/Dec/2019 19:27:49] "GET /static/helloworld/img/tweet.svg HTTP/1.1" 200 1418

9. Commit to feature, merge into develop, merge from develop to master

(django2) $ git status
On branch feature
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   ec2django/settings.py
        modified:   ec2django/urls.py
        deleted:    helloworld/templates/index.html
        modified:   helloworld/urls.py
        modified:   helloworld/views.py
        modified:   requirements/common.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        helloworld/templates/helloworld/

no changes added to commit (use "git add" and/or "git commit -a")
(django2) $ git add .
(django2) $ git commit -m "django1.11.18 -> django2.2.9"
[feature 2226d88] django1.11.18 -> django2.2.9
 6 files changed, 12 insertions(+), 12 deletions(-)
 rename helloworld/templates/{ => helloworld}/index.html (100%)
(django2) $ git push origin feature
Enumerating objects: 21, done.
Counting objects: 100% (21/21), done.
Delta compression using up to 8 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (11/11), 1.27 KiB | 648.00 KiB/s, done.
Total 11 (delta 5), reused 0 (delta 0)
To https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/helloworld
 * [new branch]      feature -> feature

Ten. Check the page when CodePipline is complete

11. The site is no longer accessible

12. Open EC2

13. SSH connection to EC2 using TeraTerm (For PuTTY, use PuTTY gen to convert key pair .pem to .ppk before connecting)

14. Click Continue for security alerts

15. 15. Specify ec2-user and key pair as user name

16. Connect

17. Launch Django

[ec2-user@ip-172-31-14-214 ~]$ sudo su
[root@ip-172-31-14-214 ec2-user]# source /home/ec2-user/environment/bin/activate
(environment) [root@ip-172-31-14-214 ec2-user]# /usr/local/bin/supervisord -c /home/ec2-user/supervisord.conf
Error: Another program is already listening on a port that one of our HTTP servers is configured to use.  Shut this program down first before starting supervisord.
For help, use /usr/local/bin/supervisord -h
(environment) [root@ip-172-31-14-214 ec2-user]# pkill supervisord
(environment) [root@ip-172-31-14-214 ec2-user]# /usr/local/bin/supervisord -c /home/ec2-user/supervisord.conf

18. The page remains invisible, so check the log

(environment) [root@ip-172-31-14-214 ec2-user]# cat /var/log/django-application-stderr.log
 ・ ・ ・
File "/home/ec2-user/environment/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 63, in check_sqlite_version
    raise ImproperlyConfigured('SQLite 3.8.3 or later is required (found %s).' % Database.sqlite_version)
django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17).
[2019-12-23 19:54:59 +0900] [25438] [INFO] Worker exiting (pid: 25438)
[2019-12-23 10:54:59 +0000] [25435] [INFO] Shutting down: Master
[2019-12-23 10:54:59 +0000] [25435] [INFO] Reason: Worker failed to boot.

Since I updated to Django2, I need to update the version of SQLite3.

19. SQLite3 update

Refer to the following article What to do if SQLite3 error occurs when starting development server in Django 2.2

(environment) [root@ip-172-31-14-214 ec2-user]# wget https://www.sqlite.org/2019/sqlite-autoconf-3280000.tar.gz
(environment) [root@ip-172-31-14-214 ec2-user]# tar xvfz sqlite-autoconf-3280000.tar.gz
(environment) [root@ip-172-31-14-214 ec2-user]# cd sqlite-autoconf-3280000
(environment) [root@ip-172-31-14-214 ec2-user]# ./configure --prefix=/usr/local
(environment) [root@ip-172-31-14-214 ec2-user]# make
(environment) [root@ip-172-31-14-214 ec2-user]# sudo make install
(environment) [root@ip-172-31-14-214 ec2-user]# sudo find /usr/ -name sqlite3
(environment) [root@ip-172-31-14-214 ec2-user]# cd ../
(environment) [root@ip-172-31-14-214 ec2-user]# rm sqlite-autoconf-3280000.tar.gz
(environment) [root@ip-172-31-14-214 ec2-user]# rm -rf ./sqlite-autoconf-3280000
(environment) [root@ip-172-31-14-214 ec2-user]# /usr/local/bin/sqlite3 --version
3.28.0 2019-04-16 19:49:53 884b4b7e502b4e991677b53971277adfaf0a04a284f8e483e2553d0f83156b50
(environment) [root@ip-172-31-14-214 ec2-user]# /usr/bin/sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668
(environment) [root@ip-172-31-14-214 ec2-user]# sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668
(environment) [root@ip-172-31-14-214 ec2-user]# sudo mv /usr/bin/sqlite3 /usr/bin/sqlite3_old
(environment) [root@ip-172-31-14-214 ec2-user]# sudo ln -s /usr/local/bin/sqlite3 /usr/bin/sqlite3
(environment) [root@ip-172-31-14-214 ec2-user]# export LD_LIBRARY_PATH="/usr/local/lib"
(environment) [root@ip-172-31-14-214 ec2-user]# python
Python 3.6.8 (default, Oct 14 2019, 21:22:53)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.28.0'
>>> exit()
(environment) [root@ip-172-31-14-214 ec2-user]# 

OK if you start Django and see the page

(environment) [root@ip-172-31-14-214 ec2-user]# pkill supervisord
(environment) [root@ip-172-31-14-214 ec2-user]# /usr/local/bin/supervisord -c /home/ec2-user/supervisord.conf

Set Django to start when EC2 restarts

1. 1. Modify supervisord.conf (edit local as well and later merge into master) Change before

[program:djangoproject]
command = environment/bin/gunicorn -b 0.0.0.0:80 ec2django.wsgi

After change

[program:djangoproject]
command = /home/ec2-user/environment/bin/gunicorn -b 0.0.0.0:80 ec2django.wsgi

2. Service registration

(environment) [root@ip-172-31-14-214 ec2-user]# vi /etc/init.d/helloworld

!/bin/sh
 chkconfig: 2345 99 10
 description: start helloworld
 processname: helloworld

start() {
       echo "start"
       source /home/ec2-user/environment/bin/activate
       export LD_LIBRARY_PATH="/usr/local/lib"
       /usr/local/bin/supervisord -c /home/ec2-user/supervisord.conf
}

stop() {
       echo "stop"
       pkill supervisord
}
case "$1" in
 start)
       start
       ;;
 stop)
       stop
       ;;
 restart)
       stop
       start
       ;;
esac

exit 0

(environment) [root@ip-172-31-14-214 ec2-user]# chkconfig --add helloworld
(environment) [root@ip-172-31-14-214 ec2-user]# chkconfig helloworld on
(environment) [root@ip-172-31-14-214 ec2-user]# chmod u+x /etc/init.d/helloworld
(environment) [root@ip-172-31-14-214 ec2-user]# service helloworld restart

3. 3. If you can see the page even after restarting, it's OK (Django starts even if EC2 is stopped / started)

(environment) [root@ip-172-31-14-214 ec2-user]# sudo reboot

• Note that the IP will change when EC2 is stopped or started.

Four. Modify feature supervisord.conf and commit, merge feature to develop, merge from develop to master

(django2) $ git status
On branch feature
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   supervisord.conf

no changes added to commit (use "git add" and/or "git commit -a")
(django2) $ git add .
(django2) D:\Users\s-fujimoto\CodeStar\helloworld>git commit -m "modified supervisord.conf"
[feature 38c0e8c] modified supervisord.conf
 1 file changed, 1 insertion(+), 1 deletion(-)
(django2) D:\Users\s-fujimoto\CodeStar\helloworld>git push origin feature
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 325 bytes | 325.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
To https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/helloworld
 * [new branch]      feature -> feature

Five. OK if CodePipline is complete and you can see the page

(6. If you can't see it, restart EC2 or connect with SSH to check the log and restart the service)

(environment) [root@ip-172-31-14-214 ec2-user]# cat /var/log/django-application-stderr.log
(environment) [root@ip-172-31-14-214 ec2-user]# cat /var/log/django-application-stdout.log
(environment) [root@ip-172-31-14-214 ec2-user]# service helloworld restart

Forced deployment

--If you forget the .gitignore setting and merge, you will be angry that pycache is already there and the deployment will fail.

--Deployment failed when merging while EC2 was stopped. Even if I start EC2 after a failure and try again, it fails.

Even if you try again from the state where the deployment failed once, it will fail again. So if you force deploy and then merge into master, it will succeed.

I was addicted to the swamp because I didn't add the file to appspec.yml and buildspec.yml

1. 1. Select a target from the CodeDeploy application
2. Specify the location of the failed revision
3. 3. Check Do not fail deployment, select Overwrite content, Create deployment to Deploy

Use a custom user model (omitted)

db.sqlite3 is .gitignore, so send it with Tera Term AWS EC2 (Linux system) connection method and file transfer method

DB needs to be updated

python manage.py makemigrations helloworld
python manage.py migrate

Delete project

1. 1. Removed from CodeStar project list
2. Enter the project name and delete

Summary

I feel that Django app development has become much easier. CICD Good luck.