[docker] [nginx] Make a simple ALB with nginx
I wanted to launch multiple apps for development and testing, so I made a simple ALB tool. (Environment is AWS) I decided to make a tool because of the following restrictions.
Constraints: --Any port cannot be published outside the server (80/443 only) --ALB launch prohibited (normally this)
As an approach, since we are using docker, we will consider using the default container of nginx. Originally, I want to run multiple apps in parallel on the host, so I will allocate different ports for each and start them up, and aim for a form that sorts by host name with nginx.
Since the distribution destination is another container running on the same host, from the viewpoint of nginx, it will connect to the host from inside the container.
Since it is a simplified version as a tool, we will proceed as long as we do not create a new docker image or add a plugin.
I put the tool I created on GitHub. https://github.com/batatch/simple-alb
Overview
Although it is a simplified version, the setting contents are similar to those of AWS ALB, prepare a definition file, generate nginx settings from there → aim for a configuration like docker execution.
procedure:
$ make build #Generate nginx configuration file from definition file
$ make up # docker-Allocate a config file with compose and start an nginx container
$ make down # docker-Stop nginx container with compose
Definition file:
alb.yml
---
http:
listen: 80 #Listener, this time HTTP only
rules:
- if: #IF condition of ALB
host: app01.example.com #Hostname matching
pathes: [ "/" ] #Path matching
then: #ALB THEN statement
forward: #Transfer settings
name: tg-app01
targets: #Forwarding destination(Multiple), Image like target group
- target: http://docker0:21080
weight: 30
- target: http://docker0:22080
stickiness: true
:
The procedure is summarized in a Makefile. I like it, but I think this is the easiest and easiest to understand. Then, based on the above definition file, create the following nginx configuration file.
nginx/conf.d/default.conf
upstream target1 {
server http://docker0:21080;
server http://docker0:22080;
}
server {
listen 80;
server_name app01.example.com;
:
location / {
proxy_pass http://target1;
}
}
The framework looks like this.
Template engine
Since the configuration file is automatically generated, if you want some kind of template engine, try using Jinja2 of Python used in Ansible etc.
I made it possible to get the conversion result from the template file and the YAML file of the configuration file with a simple Python script like the following.
j2.py
import sys
import yaml
from jinja2 import Template, Environment, FileSystemLoader
def _j2(templateFile, configFile):
env = Environment(loader=FileSystemLoader('.', encoding='utf_8'))
tpl = env.get_template(templateFile)
with open(configFile) as f:
conf = yaml.load(f)
ret = tpl.render(conf)
print(ret)
if __name__ == '__main__':
if (len(sys.argv) <= 2):
print("Usage: j2.pl <template file> <config file>")
sys.exit(-1)
_j2(sys.argv[1], sys.argv[2])
The command line looks like this.
$ python j2.pl template.conf.j2 param.yml > output.conf
Communication from docker container to host
This is quite troublesome, and in the Docker environment of Windows or Mac, it seems that you can connect from inside the container to the host with host.docker.internal, but Linux does not have such a method.
In Linux, it seems that the host / container is connected by an interface called docker0, so I got the IP address assigned to docker0 on the host side, made it an environment variable, and passed it when docker started.
$ env DOCKER0_ADDRESS=$( ip route | awk '/docker0/ {print $9}' ) \
docker-compose up -d
docker-compose.yml
version: '3'
services:
alb:
image: nginx:stable
:
extra_hosts:
- "docker0:${DOCKER0_ADDRESS}"
If you write the mapping in extra_hosts in docker-compose.yml, the mapping of the host name and IP address will be added to / etc / hosts in the container when the container is started, so it seems that you can refer to the name in the nginx settings.
/etc/hosts
----
172.17.0.1 docker0
Load balancer with nginx
To set up a load balancer with nginx, define a group of targets in http / upstream and specify the upstream name in proxy_pass in http / server / location. .. I expected it, but it doesn't start with an error.
Apparently, the free version of nginx doesn't allow DNS resolver for host names written in upstream. (I learned for the first time that nginx has a paid / free version.)
This is an article summarizing this matter. The Qiita article below shows how to use UNIX sockets. I chose this one because it met the restrictions of not using plugins or creating docker images. (Although the configuration file will be longer)
Summary of Nginx name resolution https://ktrysmt.github.io/blog/name-specification-of-nginx/ Dynamic DNS resolution in nginx upstream context without paid resolve option https://qiita.com/minamijoyo/items/183e51a28a3a9d79182f
nginx/conf.d/default.conf
upstream tg-app01 {
server unix:/var/run/nginx_tg-app01_1; # (2-1) tg-The first target of app01
server unix:/var/run/nginx_tg-app01_2; # (2-2) tg-Second target for app01
}
server {
listen 80;
server_name app01.example.com;
:
location / {
proxy_pass http://tg-app01; # (1) upstream tg-See app01
}
}
server {
listen unix:/var/run/nginx_tg-app01_1; # (2-1)Reference for the first target
server_name app01.example.com;
:
location / {
proxy_pass http://docker0:21080;
}
}
server {
listen unix:/var/run/nginx_tg-app01_2; # (2-2)Second target reference
server_name app01.example.com;
:
location / {
proxy_pass http://docker0:22080;
}
}
Websocket communication
Since it was necessary to pass through Websocket this time, make the following settings. Seems to be needed for each server block.
nginx/conf.d/default.conf
#With this
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
:
server {
listen 80;
server_name app02.example.com;
#from here
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
#So far
location / {
proxy_pass http://tg-app02;
}
}
nginx config file template
Based on the contents so far, the template definition is as follows. It's fine, but + α also includes the default pattern specification and fixed response setting.
src/default.conf.j2
## http listener settings
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
{% for rule in http.rules %}
{%- if rule.then.forward %}
upstream {{ rule.then.forward.name }} {
{%- if rule.then.forward.stickiness %}
ip_hash;
{%- endif %}
{%- for tg in rule.then.forward.targets %}
server {{ 'unix:/var/run/nginx_%s_%d' % (rule.then.forward.name, loop.index) }}{{ ' weight=%d' % tg.weight if tg.weight else '' }};
{%- endfor %}
}
{%- endif %}
server {
listen {{ http.listen }}{{ ' default_server' if rule.if.default_server }};
server_name {{ rule.if.host }};
{% if rule.then.forward %}
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
{% endif %}
{%- for path in rule.if.pathes %}
location {{ path }} {
{%- if rule.if.headers %}
{%- for header in rule.if.headers %}
if ($http_{{ header|replace('-','_')|lower() }} = "{{ rule.if.headers[header] }}") {
proxy_pass http://{{ rule.then.forward.name }};
break;
}
{%- endfor %}
{%- else %}
proxy_pass http://{{ rule.then.forward.name }};
{%- endif %}
}
{%- endfor %}
{%- if rule.then.response %}
location / {
{%- if rule.then.response.content_type %}
default_type {{ rule.then.response.content_type }};
{%- endif %}
return {{ rule.then.response.code }}{{ ' \'%s\'' % rule.then.response.message if rule.then.response.message }};
}
{%- endif %}
}
{%- if rule.then.forward %}
{%- for tg in rule.then.forward.targets %}
server {
listen {{ 'unix:/var/run/nginx_%s_%d' % (rule.then.forward.name, loop.index) }};
server_name {{ rule.if.host }};
{% if rule.then.forward %}
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
{% endif %}
location / {
proxy_pass {{ tg.target }};
}
}
{%- endfor %}
{%- endif %}
{% endfor %}
The docker-compose settings are as follows. Mount the nginx settings folder on an external volume for the settings to take effect.
docker-compose.yml
version: '3'
services:
alb:
image: nginx:stable
ports:
- "80:80"
volumes:
- ./conf.d:/etc/nginx/conf.d
extra_hosts:
- "docker0:${DOCKER0_ADDRESS}"
restart: always
The operation check is as follows.
$ make build #Configuration file conversion
$ make up #nginx container start
$ curl http://localhost:80 -i -H "Host:app01.example.com" #Access by setting the host name
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Sat, 29 Aug 2020 17:26:23 GMT
Content-Type: text/html
Content-Length: 1863
Connection: keep-alive
Last-Modified: Wed, 11 Mar 2020 05:22:13 GMT
ETag: "747-5a08d6b34ab40"
Accept-Ranges: bytes
<!DOCTYPE html>
<html>
:
Summary
I think it will be easier to manage because you don't have to write AWS ALB-like components in the YAML definition and write complicated nginx settings. You don't need to build or install any plugins, so it's easy to feel like ALB as long as you get the nginx image from DockerHub.
I also learned how to set up nginx. (I also learned about the free version / paid version.)
If you investigate further and do your best, you may be able to realize the setting pattern of the ALB head family. (This time I wanted to make it as easy to use as possible, so I did not make it)
Recommended Posts