Create a development environment for Go + MySQL + nginx with Docker (docker-compose)

Thing you want to do

--Create a development environment for Go, MySQL, nginx with Docker (docker-compose) --Go external packages are managed by Go Modules --Make Go's project practical --DB table management uses migration --test uses DB for test

Recommended for people like this

--I want to develop an API server with Go and MySQL ――I want to quickly build the environment with Docker ――I want a Go project that is easy to expand

Framework and version to use

Go framework uses Gin and ORM uses GORM.

Directory structure

├── docker-compose.yml
├── Dockerfile
├── app
│   ├── cmd
│   │   ├── migrate
│   │   │   └── main.go
│   │   └── server
│   │       └── main.go
│   ├── db
│   │   └── migrations
│   │       ├── 1_create_users.down.sql
│   │       └── 1_create_users.up.sql
│   ├── go.mod
│   ├── go.sum
│   └── pkg
│       ├── connecter
│       │   └── connecter.go
│       ├── controller
│       │   ├── router.go
│       │   └── users.go
│       └── model
│           ├── main_test.go
│           ├── user.go
│           └── user_test.go
├── mysql
│   ├── Dockerfile
│   ├── docker-entrypoint-initdb.d
│   │   └── init.sql
│   └── my.cnf
└── nginx
    ├── Dockerfile
    └── default.conf

The Dockerfile in the root directory is for the Go container.

How to use

The GitHub repository can be found here. https://github.com/fuhiz/docker-go-sample

First, launch the container in the directory where docker-compose.yml is located.

$ docker-compose up -d --build

Enter the Go container.

$ docker-compose exec web bash

Perform the migration in/app of the Go container. The SQL to be executed is a file in app/db/migrations. Schema_migrations are created for users and migration management.

$ go run cmd/migrate/main.go -exec up

For users, we have prepared name, age, and date and time columns.

sql:/db/migrations/1_create_users.up.sql

CREATE TABLE `users` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `name` varchar(50) NOT NULL,
  `age` int NOT NULL,
  `created_at` datetime NOT NULL,
  `updated_at` datetime NOT NULL,
  `deleted_at` datetime,
  PRIMARY KEY (`id`)
) DEFAULT CHARSET=utf8mb4;

Start the server.

$ go run cmd/server/main.go

Now you can connect to Go's API on localhost: 8082.

Try using the API

There is a users table in the DB, and the CRUD function of the user can be used, so check it with curl.

--User created

$ curl localhost:8082/api/v1/users -X POST -H "Content-Type: application/json" -d '{"name": "test", "age":30}'

--User list

$ curl localhost:8082/api/v1/users
{"users":[{"ID":1,"CreatedAt":"2021-01-09T11:09:31+09:00","UpdatedAt":"2021-01-09T11:09:31+09:00","DeletedAt":null,"name":"test","age":30}]}%

You can get the user you created earlier.

--User update

$ curl localhost:8082/api/v1/users/1 -X PATCH -H "Content-Type: application/json" -d '{"name": "update", "age":31}'

--User deletion

$ curl localhost:8082/api/v1/users/1 -X DELETE

docker-compose.yml

From here, we will look at the details of environment construction.

I will not touch on the basic way of writing docker-compose.yml, so if you want to refer to it, click here. https://qiita.com/zembutsu/items/9e9d80e05e36e882caaa

It is like this for each service.

db

--MySQL container --Define user names, passwords, etc. with environment variables. --Mount docker-entrypoint-initdb.d so that docker-entrypoint-initdb.d/init.sql is executed when the container starts. Create databases go_sample and go_sample_test in init.sql. /docker-entrypoint-initdb.d can create initial data in the directory provided in the MySQL Docker image. --The host side port is 3310 so that it does not overlap with MySQL running locally. --This is what happens when connecting with Sequel Pro.

• Password is localpass. The database can be empty.

web

--Go container --Persist the container with tty: true so that the container does not close immediately after startup. (Because the server startup is not written in the Dockerfile, it is executed manually in the container) --Define environment variables used in Go projects. You can read this value with os.Getenv ("DB_PASSWORD ") in your Go code. DB in DB_HOST is the service name of the MySQL container. You can connect with this service name when connecting to the DB with GORM. --Mount the ./app (host) with the Go project on/app (container)./App in the container is created when specified by WORKDIR in Dockerfile.

proxy --nginx forwards URLs by reverse proxy. Here, it is used for the purpose of setting http: // localhost to be Go's API. --Specified 8082 as the port on the host side.

docker-compose.yml

version: "3"

services:
  db:
    build: ./mysql
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: localuser
      MYSQL_PASSWORD: localpass
      TZ: Asia/Tokyo
    volumes:
      - ./mysql/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
    ports:
      - "3310:3306"

  web:
    build: .
    tty: true
    environment:
      APP_MODE: local
      DB_PASSWORD: localpass
    volumes:
      - "./go:/app"
    depends_on:
      - db

  proxy:
    build: ./nginx
    ports:
      - 8082:80
    depends_on:
      - web

MySQL container

The MySQL Dockerfile is a common setting for these.

Set the time zone to Asia/Tokyo. Copy the configuration file my.cnf. Copy init.sql to be executed at startup.

mysql/Dockerfile

FROM mysql:5.7

ENV TZ Asia/Tokyo
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && chown -R mysql:root /var/lib/mysql/

COPY my.cnf /etc/mysql/conf.d/my.cnf
COPY docker-entrypoint-initdb.d/init.sql /docker-entrypoint-initdb.d/

CMD ["mysqld"]

EXPOSE 3306

In init.sql, create go_sample and go_sample_test for testing and set user privileges.

sql:mysql/docker-entrypoint-initdb.d/init.sql

CREATE DATABASE IF NOT EXISTS `go_sample` COLLATE 'utf8mb4_general_ci' ;
CREATE DATABASE IF NOT EXISTS `go_sample_test` COLLATE 'utf8mb4_general_ci' ;

GRANT ALL ON `go_sample`.* TO 'localuser'@'%' ;
GRANT ALL ON `go_sample_test`.* TO 'localuser'@'%' ;

FLUSH PRIVILEGES ;

Go container

The content is as commented, and go.mod and go.sum are pre-copied to download the external package.

Dockerfile

FROM golang:1.15

##Working directory
WORKDIR /app

#Copy module management files
COPY go/go.mod .
COPY go/go.sum .

#Download external package
RUN go mod download

EXPOSE 9000

nginx container

Copy the default.conf that you read in nginx.conf.

nginx/Dockerfile

FROM nginx:1.19-alpine

COPY ./default.conf /etc/nginx/conf.d/default.conf

EXPOSE 80

nginx/default.conf

server {
    listen 80;
    server_name localhost;

    location / {
        proxy_pass http://web:9000;
    }
}

Since the nginx configuration file /etc/nginx/nginx.conf is designed to read * .conf under /etc/nginx/conf.d/, only the part to be read is created.

Here is a reference for nginx settings. https://qiita.com/morrr/items/7c97f0d2e46f7a8ec967

The important part is the proxy_pass http: // web: 9000;, where we replace http: // localhost with http: // web: 9000. web is the service name of the Go container defined in docker-compose.yml. Since docker-compose allows network communication between services, you can specify this. The Go project is launching a server with port 9000, so the port should match.

Also, since the port is set to 8082: 80 in the nginx container of docker-compose.yml, access from the host with http: // localhost: 8082.

It's confusing, but in other words, you can hit Go's API with http: // localhost: 8082.

Go project overview

The Go code was created with an awareness of what can be used practically as much as possible. Refer to here for the directory structure. https://qiita.com/sueken/items/87093e5941bfbc09bea8

cmd Application entry point. Deploy server startup and migration functions.

db Place the sql file you want to execute in the migration.

pkg The part related to the behavior of the application. Create a model, controller, and connector.

migration

I referred to here for the migration area. https://qiita.com/tanden/items/7b4fb1686a61dd5f580d

Use golang-migrate to manage your DB with a sql file located in db/migrations.

As for the file name rule, if {version} is in ascending order, it doesn't matter whether it is a number or a time stamp. https://github.com/golang-migrate/migrate/blob/master/MIGRATIONS.md

{version}_{title}.up.{extension}
{version}_{title}.down.{extension}

Here we have created 1_create_users.up.sql to create the users table and 1_create_users.down.sql to create the table.

The migration management files are located in cmd/migrate/main.go. The content is almost a copy and paste of Reference Site.

If you execute this file, only the added * .up.sql will run.

$ go run cmd/migrate/main.go -exec up

If you want to go back, set the option to down and all * .down.sql will be executed.

$ go run cmd/migrate/main.go -exec down

If you want to connect to the database for test, execute it with APP_MODE = test with environment variables.

$ APP_MODE=test go run cmd/migrate/main.go -exec up

If APP_MODE is test in init () of cmd/migrate/main.go, the database uses DB_NAME_TEST.

cmd/migrate/main.go

func init() {
	// database name decide by APP_MODE
	dbName := os.Getenv("DB_NAME")
	if os.Getenv("APP_MODE") == "test"{
		dbName = os.Getenv("DB_NAME_TEST")
	}

	Database = fmt.Sprintf("mysql://%s:%s@tcp(%s:%s)/%s",
		os.Getenv("DB_USER"),
		os.Getenv("DB_PASSWORD"),
		os.Getenv("DB_HOST"),
		os.Getenv("DB_PORT"),
		dbName)
}

Go API processing flow

The entry point file is cmd/server/main.go. I'm using gin to launch a server on port 9000.

cmd/server/main.go

package main

import (
	"net/http"

	"github.com/gin-gonic/gin"

	"github.com/fuhiz/docker-go-sample/app/pkg/connecter"
	"github.com/fuhiz/docker-go-sample/app/pkg/controller"
)

func main() {
	//gorm DB connection
	connecter.Setup()

	router := gin.Default()

	//For confirming communication of api
	router.GET("/", func(c *gin.Context) {
		c.String(http.StatusOK, "Response OK")
	})

	// routing
	r := router.Group("/api/v1")
	controller.Setup(r)

	router.Run(":9000")
}

For detailed processing, use the controller of/pkg.

DB connection of gorm is done by pkg/connector/connector.go.

• Gorg.DB is stored in the variable db and can be called by DB ().

You can get a general idea of ​​how to connect by looking at the formula. https://gorm.io/docs/connecting_to_the_database.html

Each parameter of the database is obtained from the environment variable defined in docker-compose.yml. Again, if APP_MODE is test, use DB_NAME_TEST.

pkg/connecter/connecter.go

package connecter

import (
	"fmt"
	"os"

	"gorm.io/driver/mysql"
	"gorm.io/gorm"
)

var db *gorm.DB

func Setup() {
	// APP_Determine the database name from MODE
	dbName := os.Getenv("DB_NAME")
	if os.Getenv("APP_MODE") == "test"{
		dbName = os.Getenv("DB_NAME_TEST")
	}

	//DB connection(https://gorm.io/docs/connecting_to_the_database.html)
	dsn := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s?charset=utf8mb4&parseTime=true&loc=%s",
		os.Getenv("DB_USER"),
		os.Getenv("DB_PASSWORD"),
		os.Getenv("DB_HOST"),
		os.Getenv("DB_PORT"),
		dbName,
		os.Getenv("DB_LOC"))
	gormDB, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})

	if err != nil {
		panic(err)
	}

	db = gormDB
}

func DB() *gorm.DB {
	return db
}

The routing is written in pkg/controllers/router.go. Call func in pkg/controllers/users.go respectively.

pkg/controllers/router.go

package controller

import (
	"github.com/gin-gonic/gin"
)

func Setup(r *gin.RouterGroup) {
	users := r.Group("/users")
	{
		u := UserController{}
		users.GET("", u.Index)
		users.GET("/:id", u.GetUser)
		users.POST("", u.CreateUser)
		users.PATCH("/:id", u.UpdateUser)
		users.DELETE("/:id", u.DeleteUser)
	}
}

In pkg/controllers/users.go, func of /pkg/model/user.go is called according to the process.

pkg/controllers/users.go

package controller

import (
	"net/http"
	"strconv"

	"github.com/gin-gonic/gin"

	"github.com/fuhiz/docker-go-sample/app/pkg/connecter"
	"github.com/fuhiz/docker-go-sample/app/pkg/model"
)

type UserController struct{}

type UserParam struct {
	Name string `json:"name" binding:"required,min=1,max=50"`
	Age  int    `json:"age" binding:"required,number"`
}

//User acquisition
func (self *UserController) GetUser(c *gin.Context) {
	ID := c.Params.ByName("id")
	userID, _ := strconv.Atoi(ID)
	user, err := model.GetUserById(connecter.DB(), userID)

	if err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "user not found"})
		return
	}

	c.JSON(http.StatusOK, gin.H{"user": user})
}

//User list
func (self *UserController) Index(c *gin.Context) {
	users, err := model.GetUsers(connecter.DB())

	if err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "user search failed"})
		return
	}

	c.JSON(http.StatusOK, gin.H{"users": users})
}

//User created
func (self *UserController) CreateUser(c *gin.Context) {
	var param UserParam
	if err := c.BindJSON(&param); err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
		return
	}

	newUser := model.NewUser(param.Name, param.Age)
	user, err := model.CreateUser(connecter.DB(), newUser)

	if err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "user create failed"})
		return
	}

	c.JSON(http.StatusOK, gin.H{"user": user})
}

//User update
func (self *UserController) UpdateUser(c *gin.Context) {
	ID := c.Params.ByName("id")
	userID, _ := strconv.Atoi(ID)
	user, err := model.GetUserById(connecter.DB(), userID)

	if err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "user not found"})
		return
	}

	var param UserParam
	if err := c.BindJSON(&param); err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
		return
	}

	updateParam := map[string]interface{}{
		"name": param.Name,
		"age":  param.Age,
	}

	_, err = user.Update(connecter.DB(), updateParam)

	if err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "user update failed"})
		return
	}

	c.JSON(http.StatusOK, gin.H{"user": user})
}

//User deletion
func (self *UserController) DeleteUser(c *gin.Context) {
	ID := c.Params.ByName("id")
	userID, _ := strconv.Atoi(ID)
	user, err := model.GetUserById(connecter.DB(), userID)

	if err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "user not found"})
		return
	}

	_, err = user.Delete(connecter.DB())

	if err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "user delete failed"})
		return
	}

	c.JSON(http.StatusOK, gin.H{"deleted": true})
}

/pkg/model/user.go

package model

import (
	"gorm.io/gorm"
)

type User struct {
	gorm.Model
	Name string `json:"name"`
	Age  int    `json:"age"`
}

func NewUser(name string, age int) *User {
	return &User{
		Name: name,
		Age:  age}
}

func CreateUser(db *gorm.DB, user *User) (*User, error) {
	result := db.Create(&user)

	return user, result.Error
}

func GetUsers(db *gorm.DB) ([]*User, error) {
	users := []*User{}
	result := db.Find(&users)

	return users, result.Error
}

func GetUserById(db *gorm.DB, ID int) (*User, error) {
	user := User{}
	result := db.First(&user, ID)

	return &user, result.Error
}

func (user *User) Update(db *gorm.DB, param map[string]interface{}) (*User, error) {
	result := db.Model(&user).Updates(param)

	return user, result.Error
}

func (user *User) Delete(db *gorm.DB) (*User, error) {
	result := db.Delete(&user)

	return user, result.Error
}

test

The test is run inside the Go container with APP_MODE = test. Follow the steps below.

Create a table in go_sample_test by migration (up).

$ APP_MODE=test go run cmd/migrate/main.go -exec up

Test/pkg.

$ APP_MODE=test go test -v ./pkg/...

Return go_sample_test for the next test.

$ APP_MODE=test go run cmd/migrate/main.go -exec down

The test files have main_test.go and user_test.go in/pkg/model. TestMain is executed first, so when connecting to the DB there.

/pkg/model/main_test.go

package model_test

import (
	"testing"

	"github.com/fuhiz/docker-go-sample/app/pkg/connecter"
)

func TestMain(m *testing.M) {
	connecter.Setup()
	m.Run()
}

User-created test.

/pkg/model/user_test.go

package model_test

import (
	"testing"

	"github.com/fuhiz/docker-go-sample/app/pkg/connecter"
	"github.com/fuhiz/docker-go-sample/app/pkg/model"
)

func TestCreateUser(t *testing.T) {
	newUser := model.NewUser("test_user", 30)
	user, _ := model.CreateUser(connecter.DB(), newUser)

	if user.Name != "test_user" {
		t.Fatal("model.CreateUser Failed")
	}
}

Summary

As a local environment, I think that an environment that can be used as it is has been prepared. I would like to verify in the future whether it can support automated testing and deployment.

Go seems to have not yet established the best plexis, so it was difficult to isolate the test environment. I once again felt the greatness of all-inclusive frameworks like Laravel and Rails.

It was a long article, but I would appreciate it if you could refer to it!