[Oracle Cloud] Build a 4-Node RAC environment of Oracle Database 19c with Docker on OCI Compute

1.First of all

I see an article about building a single instance environment of Oracle Database with Docker, but there seemed to be few articles about building a RAC (Real Application Clusters) environment, so I wanted to write an article, so I built it again with the meaning of checking the procedure I would like to do it.

This time, we will create 4 containers with Docker on one Compute created with Oracle Cloud Infrastructure (hereinafter, OCI) and build a RAC environment of 4 Nodes. Of course, the number of Nodes is adjustable.

1-1. Attention

--The Oracle Database configured with Docker can only be used in the verification and development environments. Please note that it cannot be used in a production environment.

――Although it is different from this environment, services such as DBCS, ExaCS, and ExaCC originally support RAC configuration in OCI. Please note that you cannot configure RAC with multiple Computes.

1-2. Environment

• Oracle Cloud Infrastructure Compute
• Oracle Linux 7.9
• Oracle Database 19c (19.3)
• Oracle Database 19c Grid Infrastructure（19.3）
• Docker

1-3. Reference document

Follow the steps on the official Oracle Github to build it.

Oracle RAC Database on Docker

2. Create Compute

First, let's create Compute with OCI. Click the hamburger menu in the upper left of the OCI Console, then click Compute-Instance.

Click Create Instance.

Please enter the instance name. It doesn't matter which AD you choose.

The image is the default selected version of Oracle Linux. Since the shape is 4 Node RAC this time, I thought that I would like about 4 cores somehow, so I chose VM.Standard 2.4.

It is assumed that the network (VCN) has been created in advance. This time, create it on a Public Subnet that can be connected from the Internet. Also select Assign Public IP Address to connect via the Internet.

You can add the SSH key the way you like. This time, I registered the SSH key that was created in advance in [Select SSH key file].

By default, the boot volume is only around 46GB, which is a bit insufficient to configure Oracle Database with Docker. Probably about 100GB is fine, but this time I will set it to 200GB with a margin. Note that the size specified here does not include the capacity of the ASM disk that stores the Oracle Database data files. ASM will be added later, so you don't have to consider it once.

Click Create when you have finished entering each item.

Creation is complete when the instance becomes [Running]. [Public IP Address] will be displayed. Log in to this IP address with SSH.

3. Log in to Compute

Log in to the created Compute with an SSH client such as Teraterm. --Destination: [Public IP address] --Username: opc --Passphrase: If specified when creating the SSH key, specify the same here as well --Private key: Specify the private key that is paired with the SSH public key specified when creating Compute.

I was able to log in.

4. Compute OS settings

After creating Compute, there are some things to do first, so I will do it.

4-1. Change time zone

Change the time zone to Asia/Tokyo.

[opc@rac-instance01 ~]$ sudo timedatectl set-timezone Asia/Tokyo
[opc@rac-instance01 ~]$ timedatectl
      Local time: Fri 2021-01-01 16:18:26 JST
  Universal time: Fri 2021-01-01 07:18:26 UTC
        RTC time: Fri 2021-01-01 07:18:27
       Time zone: Asia/Tokyo (JST, +0900)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a
[opc@rac-instance01 ~]$

4-2. Locale change

Change the locale to ja_JP.utf8. This is your choice.

[opc@rac-instance01 ~]$ sudo localectl set-locale LANG=ja_JP.utf8
[opc@rac-instance01 ~]$ localectl
   System Locale: LANG=ja_JP.utf8
       VC Keymap: us
      X11 Layout: us
[opc@rac-instance01 ~]$ cat /etc/locale.conf
LANG=ja_JP.utf8
[opc@rac-instance01 ~]$

4-3. Expansion of boot volume

The 200GB specified when creating the Compute is 200GB as a device, but it is still the same as the file system, so extend/dev/sda3 that mounts /.

First, check the status in advance.

[opc@rac-instance01 ~]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

sda 8: 0 0 200G 0 disk ★ 200GB -sda2 8:2 0 8G 0 part [SWAP] -sda3 8:3 0 38.4G 0 part / ★ Only about 39GB -sda1 8:1 0 200M 0 part /boot/efi

[opc@rac-instance01 ~]$ df -h /

File system size used Remaining used% Mount position / dev / sda3 39G 11G 29G 27% / ★ Only 39GB

For extension, use the OCI utility oci-growfs.

oci-growfs https://docs.oracle.com/ja-jp/iaas/Content/Compute/References/oci-growfs.htm

Extend the root file system of the instance to the configured size. This command must be run as root.

It is very convenient because you can extend the boot volume just by running oci-growfs.

[opc@rac-instance01 ~]$ sudo /usr/libexec/oci-growfs
CHANGE: partition=3 start=17188864 old: size=80486400 end=97675264 new: size=402241502 end=419430366

Confirm? [y/n] y ★ Enter y and enter CHANGED: partition=3 start=17188864 old: size=80486400 end=97675264 new: size=402241502 end=419430366 meta-data=/dev/sda3 isize=256 agcount=4, agsize=2515200 blks = sectsz=4096 attr=2, projid32bit=1 = crc=0 finobt=0, sparse=0, rmapbt=0 = reflink=0 data = bsize=4096 blocks=10060800, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=4912, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 data blocks changed from 10060800 to 50280187

[opc@rac-instance01 ~]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   200G  0 disk
 -sda2   8:2    0     8G  0 part [SWAP]

-sda3 8:3 0 191.8G 0 part / ★ All remaining capacity has been allocated -sda1 8:1 0 200M 0 part /boot/efi

[opc@rac-instance01 ~]$ df -h /

File system size used Remaining used% Mount position / dev / sda3 192G 11G 182G 6% / ★ All remaining capacity was allocated

5. Compute Docker host construction

We will set it as a Docker host.

5-1. yum update

Yum update. Maybe it will take some time as it will be updated in various ways.

[opc@rac-instance01 ~]$ sudo yum update

5-2. git installation

Install git to clone the official Oracle Docker Image.

[opc@rac-instance01 ~]$ sudo yum install git

5-3. Clone Docker image

Clone the Oracle Docker image.

[opc@rac-instance01 ~]$ git clone https://github.com/oracle/docker-images.git
Cloning into 'docker-images'...
remote: Enumerating objects: 13371, done.
remote: Total 13371 (delta 0), reused 0 (delta 0), pack-reused 13371
Receiving objects: 100% (13371/13371), 9.75 MiB | 0 bytes/s, done.
Resolving deltas: 100% (7857/7857), done.

5-4. Oracle Database Software Preparation

Prepare a zip file of the Oracle Database software.

Oracle Database 19c (19.3) https://www.oracle.com/database/technologies/oracle19c-linux-downloads.html

• Please use within the range of OTN license.

You can usually download it with a browser and upload it to Compute via SCP or FTP, but that would be time consuming and cumbersome, so this time I will download it directly on Compute. However, you can't download the file directly from the URL, so I'll do a little trick.

Click the Download link for 19.3.

Check the license and click the Download button.

As soon as the download starts, it will pause on your browser.

Opens the screen for downloading Blanca.

Copy the URL of the target file.

Go to the location where you want to place the zip on Compute and download it directly with the copied URL.

[opc@rac-instance01 ~]$ cd ~/docker-images/OracleDatabase/RAC/OracleRealApplicationClusters/dockerfiles/19.3.0/
[opc@rac-instance01 19.3.0]$ curl -0 https://download.oracle.com/otn/linux/oracle19c/190000/LINUX.X64_193000_db_home.zip?AuthParam=xxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > LINUX.X64_193000_db_home.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 58 2917M   58 1694M    0     0  75.0M      0  0:00:38  0:00:22  0:00:16 6193k

Similarly, download the Grid Infrastructure zip to the same location.

5-5. Preparing the yum repository

When I built it about a year ago, the default image for OCI Compute was Oracle Linux 7.7, which required the addition of a yum repository to install docker-engine. This time, the default image is Oracle Linux 7.9, and the required repository existed by default, so it was unnecessary, but I will describe the procedure just in case.

[opc@rac-instance01 ~]$ cd /etc/yum.repos.d/
[opc@rac-instance01 yum.repos.d]$ grep 'ol7_addons' *.repo

oracle-linux-ol7.repo: [ol7_addons] ★ ol7_addons exists

If the repository file containing ol7_addons exists, this step is not necessary. Please SKIP. If it does not exist, add it according to the following procedure.

[opc@rac-instance01 yum.repos.d]$ sudo wget http://yum.oracle.com/public-yum-ol7.repo
[opc@rac-instance01 yum.repos.d]$ vi public-yum-ol7.repo
-------------------------
[ol7_addons]
name=Oracle Linux $releasever Add ons ($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/addons/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1

enabled = 1 <--- ★ Change here from 0 to 1 -------------------------

5-6. docker-engine installation

Install docker-engine.

[opc@rac-instance01 ~]$ sudo yum install docker-engine

Start Docker.

[opc@rac-instance01 ~]$ sudo systemctl start docker
[opc@rac-instance01 ~]$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)

Active: active (running) since month 2021-01-04 19:45:07 JST; 3s ago

5-7. Kernel parameter settings

Add the kernel parameters as per the steps in Official Github.

[opc@rac-instance01 ~]$ sudo vi /etc/sysctl.conf

---- ★ Add all the following fs.file-max = 6815744 net.core.rmem_max = 4194304 net.core.rmem_default = 262144 net.core.wmem_max = 1048576 net.core.wmem_default = 262144 net.core.rmem_default = 262144 ---- [opc@rac-instance01 ~]$ sysctl -a [opc@rac-instance01 ~]$ sudo sysctl -p

5-8. Docker startup settings

Set the startup of docker.service. Replace ExecStart in /usr/lib/systemd/system/docker.service with the following:

 [opc@rac-instance01 ~]$ sudo vi /usr/lib/systemd/system/docker.service
---
ExecStart=/usr/bin/dockerd --cpu-rt-runtime=950000 --cpu-rt-period=1000000 --exec-opt=native.cgroupdriver=systemd
----

Next, set the cgroup. The third is a value greater than or equal to 95000 x number of nodes. This time it's 4Node so it's 95000 X 4 It is 380000, but let's set it to 400000 with a margin. If you do not do this, you will get an error when starting the container in the second half and it will fail, so be sure to set it. Also, this setting disappears when you restart the Docker host server, so it is recommended that you set it every time you start the host server.

[root@rac-instance01 opc]# echo 950000 > /sys/fs/cgroup/cpu/cpu.rt_runtime_us
[root@rac-instance01 opc]# echo 1000000 > /sys/fs/cgroup/cpu/cpu.rt_period_us
[root@rac-instance01 opc]# echo 400000 > /sys/fs/cgroup/cpu,cpuacct/system.slice/cpu.rt_runtime_us

After completing the settings so far, restart Docker.

[opc@rac-instance01 ~]$ sudo systemctl daemon-reload
[opc@rac-instance01 ~]$ sudo systemctl restart docker
[opc@rac-instance01 ~]$ sudo systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[opc@rac-instance01 ~]$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)

Active: active (running) since Wed 2021-01-06 21:35:11 JST; 9s ago Docs: https://docs.docker.com Main PID: 11610 (dockerd) CGroup: /system.slice/docker.service mq11610 /usr/bin/dockerd --cpu-rt-runtime=950000 --cpu-rt-period=1000000 --exec-opt=native.cgroupdriver=systemd

6. Docker build

We will build Docker.

6-1. Building the installation image

Build a Docker installation image for the Oracle RAC Database.

[opc@rac-instance01 ~]$ cd ~/docker-images/OracleDatabase/RAC/OracleRealApplicationClusters/dockerfiles
[opc@rac-instance01 dockerfiles]$ sudo ./buildDockerImage.sh -v 19.3.0
Checking if required packages are present and valid...

LINUX.X64_193000_grid_home.zip: Done

(Omitted)

  Oracle Database Docker Image for Real Application Clusters (RAC) version 19.3.0 is ready to be extended:

    --> oracle/database-rac:19.3.0

  Build completed in 988 seconds.

You may get this error along the way (I encounter 100% ...), but you can safely ignore it. (Reference: https://github.com/oracle/docker-images/issues/1416)

/opt/scripts/install/installGridBinaries.sh: line 57:  : command not found

6-2. Docker network settings

Set up the network between Docker containers. rac_pub1_nw is the public LAN and rac_priv1_nw is the private LAN.

[opc@rac-instance01 dockerfiles]$ sudo docker network create --driver=bridge --subnet=172.16.1.0/24 rac_pub1_nw
b6d7984df77c6e42705bd242ee2e790882b2d9f5ff2ef20cf369b7105238adf4
[opc@rac-instance01 dockerfiles]$ sudo docker network create --driver=bridge --subnet=192.168.17.0/24 rac_priv1_nw
a47b39a3b43bebcc1d8a7676ac6786a581919f0dd20cab79c0ffbcb7005153ea

6-3. Creating a shared host file

All containers use shared host files for host name resolution. Create the file on the Docker host because the shared host file must be available in all containers. At the moment, an empty file is fine. The contents of hosts are automatically written when the node is created.

[opc@rac-instance01 dockerfiles]$ sudo mkdir /opt/containers
[opc@rac-instance01 dockerfiles]$ sudo touch /opt/containers/rac_host_file
[opc@rac-instance01 dockerfiles]$ ls -l /opt/containers/rac_host_file

-rw-r--r--. 1 root root 0 January 6 22:12/opt/containers/rac_host_file

6-4. Creating a password file

You will need the grid / oracle and database passwords while creating or adding nodes, so create a pre-encrypted file of passwords. You can set passwords individually for each, but this time we will set them all with a common password.

[opc@rac-instance01 dockerfiles]$ sudo su -
[root@rac-instance01 ~]# mkdir /opt/.secrets/
[root@rac-instance01 ~]# openssl rand -hex 64 -out /opt/.secrets/pwd.key

Use echo to redirect the password string to a file. This time it is "oracle".

[root@rac-instance01 ~]# echo oracle > /opt/.secrets/common_os_pwdfile
[root@rac-instance01 ~]# openssl enc -aes-256-cbc -salt -in /opt/.secrets/common_os_pwdfile -out /opt/.secrets/common_os_pwdfile.enc -pass file:/opt/.secrets/pwd.key

Once you have created the encrypted file, delete the plaintext file.

[root@rac-instance01 ~]# rm -f /opt/.secrets/common_os_pwdfile

6-5. Creating a shared DISK as an ASM area

Create a shared DISK area on the block device to be the ASM area. (There is also a method of setting up a container for NAS called RAC Storage Container and making it a shared DISK, but this time we will use the block device method)

Since the boot volume expanded when the OCI Compute was created is already allocated to the existing file system, the boot volume is expanded separately and a new partition is created to use it as an ASM area. Be sure to do this after expanding the boot volume in 4.3. This is because the extension with oci-growfs is only valid for the partition with the final partition number. (By default, the / dev / sda3 partition on which / is mounted is the last, so / can be easily extended with oci-growfs)

Now let's expand the block device online in the OCI Console. Click the hamburger menu in the upper left of the OCI Console and select Compute-Boot Volume.

Select the target boot volume.

Click Edit.

In the [Volume Size (GB)] input field, enter the value obtained by adding the desired ASM size. Since 100GB is used as ASM this time, add 100GB from the original 200GB to make 300GB. After entering, click Save Changes.

Then you will see something like this, so make a note of the [Rescan Command].

When the icon changes from orange to green Enabled, you're done with the OCI Console.

Check the current status of the blocking device. It hasn't changed in particular.

[opc@rac-instance01 dockerfiles]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

sda 8: 0 0 200G 0 disk ★ 200GB remains tqsda2 8:2 0 8G 0 part [SWAP] tqsda3 8:3 0 191.8G 0 part / mqsda1 8:1 0 200M 0 part /boot/efi

Run the rescan command you wrote down earlier in the OCI Console to make the OS aware of the block device extensions.

[opc@rac-instance01 dockerfiles]$ sudo dd iflag=direct if=/dev/oracleoci/oraclevda of=/dev/null count=1
 echo "1" | sudo tee /sys/class/block/`readlink /dev/oracleoci/oraclevda | cut -d'/' -f 2`/device/rescan1+0 Record input

1 + 0 record output 512 bytes (512 B) copied, 0.000896604 seconds, 571 kB/sec [opc@rac-instance01 dockerfiles]$ echo "1" | sudo tee /sys/class/block/readlink /dev/oracleoci/oraclevda | cut -d'/' -f 2/device/rescan 1

Check the status of the block device again. You can see that it has been expanded to 300GB.

[opc@rac-instance01 dockerfiles]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

sda 8: 0 0 300G 0 disk ★ It is 300GB tqsda2 8:2 0 8G 0 part [SWAP] tqsda3 8:3 0 191.8G 0 part / mqsda1 8:1 0 200M 0 part /boot/efi

Add a partition to the extended block device. An error or warning will appear immediately after executing parted, but fix it.

[opc@rac-instance01 ~]$ sudo -s
[root@rac-instance01 opc]# parted /dev/sda
(parted) print

Error: There is no backup of the GPT table that should be at the end of the disk. Other operating systems may think that the disk is smaller. Would you like to bring the backup last (delete the old backup) and repair it? Fix/Fix/Ignore/Ignore/Cancel (C)/Cancel? Fix Warning: Some of the space available in/dev/sda is not being used. You can either modify the GPT to make all the space available (more 209715200 blocks) or continue as it is, what do you do? Fix/Fix/Ignore? Fix

Number Start End Size File System Name Flag 1 1049kB 211MB 210MB fat16 EFI System Partition boot 2 211MB 8801MB 8590MB linux-swap(v1) 3 8801MB 215GB 206GB xfs

Allocate 100% from the end size of the last partition number. This time, the starting point is 215GB, which is the end of number 3.

(parted) mkpart gpt 215GB 100%

If you check, partition number 4 is created.

(parted) print

Number Start End Size File System Name Flag 1 1049kB 211MB 210MB fat16 EFI System Partition boot 2 211MB 8801MB 8590MB linux-swap(v1) 3 8801MB 215GB 206GB xfs 4 215GB 322GB 107GB gpt

(parted) q

The created partition is/dev/sda4.

[root@rac-instance01 opc]# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   300G  0 disk

-sda4 8:4 0 100G 0 part ★ Created partition -sda2 8:2 0 8G 0 part [SWAP] -sda3 8:3 0 191.8G 0 part / -sda1 8:1 0 200M 0 part /boot/efi

Initialize the added partition.

[root@rac-instance01 opc]# dd if=/dev/zero of=/dev/sda4  bs=8k count=100000

100000 + 0 record input 100000 + 0 record output 819200000 bytes (819 MB) copied, 4.09183 seconds, 200 MB/sec

7. Creating Oracle Database RAC Node1

7-1. Creating a Docker container

RAC node Create the first container. It can be changed in various ways as an option. Let's go to the end of the procedure on Github describes various options.

[opc@rac-instance01 ~]$ sudo docker create -t -i \
    --hostname racnode1 \
    --volume /boot:/boot:ro \
    --volume /dev/shm \
    --tmpfs /dev/shm:rw,exec,size=4G \
    --volume /opt/containers/rac_host_file:/etc/hosts  \
    --volume /opt/.secrets:/run/secrets \
    --device=/dev/sda4:/dev/asm_disk1  \
    --privileged=false  \
    --cap-add=SYS_NICE \
    --cap-add=SYS_RESOURCE \
    --cap-add=NET_ADMIN \
    -e NODE_VIP=172.16.1.160 \
    -e VIP_HOSTNAME=racnode1-vip  \
    -e PRIV_IP=192.168.17.150 \
    -e PRIV_HOSTNAME=racnode1-priv \
    -e PUBLIC_IP=172.16.1.150 \
    -e PUBLIC_HOSTNAME=racnode1  \
    -e SCAN_NAME=racnode-scan \
    -e SCAN_IP=172.16.1.70  \
    -e OP_TYPE=INSTALL \
    -e DOMAIN=example.com \
    -e ASM_DEVICE_LIST=/dev/asm_disk1 \
    -e ORACLE_SID=ORCL \
    -e ASM_DISCOVERY_DIR=/dev \
    -e CMAN_HOSTNAME=racnode-cman1 \
    -e CMAN_IP=172.16.1.15 \
    -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc \
    -e PWD_KEY=pwd.key \
    --restart=always --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    --cpu-rt-runtime=95000 --ulimit rtprio=99  \
    --name racnode1 \
    oracle/database-rac:19.3.0

7-2. Assign a network to a container

Assign the network to the created container.

[opc@rac-instance01 ~]$ sudo docker network disconnect bridge racnode1
[opc@rac-instance01 ~]$ sudo docker network connect rac_pub1_nw --ip 172.16.1.150 racnode1
[opc@rac-instance01 ~]$ sudo docker network connect rac_priv1_nw --ip 192.168.17.150  racnode1

7-3. Start RAC container (RAC Node 1)

From here is the production. Start the container on the first RAC node. By starting this container, GI/DB is installed, DB is created, and RAC is configured.

[opc@rac-instance01 ~]$ sudo docker start racnode1
racnode1

The start command itself returns immediately. Check the log with the following command and make sure that no error occurs. It will take about 40 minutes to 1 hour at the earliest to complete.

[opc@rac-instance01 ~]$ sudo docker logs -f racnode1

If successful, the following message will be displayed at the end.

####################################
ORACLE RAC DATABASE IS READY TO USE!
####################################

If some error occurs on the way and the process stops, log in to the container with the following command and check the log output to /tmp/orod.log or $ GRID_BASE/diag/crs etc. please.

[opc@rac-instance01 ~]$ sudo docker exec -i -t racnode1 /bin/bash

Since the startup was successful, log in to the container and check it.

[opc@rac-instance01 ~]$ sudo docker exec -i -t racnode1 /bin/bash
[grid@racnode1 ~]$ sudo su - oracle
[oracle@racnode1 ~]$ /u01/app/19.3.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racnode1                 STABLE
ora.chad
               ONLINE  ONLINE       racnode1                 STABLE
ora.net1.network
               ONLINE  ONLINE       racnode1                 STABLE
ora.ons
               ONLINE  ONLINE       racnode1                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 Started,STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.orcl.db
      1        ONLINE  ONLINE       racnode1                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
--------------------------------------------------------------------------------
[oracle@racnode1 ~]$ exit
logout
[grid@racnode1 ~]$ exit
exit
[opc@rac-instance01 ~]$

There seems to be no problem, so we will continue to create the second and subsequent nodes.

8. Addition of Oracle Database RAC Node2

8-1. Creating a Docker container

Add a second node to the RAC environment you just created. The procedure is basically the same, but the options specified are slightly different. (Host name, node name, IP address, CLS existing node specification, node addition option, etc.)

[opc@rac-instance01 ~]$ sudo docker create -t -i \
    --hostname racnode2 \
    --volume /dev/shm \
    --tmpfs /dev/shm:rw,exec,size=4G  \
    --volume /boot:/boot:ro \
    --dns-search=example.com  \
    --volume /opt/containers/rac_host_file:/etc/hosts \
    --volume /opt/.secrets:/run/secrets \
    --device=/dev/sda4:/dev/asm_disk1  \
    --privileged=false \
    --cap-add=SYS_NICE \
    --cap-add=SYS_RESOURCE \
    --cap-add=NET_ADMIN \

-e EXISTING_CLS_NODES = racnode1 \ ★ Specify an existing node -e NODE_VIP=172.16.1.161
-e VIP_HOSTNAME=racnode2-vip
-e PRIV_IP=192.168.17.151
-e PRIV_HOSTNAME=racnode2-priv
-e PUBLIC_IP=172.16.1.151
-e PUBLIC_HOSTNAME=racnode2
-e DOMAIN=example.com
-e SCAN_NAME=racnode-scan
-e SCAN_IP=172.16.1.70
-e ASM_DISCOVERY_DIR=/dev
-e ASM_DEVICE_LIST=/dev/asm_disk1
-e ORACLE_SID=ORCL
-e OP_TYPE = ADDNODE \ ★ Add node option -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc
-e PWD_KEY=pwd.key
--tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro
--cpu-rt-runtime=95000 --ulimit rtprio=99
--restart=always
--name racnode2
oracle/database-rac:19.3.0

8-2. Assign network to container

Assign the network to the created container.

[opc@rac-instance01 ~]$ sudo docker network disconnect bridge racnode2
[opc@rac-instance01 ~]$ sudo docker network connect rac_pub1_nw --ip 172.16.1.151 racnode2
[opc@rac-instance01 ~]$ sudo docker network connect rac_priv1_nw --ip 192.168.17.151  racnode2

8-3. Starting the RAC container (RAC Node 2)

Start the container on the second RAC node. What you do is the same as for the first node.

[opc@rac-instance01 ~]$ sudo docker start racnode2
racnode2

Check the log with the following command and make sure that no error occurs.

[opc@rac-instance01 ~]$ sudo docker logs -f racnode2

If successful, the following message will be displayed at the end. The second and subsequent nodes take less time than the first node. In this environment, it took about 10 minutes.

####################################
ORACLE RAC DATABASE IS READY TO USE!
####################################

Since the startup was successful, log in to check.

[opc@rac-instance01 ~]$ sudo docker exec -i -t racnode2 /bin/bash
[grid@racnode2 ~]$ sudo su - oracle
[oracle@racnode2 ~]$ /u01/app/19.3.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
ora.chad
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
ora.net1.network
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
ora.ons
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 Started,STABLE
      2        ONLINE  ONLINE       racnode2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.orcl.db
      1        ONLINE  ONLINE       racnode1                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       racnode2                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode2.vip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
--------------------------------------------------------------------------------

The second node was added without any problem. Continue to add the third node.

9. Addition of Oracle Database RAC Node3

9-1. Creating a Docker container

Add a third node to the RAC environment. The procedure and changes are exactly the same as the second node.

[opc@rac-instance01 ~]$ sudo docker create -t -i \
  --hostname racnode3 \
  --volume /dev/shm \
  --tmpfs /dev/shm:rw,exec,size=4G  \
  --volume /boot:/boot:ro \
  --dns-search=example.com  \
  --volume /opt/containers/rac_host_file:/etc/hosts \
  --volume /opt/.secrets:/run/secrets \
  --device=/dev/sda4:/dev/asm_disk1  \
  --privileged=false \
  --cap-add=SYS_NICE \
  --cap-add=SYS_RESOURCE \
  --cap-add=NET_ADMIN \

-e EXISTING_CLS_NODES = racnode1, racnode2 \ ★ Specify an existing node -e NODE_VIP=172.16.1.162
-e VIP_HOSTNAME=racnode3-vip
-e PRIV_IP=192.168.17.152
-e PRIV_HOSTNAME=racnode3-priv
-e PUBLIC_IP=172.16.1.152
-e PUBLIC_HOSTNAME=racnode3
-e DOMAIN=example.com
-e SCAN_NAME=racnode-scan
-e SCAN_IP=172.16.1.70
-e ASM_DISCOVERY_DIR=/dev
-e ASM_DEVICE_LIST=/dev/asm_disk1
-e ORACLE_SID=ORCL
-e OP_TYPE = ADDNODE \ ★ Add node option -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc
-e PWD_KEY=pwd.key
--tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro
--cpu-rt-runtime=95000 --ulimit rtprio=99
--restart=always
--name racnode3
oracle/database-rac:19.3.0

9-2. Allocate a network to a container

Assign the network to the created container.

[opc@rac-instance01 ~]$ sudo docker network disconnect bridge racnode3
[opc@rac-instance01 ~]$ sudo docker network connect rac_pub1_nw --ip 172.16.1.152 racnode3
[opc@rac-instance01 ~]$ sudo docker network connect rac_priv1_nw --ip 192.168.17.152  racnode3

9-3. Starting the RAC container (RAC Node 3)

Start the container on the third RAC node. What you do is the same as for the second node.

[opc@rac-instance01 ~]$ sudo docker start racnode3
racnode3

Check the log with the following command and make sure that no error occurs.

[opc@rac-instance01 ~]$ sudo docker logs -f racnode3

If successful, the following message will be displayed at the end. The second and subsequent nodes take less time than the first node. In this environment, it took about 10 minutes.

####################################
ORACLE RAC DATABASE IS READY TO USE!
####################################

Since the startup was successful, log in to check.

[opc@rac-instance01 ~]$ sudo docker exec -i -t racnode3 /bin/bash
[grid@racnode3 ~]$ sudo su - oracle
[oracle@racnode3 ~]$ /u01/app/19.3.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
ora.chad
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
ora.net1.network
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
ora.ons
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        ONLINE  ONLINE       racnode3                 STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        ONLINE  ONLINE       racnode3                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 Started,STABLE
      2        ONLINE  ONLINE       racnode2                 Started,STABLE
      3        ONLINE  ONLINE       racnode3                 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        ONLINE  ONLINE       racnode3                 STABLE
ora.cvu
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.orcl.db
      1        ONLINE  ONLINE       racnode1                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       racnode2                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
      3        ONLINE  ONLINE       racnode3                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode2.vip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.racnode3.vip
      1        ONLINE  ONLINE       racnode3                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
--------------------------------------------------------------------------------

The third node was added without any problem. Continue to add the last 4th node.

10. Addition of Oracle Database RAC Node4

10-1. Creating a Docker container

Add the 4th node to the RAC environment. The procedure and changes are exactly the same as the 2nd and 3rd nodes.

[opc@rac-instance01 ~]$ sudo docker create -t -i \
    --hostname racnode4 \
    --volume /dev/shm \
    --tmpfs /dev/shm:rw,exec,size=4G  \
    --volume /boot:/boot:ro \
    --dns-search=example.com  \
    --volume /opt/containers/rac_host_file:/etc/hosts \
    --volume /opt/.secrets:/run/secrets \
    --device=/dev/sda4:/dev/asm_disk1  \
    --privileged=false \
    --cap-add=SYS_NICE \
    --cap-add=SYS_RESOURCE \
    --cap-add=NET_ADMIN \
    -e EXISTING_CLS_NODES=racnode1,racnode2,racnode3 \
    -e NODE_VIP=172.16.1.163  \
    -e VIP_HOSTNAME=racnode4-vip  \
    -e PRIV_IP=192.168.17.153  \
    -e PRIV_HOSTNAME=racnode4-priv \
    -e PUBLIC_IP=172.16.1.153  \
    -e PUBLIC_HOSTNAME=racnode4  \
    -e DOMAIN=example.com \
    -e SCAN_NAME=racnode-scan \
    -e SCAN_IP=172.16.1.70 \
    -e ASM_DISCOVERY_DIR=/dev \
    -e ASM_DEVICE_LIST=/dev/asm_disk1\
    -e ORACLE_SID=ORCL \
    -e OP_TYPE=ADDNODE \
    -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc \
    -e PWD_KEY=pwd.key \
    --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    --cpu-rt-runtime=95000 --ulimit rtprio=99  \
    --restart=always \
    --name racnode4 \
    oracle/database-rac:19.3.0

10-2. Allocate a network to a container

Assign the network to the created container.

[opc@rac-instance01 ~]$ sudo docker network disconnect bridge racnode4
[opc@rac-instance01 ~]$ sudo docker network connect rac_pub1_nw --ip 172.16.1.153 racnode4
[opc@rac-instance01 ~]$ sudo docker network connect rac_priv1_nw --ip 192.168.17.153  racnode4

10-3. Starting the RAC container (RAC Node 4)

Start the container on the 4th RAC node. What you do is the same as for the second and third nodes.

[opc@rac-instance01 ~]$ sudo docker start racnode4
racnode4

Check the log with the following command and make sure that no error occurs.

[opc@rac-instance01 ~]$ sudo docker logs -f racnode4

If successful, the following message will be displayed at the end.

####################################
ORACLE RAC DATABASE IS READY TO USE!
####################################

Since the startup was successful, log in to check.

[opc@rac-instance01 ~]$ sudo docker exec -i -t racnode4 /bin/bash
[grid@racnode4 ~]$ sudo su - oracle
[oracle@racnode4 ~]$ /u01/app/19.3.0/grid/bin/crsctl stat res -t

--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
               ONLINE  ONLINE       racnode4                 STABLE
ora.chad
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
               ONLINE  ONLINE       racnode4                 STABLE
ora.net1.network
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
               ONLINE  ONLINE       racnode4                 STABLE
ora.ons
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
               ONLINE  ONLINE       racnode3                 STABLE
               ONLINE  ONLINE       racnode4                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        ONLINE  ONLINE       racnode3                 STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        ONLINE  ONLINE       racnode3                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 Started,STABLE
      2        ONLINE  ONLINE       racnode2                 Started,STABLE
      3        ONLINE  ONLINE       racnode3                 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
      3        ONLINE  ONLINE       racnode3                 STABLE
ora.cvu
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.orcl.db
      1        ONLINE  ONLINE       racnode1                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       racnode2                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
      3        ONLINE  ONLINE       racnode3                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
      4        ONLINE  ONLINE       racnode4                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /dbhome_1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode2.vip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.racnode3.vip
      1        ONLINE  ONLINE       racnode3                 STABLE
ora.racnode4.vip
      1        ONLINE  ONLINE       racnode4                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
--------------------------------------------------------------------------------

The 4th node was added without any problem.

11. Database login confirmation

Check if you can log in to the DB.

[opc@rac-instance01 ~]$ sudo docker exec -i -t racnode1 /bin/bash
[grid@racnode1 ~]$ export ORACLE_HOME=`echo ${DB_HOME}`
[grid@racnode1 ~]$ export ORACLE_SID=ORCL1
[grid@racnode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jan 6 16:12:43 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 ORCLPDB                        READ WRITE NO

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

I was able to log in. A PDB has also been created.

that's all.

12. Postscript

You have now built an Oracle Database 19c 4 Node RAC environment with Docker on OCI Compute.

I could have created an Oracle Data Guard Physical Standby container in this environment, but I'll write that at another time.

Although the environment was built on OCI Compute this time, it can be built on Virtual Box instead of the cloud environment, and we have a track record. With the new Virtual Box, I think it is possible to transfer the image built with Virtual Box to OCI as it is.

Finally, as mentioned at the beginning, the Oracle Database environment built on Docker can be used only for verification and development purposes. Please do not use it in a production environment.