Set client-specific ssh host keys on multiple diskless clients that utilize the exact same root file system

Foreword: Target readers of this article

• Anyone who wants to build some kind of cluster on multiple diskless Linux
• When building a cluster with 10 to 20 or more nodes, it is overwhelmingly easier to manage users / hosts by setting up one LDAP server, so people who do not have the number of nodes until setting up an LDAP server
• People who want to use an OS image on the same NFS share on all nodes because they don't want to overwhelm the NFS server disk
• People who want to start an SSH server on all nodes but do not want to use a common SSH host key on all nodes (= want to use a host key unique to each node)

It is information for people who meet all requirements.

The essence of the problem

1. Since multiple machines use the same root file system, using the default sshd_config file causes sshd on all machines to read the same host key file (/ etc / ssh / ssh_host_ * _key) and start it.
2. It is impossible for Linux to separate / etc from the root and another partition.
3. It is almost impossible to use overlayfs with a diskless client that uses NFS (overlayfs does not support NFSv3).
4. If you change the host key under / etc / ssh before starting sshd, the keys of other hosts will also change. Since it is a diskless client in the first place, it is impossible because the files under / etc cannot be changed.

solution

You can solve the above problem by the following methods.

1. It doesn't matter where you save it, so save all the node-specific host keys in advance (create a subdirectory with the node host name and put them all in it).
2. Prepare one writable directory (eg create / var / lib / sshd and mount it as tmpfs in / etc / fstab)
3. Set sshd_config to read the host key under the directory created in 2.
4. At startup, after acquiring the IP (the local host name is known), and just before sshd starts, change the host key stored in the subdirectory for the local host from 1. to the directory 2. copy

This technique can be used not only for sshd, but also for starting other daemons that utilize host-specific information. (However, you can use DNS or DHCP for IP and host name.)

Specific method

All work is done on an NFS server, chrooting to the OS image distributed to the nodes. Below, it is assumed that the host name of the diskless node is host1 to host4.

1. How to generate an individual host key

For security, keep all keys in / root / host_keys.

# mkdir /root/host_keys; cd /root/host_keys
# for N in `seq 1 4`; do HN="host${N}"; mkdir ${HN}; for T in rsa ecdsa ed25519; do ssh-keygen -h -t ${T} -C ${HN} -N '' -f ${HN}/ssh_host_${T}_key; done; done

2. Create a writable directory for diskless clients

# mkdir /var/lib/sshd
# vim /etc/fstab
(The changes are as follows)
#

Add the following line to / etc / fstab

tmfps /var/lib/sshd tmpfs defaults,mode=700 0 0

3. Modify the sshd_config file

# vi /etc/ssh/sshd_config
(The changes are as follows)
#

Changes (enabled and changed HostKey entry commented out by default)

(abridgement)
HostKey /var/lib/sshd/ssh_host_rsa_key
HostKey /var/lib/sshd/ssh_host_ecdsa_key
HostKey /var/lib/sshd/ssh_host_ed25519_key
(abridgement)

4. Create a process to copy the host-specific key file at the right time

In order to handle the timing properly, create a new systemd unit file and format it to call a dedicated script from here.

4.1 Creating a dedicated shell script

The file name can be anything, but it should be / usr / local / sbin / copy_sshkeys.

# vim /usr/local/sbin/copy_sshkeys
# chmod 700 /usr/local/sbin/copy_sshkeys

The contents of the script are as follows. Set the return value so that systemd can determine the error.

#!/bin/bash
HN=`hostname`
RETVAL=0

if [ "${HN}" != "" ]
then
  cp -pf /root/host_keys/${HN}/ssh_host* /var/lib/sshd/
  RETVAL=$?
else
  #Failed to get the host name
  RETVAL=2
fi

exit $RETVAL

4.2. Creating a systemd Unit file

Create a new Unit file under / etc / systemd / system. The name of this file can be anything, as long as you follow the rules for the extension .service, but for clarity, name it ssh_hostkey_copy.service.

# vi /etc/systemd/system/ssh_hostkey_copy.service
(The contents are as follows)
# systemctl enable ssh_hostkey_copy.service
(Output result omitted)
#

The contents of the Unit file are as follows.

[Unit]
Description=SSH Hostkey copy process for diskless clients
After=network.target local-fs.target
#Sshd depending on the distribution.It may be service
Before=ssh.service

[Service]
ExecStart=/usr/local/sbin/copy_sshkeys
Type=oneshot

[Install]
WantedBy=multi-user.target
RequiredBy=ssh.service

Now, even diskless clients can start sshd using the unique host key.