[DOCKER] Settings when installing Mirantis Kubernetes

Introduction

Create a cluster configuration file to build MirantisKubernetes. In the previous two times, I created a cluster configuration file in advance, but this time I would like to confirm the setting items.

-Mirantis Kubernetes construction (formerly Docker Enterprise) -Istio Ingress on Mirantis Kubernetes

Setting items at the time of installation

launchpad.You can check the yaml setting items from the following.



> Mirantis Launchpad Configuration File
> https://github.com/Mirantis/launchpad/blob/master/docs/configuration-file.md

 I will pick up and explain the parameters that are likely to be used frequently.
 * This time, it will be the setting item in launchpad 1.1.1.

 ** Common **: Common settings

 --kind: Set with ``` mke``` if only MKE (k8s), and with ``` mke + msr``` if you also use MSR (Registory).
- metadata
 --name: The name of the cluster. For now, it is used internally for logs, client settings, etc.

 ** hosts **: Settings for each host

 --address: IP address when connecting to each node
 --privateInterface: NIC can be specified when configuring the network (initial value: `` `eth0```)
 --role: You can specify the role in the cluster (`` `manager```, `` `worker```, `` `msr```)
 --engineConfig: Mapped as a Docker Engine config file in the lower hierarchy (converted to daemon.json) <br> The above URL example specifies debug mode and log options (file size and rotation).
 --ssh: This is the setting when the connection method is SSH.
 --user: User name for SSH connection (initial value: ``` root```)
 --port: Port number for SSH connection (initial value: 22)
 --keypath: Private key path for SSH connection (initial value: `` `~/.ssh/id_rsa```)

 ** MKE (mke) **: MKE settings

 * version: Version of MKE to be installed (initial value: 3.3.3)
 * adminUsername: Administrator account name (initial value: admin)
 * adminPassword: Administrator account password (default: generate a random character string without specifying)
 * installFlags: Explained in InstallFlags
 * configFile: Explained in config.toml
 * configData: Explained in config.toml

 ** DockerEngine (engine) **: DockerEngine settings

 * version: Docker Engine version

 InstallFlags

 You can make detailed settings by setting the installFlags item.
 You can check the items by running the following on the VM where docker is installed.

$ docker run -t -i --rm docker/ucp:3.3.0 install --help Docker Universal Control Plane Tool

install - Install UCP on this node

USAGE: docker run --rm -it
-v /var/run/docker.sock:/var/run/docker.sock
docker/ucp
install [command options]

DESCRIPTION:

This command initializes a new swarm, turns this node into a manager, and installs Docker Universal Control Plane (UCP).

When installing UCP you can customize:

If you're joining more nodes to this swarm, open the following ports in your firewall:

If you have SELinux policies enabled for your Docker install, you will need to use 'docker run --rm -it --security-opt label=disable ...' when running this command.

OPTIONS: --debug, -D Enable debug mode --jsonlog Produce json formatted output for easier parsing --interactive, -i Run in interactive mode and prompt for configuration values --admin-username value The UCP administrator username [UCP_ADMIN_USER] --admin-password value The UCP administrator password [UCP_ADMIN_PASSWORD] --swarm-grpc-port value Port for communication between nodes (default: 2377) --san value Add subject alternative names to certificates (e.g. --san www1.acme.com --san www2.acme.com) [UCP_HOSTNAMES] --host-address value The network address to advertise to other nodes. Format: IP address or network interface name [UCP_HOST_ADDRESS] --data-path-addr value Address or interface to use for data path traffic. Format: IP address or network interface name [UCP_DATA_PATH_ADDR] --jwk-url value JWK url of external OpenID provider [JWK_URL] --force-minimums Force the install/upgrade even if the system does not meet the minimum requirements --controller-port value Port for the web UI and API (default: 443) --kube-apiserver-port value Port for the Kubernetes API server (default: 6443) --swarm-port value Port for the Docker Swarm manager. Used for backwards compatibility (default: 2376) --cni-installer-url value A URL pointing to a kubernetes YAML file to be used as an installer for the CNI plugin of the cluster. If specified, the default CNI plugin will not be installed. If the URL is using the HTTPS scheme, no certificate verification will be performed --unmanaged-cni Flag to indicate if cni provider is calico and managed by UCP (calico is the default CNI provider) --secure-overlay Flag to enable IPSec network encryption using SecureOverlay in Kubernetes --calico-vxlan Flag to indicate if calico CNI should choose VXLAN as default dataplane. Default is vxlan --istio Flag to indicate if Istio Ingress should be enabled upon startup --vxlan-vni value Flag allows to choose vxlan-vni ID (10000-20000) if VXLAN dataplane mode is selected (Default: 10000) (default: 10000) --pod-cidr value Kubernetes cluster IP pool for the pods to allocated IP from (Default: 192.168.0.0/16) (default: "192.168.0.0/16") --service-cluster-ip-range value Kubernetes Cluster IP Range for Services (Default: 10.96.0.0/16) (default: "10.96.0.0/16") --nodeport-range value Allowed port range for Kubernetes services of type NodePort (Default: 32768-35535) (default: "32768-35535") --cloud-provider value The cloud provider for the cluster --default-node-orchestrator value The default orchestrator for the nodes in cluster. Can be either swarm or kubernetes. By default, UCP nodes use swarm as the orchestrator. --azure-ip-count value Configure the Number of IP Address to be provisioned for each Azure Virtual Machine (Default: 64) (default: "64") --storage-iscsi Enable ISCSI based Persistent Volumes in Kubernetes --iscsiadm-path value Absolute path to host iscsiadm binary. Option applicable only when --storage-iscsi is specified. Symlinks not allowed --iscsidb-path value Absolute path to host iscsi DB. Option applicable only when --storage-iscsi is specified. Symlinks not allowed --storage-expt-enabled Flag to enable experimental features in Kubernetes storage --dns value Set custom DNS servers for the UCP containers [DNS] --dns-opt value Set DNS options for the UCP containers [DNS_OPT] --dns-search value Set custom DNS search domains for the UCP containers [DNS_SEARCH] --unlock-key value The unlock key for this swarm-mode cluster, if one exists. [UNLOCK_KEY] --existing-config Use the latest existing UCP config during this installation. The install will fail if a config is not found --skip-cloud-provider-check Disables checks which rely on detecting which (if any) cloud provider the cluster is currently running on --pull value Pull UCP images: 'always', when 'missing', or 'never' (default: "missing") --registry-username value Username to use when pulling images [REGISTRY_USERNAME] --registry-password value Password to use when pulling images [REGISTRY_PASSWORD] --kv-timeout value Timeout in milliseconds for the key-value store (default: 5000) [KV_TIMEOUT] --kv-snapshot-count value Number of changes between key-value store snapshots (default: 20000) [KV_SNAPSHOT_COUNT] --disable-tracking Disable anonymous tracking and analytics --disable-usage Disable anonymous usage reporting --external-server-cert Customize the certificates used by the UCP web server --preserve-certs Don't generate certificates if they already exist --binpack Set the Docker Swarm scheduler to binpack mode. Used for backwards compatibility --random Set the Docker Swarm scheduler to random mode. Used for backwards compatibility --external-service-lb value Set the IP address of the load balancer that published services are expected to be reachable on --enable-profiling Enable performance profiling --license value Add a license: e.g. --license "(cat license.lic)" [UCP_LICENSE] --force-insecure-tcp Force install to continue even with unauthenticated Docker Engine ports.


 config.toml
 You can also pass the MKE installation settings as a configuration file.
 Create a sample file with the following command and specify the path in launchpad.yaml.
 * You can also pass the value in the configuration file directly to "configData".
 * Item explanations are also explained on the following pages.
https://docs.mirantis.com/docker-enterprise/v3.1/dockeree-products/mke/mke-configure/mke-configuration-file.html


#### **`shell-sessionconfig.toml creation`**
```toml creation

$ docker container run --rm mirantis/ucp:3.3.2 example-config

config.toml


# Example UCP configuration file
[auth]

  # The name of the auth backend to use, either "managed" or "ldap".
  backend = "managed"

  # NOTE: LDAP options may only be configured via the UCP UI or API.

  # Indicates if SAML is enabled on this UCP.
  samlEnabled = false

  # The customized SAML login button text
  samlLoginText = ""

  # Indicates if SCIM is enabled on this UCP.
  scimEnabled = false

  # Indicates if managed password is disabled.
  managedPasswordDisabled = false

  # The fallback user when the managed password authentication is disabled.
  managedPasswordFallbackUser = ""

  # The role that new users will get for their private collections. Can be
  # "admin", "viewonly", "scheduler", "restrictedcontrol", or "fullcontrol".
  default_new_user_role="restrictedcontrol"

  # JSON Web Keys url of external OpenID provider.
  jwkUrl="http:///www.example.service/openid-connect/certs"

  [auth.sessions]
    # Specifies the initial lifetime (in minutes) of a session from the moment
    # it is generated.
    lifetime_minutes = 60

    # Indicates a period of time (in minutes) before the expiration of a session
    # where, if used, a session will be extended by the current configured lifetime
    # from then, a zero value disables session extension.
    renewal_threshold_minutes = 20

    # Indicates the maximum number of sessions that any user can have active at any
    # given time. If creating a new session would put a user over this limit then
    # the least recently used session will be deleted. A value of zero disables
    # limiting the number of sessions that users may have.
    per_user_limit = 10

    # Set token to be stored in sessionStorage instead of localStorage
    store_token_per_session = false

  # SAML related settings
  [auth.saml]

    # Service Provider Host
    spHost = "52.38.7.31"

    # Identity Provider URL
    idpMetadataURL = "https://www.testshib.org/metadata/testshib-providers.xml"

    # Root Certs for IdP metadata
    rootCerts = ""

    # TLS Skip verify for IdP Metadata
    tlsSkipVerify = false

  # SCIM related settings
  [auth.scim]

    # SCIM API token
    token = "4e102f46-6258-4501-9a54-4e5993fabcf5"

    # Organization name
    orgName = "docker-datacenter"

[[registries]]
  # The address used to connect to the DTR instance tied to this UCP cluster
  host_address = ""

  # The DTR instance's OpenID Connect Client ID, as registered with our auth provider.
  service_id = ""

  # The root CA bundle for the DTR instance (if using a custom CA).
  ca_bundle = ""

  # Whether or not this DTR can look up image scanning data in batches
  batch_scanning_data_enabled = true

[scheduling_configuration]

  # Allow admins to schedule containers on managers
  # Set to true to allow admins to schedule on manager
  enable_admin_ucp_scheduling = true

  # The type of orchestrator to use for new nodes that are joined to the
  # cluster. Can be either "swarm" or "kubernetes".
  default_node_orchestrator = "swarm"


[tracking_configuration]

  # Disable analytics of usage information
  # Set to true to disable analytics
  disable_usageinfo = false

  # Disable analytics of API call information
  # Set to true to disable analytics
  disable_tracking = false

  # Anonymize analytic data
  # Set to true to hide your license ID
  anonymize_tracking = false

  # Set an optional label to be included with analytics that are sent to
  # Docker indicating whether this UCP cluster is a production cluster or a
  # development cluster. This allows Docker to analyze the data in UCP
  # analytics more accurately.
  cluster_label = "prod"


[trust_configuration]

  # Require images be signed by content trust
  require_content_trust = false

  # Specify users or teams which must sign images
  require_signature_from = ["team1", "team2"]


[log_configuration]

  # Set the logging level for UCP components - uses syslog levels
  level = "DEBUG"

[audit_log_configuration]

  # Specify the audit logging level. Leave empty for disabling audit logs. Other
  # legal values are "metadata" and "request". TODO: add link to level descriptions
  level = ""

  # When set to true, support dumps will include audit logs in the logs
  # of the ucp-controller container of each manager node.
  support_dump_include_audit_logs = false

[license_configuration]

  # Enable attempted automatic license renewal when the license nears expiration
  # If disabled, you must manually upload renewed licesnse after expiration.
  auto_refresh = true

[[custom_api_server_headers]]
  # The name of the HTTP header.
  name = "X-Some-Custom-Name"
  # The value of the HTTP header.
  value = "Some Custom Value"

[pki]

  # A custom Root CA bundle to use for client TLS certificate authentication in
  # addition to UCP's internal cluster CAs.
  client_ca_bundle = ""

[user_workload_defaults]
  # A map describing default values to set on Swarm services at creation time
  # if those fields are not explicitly set in the service spec.
  [user_workload_defaults.swarm_defaults]
    [user_workload_defaults.swarm_defaults."tasktemplate.restartpolicy.delay"]
      value = "5m"
    [user_workload_defaults.swarm_defaults."tasktemplate.restartpolicy.maxattempts"]
      value = "3"

[cluster_config]

  # Configures the port the ucp-controller listens to
  controller_port = 443

  # Configures the port the Kubernetes API server listens to
  kube_apiserver_port = 6443

  # Configures the port the ucp-swarm-manager listens to
  swarm_port = 2376

  # Configures Swarm scheduler strategy for container scheduling
  # This does not affect swarm-mode services
  swarm_strategy = "spread"

  # Configures DNS settings for the UCP components
  dns = []
  dns_opt = []
  dns_search = []

  # Turn on specialized debugging endpoints for profiling UCP performance
  profiling_enabled = false

  # Turn off Swarm API polling to increase performance of UCP
  swarm_polling_disabled = false

  # Enable aggregation of DTR image vulnerability scans
  image_scan_aggregation_enabled = false

  # Tune the KV store timeout and snapshot settings
  kv_timeout = 5000
  kv_snapshot_count = 20000

  # Specify an optional external LB for default links to services with expose ports in the UI
  external_service_lb = ""

  # Specify the URL of a kubernetes YAML file to be used for installing a CNI
  # plugin. Only applies during initial installation. If empty, the default CNI
  # plugin will be used
  cni_installer_url = ""

  # Set to true if ucp should not manage the cni provider lifecycle.
  # UCP only manages Calico plugin if the unmanaged-flag is set to false.
  unmanaged_cni = false

  # Set to true if ucp should deploy secure-overlay which enables IPsec
  # Network Encryption in Kubernetes
  secure_overlay = false

  # Set to true if cluster should use vxlan as dataplane.
  calico_vxlan = false

  # Set VXLAN VNI id for vxlan data plane.
  vxlan_vni = 10000

  # Adjust the metrics retention time
  metrics_retention_time = ""

  # Set the interval for how frequently managers gather metrics from nodes in the cluster
  metrics_scrape_interval = "1m"

  # Set the size of the cache used by UCP's RethinkDB servers. The installed
  # default is 1GB, but leaving this field empty or specifying the special
  # value "auto" will instruct RethinkDB to determine a cache size
  # automatically (about half of currently available memory) which ensures a
  # reliable operation in most scenarios, but the cache size can be tuned
  # manually to accommodate increased memory usage by other processes or to
  # maximize query performance. A larger cache improves the database’s
  # performance, but you must consider other sources of memory consumption to
  # avoid out of memory conditions.
  # https://www.rethinkdb.com/docs/memory-usage/#cache-size
  rethinkdb_cache_size = "16GB"

  # If true, disable headers in UCP API responses that indicate which
  # UCP controller is returning the response
  exclude_server_identity_headers = true

  # Set the cloud provider for the kubernetes cluster. Depending on the cloud provider
  # different configurations to kubernetes plugin might be need.For eg This could be used
  # for cases where we need specific CNI plugin depending on the cloud provider to support
  # pod networking
  cloud_provider = ""

  # Enable the cluster's ISCSI storage backend. Enabling this prepares the
  # Kubernetes worker nodes for creation and management of ISCSI based
  # persistent volumes.
  iscsi_enabled = false

  # iscsiadm is the absolute path to the host's iscsiadm binary. This is used when
  # iscsi_enabled is set to true. This is necessary for the containerized
  # kubelet to act as a iscsi initiator. Defaults to "/usr/sbin/iscsiadm"
  # Symlinks not allowed.
  iscsiadm_path = ""

  # iscsidb is the absolute path to the host's iscsi database. This is used when
  # iscsi_enabled is set to true. This is necessary for the containerized
  # kubelet to act as a iscsi initiator. Defaults to "/etc/iscsi". Symlinks not allowed.
  iscsidb_path = ""

  # Set cluster Pod subnet pool. This will be the subnet pool from which the IP for the pod
  # would be allocated by the CNI ipam plugin
  pod_cidr = "192.168.0.0/16"

  # storage_expt_enabled flag is used to enable experimental storage features in the cluster.
  storage_expt_enabled = false

  # Set the MTU (maximum transmission unit) size for the Calico plugin.
  calico_mtu = "1480"

  # Set the VXLAN MTU (maximum transmission unit) size for the Calico plugin.
  calico_vxlan_mtu = "1450"

  # Set the VXLAN Port for the Calico plugin.
  calico_vxlan_port = "4789"

  # Set the IPIP MTU size for the calico IPIP tunnel interface
  ipip_mtu = "1480"

  # Set the IP count for azure allocator to allocate IPs per azure VM
  azure_ip_count = "64"

  # Set the ServiceClusterIPRange used by Kubernates for Services
  service_cluster_ip_range = "10.96.0.0/16"

  # Set the port range that for Kubernetes services of type NodePort can be
  # exposed in.
  nodeport_range = "32768-35535"

  # Enable the use of a KMS (Key Management Service) plugin. The KMS plugin
  # must be deployed on every manager node. UCP will configure the API server
  # with the specified endpoint socket.
  kms_enabled = false

  # Name of the KMS resource
  kms_name = ""

  # Host path of the KMS endpoint socket
  kms_endpoint = ""

  # Number of data encryption keys to be cached in the clear by the KMS.
  kms_cachesize = 0

  # ADVANCED: Set any additional flags on the Kubernetes system components that
  # you may require. You must ensure that any custom flags do not conflict with
  # configuration options already set by UCP.
  custom_kube_api_server_flags = ["--event-ttl=1h0m0s", "--service-node-port-range=30000-32767"]
  custom_kube_controller_manager_flags = ["--service-sync-period=5m0s"]
  custom_kubelet_flags = ["--http-check-frequency=20s", "--kube-api-qps=200", "--kube-api-burst=200"]
  custom_kube_scheduler_flags = ["--algorithm-provider=DefaultProvider", "--kube-api-qps=200", "--kube-api-burst=200"]
  custom_kube_proxy_flags = ["--conntrack-min=300000", "--conntrack-max-per-core=200000"]

  # Store data about collections for volumes in UCP's local KV store instead of
  # on the volume labels. This is used for enforcing access control on volumes.
  # Most users will not need to turn this on; this is only necessary if you
  # use third-party volume drivers.
  local_volume_collection_mapping = false

  # Reserve resources for Docker UCP and Kubernetes components which are
  # running on manager nodes.
  # For more details, see:
  #   https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved
  manager_kube_reserved_resources = "cpu=1,memory=2Gi,ephemeral-storage=4Gi"

  # Reserve resources for Docker UCP and Kubernetes components which are
  # running on worker nodes.
  # For more details, see:
  #   https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved
  worker_kube_reserved_resources = "cpu=100m,memory=300Mi,ephemeral-storage=500Mi"

  # Set Number of Pods that can run on this Kubelet.
  kubelet_max_pods = 110

  # Set pre logon message to alert users before they proceed with login.
  pre_logon_message = ""

  # Enable Windows GMSA support in Kubernetes. Enabling this deploys the webhook
  # admission controller, and creates the Kubernetes resources, required for GMSA
  # workloads in Kubernetes.
  windows_gmsa = false

  [cluster_config.service_mesh]
    # Disable Istio Ingress and Service Mesh
    enabled = false

    # Set the Number of Ingress Replicas
    ingress_num_replicas = 2

    # Set the List of External IPs for Ingress Gateway Service
    ingress_external_ips = []

    # Enable external load balancer
    ingress_enable_lb = false

    # Enable preserving inbound traffic source IP
    ingress_preserve_client_ip = false

    # Set exposed ports
    [[cluster_config.service_mesh.ingress_exposed_ports]]
      name = "name"
      port = 1
      target_port = 2
      node_port = 3

    # Set node affinity
    [cluster_config.service_mesh.ingress_node_affinity]
       nodetype = "ingress"

    # Set node toleration
    [[cluster_config.service_mesh.ingress_node_toleration]]
      key = "com.docker.ucp.manager"
      value = "dummy"
      operator = "Exists"
      effect = "NoSchedule"

Summary

This time, I checked the setting method and setting values ​​at the time of installation. Since there are core settings from the required items, I think it is better to set from the required place and apply the settings as appropriate.

Recommended Posts

Settings when installing Mirantis Kubernetes
Setting maintenance when installing Eclipse
Ansible: Allow downgrade when installing Docker
[Practice! ] Minimum settings when using MyBatis