Collection to deploy OKD/OCP on baremetal

This Project provides CLI tools to help OKD/OCP deployment with a special focus on baremetal.

1. Type of Architecture and Requirements

Add to above list, an helper node or pfsense to provide following services: DNS / DHCP / PXE boot / LoadBalancer (+ eventually registry). In case of deployment on KVM, the DNS, DHCP and TFTP are embeded in KVM to avoid changes on the host's network config.

NB: SNO installation provide a bootstrap-in-place-for-live-iso.ign which works only for iso boot (but not for pxeboot). The UPI require only 4 CPU when ABI require 8 cpu. For an SNO install, it understandable to have a usb drive to boot on baremetal. But it become a constraint when there is 3 masters with the Agent Based Install which need to booted in the same time.

1. Get a Pull Secret and set it in .docker/config.json

{
  "auths": {
    "cloud.openshift.com": {
      "auth": "<snip>",
      "email": "<user's email>"
    },
    "quay.io": {
      "auth": "<snip>",
      "email": "<user's email>"
    },
    "registry.connect.redhat.com": {
      "auth": "<snip>",
      "email": "<user's email>"
    },
    "registry.redhat.io": {
      "auth": "<snip>",
      "email": "<user's email>"
    }
  }
}

if KVM is used

sudo apt update
sudo apt install cpu-checker
sudo kvm-ok

#Basic
sudo apt -y install libvirt-daemon-system bridge-utils qemu-kvm libvirt-daemon

#Extra tools
sudo apt install virtinst libosinfo-bin virt-top libguestfs-tools 
sudo apt install xsltproc uidmap

# GUI tools
sudo apt -y install qemu-system virt-manager 

sudo addgroup kvm
sudo adduser `id -un` libvirt-qemu
sudo adduser `id -un` kvm
newgrp libvirt

2. Clone this project and get inside

git clone https://github.com/mozebaltyk/Okub.git

• get a first look a the project:

# Get a first look
cd Okub
just

• init a project

export PRODUCT="okd"
export RELEASE_VERSION="4.15"
export MASTERS=1
export DHCP_BOOL=true
export TYPE_OF_INSTALL="iso"

just init iso

• deploy OCP/OKD on KVM

just ocp create

just ocp destroy

Few tips for troubleshooting:

cd ${OKUB_INSTALL_PATH}
export KUBECONFIG=./auth/kubeconfig
# Agent based install
./bin/openshift-install --dir . agent wait-for bootstrap-complete --log-level=info
./bin/openshift-install --dir . agent wait-for install-complete --log-level=info
# SNO and other type of install 
./bin/openshift-install --dir . wait-for bootstrap-complete --log-level=info
./bin/openshift-install --dir . wait-for install-complete --log-level=info

/bin/oc get co

# Connect in ssh to first master node
journalctl -u bootkube --no-pager | tail -50
journalctl -u kubelet --no-pager | tail -50

systemctl status bootkube -l
systemctl status kubelet -l

# Check if the certificate is still valid
[[ $(jq -r '.. | objects | select(.Filename? == "tls/root-ca.crt") | .Data' .openshift_install_state.json  | base64 -d | openssl x509 -noout -startdate | cut -d= -f2 | xargs -I{} date -d {} +%s) -le $(date -d "24 hours" +%s) ]] && echo OK || echo NOK

After first reboot, fs should be setup:

lsblk

Once install is finished

oc whoami --show-console

1. Diverse installation methods

We should normally count one extra bootstrap node, but with Single-node installer and Agent-based Installer bootstraping is handled by one master node. Since this project focus on baremetal installation. So there is a benefice to not use one baremetal for bootstraping which then need to be erase and reuse as a worker but added manually.

The Single-node installer will have an ignition file named bootstrap-in-place-for-live-iso.ign. This install method could seems outdated and the Agent-based Installer a better approach. But there are still some advantage left to use it, the install is completed as bootstrap-in-place and require only 4 vcpu instead of 8 vcpu for Agent-based Installer. In case of resources scarcity like running locally on laptop, it makes sense to keep this option available.

The Agent-based Installer will require an extra agent-config.yaml to set up the rendezvousIP. In the case of DHCP, this will be the control-plane IP. In an environment without a DHCP server, you can define IP addresses statically. This method seems to work for OKD even though it is not present in the documentation.

2. plateform options

Agent-based Installer support only those 3 plateforms options below:

• none, the only possible option for single-node installer but works also on all raw install (like for a baremetal without BMC).

  Requirements for plateform: none{}:

  - `networkType: OVNKubernetes`
  
  - DNS for `*.api.<domain>` and `apps.<domain>` pointing to the Loadbalancer.
  
  - DNS and reverse DNS (PTR) for all masters and workers is required
  
  - DHCP services to provide IP addresses to nodes during installation.
  
  - Loadbalancer for 6443 and 22623 if not standalone install ( since `apiVIPs` and `ingressVIPs` are not defined in *none* block )
  

• baremetal, for hardware with BMC or for configuring dual stacks network.

  Requirements for plateform: baremetal{}:

  - if `apiVIPs` and `ingressVIPs` are defined in the config, no need for loadbalancing
  
  - if static IP defined then no DHCP
  
  - if Outcome iso - no PXE boot server
  
  - only DNS for `*.api.<domain>` and `apps.<domain>` is required
  

• vsphere, does not concern us since this project focus mainly on baremetal.

4. Diverse "Helpers" are present as ansible roles but the best would be to use pfsense vm or router:

• DNS = Bind server.

• DHCP = DHCP server.

• PXE server = TFTP server.

• Loadbalancer = HAproxy server.

5. Diverse Installer Outcome

• an bootable iso to burn on USB stick

• pxe boot to push on helper server or in the KVM embended TFTP server.

• Baremetal

https://github.com/ryanhay/ocp4-metal-install/tree/master

• Agent-based

https://www.redhat.com/en/blog/meet-the-new-agent-based-openshift-installer-1

https://kapilrajyaguru.medium.com/agent-based-red-hat-openshift-cluster-install-ee33d3b9fe0e

https://docs.openshift.com/container-platform/4.14/installing/installing_with_agent_based_installer/preparing-to-install-with-agent-based-installer.html#static-networking

https://github.com/cgruver/kamarotos/blob/main/agent-install.md

• KVM

https://github.com/jmutai/ocp4_ansible

https://github.com/lgchiaretto/ocp4_setup_upi_kvm_ansible/tree/master

https://fajlinuxblog.medium.com/openshift-running-as-single-node-with-libvirt-kvm-cb615d2c43e6

• Vsphere

https://guifreelife.com/blog/2022/05/13/OpenShift-Virtualization-on-vSphere/

• Airgap

https://www.redhat.com/en/blog/red-hat-openshift-disconnected-installations

https://two-oes.medium.com/openshift-4-in-an-air-gap-disconnected-environment-part-2-installation-1dd8bf085fdd

milestones:

• To deploy sone staffs
• To add flavors

Improvment:

• Add a option to chooce by url or by copy

morze.baltyk@proton.me

Still on developement