Enable Overlay Network

Edge clusters are often deployed in locations where network environments are not managed by teams that maintain the Edge deployments. However, a Kubernetes cluster, specifically several control plane components, requires stable IP addresses. In the case of an extended network outage, it is possible that your cluster components would lose their original IP addresses when the cluster expects them to remain stable, causing the cluster to experience degraded performance or become non-operational.

Palette allows you to create a virtual overlay network on top of the physical network, where the virtual IP addresses of all cluster components are managed by Palette. Inside the cluster, the different components use the virtual IP addresses to communicate with each other instead of the underlying IP addresses that could change due to external factors. If the cluster experiences an outage with the overlay network enabled, components inside the cluster retain their virtual IP addresses in the overlay network, even if their IP addresses in the underlying physical network change, protecting the cluster from an outage.

VxLAN Overlay Architecture

tech preview

This is a Tech Preview feature and is subject to change. Do not use this feature in production workloads.

When Should You Consider Enabling Overlay Network?

If your Edge clusters are deployed in network environments that fit any of the following descriptions, you should consider enabling an overlay network for your cluster:

Network environments with dynamic IP address management, such as a Dynamic Host Configuration Protocol (DHCP) network.
Unstable network environments or environments that are out of your control. For example, you are deploying an Edge host in a restaurant located in a commercial building, where the network is managed by the building and cannot be easily altered by your staff.
Environments where you expect your Edge hosts to move from one physical location to another.

Example Scenario

The Analytics team of a manufacturing company is deploying an Edge host to their assembly line to collect metrics from the manufacturing process. The building in which the Edge host is deployed has a network that is managed by a DHCP server. The region experiences a bad weather event that causes a sustained outage.

Without Overlay Network	With Overlay Network
Upon recovery, each Kubernetes component inside in the Edge host requests an IP address from the DHCP server and receives a different IP address than their original IP address before the outage happened. Since Kubernetes expects several components in the control plane to have stable IP addresses, the cluster becomes non-operational, and the assembly line is unable to resume operations.	Each Kubernetes component inside in the Edge host has a virtual IP address in the overlay network. Upon recovery, the IP addresses in the overlay network remain the same despite the IP addresses changing in the underlying DHCP network. The Edge host is able to resume its workload, and the assembly line resumes operations.

Limitations

When adding multiple Edge hosts to an existing cluster with overlay enabled, failure to add one host will block the addition of the other hosts.
If you configured static IP on any host of your cluster using the Terminal User Interface (TUI), enabling network overlay is not supported. Static IPs reduce the need to configure an overlay network, but if you must use network overlay on top of static IPs, you can configure the static IPs using the user data network block instead.

Prerequisites

At least one Edge host registered with your Palette account.
Your cluster profile must have K3s, RKE2, or PXK-E as its Kubernetes distribution.
All Edge hosts must be on the same Layer-2 network.
Broadcast messages must be allowed between all Edge hosts participating in the cluster.
- For Virtual Machine (VM) Edge hosts in VMware, this means features such as promiscuous mode must be enabled to allow broadcasts between hosts.
- Switches cannot implement features that block broadcasts between ports where Edge hosts are connected.
If your host is a virtual machine using a VMXNET3 adapter, ensure that you add the following commands in the user-data file at the boot stage. Replace <interface-name> with the name of the network interface on your Edge host.
```
stages:
  initramfs:
    - name: "Disable UDP segmentation"
      commands:
        - ethtool --offload <interface-name> tx-udp_tnl-segmentation off
        - ethtool --offload <interface-name> tx-udp_tnl-csum-segmentation off
```
This is related to a known issue with VMware's VMXNET3 adapter, which is widely used in different virtual machine management services, including VMware vSphere and Hyper-V.
If your host's physical IP address is static, ensure that you configure the IP address using the network block in your user-data file.
If your hosts are deployed in agent mode, ensure that your hosts use systemd-networkd and systemd-resolved for interface and DNS management. Refer to Configure networkd to Prepare Host for Overlay Network for more information.