Kubernetes

Palette supports various configuration options for the EKS Kubernetes pack. These are explained in the following sections.

Configure Managed Control Plane Settings

The following sections relate to the customizable options within the managedControlPlane.* section.

Disable IAM OIDC Identity Provider

Use this parameter to disable creation of the Identity and Access Management (IAM) OpenID Connect (OIDC) identity provider.

Parameter	Description	Default
disableAssociateOIDCProvider	When set to true, this parameter prevents the creation of an IAM OIDC identity provider for the EKS cluster. Once the cluster is created, changing this field has no effect.	false

Example

disableAssociateOIDCProvider: true

Configure Logging

Use the following parameters to control the logging for the managed control plane services.

tip

You can use the Control Plane Logging preset to enable or disable all of these parameters.

Parameter	Description	Default
logging.apiServer	Toggles logging for the Kubernetes API server.	false
logging.audit	Toggles Kubernetes API audit logging.	false
logging.authenticator	Toggles cluster authenticator logging.	false
logging.controllerManager	Toggles logging for the Kubernetes controller manager.	false
logging.scheduler	Toggles logging for the Kubernetes scheduler.	false

Example

managedControlPlane:
  logging:
    apiServer: true
    audit: true
    authenticator: true
    controllerManager: true
    scheduler: false

Configure IAM Roles for Service Accounts

Use the following parameters to assign IAM roles to Kubernetes service accounts using IAM Roles for Service Accounts (IRSA).

Parameter	Description
irsaRoles.name	The name of the IAM role to be created for IRSA. This typically includes the cluster name for uniqueness.
irsaRoles.policies	A list of AWS Managed Policy Amazon Resource Names (ARNs) or custom policy ARNs that should be attached to the role.
irsaRoles.serviceAccount	Optional object specifying the service account name and namespace that the role should trust. If omitted, it defaults to the audience sts.amazonaws.com.

Example

irsaRoles:
  - name: "{{.spectro.system.cluster.name}}-irsa-cni"
    policies:
      - arn:aws:iam::123456789012:policy/MyCustomCNIForEKS
    serviceAccount:
      name: aws-node
      namespace: kube-system

  - name: "mycluster-irsa-csi"
    policies:
      - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

Configure Security Group Overrides

Use the following parameters to override the default security groups for the Amazon EKS components.

Parameter	Description
securityGroupOverrides.controlplane	The security group ID for the EKS control plane instances. If not provided, one is automatically created.
securityGroupOverrides.bastion	The security group ID for the bastion host, if configured.
securityGroupOverrides.node	The primary security group for worker nodes, either self-managed or managed.
securityGroupOverrides.node-eks-additional	Additional security group for worker nodes, if needed.
securityGroupOverrides.apiserver-lb	The security group for the API server load balancer, which is the public endpoint to your cluster.
securityGroupOverrides.lb	The security group for any other load balancers used by the cluster.

Example

securityGroupOverrides:
  controlplane: "sg-11111111"
  bastion: "sg-22222222"
  node: "sg-33333333"
  node-eks-additional: "sg-44444444"
  apiserver-lb: "sg-55555555"
  lb: "sg-66666666"

Configure OIDC Identity Provider

Use these parameters to configure an external OIDC identity provider for Amazon EKS, such as Okta or Auth0.

Parameter	Description
oidcIdentityProvider.identityProviderConfigName	The display name for the OIDC provider config in EKS.
oidcIdentityProvider.issuerUrl	The base URL of the external OIDC identity provider.
oidcIdentityProvider.clientId	The OIDC client application identifier.
oidcIdentityProvider.usernameClaim	Specifies which JWT claim is mapped to the Kubernetes username.
oidcIdentityProvider.usernamePrefix	An optional string prepended to the username. This helps avoid collision with existing names.
oidcIdentityProvider.groupsClaim	Specifies which JWT claim is mapped to the Kubernetes groups.
oidcIdentityProvider.groupsPrefix	An optional string prepended to group names.
oidcIdentityProvider.requiredClaims	A map of key-value pairs that must be present in the token. The token is only accepted if all these claim requirements are met.

Example

oidcIdentityProvider:
  identityProviderConfigName: "eks-oidc"
  issuerUrl: "https://tenant.okta.com"
  clientId: "my-oidc-client-id"
  usernameClaim: "email"
  usernamePrefix: "-"
  groupsClaim: "groups"
  groupsPrefix: "-"
  requiredClaims:
    email_verified: "true"

Configure Additional Control Plane Policies

Use this parameter to attach extra policies in ARN format to the Amazon EKS control plane role.

Parameter	Description
roleAdditionalPolicies	A list of AWS IAM Policy ARNs to attach to the control plane's IAM role in addition to the default AWS-managed ones.

Example

roleAdditionalPolicies:
  - "arn:aws:iam::123456789012:policy/MyCustomPolicy"
  - "arn:aws:iam::123456789012:policy/AnotherCustomPolicy"

Map IAM Identities to Kubernetes RBAC Groups

Use the following parameters to map IAM users and roles to Kubernetes Role-based Access Control (RBAC) groups, enabling fine-grained access control.

Parameter	Description
iamAuthenticatorConfig.mapRoles	A list that maps one or more IAM roles to Kubernetes usernames and groups.
iamAuthenticatorConfig.mapUsers	A list that maps one or more IAM users to Kubernetes usernames and groups.

Example

iamAuthenticatorConfig:
  mapRoles:
    - rolearn: arn:aws:iam::000000000000:role/KubernetesNode
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers:
    - userarn: arn:aws:iam::000000000000:user/Alice
      username: alice
      groups:
        - system:masters

Configure Managed Machine Pool Settings

The following sections relate to the customizable options within the managedMachinePool.* section.

Configure Custom IAM Role and Policies for Worker Nodes

Parameter	Description
roleName	A custom IAM role name for the worker node group. Must be a self-managed role with EKS worker policies attached. The role name cannot start with ng-role_ as this is reserved for roles generated by Spectro Cloud.
roleAdditionalPolicies	A list of additional policy ARNs to attach to the worker node group's IAM role in addition to the required EKS worker policies.

Example

managedMachinePool:
  roleName: "eks-workers-mycluster"
  roleAdditionalPolicies:
    - "arn:aws:iam::123456789012:policy/MyExtraPolicy"
    - "arn:aws:iam::123456789012:policy/AnotherPolicy"

Configure Client Configuration Settings

The following sections relate to the customizable options within the clientConfig.* section.

Configure OIDC-Based Authentication in Kubeconfig

Use the following parameters to configure OIDC-based authentication when generating a kubeconfig for CLI access and other client tools.

Parameter	Description
oidc-issuer-url	The OIDC issuer URL used in the generated kubeconfig.
oidc-client-id	The OIDC client ID. This should match the Amazon EKS OIDC identity provider client ID.
oidc-client-secret	The OIDC client secret. This is only required if the provider requires client_secret authentication.
oidc-extra-scope	The additional OAuth scopes to request. These are specific categories of user data, such as email or user profile.

Example

clientConfig:
  oidc-issuer-url: "https://tenant.okta.com"
  oidc-client-id: "my-oidc-client-id"
  oidc-client-secret: "my-oidc-client-secret"
  oidc-extra-scope: "profile,email"

Node Customization

The following sections relate to the customizable options within the nodeCustomization.* section. These options only apply when you have selected a custom Amazon Machine Image (AMI) Type for worker node pools.

You can configure both Amazon Linux 2 (AL2) and Amazon Linux 2023 (AL2023) worker nodes using the nodeCustomization section. The configuration options differ between the two AMI types, as described in the following sections.

info

The nodeCustomization section is not included by default in the Kubernetes pack manifest. You can add it to your manifest in the top-level of the YAML file. Ensure it is not nested under any other section. Examples are provided in the following sections for the Amazon Linux 2 and Amazon Linux 2023 AMI types.

Day-2 Usage Considerations

Any Day-2 additions or changes to the nodeCustomization section will invoke a worker node pool repave.

Configure Amazon Linux 2 AMI Nodes

If you have selected an AL2 AMI for one or more of your worker node groups, you can provide additional configuration and command overrides as described in the following table.

Parameter	Description
preKubeadmCommands	A list of shell commands to execute before the kubeadm command is executed. These commands are issued on each worker node before they are bootstrapped to the cluster.
postKubeadmCommands	A list of shell commands to execute after the kubeadm command is executed. These commands are issued on each worker node after they are bootstrapped to the cluster. This can be useful for additional setup or configuration tasks that need to be performed after the node joins the cluster.

Example

## EKS settings
managedControlPlane:
---
managedMachinePool:
---
nodeCustomization:
  al2:
    preKubeadmCommands:
      - |
        echo "[pre-kubeadm] ===== Setting sysctl params =====" | tee --append /var/log/kube_bootstrap.log
        sysctl --write net.bridge.bridge-nf-call-iptables=1
        sysctl --write net.ipv4.ip_forward=1
    postKubeadmCommands:
      - |
        echo "[post-kubeadm] ===== Pulling troubleshooting tools =====" | tee --append /var/log/kube_bootstrap.log
        crictl pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
        echo "[post-kubeadm] Done." | tee --append /var/log/kube_bootstrap.log

Configure Amazon Linux 2023 AMI Nodes

If you have selected an AL2023 AMI for one or more of your worker node groups, you can provide additional configuration and command overrides as described in the following table.

Parameter	Type	Description
userDataScript	String (block scalar)	A custom multi-line Bash script that is executed as the root user on every worker node after all nodeadm operations are complete. You can use it to execute any extra bootstrap logic, such as install additional packages, pre-pull images, configure HTTP/HTTPS proxies, or inject Certificate Authority (CA) certificates into the system trust store.

Example of proxy configuration and self-signed CA certificate injection

The following example configures HTTP/HTTPS proxy settings and installs a custom CA certificate, which is useful for private registries or internal services secured with non-public CA certificates.

Example command to create a self-signed CA certificate that expires in one year

openssl req -x509 -newkey rsa:4096 -nodes -keyout my-ca.key \
  -sha256 -days 365 -out my-ca.crt \
  -subj "/C=GB/O=Example Ltd/CN=Example Test Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -addext "subjectKeyIdentifier=hash" \
  -addext "authorityKeyIdentifier=keyid:always" \
  -set_serial 0x01

Example userDataScript for AL2023 nodes

## EKS settings
managedControlPlane:
---
managedMachinePool:
---
nodeCustomization:
  al2023:
    userDataScript: |
      #!/bin/bash
      set -euo pipefail

      # Configure HTTP/HTTPS proxy if needed
      cat > /etc/profile.d/http-proxy.sh << 'EOF'
      export HTTP_PROXY="http://proxy.example.com:3128"
      export HTTPS_PROXY="http://proxy.example.com:3128"
      export NO_PROXY="localhost,127.0.0.1,10.11.12.13,.internal"
      EOF

      # Inject a custom CA certificate
      cat <<EOF > /etc/pki/ca-trust/source/anchors/my-ca.crt
      -----BEGIN CERTIFICATE-----
      ...PEM-encoded root/intermediate CA...
      -----END CERTIFICATE-----
      EOF

      # Make sure file permissions are correct
      chmod 0644 /etc/pki/ca-trust/source/anchors/my-ca.crt

      # Ensure the CA trust bundle is updated system-wide
      update-ca-trust extract

      # Verify the custom CA certificate
      openssl verify -CAfile /etc/ssl/certs/ca-bundle.crt /etc/pki/ca-trust/source/anchors/my-ca.crt

      # Expected output
      # /etc/pki/ca-trust/source/anchors/my-ca.crt: OK

      echo "[user-data] Custom CA installed and system trust store updated."

Example of containerd mirror registry configuration

The following example configures containerd to use an internal registry mirror for upstream container images. This is useful for environments with restricted internet access or when you want to cache images locally. In this example, the internal mirror registry supports authentication headers and skipping TLS verification.

Example userDataScript for AL2023 nodes

## EKS settings
managedControlPlane:
---
managedMachinePool:
---
nodeCustomization:
  al2023:
    userDataScript: |
      #!/bin/bash
      set -o errexit
      set -o pipefail
      set -o nounset

      echo "Creating containerd mirror configuration directory"
      mkdir -p /etc/containerd/certs.d

      # Your internal registry/gateway that fronts upstream registries.
      INTERNAL_REGISTRY="my-subdomain.company.com/my-registry"
      # Access credentials for the internal registry
      AUTH_HEADER="Basic ...base64-encoded-credentials..."

      # Upstream registries you want to mirror via the internal gateway
      REGISTRIES=(
        "docker.io"
        "gcr.io"
        "ghcr.io"
        "k8s.gcr.io"
        "registry.k8s.io"
        "quay.io"
        "us-docker.pkg.dev"
        "us-east1-docker.pkg.dev"
      )

      # For each upstream registry listed, write
      # /etc/containerd/certs.d/<registry>/hosts.toml
      # that redirects pulls and resolves to https://${INTERNAL_REGISTRY}/<registry>
      # using the provided Authorization header (AUTH_HEADER). This ensures all
      # containerd traffic goes through your internal registry/gateway.
      for REGISTRY in "${REGISTRIES[@]}"; do
        MIRROR_PATH="/etc/containerd/certs.d/${REGISTRY}"
        echo "Configuring mirror for ${REGISTRY}"
        mkdir -p "${MIRROR_PATH}"
        cat <<EOF > "${MIRROR_PATH}/hosts.toml"
        server = "https://${REGISTRY}"
        [host."https://${INTERNAL_REGISTRY}/${REGISTRY}"]
          capabilities = ["pull", "resolve"]
          skip_verify = true
        [host."https://${INTERNAL_REGISTRY}/${REGISTRY}".header]
          Authorization = ["${AUTH_HEADER}"]
        EOF
      done

      echo "Restarting containerd to apply mirror configuration"
      systemctl restart containerd

      echo "Containerd mirror configuration complete."

Node Customization Layout for AL2 and AL2023 Combined

## EKS settings
managedControlPlane:
...
managedMachinePool:
...
nodeCustomization:
  al2:
    preKubeadmCommands:
      - |
        ...
    postKubeadmCommands:
      - |
        ...
  al2023:
    userDataScript: |
    ...

Required IAM Permissions for Configuration

You must ensure the AWS IAM user or role performing these actions has sufficient privileges. The following table is an overview of the permissions required for each configuration section.

Configuration Section	IAM Permissions Required
Disable IAM OIDC Identity Provider	iam:CreateOpenIDConnectProvider iam:DeleteOpenIDConnectProvider iam:GetOpenIDConnectProvider iam:TagOpenIDConnectProvider
Configure Logging	logs:CreateLogGroup logs:CreateLogStream logs:PutLogEvents eks:UpdateClusterConfig
Configure IAM Roles for Service Accounts	iam:CreateRole iam:AttachRolePolicy iam:PassRole iam:GetRole iam:ListRoles
Configure Security Group Overrides	ec2:DescribeSecurityGroups ec2:CreateSecurityGroup ec2:AuthorizeSecurityGroupIngress ec2:AuthorizeSecurityGroupEgress ec2:DeleteSecurityGroup
Configure OIDC Identity Provider	eks:AssociateIdentityProviderConfig eks:DisassociateIdentityProviderConfig eks:DescribeIdentityProviderConfig iam:CreateOpenIDConnectProvider iam:DeleteOpenIDConnectProvider iam:TagOpenIDConnectProvider
Configure Additional Control Plane Policies	iam:AttachRolePolicy iam:DetachRolePolicy iam:PassRole
Map IAM Identities to Kubernetes RBAC Groups	Depending on the cluster authentication mode, EKS API mode may require eks:DescribeCluster and eks:UpdateClusterConfig.
Configure Custom IAM Role and Policies for Worker Nodes	iam:CreateRole iam:AttachRolePolicy iam:PassRole iam:GetRole iam:ListRoles
Configure OIDC-Based Authentication in Kubeconfig	eks:DescribeCluster

Core IAM Policies Coverage

If using the Core IAM Policies, including the Controllers EKS Policy, you may need the following additional IAM permissions.

Configuration Section	Missing Permissions
Configure Logging	logs:CreateLogGroup logs:CreateLogStream logs:PutLogEvents
Configure IAM Roles for Service Accounts	iam:ListRoles
Configure Security Group Overrides	ec2:AuthorizeSecurityGroupEgress
Configure Custom IAM Role and Policies for Worker Nodes	iam:ListRoles

Minimum Permissions Coverage

If using Minimum Permissions (dynamic or static), including the Controllers EKS Policy, you may need the following additional IAM permissions.

Configuration Section	Remaining Missing Permissions
Disable IAM OIDC Identity Provider	iam:GetOpenIDConnectProvider
Configure Logging	logs:CreateLogGroup logs:CreateLogStream logs:PutLogEvents
Configure IAM Roles for Service Accounts	iam:ListRoles
Configure Security Group Overrides	ec2:AuthorizeSecurityGroupEgress
Configure Custom IAM Role and Policies for Worker Nodes	iam:ListRoles

Terraform

You can retrieve details about the Kubernetes pack for AWS EKS by using the following Terraform code.

data "spectrocloud_registry" "public_registry" {
  name = "Public Repo"
}

data "spectrocloud_pack_simple" "k8s" {
  name    = "kubernetes-eks"
  version = "1.29"
  type = "helm"
  registry_uid = data.spectrocloud_registry.public_registry.id
}