[PATCH] - Cloud-Init temporary fix

[K-MeLeOn]

Hey,

After several creations, I saw the recent problem with cloud-init.
For several months I've been waiting for the patch to be merged, but it's still pending and I'm delivering cloud-init 23.3-x instead of 24.4.1 or a patched version.

In fact a bug appeared when resizing btrfs with the python script on the cloud-init side when the kernel is higher than a specific version.
The problem:

2025-01-21 09:46:00,224 - util.py[WARNING]: Running module resizefs (<module 'cloudinit.config.cc_resizefs' from '/usr/lib/python3.11/site-packages/cloudinit/config/cc_resizefs.py'>) failed

It's due to cloud-init < 24.4.1 iwith kernel >= 6.10.

To summarize, the problem comes from the btrfs --version command, which returns more data than the previous version and breaks the python script on cloud-init side.

If it's of any help to someone who needs it, I've solved the problem by creating a temporary patch:

• Create a substitution btrfs file that adapts the --version command only when it comes from cloud-init.
• Modify the shell PATH directly for the current shell session and execute cloud-init correctly.

./locals.tf

# Error introduced by btrfs update
# cloud-init < 24.4 && btrfs >= 6.10
cloudinit_write_btrfs_patch_files = <<EOT
# Script to patch btrfs to work with cloud-init < 24.4 && btrfs >= 6.10
- path: /usr/local/bin/btrfs
  permissions: "0755"
  content: |
    #!/bin/bash
    if [[ $1 == "--version" || $1 == "version" ]]; then
        if [[ $(ps -o comm= $PPID) == "cloud-init" ]]; then
            /usr/sbin/btrfs --version | head -n 1 | awk '{print $2}'
        else
            /sbin/btrfs --version
        fi
    else
        /sbin/btrfs "$@"
    fi
EOT

# Error introduced by btrfs update
# Fix path for btrfs substitution
# cloud-init < 24.4 && btrfs >= 6.10
cloudinit_runcmd_btrfs_patch = <<EOT
- ['export', 'PATH=/usr/local/bin:/usr/sbin:/usr/local/sbin:/sbin:/root/bin:/usr/bin:/bin']
EOT

./control_planes.tf

• Add the two vars

module "control_planes" {
  source = "./modules/host"

  providers = {
    hcloud = hcloud,
  }

  for_each = local.control_plane_nodes

  name                               = "${var.use_cluster_name_in_node_name ? "${var.cluster_name}-" : ""}${each.value.nodepool_name}"
  microos_snapshot_id                = substr(each.value.server_type, 0, 3) == "cax" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id
  base_domain                        = var.base_domain
  ssh_keys                           = length(var.ssh_hcloud_key_label) > 0 ? concat([local.hcloud_ssh_key_id], data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.id) : [local.hcloud_ssh_key_id]
  ssh_port                           = var.ssh_port
  ssh_public_key                     = var.ssh_public_key
  ssh_private_key                    = var.ssh_private_key
  ssh_additional_public_keys         = length(var.ssh_hcloud_key_label) > 0 ? concat(var.ssh_additional_public_keys, data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.public_key) : var.ssh_additional_public_keys
  firewall_ids                       = [hcloud_firewall.k3s.id]
  placement_group_id                 = var.placement_group_disable ? null : (each.value.placement_group == null ? hcloud_placement_group.control_plane[each.value.placement_group_compat_idx].id : hcloud_placement_group.control_plane_named[each.value.placement_group].id)
  location                           = each.value.location
  server_type                        = each.value.server_type
  backups                            = each.value.backups
  ipv4_subnet_id                     = hcloud_network_subnet.control_plane[[for i, v in var.control_plane_nodepools : i if v.name == each.value.nodepool_name][0]].id
  dns_servers                        = var.dns_servers
  k3s_registries                     = var.k3s_registries
  k3s_registries_update_script       = local.k3s_registries_update_script
  cloudinit_write_btrfs_patch_files  = local.cloudinit_write_btrfs_patch_files
  cloudinit_write_files_common       = local.cloudinit_write_files_common
  cloudinit_runcmd_btrfs_patch       = local.cloudinit_runcmd_btrfs_patch
  cloudinit_runcmd_common            = local.cloudinit_runcmd_common
  swap_size                          = each.value.swap_size
  zram_size                          = each.value.zram_size
  keep_disk_size                     = var.keep_disk_cp

  # We leave some room so 100 eventual Hetzner LBs that can be created perfectly safely
  # It leaves the subnet with 254 x 254 - 100 = 64416 IPs to use, so probably enough.
  private_ipv4 = cidrhost(hcloud_network_subnet.control_plane[[for i, v in var.control_plane_nodepools : i if v.name == each.value.nodepool_name][0]].ip_range, each.value.index + 101)

  labels = merge(local.labels, local.labels_control_plane_node)

  automatically_upgrade_os = var.automatically_upgrade_os

  depends_on = [
    hcloud_network_subnet.control_plane,
    hcloud_placement_group.control_plane,
  ]
}

./agents.tf

module "agents" {
  source = "./modules/host"

  providers = {
    hcloud = hcloud,
  }

  for_each = local.agent_nodes

  name                               = "${var.use_cluster_name_in_node_name ? "${var.cluster_name}-" : ""}${each.value.nodepool_name}${try(each.value.node_name_suffix, "")}"
  microos_snapshot_id                = substr(each.value.server_type, 0, 3) == "cax" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id
  base_domain                        = var.base_domain
  ssh_keys                           = length(var.ssh_hcloud_key_label) > 0 ? concat([local.hcloud_ssh_key_id], )data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.id) : [local.hcloud_ssh_key_id]
  ssh_port                           = var.ssh_port
  ssh_public_key                     = var.ssh_public_key
  ssh_private_key                    = var.ssh_private_key
  ssh_additional_public_keys         = length(var.ssh_hcloud_key_label) > 0 ? concat(var.ssh_additional_public_keys, data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.public_key) : var.ssh_additional_public_keys
  firewall_ids                       = [hcloud_firewall.k3s.id]
  placement_group_id                 = var.placement_group_disable ? null : (each.value.placement_group == null ? hcloud_placement_group.agent[each.value.placement_group_compat_idx].id : hcloud_placement_group.agent_named[each.value.placement_group].id)
  location                           = each.value.location
  server_type                        = each.value.server_type
  backups                            = each.value.backups
  ipv4_subnet_id                     = hcloud_network_subnet.agent[[for i, v in var.agent_nodepools : i if v.name == each.value.nodepool_name][0]].id
  dns_servers                        = var.dns_servers
  k3s_registries                     = var.k3s_registries
  k3s_registries_update_script       = local.k3s_registries_update_script
  cloudinit_write_btrfs_patch_files  = local.cloudinit_write_btrfs_patch_files
  cloudinit_write_files_common       = local.cloudinit_write_files_common
  cloudinit_runcmd_btrfs_patch       = local.cloudinit_runcmd_btrfs_patch
  cloudinit_runcmd_common            = local.cloudinit_runcmd_common
  swap_size                          = each.value.swap_size
  zram_size                          = each.value.zram_size
  keep_disk_size                     = var.keep_disk_agents

  private_ipv4 = cidrhost(hcloud_network_subnet.agent[[for i, v in var.agent_nodepools : i if v.name == each.value.nodepool_name][0]].ip_range, each.value.index + 101)

  labels = merge(local.labels, local.labels_agent_node)

  automatically_upgrade_os = var.automatically_upgrade_os

  depends_on = [
    hcloud_network_subnet.agent,
    hcloud_placement_group.agent
  ]
}

./autoscaler-agents.tf

data "cloudinit_config" "autoscaler_config" {
  count = length(var.autoscaler_nodepools)

  gzip          = true
  base64_encode = true

  # Main cloud-config configuration file.
  part {
    filename     = "init.cfg"
    content_type = "text/cloud-config"
    content = templatefile(
      "${path.module}/templates/autoscaler-cloudinit.yaml.tpl",
      {
        hostname          = "autoscaler"
        sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)
        k3s_config = yamlencode({
          server        = "https://${var.use_control_plane_lb ? hcloud_load_balancer_network.control_plane.*.ip[0] : module.control_planes[keys(module.control_planes)[0]].private_ipv4_address}:6443"
          token         = local.k3s_token
          kubelet-arg   = concat(local.kubelet_arg, var.k3s_global_kubelet_args, var.k3s_autoscaler_kubelet_args, var.autoscaler_nodepools[count.index].kubelet_args)
          flannel-iface = local.flannel_iface
          node-label    = concat(local.default_agent_labels, [for k, v in var.autoscaler_nodepools[count.index].labels : "${k}=${v}"])
          node-taint    = concat(local.default_agent_taints, [for taint in var.autoscaler_nodepools[count.index].taints : "${taint.key}=${taint.value}:${taint.effect}"])
          selinux       = true
        })
        install_k3s_agent_script           = join("\n", concat(local.install_k3s_agent, ["systemctl start k3s-agent"]))
        cloudinit_write_btrfs_patch_files  = local.cloudinit_write_btrfs_patch_files
        cloudinit_write_files_common       = local.cloudinit_write_files_common
        cloudinit_runcmd_btrfs_patch       = local.cloudinit_runcmd_btrfs_patch
        cloudinit_runcmd_common            = local.cloudinit_runcmd_common
      }
    )
  }
}

data "cloudinit_config" "autoscaler_legacy_config" {
  count = length(var.autoscaler_nodepools) > 0 && local.isUsingLegacyConfig ? 1 : 0

  gzip          = true
  base64_encode = true

  # Main cloud-config configuration file.
  part {
    filename     = "init.cfg"
    content_type = "text/cloud-config"
    content = templatefile(
      "${path.module}/templates/autoscaler-cloudinit.yaml.tpl",
      {
        hostname          = "autoscaler"
        sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)
        k3s_config = yamlencode({
          server        = "https://${var.use_control_plane_lb ? hcloud_load_balancer_network.control_plane.*.ip[0] : module.control_planes[keys(module.control_planes)[0]].private_ipv4_address}:6443"
          token         = local.k3s_token
          kubelet-arg   = local.kubelet_arg
          flannel-iface = local.flannel_iface
          node-label    = concat(local.default_agent_labels, var.autoscaler_labels)
          node-taint    = concat(local.default_agent_taints, var.autoscaler_taints)
          selinux       = true
        })
        install_k3s_agent_script           = join("\n", concat(local.install_k3s_agent, ["systemctl start k3s-agent"]))
        cloudinit_write_btrfs_patch_files  = local.cloudinit_write_btrfs_patch_files
        cloudinit_write_files_common       = local.cloudinit_write_files_common
        cloudinit_runcmd_btrfs_patch       = local.cloudinit_runcmd_btrfs_patch
        cloudinit_runcmd_common            = local.cloudinit_runcmd_common
      }
    )
  }
}

./modules/host/variables.tf

variable "cloudinit_write_btrfs_patch_files" {
  default = ""
  type    = string
}

variable "cloudinit_runcmd_btrfs_patch" {
  default = ""
  type    = string
}

./modules/host/main.tf

• Add the two vars

data "cloudinit_config" "config" {
  gzip          = true
  base64_encode = true

  # Main cloud-config configuration file.
  part {
    filename     = "init.cfg"
    content_type = "text/cloud-config"
    content = templatefile(
      "${path.module}/templates/cloudinit.yaml.tpl",
      {
        hostname                           = local.name
        sshAuthorizedKeys                  = concat([var.ssh_public_key], var.ssh_additional_public_keys)
        cloudinit_write_btrfs_patch_files  = var.cloudinit_write_btrfs_patch_files
        cloudinit_write_files_common       = var.cloudinit_write_files_common
        cloudinit_runcmd_btrfs_patch       = var.cloudinit_runcmd_btrfs_patch
        cloudinit_runcmd_common            = var.cloudinit_runcmd_common
        swap_size                          = var.swap_size
      }
    )
  }
}

./modules/host/templates/cloud-init.yaml.tpl

• Add the two vars

#cloud-config

debug: True

write_files:

${cloudinit_write_btrfs_patch_files}

${cloudinit_write_files_common}

# Add ssh authorized keys
ssh_authorized_keys:
%{ for key in sshAuthorizedKeys ~}
  - ${key}
%{ endfor ~}

# Resize /var, not /, as that's the last partition in MicroOS image.
growpart:
    devices: ["/var"]

# Make sure the hostname is set correctly
hostname: ${hostname}
preserve_hostname: true

runcmd:

${cloudinit_runcmd_btrfs_patch}

${cloudinit_runcmd_common}

%{if swap_size != ""~}
- |
  btrfs subvolume create /var/lib/swap
  chmod 700 /var/lib/swap
  truncate -s 0 /var/lib/swap/swapfile
  chattr +C /var/lib/swap/swapfile
  fallocate -l ${swap_size} /var/lib/swap/swapfile
  chmod 600 /var/lib/swap/swapfile
  mkswap /var/lib/swap/swapfile
  swapon /var/lib/swap/swapfile
  echo "/var/lib/swap/swapfile none swap defaults 0 0" | sudo tee -a /etc/fstab
  cat << EOF >> /etc/systemd/system/swapon-late.service
  [Unit]
  Description=Activate all swap devices later
  After=default.target

  [Service]
  Type=oneshot
  ExecStart=/sbin/swapon -a

  [Install]
  WantedBy=default.target
  EOF
  systemctl daemon-reload
  systemctl enable swapon-late.service
%{endif~}

./templates/autoscaler-cloudinit.yaml.tpl

• Add the two vars

#cloud-config

debug: True

write_files:

${cloudinit_write_btrfs_patch_files}

${cloudinit_write_files_common}

- content: ${base64encode(k3s_config)}
  encoding: base64
  path: /tmp/config.yaml

- content: ${base64encode(install_k3s_agent_script)}
  encoding: base64
  path: /var/pre_install/install-k3s-agent.sh

# Add ssh authorized keys
ssh_authorized_keys:
%{ for key in sshAuthorizedKeys ~}
  - ${key}
%{ endfor ~}

# Resize /var, not /, as that's the last partition in MicroOS image.
growpart:
    devices: ["/var"]

# Make sure the hostname is set correctly
hostname: ${hostname}
preserve_hostname: true

runcmd:

${cloudinit_runcmd_btrfs_patch}

${cloudinit_runcmd_common}

# Start the install-k3s-agent service
- ['/bin/bash', '/var/pre_install/install-k3s-agent.sh']

Now don't forget to specify your good dev folder in the kube.tf file :

 # * source can be specified in multiple ways:
  # 1. For normal use, (the official version published on the Terraform Registry), use
  # source = "kube-hetzner/kube-hetzner/hcloud"
  #    When using the terraform registry as source, you can optionally specify a version number.
  #    See https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud for the available versions
  # version = "2.15.3"
  # 2. For local dev, path to the git repo
  source = "./kube-hetzner-dev"
  # 3. If you want to use the latest master branch (see https://developer.hashicorp.com/terraform/language/modules/sources#github), use
  # source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner"

Execute the classic command loop:

terraform init --upgrade

terraform validate

terraform apply -auto-approve

Enjoy 👍