Skip to content

Latest commit

 

History

History
380 lines (275 loc) · 11.4 KB

README.md

File metadata and controls

380 lines (275 loc) · 11.4 KB

Setup k8s-at-home on Ubuntu 22.04

Disclaimer

  • this is for learning first and getting something useful later
  • security and HA are not in scope for this exercise

DNS records used in this setup

  • sync.lt - all other entries are CNAME records pointing to this parent domain
  • login
  • prowlarr
  • sonarr
  • radarr
  • qbittorrent

Fork and clone

  • if you want to re-use this for you domain, fork it and replace all references to sync.lt domain to your own domain

  • clone k8s-at-home repository

    git clone https://github.com/AurimasNav/k8s-at-home.git ~/k8s-at-home

Uninstall k3s

  • during setup one might find himself wanting to start over

    /usr/local/bin/k3s-uninstall.sh

Setup k3s on hosting machine (further referred as k3s_host)

  • install k3s

    curl -sfL https://get.k3s.io | sh -s - --disable=servicelb --disable=traefik --write-kubeconfig-mode 644
    # Check for Ready node, takes ~30 seconds 
    k3s kubectl get node 

Setup tools on management host (tested on ubuntu 22.04 wsl)

  • install powershell | doc

    sudo apt-get update
    sudo apt-get install -y wget apt-transport-https software-properties-common
    wget -q "https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb"
    sudo dpkg -i packages-microsoft-prod.deb
    sudo apt-get update
    sudo apt-get install -y powershell
    • verify
    $ pwsh -version
    PowerShell 7.3.1
  • install jq

    sudo apt install jq
  • copy kubeconfig (run command from management host)

    mkdir .kube
    scp <remote_user>@<k3s_host>:/etc/rancher/k3s/k3s.yaml ~/.kube/config
  • modify permissions on the config file

    sudo chmod 600 ~/.kube/config
  • replace k3s ip in config from 127.0.0.1 to k3s_host ip

    sed -i 's/127.0.0.1/<k3s_host_ip>/' .kube/config
  • install kubectl | doc

    sudo apt-get install -y ca-certificates curl
    sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    sudo apt-get update
    sudo apt-get install -y kubectl
    • verify
    $ kubectl version -o yaml
    clientVersion:
        buildDate: "2022-12-08T19:58:30Z"
        compiler: gc
        gitCommit: b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d
        gitTreeState: clean
        gitVersion: v1.26.0
        goVersion: go1.19.4
        major: "1"
        minor: "26"
        platform: linux/amd64
    kustomizeVersion: v4.5.7
    serverVersion:
        buildDate: "2022-12-21T00:06:36Z"
        compiler: gc
        gitCommit: 48e5d2af5bfc69db051d46b6c6b83c46d15a9da5
        gitTreeState: clean
        gitVersion: v1.25.5+k3s1
        goVersion: go1.19.4
        major: "1"
        minor: "25"
        platform: linux/amd64
  • install kubectx

    echo "deb [trusted=yes] http://ftp.de.debian.org/debian buster main" | sudo tee -a /etc/apt/sources.list
    sudo apt-get update
    sudo apt install kubectx
    • verify
    $ kubectx
    default
  • verify that kubens command works

    $ kubens
    default
    kube-system
    kube-public
    kube-node-lease

Install helm

  • add apt repository and install helm

    curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
    sudo apt-get update
    sudo apt-get install helm
  • verify helm is working

    $ helm list -A -o yaml
    - app_version: v2.9.4
      chart: traefik-20.3.1+up20.3.0
      name: traefik
      namespace: kube-system
      revision: "1"
      status: deployed
      updated: 2023-01-05 19:13:06.321207336 +0000 UTC
    - app_version: v2.9.4
      chart: traefik-crd-20.3.1+up20.3.0
      name: traefik-crd
      namespace: kube-system
      revision: "1"
      status: deployedTH
    
  • install kustomize | doc

    cd /usr/local/bin
    curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | sudo bash
  • verify installation

    $ kustomize version
    {kustomize/v4.5.7  2022-08-02T16:35:54Z  }

Setup directory structure for media

  • create directories on k3s_host to map to sonarr/radarr etc.

    sudo mkdir -p /data/torrents/movies /data/torrents/tv \
    /data/media/movies /data/media/tv
  • this assumes that your storage volume is mounted to /data

Secret store setup - Doppler

Setup saas part

  • create account https://www.doppler.com/

  • create workspace - k8s-at-home

  • create project - k8s-at-home

  • from Production env navigate to ACCESS and generate service token named external-secrets - save the generated token.

  • add secrets to Production environment

Prepare external-secrets for doppler secret store

  • update doppler secret template

    cp ~/k8s-at-home/src/doppler-secret.yaml ~
    read -p "Enter doppler service token: " SERVICE_TOKEN
    
    $ Enter doppler service token: <you_service_token_here>
    
    sed -i 's/dopler-service-token/'"$(echo $SERVICE_TOKEN|base64)"/ ~/doppler-secret.yaml
  • create secret

    kubectl create namespace external-secrets
    kubectl apply -f ~/doppler-secret.yaml -n external-secrets

OAuth with google

  • Follow Google Auth Provider instructions (steps 1-7)

  • generate oauth2-proxy cookie secret

    dd if=/dev/urandom bs=32 count=1 2>/dev/null | base64 | tr -d -- '\n' | tr -- '+/' '-_'; echo
  • add secrets for OAuth to Doppler secret store

    • GOOGLE_OAUTH_CLIENT_ID: <client_id_value> (from Google Auth Provider step)
    • GOOGLE_OAUTH_CLIENT_SECRET: <client_secret_value> (from Google Auth Provider step)
    • OAUTH2_PROXY_COOKIE_SECRET: <value_from_previous_step>
    • AUTHENTICATED_EMAILS: [email protected] (list of allowed emails to login - one per line)

Install argo-cd

  • create namespace for argocd

    kubectl create namespace argocd
  • deploy resources from kustomization file

    kustomize build ~/k8s-at-home/gitops/argocd | kubectl apply -f -
  • verify that all pods are ready/running

    watch kubectl get pods -n argocd
  • ignore the errors:

    these will be taken care of automatically once argocd deploys missing dependencies

    resource mapping not found for name: "argocd" namespace: "argocd" from "STDIN": no matches for kind "ExternalSecret" in version "external-secrets.io/v1beta1"
    ensure CRDs are installed first
    Error from server (BadRequest): error when creating "STDIN": Service in version "v1" cannot be handled as a Service: strict decoding error: unknown field "spec.loadBalancerIp
    

Deploy argo-cd applications

  • modify flannel network to contain podcidr if not default

    podcidr=$(kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}')
    sed -i -r "s/\b([0-9]{1,3}\.){3}[0-9]{1,3}\/[0-9]{1,2}\b/${podcidr/\//\\/}/g" ~/k8s-at-home/gitops/flannel/patch.configmap.yaml
  • update unbound config to allow_snoop from kubernetes interal address range (allows blocky to use unbound as upstream)

    sed -i -r "s/\b([0-9]{1,3}\.){3}[0-9]{1,3}\/[0-9]{1,2}\b/${podcidr/\//\\/}/g" ~/k8s-at-home/gitops/unbound/configmap.yaml
  • commit and push changes back to remote

    git add .
    git commit -m "update kubernetes pod cidr references"
    git push
  • create root application

    kubectl apply -f ~/k8s-at-home/argocd-apps/base/root-app.yaml
  • get inital argo-cd admin password

    kubectl get secret/argocd-initial-admin-secret -n argocd -ojsonpath="{.data.password}" | base64 -d
  • login to argocd and verify you can see deployments

    kubectl port-forward service/argocd-server -n argocd 8080:80

Update argo-cd password

  • verify that external-secrets is up and running

  • encrypt your password with bcrypt (use sdk of your preferred language or do it online)

  • add generated bcrypt hash to doppler secrets

    • ARGOCD_ADMIN_PASSWORD: <your bcrypt hash>

Pi-Hole / Unbound config

  • add pihole webui password to doppler secrets
    • PIHOLE_WEBPASSWORD: <your pihole admin password>

Alternative UI for qBittorrent | Mobile friendly iQbit

  • get pvc name (ssh to k3s_host)

    VOL = $(kubectl get pvc qbittorrent-config -ojsonpath="{.spec.volumeName}")
    cd /opt/local-path-provisioner/${VOL}_qbittorrent_qbittorrent-config
    git clone https://github.com/ntoporcov/iQbit.git
  • change qbittorent setttings to use alternative webUI

    • login to qbittorent webui
    • go to options (gear icon)
    • navigate to Web UI tab
    • Use alternative Web UI
    • Files location: /config/iQbit/release