Skip to content

Homelab: Commiting my secrets to git

I spent some time tonight setting configuring some services on my home kubernetes cluster. See this post for more details on how I set up the cluster. So far it's been a fun experiment to see if I can avoid anything spontanously catching file. At work there is a full team of experts dedicated to keep our cluster running smoothly. At home, there is... me.

Today I managed to get FreshRSS and Atuin Sync running.

I've been using Cursor as a guide generating the yaml files and asking questions about how Kubernetes works. I think I am a decent user of Kubernetes clusters, but a rank novice as an operator of a cluster.

FreshRSS

I want to try to get away from doomscrolling, and being caught in some algorithmically generated news feed. I'll try FreshRSS for a while at least.

To get started I asked Cursor to generate a deployment, giving it a link to the FreshRSS documentation.

I had to go back and forth a few times to understand how to get a URL to resolve on my home network. The kubernetes cluster is running on the host tyr, so I can ping that from my home network on tyr.local.

Initially I wanted to host FreshRSS at rss.tyr.local, but I didn't figure out how to do that. Instead I hosted it at tyr.local/rss and then added Middleware to strip the /rss path before sending the traffic to the Service.

Complete manifest 
---
# deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: freshrss
namespace: freshrss
labels:
    app: freshrss
spec:
replicas: 1
selector:
    matchLabels:
    app: freshrss
template:
    metadata:
    labels:
        app: freshrss
    spec:
    containers:
        - name: freshrss
        image: freshrss/freshrss:latest
        ports:
            - containerPort: 80
        env:
            - name: TZ
            value: "Europe/Copenhagen"
            - name: CRON_MIN
            value: "13,43"
        volumeMounts:
            - name: data
            mountPath: /var/www/FreshRSS/data
            - name: extensions
            mountPath: /var/www/FreshRSS/extensions
        resources:
            requests:
            memory: "128Mi"
            cpu: "100m"
            limits:
            memory: "256Mi"
            cpu: "500m"
    volumes:
        - name: data
        persistentVolumeClaim:
            claimName: freshrss-data
        - name: extensions
        persistentVolumeClaim:
            claimName: freshrss-extensions

---
# ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: freshrss
namespace: freshrss
annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    traefik.ingress.kubernetes.io/router.middlewares: "freshrss-strip-prefix@kubernetescrd"
    traefik.ingress.kubernetes.io/router.tls: "false"
spec:
rules:
    - host: tyr.local
    http:
        paths:
        - path: /rss
            pathType: Prefix
            backend:
            service:
                name: freshrss
                port:
                number: 80

---
# middleware.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: strip-prefix
namespace: freshrss
spec:
stripPrefix:
    prefixes:
    - /rss
    forceSlash: true

---
# namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
name: freshrss

---
# pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: freshrss-data
namespace: freshrss
spec:
accessModes:
    - ReadWriteOnce
resources:
    requests:
    storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: freshrss-extensions
namespace: freshrss
spec:
accessModes:
    - ReadWriteOnce
resources:
    requests:
    storage: 100Mi

---
# service.yaml

apiVersion: v1
kind: Service
metadata:
name: freshrss
namespace: freshrss
spec:
selector:
    app: freshrss
ports:
    - port: 80
    targetPort: 80
type: ClusterIP

Atuin

Again, I just asked Cursor to generate the Kubernetes manifests after linking to the self-hosting docs.

One issue I had here, was that I want to commit the manifests to git, but Atuin requires a password to communicate with the local Postgres database.

secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: atuin-secrets
type: Opaque
stringData:
  ATUIN_DB_USERNAME: atuin
  ATUIN_DB_PASSWORD: seriously-insecure
  ATUIN_HOST: "127.0.0.1"
  ATUIN_PORT: "8888"
  ATUIN_OPEN_REGISTRATION: "true"
  ATUIN_DB_URI: "postgres://atuin:seriously-insecure@postgres/atuin"
immutable: true

Since both services are local and not exposed to the internet, there is probably no harm in commiting something like this.

But I had a discussion about this problem with my smart, very kubernetes savvy friend, Lasse. So I wanted to find some solution to this that did not commit any deadly sins, such as committing a secret in plaintext.

Again, I asked Cursor for suggestions, prompting it to not make any changes to my files, just list out options.

Here is the full list of suggestions:

List some ways I can add secrets here without leaking it in git

  1. Using Sealed Secrets:

    # Install sealed-secrets controller
helm install sealed-secrets sealed-secrets/sealed-secrets
# Encrypt your secrets
kubeseal < atuin/secrets.yaml > atuin/sealed-secrets.yaml

    This lets you safely commit encrypted secrets that only your cluster can decrypt.

  2. Using External Secrets Operator with a secrets manager:

    apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: atuin-secrets
namespace: atuin
spec:
refreshInterval: 1h
secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
target:
    name: atuin-secrets
data:
    - secretKey: ATUIN_DB_PASSWORD
    remoteRef:
        key: atuin/db-password

    This fetches secrets from services like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.

  3. Using Kustomize with secret generators: 

    # kustomization.yaml
secretGenerator:
- name: atuin-secrets
namespace: atuin
envs:
- .env # This file is not committed
    This generates secrets from local files that aren't committed.

  4. Using environment variables with a CI/CD pipeline: 

    apiVersion: v1
kind: Secret
metadata:
name: atuin-secrets
namespace: atuin
stringData:
ATUIN_DB_PASSWORD: ${ATUIN_DB_PASSWORD}
ATUIN_DB_URI: "postgres://atuin:${ATUIN_DB_PASSWORD}@postgres/atuin"
    Your CI/CD system (like GitHub Actions) replaces the variables before applying.

  5. Using SOPS (Secrets OPerationS): 

    # Encrypt secrets
sops -e -i atuin/secrets.yaml

# Decrypt when needed
sops -d atuin/secrets.yaml | kubectl apply -f -
    This encrypts secrets with age, PGP, or cloud KMS keys.

I chose to use Sealed Secrets. It sounded by far the simplest, and I could sort-of understand what was going on:

  • I install the SealedSecret helm chart on my cluster. This can encrypt and decrypt text using some fancy crypto magic. Basically a private/public key pair is generated and stored as Kubernetes secerets and these are used to encrypt my secrets.
  • I install kubeseal on my local machine. It can communicate with the SealedSecret controller running in the cluster.
  • I pass a local (uncommitted) kubernetes Secret to kubeseal, it encrypts it and I get back a SealedSecret.
  • I can then store, apply and commit this SealedSecret. It will get unsealed when applied to my cluster (so my services can use it), but the unsealing only happens inside the cluster. My local manifest file is encrypted.

Let's say I want to encrypt this Secret

secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: atuin-secrets
  namespace: atuin
type: Opaque
stringData:
  ATUIN_DB_USERNAME: atuin
  ATUIN_DB_PASSWORD: "123"
  ATUIN_DB_URI: "postgres://username:123@postgres/atuin" # Match the password here

I can run kubeseal to encrypt: 

kubeseal < secrets.yaml > sealed-secrets.yaml

and I get back

sealed-secrets.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: atuin-secrets
  namespace: atuin
spec:
  encryptedData:
    ATUIN_DB_PASSWORD: AgBKfphBarMiNX8CIsvjAXqEtRp/Bq+a4y67k/M6bxMm1w/[TRUNCATED FOR SPACE]
    ATUIN_DB_URI: AgCfm2AisGVBlMrOqPvMWOor0e0UXDruZnWVG3klrfSzbtZfrzYF4x[TRUNCATED FOR SPACE]
    ATUIN_DB_USERNAME: AgAt8yDkKRjmvJtB4ecxOOcuEm1Zcoa8pX1UvtvwAAT4M18PN3JK[TRUNCATED FOR SPACE]
  template:
    metadata:
      creationTimestamp: null
      name: atuin-secrets
      namespace: atuin
    type: Opaque

Pretty cool! I have also backed up the Sealed Secrets private key in my 1Password. 

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-master.key

If my cluster suddenly catches fire, I can recreate my deployments in a new cluster by adding the key to that cluster 

kubectl apply -f sealed-secrets-master.key
kubectl delete pod -n kube-system -l name=sealed-secrets-controller

Here is the complete manifest

Complete manifest 
# config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: atuin-config
namespace: atuin
data:
ATUIN_HOST: "0.0.0.0"
ATUIN_PORT: "8888"
ATUIN_OPEN_REGISTRATION: "true"

---
# deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: atuin
spec:
replicas: 1
strategy:
    type: Recreate # Prevent data corruption by ensuring only one pod runs
selector:
    matchLabels:
    app: postgres
template:
    metadata:
    labels:
        app: postgres
    spec:
    containers:
        - name: postgresql
        image: postgres:14
        ports:
            - containerPort: 5432
        env:
            - name: POSTGRES_DB
            value: atuin
            - name: POSTGRES_PASSWORD
            valueFrom:
                secretKeyRef:
                name: atuin-secrets
                key: ATUIN_DB_PASSWORD
            - name: POSTGRES_USER
            valueFrom:
                secretKeyRef:
                name: atuin-secrets
                key: ATUIN_DB_USERNAME
        lifecycle:
            preStop:
            exec:
                command:
                [
                    "/usr/local/bin/pg_ctl",
                    "stop",
                    "-D",
                    "/var/lib/postgresql/data",
                    "-w",
                    "-t",
                    "60",
                    "-m",
                    "fast",
                ]
        resources:
            requests:
            cpu: 100m
            memory: 100Mi
            limits:
            cpu: 250m
            memory: 600Mi
        volumeMounts:
            - mountPath: /var/lib/postgresql/data/
            name: database
    volumes:
        - name: database
        persistentVolumeClaim:
            claimName: database
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: atuin
namespace: atuin
spec:
replicas: 1
selector:
    matchLabels:
    app: atuin
template:
    metadata:
    labels:
        app: atuin
    spec:
    containers:
        - name: atuin
        image: ghcr.io/atuinsh/atuin:v18.4.0 # Using a specific version as recommended
        args:
            - server
            - start
        env:
            - name: ATUIN_DB_URI
            valueFrom:
                secretKeyRef:
                name: atuin-secrets
                key: ATUIN_DB_URI
            - name: ATUIN_HOST
            valueFrom:
                configMapKeyRef:
                name: atuin-config
                key: ATUIN_HOST
            - name: ATUIN_PORT
            valueFrom:
                configMapKeyRef:
                name: atuin-config
                key: ATUIN_PORT
            - name: ATUIN_OPEN_REGISTRATION
            valueFrom:
                configMapKeyRef:
                name: atuin-config
                key: ATUIN_OPEN_REGISTRATION
        ports:
            - containerPort: 8888
        resources:
            limits:
            cpu: 250m
            memory: 1Gi
            requests:
            cpu: 250m
            memory: 1Gi
        volumeMounts:
            - mountPath: /config
            name: atuin-config
    volumes:
        - name: atuin-config
        persistentVolumeClaim:
            claimName: atuin-config

---
# ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: atuin
namespace: atuin
annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    traefik.ingress.kubernetes.io/router.middlewares: "atuin-strip-prefix@kubernetescrd"
spec:
rules:
    - host: tyr.local
    http:
        paths:
        - path: /atuin
            pathType: Prefix
            backend:
            service:
                name: atuin
                port:
                number: 8888

---
# middleware.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: strip-prefix
namespace: atuin
spec:
stripPrefix:
    prefixes:
    - /atuin
    forceSlash: true

---
# namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
name: atuin

---
# sealed-secrets.yaml

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: atuin-secrets
namespace: atuin
spec:
encryptedData:
    ATUIN_DB_PASSWORD: AgBKfphBarMiNX8CIsvjAXqEtRp/Bq+a4y67k/M6bxMm1w/fJUERNqBKaPWqaABfHR4WEk9ePj4CWcVbHb2xVCviX4zYE4pZ9onMvzRGJa2UUl1qRsJGN/ooMRJux+ztfSXJfRzzZxt1QjBlJOmMxG0XjKu0TdahXnI4BMJ2rrBPPmWx9sr4z8YxG8BU/TL8DiJGiD2DtarQWmqSogueGpsOE/9hdeWvW4E7RNlcd7JJ0Hv/nELlhVIUB9fzGoaioDJO6qodYBWNtt2ckyNp3KwoOKXddwRV5tq1ggPKnZOqlHpDgmTaYAFNPXVGIpMNxzUfs+CU0VdT60hx5e3qMbVD86NrnqmbQ38GYc/A7TDrWImSEPjkweLPSTgK5YuQEHJBGYDy9jNNVTMHwfcXkAZkD8swu8+2Whw6No1D2WO2LwewVdTDOynjVhekGk3UF6B2lqIn9TowkIBbZZ6mYYK4VzXRCRXmo2ZiEqDMQK78ejUHdK5m43cZ9M+BEmE3lKzAmgZt+xons/xcisI63pff31urXWZsFylZvnVUnR/l0cp5jmr8KDnMp1WDPf+UyhSlxVvnfAKRyXIGi6jpMQluXVvx/waX4MdqgJMfyn3cQ6tFH4YiZCX6kdNNWjJp5lYxmhRdqWRznCB1vxuWIfXCc9eUT8Kz0Houmw/S8HR11ApNoxopbalC23wdTa9ZXlJdC4bXElfdC8HHwjTcNezDN9mc+4e+WdaKkbuYZljP
    ATUIN_DB_URI: AgCfm2AisGVBlMrOqPvMWOor0e0UXDruZnWVG3klrfSzbtZfrzYF4x+sY7fVLsfUY3RSRF84m13hIJPBxhiO3pFPAs6e6zm5GH7B+8Iem1ijIXWNVW5oc7h/Kas77k1h+TcJTVyZ4gL52oqzZM3cwAX0UdE/enNrvYWoeTsJ0UMbNw3bKZ9Ll0BPfdirdHT8Ve7jMzaDF+d11difPOhyZ7wgK3ykzOGu9G8LbzJ8IwUYYFK/1DETYU76XC/d79tUOwSYxGwf88/r2zjn9ZFA7rnzzEnV7ECR33fSoRJALZMyHMUOp8cxa1rYGPrBRyHhivdhhUnyRgXqAq/oymQo4+cwBHZFSpmtEqafQ8RpuOr2ymRgrxBGfe4n4eLprzY5EUZpFRhgxonb10YL16vg/oAlWObdYkS17ZayQtsfbHBD2udjljQXrjWNIWlT6fXG8JeJth+kFewr9+2c0Rfh9sQJ+F2otBk5x+dbt5xTKppAsAEHIy9lN8/Gbh+U+woCxgP11x+w/HYX9KXDkGHcOiAteYEI7Cf2Eo1TKD7ICVTVfReETWxAzSpKMabltNuM8fuLj6dHakvkQ6PgS537ShhyGofbLQaWTB8AMpwRCIUZme6EkfZuoO2CBt8gCnL3U6geDhHUB4ZGU4g9wPL/FlIqSPaWhafwbjc+PCyXqpOMNHdXtNc7D7bAsWN1Nri3Gk1D4ae0BDTunG/SgX4rlx6zc8kGgmFtJ/cnX//RO40Om2Yf36bdeb3KgDo4Ia49EZDaH7FlRn1cwUax0Gr3Jz4=
    ATUIN_DB_USERNAME: AgAt8yDkKRjmvJtB4ecxOOcuEm1Zcoa8pX1UvtvwAAT4M18PN3JK+6yOyhHuuTwWtWphlQnAjSWx6Bu8usgIxrw9dhBCRxf4pJIaW2VmszUnn1HOtdEFcU6+40PEZ8vJEqCQz/sQoilhZyH06VYecNZFtUHleFAaEFfSGPtxd73lqpjY62fOI8yoGfd/lmXays5vjSx9kUtUVd71FYEOf7P6x+OWlFWsbQ6FepiHygoCXTiCi9umbherpIHWCMZxELja/mNdVZp2wIO+NytedM47LIy2U0FP3b6quPc1H52OK/9AK9TJf/Ke8vUaRDE6TAqv1K0fT5diD4zwERzpNoHKHhnejKj1FOCm6WVcnPHk17zy9Et+kdB+feKpgbeZlolCSJ+JgNWnM2Y3WaovQI4i4yq3ipqQDI1AgY6hHMj1HGNH8gpFjHRy/+UfPd1f4aDO6hGAbL86O2y18VcqD7gESRJ7XVWikJWpU2hIp2FAEpopoqU1QPWyTGvvC46g+gfTARIphn1EzjKymdc4ICb8Viuy/B1oVuwFaD7y9FnNx3tPP4cSuODiG2u6q0j/UTMkAftGqPZUNu3yfkrJHziKUnGc9kuasgAFJKXL2qJuG4VBxNPwTmp2VnJiBysvUb1JTTYd+2uEu4woGmzVfm/9kjkP1rbRk+hAUj5fyW2Nebds9dgD2gXZ2yGOK/S1G0TXnriSQA==
template:
    metadata:
    creationTimestamp: null
    name: atuin-secrets
    namespace: atuin
    type: Opaque


---
# services.yaml

---
apiVersion: v1
kind: Service
metadata:
name: atuin
namespace: atuin
spec:
type: ClusterIP
ports:
    - port: 8888
    targetPort: 8888
selector:
    app: atuin
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: atuin
spec:
type: ClusterIP
ports:
    - port: 5432
    targetPort: 5432
selector:
    app: postgres

---
# storage.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: database
namespace: atuin
spec:
accessModes:
    - ReadWriteOnce
resources:
    requests:
    storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: atuin-config
namespace: atuin
spec:
accessModes:
    - ReadWriteOnce
resources:
    requests:
    storage: 100Mi