I spent some time tonight setting configuring some services on my home kubernetes cluster. See this post for more details on how I set up the cluster.
So far it's been a fun experiment to see if I can avoid anything spontanously catching file. At work there is a full team of experts dedicated to keep our cluster running smoothly. At home, there is... me.
Today I managed to get FreshRSS and Atuin Sync running.
I've been using Cursor as a guide generating the yaml files and asking questions about how Kubernetes works. I think I am a decent user of Kubernetes clusters, but a rank novice as an operator of a cluster.
I want to try to get away from doomscrolling, and being caught in some algorithmically generated news feed. I'll try FreshRSS for a while at least.
To get started I asked Cursor to generate a deployment, giving it a link to the FreshRSS documentation.
I had to go back and forth a few times to understand how to get a URL to resolve on my home network.
The kubernetes cluster is running on the host tyr, so I can ping that from my home network on tyr.local.
Initially I wanted to host FreshRSS at rss.tyr.local, but I didn't figure out how to do that. Instead I hosted it at tyr.local/rss and then added Middleware to strip the /rss path before sending the traffic to the Service.
Complete manifest
---
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: freshrss
namespace: freshrss
labels:
app: freshrss
spec:
replicas: 1
selector:
matchLabels:
app: freshrss
template:
metadata:
labels:
app: freshrss
spec:
containers:
- name: freshrss
image: freshrss/freshrss:latest
ports:
- containerPort: 80
env:
- name: TZ
value: "Europe/Copenhagen"
- name: CRON_MIN
value: "13,43"
volumeMounts:
- name: data
mountPath: /var/www/FreshRSS/data
- name: extensions
mountPath: /var/www/FreshRSS/extensions
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
volumes:
- name: data
persistentVolumeClaim:
claimName: freshrss-data
- name: extensions
persistentVolumeClaim:
claimName: freshrss-extensions
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: freshrss
namespace: freshrss
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
traefik.ingress.kubernetes.io/router.middlewares: "freshrss-strip-prefix@kubernetescrd"
traefik.ingress.kubernetes.io/router.tls: "false"
spec:
rules:
- host: tyr.local
http:
paths:
- path: /rss
pathType: Prefix
backend:
service:
name: freshrss
port:
number: 80
---
# middleware.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: strip-prefix
namespace: freshrss
spec:
stripPrefix:
prefixes:
- /rss
forceSlash: true
---
# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: freshrss
---
# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: freshrss-data
namespace: freshrss
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: freshrss-extensions
namespace: freshrss
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
---
# service.yaml
apiVersion: v1
kind: Service
metadata:
name: freshrss
namespace: freshrss
spec:
selector:
app: freshrss
ports:
- port: 80
targetPort: 80
type: ClusterIP
Again, I just asked Cursor to generate the Kubernetes manifests after linking to the self-hosting docs.
One issue I had here, was that I want to commit the manifests to git, but Atuin requires a password to communicate with the local Postgres database.
secrets.yamlapiVersion: v1
kind: Secret
metadata:
name: atuin-secrets
type: Opaque
stringData:
ATUIN_DB_USERNAME: atuin
ATUIN_DB_PASSWORD: seriously-insecure
ATUIN_HOST: "127.0.0.1"
ATUIN_PORT: "8888"
ATUIN_OPEN_REGISTRATION: "true"
ATUIN_DB_URI: "postgres://atuin:seriously-insecure@postgres/atuin"
immutable: true
Since both services are local and not exposed to the internet, there is probably no harm in commiting something like this.
But I had a discussion about this problem with my smart, very kubernetes savvy friend, Lasse. So I wanted to find some solution to this that did not commit any deadly sins, such as committing a secret in plaintext.
Again, I asked Cursor for suggestions, prompting it to not make any changes to my files, just list out options.
Here is the full list of suggestions:
List some ways I can add secrets here without leaking it in git
-
Using Sealed Secrets:
# Install sealed-secrets controller
helm install sealed-secrets sealed-secrets/sealed-secrets
# Encrypt your secrets
kubeseal < atuin/secrets.yaml > atuin/sealed-secrets.yaml
This lets you safely commit encrypted secrets that only your cluster can decrypt.
-
Using External Secrets Operator with a secrets manager:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: atuin-secrets
namespace: atuin
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: atuin-secrets
data:
- secretKey: ATUIN_DB_PASSWORD
remoteRef:
key: atuin/db-password
This fetches secrets from services like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.
-
Using Kustomize with secret generators:
# kustomization.yaml
secretGenerator:
- name: atuin-secrets
namespace: atuin
envs:
- .env # This file is not committed
-
Using environment variables with a CI/CD pipeline:
apiVersion: v1
kind: Secret
metadata:
name: atuin-secrets
namespace: atuin
stringData:
ATUIN_DB_PASSWORD: ${ATUIN_DB_PASSWORD}
ATUIN_DB_URI: "postgres://atuin:${ATUIN_DB_PASSWORD}@postgres/atuin"
-
Using SOPS (Secrets OPerationS):
# Encrypt secrets
sops -e -i atuin/secrets.yaml
# Decrypt when needed
sops -d atuin/secrets.yaml | kubectl apply -f -
I chose to use Sealed Secrets. It sounded by far the simplest, and I could sort-of understand what was going on:
- I install the SealedSecret helm chart on my cluster. This can encrypt and decrypt text using some fancy crypto magic. Basically a private/public key pair is generated and stored as Kubernetes secerets and these are used to encrypt my secrets.
- I install
kubeseal on my local machine. It can communicate with the SealedSecret controller running in the cluster.
- I pass a local (uncommitted) kubernetes
Secret to kubeseal, it encrypts it and I get back a SealedSecret.
- I can then store, apply and commit this
SealedSecret. It will get unsealed when applied to my cluster (so my services can use it), but the unsealing only happens inside the cluster. My local manifest file is encrypted.
Let's say I want to encrypt this Secret
secrets.yamlapiVersion: v1
kind: Secret
metadata:
name: atuin-secrets
namespace: atuin
type: Opaque
stringData:
ATUIN_DB_USERNAME: atuin
ATUIN_DB_PASSWORD: "123"
ATUIN_DB_URI: "postgres://username:123@postgres/atuin" # Match the password here
I can run kubeseal to encrypt:
kubeseal < secrets.yaml > sealed-secrets.yaml
and I get back
sealed-secrets.yamlapiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: atuin-secrets
namespace: atuin
spec:
encryptedData:
ATUIN_DB_PASSWORD: AgBKfphBarMiNX8CIsvjAXqEtRp/Bq+a4y67k/M6bxMm1w/[TRUNCATED FOR SPACE]
ATUIN_DB_URI: AgCfm2AisGVBlMrOqPvMWOor0e0UXDruZnWVG3klrfSzbtZfrzYF4x[TRUNCATED FOR SPACE]
ATUIN_DB_USERNAME: AgAt8yDkKRjmvJtB4ecxOOcuEm1Zcoa8pX1UvtvwAAT4M18PN3JK[TRUNCATED FOR SPACE]
template:
metadata:
creationTimestamp: null
name: atuin-secrets
namespace: atuin
type: Opaque
Pretty cool!
I have also backed up the Sealed Secrets private key in my 1Password.
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-master.key
If my cluster suddenly catches fire, I can recreate my deployments in a new cluster by adding the key to that cluster
kubectl apply -f sealed-secrets-master.key
kubectl delete pod -n kube-system -l name=sealed-secrets-controller
Here is the complete manifest
Complete manifest
# config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: atuin-config
namespace: atuin
data:
ATUIN_HOST: "0.0.0.0"
ATUIN_PORT: "8888"
ATUIN_OPEN_REGISTRATION: "true"
---
# deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: atuin
spec:
replicas: 1
strategy:
type: Recreate # Prevent data corruption by ensuring only one pod runs
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgresql
image: postgres:14
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: atuin
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: atuin-secrets
key: ATUIN_DB_PASSWORD
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: atuin-secrets
key: ATUIN_DB_USERNAME
lifecycle:
preStop:
exec:
command:
[
"/usr/local/bin/pg_ctl",
"stop",
"-D",
"/var/lib/postgresql/data",
"-w",
"-t",
"60",
"-m",
"fast",
]
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 250m
memory: 600Mi
volumeMounts:
- mountPath: /var/lib/postgresql/data/
name: database
volumes:
- name: database
persistentVolumeClaim:
claimName: database
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: atuin
namespace: atuin
spec:
replicas: 1
selector:
matchLabels:
app: atuin
template:
metadata:
labels:
app: atuin
spec:
containers:
- name: atuin
image: ghcr.io/atuinsh/atuin:v18.4.0 # Using a specific version as recommended
args:
- server
- start
env:
- name: ATUIN_DB_URI
valueFrom:
secretKeyRef:
name: atuin-secrets
key: ATUIN_DB_URI
- name: ATUIN_HOST
valueFrom:
configMapKeyRef:
name: atuin-config
key: ATUIN_HOST
- name: ATUIN_PORT
valueFrom:
configMapKeyRef:
name: atuin-config
key: ATUIN_PORT
- name: ATUIN_OPEN_REGISTRATION
valueFrom:
configMapKeyRef:
name: atuin-config
key: ATUIN_OPEN_REGISTRATION
ports:
- containerPort: 8888
resources:
limits:
cpu: 250m
memory: 1Gi
requests:
cpu: 250m
memory: 1Gi
volumeMounts:
- mountPath: /config
name: atuin-config
volumes:
- name: atuin-config
persistentVolumeClaim:
claimName: atuin-config
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: atuin
namespace: atuin
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
traefik.ingress.kubernetes.io/router.middlewares: "atuin-strip-prefix@kubernetescrd"
spec:
rules:
- host: tyr.local
http:
paths:
- path: /atuin
pathType: Prefix
backend:
service:
name: atuin
port:
number: 8888
---
# middleware.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: strip-prefix
namespace: atuin
spec:
stripPrefix:
prefixes:
- /atuin
forceSlash: true
---
# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: atuin
---
# sealed-secrets.yaml
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: atuin-secrets
namespace: atuin
spec:
encryptedData:
ATUIN_DB_PASSWORD: AgBKfphBarMiNX8CIsvjAXqEtRp/Bq+a4y67k/M6bxMm1w/fJUERNqBKaPWqaABfHR4WEk9ePj4CWcVbHb2xVCviX4zYE4pZ9onMvzRGJa2UUl1qRsJGN/ooMRJux+ztfSXJfRzzZxt1QjBlJOmMxG0XjKu0TdahXnI4BMJ2rrBPPmWx9sr4z8YxG8BU/TL8DiJGiD2DtarQWmqSogueGpsOE/9hdeWvW4E7RNlcd7JJ0Hv/nELlhVIUB9fzGoaioDJO6qodYBWNtt2ckyNp3KwoOKXddwRV5tq1ggPKnZOqlHpDgmTaYAFNPXVGIpMNxzUfs+CU0VdT60hx5e3qMbVD86NrnqmbQ38GYc/A7TDrWImSEPjkweLPSTgK5YuQEHJBGYDy9jNNVTMHwfcXkAZkD8swu8+2Whw6No1D2WO2LwewVdTDOynjVhekGk3UF6B2lqIn9TowkIBbZZ6mYYK4VzXRCRXmo2ZiEqDMQK78ejUHdK5m43cZ9M+BEmE3lKzAmgZt+xons/xcisI63pff31urXWZsFylZvnVUnR/l0cp5jmr8KDnMp1WDPf+UyhSlxVvnfAKRyXIGi6jpMQluXVvx/waX4MdqgJMfyn3cQ6tFH4YiZCX6kdNNWjJp5lYxmhRdqWRznCB1vxuWIfXCc9eUT8Kz0Houmw/S8HR11ApNoxopbalC23wdTa9ZXlJdC4bXElfdC8HHwjTcNezDN9mc+4e+WdaKkbuYZljP
ATUIN_DB_URI: AgCfm2AisGVBlMrOqPvMWOor0e0UXDruZnWVG3klrfSzbtZfrzYF4x+sY7fVLsfUY3RSRF84m13hIJPBxhiO3pFPAs6e6zm5GH7B+8Iem1ijIXWNVW5oc7h/Kas77k1h+TcJTVyZ4gL52oqzZM3cwAX0UdE/enNrvYWoeTsJ0UMbNw3bKZ9Ll0BPfdirdHT8Ve7jMzaDF+d11difPOhyZ7wgK3ykzOGu9G8LbzJ8IwUYYFK/1DETYU76XC/d79tUOwSYxGwf88/r2zjn9ZFA7rnzzEnV7ECR33fSoRJALZMyHMUOp8cxa1rYGPrBRyHhivdhhUnyRgXqAq/oymQo4+cwBHZFSpmtEqafQ8RpuOr2ymRgrxBGfe4n4eLprzY5EUZpFRhgxonb10YL16vg/oAlWObdYkS17ZayQtsfbHBD2udjljQXrjWNIWlT6fXG8JeJth+kFewr9+2c0Rfh9sQJ+F2otBk5x+dbt5xTKppAsAEHIy9lN8/Gbh+U+woCxgP11x+w/HYX9KXDkGHcOiAteYEI7Cf2Eo1TKD7ICVTVfReETWxAzSpKMabltNuM8fuLj6dHakvkQ6PgS537ShhyGofbLQaWTB8AMpwRCIUZme6EkfZuoO2CBt8gCnL3U6geDhHUB4ZGU4g9wPL/FlIqSPaWhafwbjc+PCyXqpOMNHdXtNc7D7bAsWN1Nri3Gk1D4ae0BDTunG/SgX4rlx6zc8kGgmFtJ/cnX//RO40Om2Yf36bdeb3KgDo4Ia49EZDaH7FlRn1cwUax0Gr3Jz4=
ATUIN_DB_USERNAME: AgAt8yDkKRjmvJtB4ecxOOcuEm1Zcoa8pX1UvtvwAAT4M18PN3JK+6yOyhHuuTwWtWphlQnAjSWx6Bu8usgIxrw9dhBCRxf4pJIaW2VmszUnn1HOtdEFcU6+40PEZ8vJEqCQz/sQoilhZyH06VYecNZFtUHleFAaEFfSGPtxd73lqpjY62fOI8yoGfd/lmXays5vjSx9kUtUVd71FYEOf7P6x+OWlFWsbQ6FepiHygoCXTiCi9umbherpIHWCMZxELja/mNdVZp2wIO+NytedM47LIy2U0FP3b6quPc1H52OK/9AK9TJf/Ke8vUaRDE6TAqv1K0fT5diD4zwERzpNoHKHhnejKj1FOCm6WVcnPHk17zy9Et+kdB+feKpgbeZlolCSJ+JgNWnM2Y3WaovQI4i4yq3ipqQDI1AgY6hHMj1HGNH8gpFjHRy/+UfPd1f4aDO6hGAbL86O2y18VcqD7gESRJ7XVWikJWpU2hIp2FAEpopoqU1QPWyTGvvC46g+gfTARIphn1EzjKymdc4ICb8Viuy/B1oVuwFaD7y9FnNx3tPP4cSuODiG2u6q0j/UTMkAftGqPZUNu3yfkrJHziKUnGc9kuasgAFJKXL2qJuG4VBxNPwTmp2VnJiBysvUb1JTTYd+2uEu4woGmzVfm/9kjkP1rbRk+hAUj5fyW2Nebds9dgD2gXZ2yGOK/S1G0TXnriSQA==
template:
metadata:
creationTimestamp: null
name: atuin-secrets
namespace: atuin
type: Opaque
---
# services.yaml
---
apiVersion: v1
kind: Service
metadata:
name: atuin
namespace: atuin
spec:
type: ClusterIP
ports:
- port: 8888
targetPort: 8888
selector:
app: atuin
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: atuin
spec:
type: ClusterIP
ports:
- port: 5432
targetPort: 5432
selector:
app: postgres
---
# storage.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: database
namespace: atuin
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: atuin-config
namespace: atuin
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
