![\"productivity](\"../images/productivity_contribution.png\")
It is hard to go more than a few days without some major story or a new hype cycle on AI. \nRecently, a lot of people have discovered that Claude Code is quite good actually.
\nA blog I really enjoy, \"Don't Worry About The Vase\" by Zvi Mowshowitz, has extensive coverage of what is going on in the AI world. \nI'm always excited to have a look at his weekly AI updates on thursdays, even if I rarely finish the whole thing. He really is a prolific writer. See the latest one here: AI #154: Claw Your Way To The Top, focused on the recent OpenClaw and Moltbook hype (small aside, has that already died down again?)
\nRecently, he has started posting about Claude Code, and about how people are using it or realizing that it can do a lot for them. I find it VERY interesting! Coding agents are really transforming the way I work, and I think they are going to be a huge part of the future of work in general.
\nThe posts primarily consists of snippets from Twitter with some commentary, but since I'm not on Twitter, this is a great way for me to get a sense of what people are saying about it.\nHere is the latest one: Claude Code #4: From The Before Times.
\nHe pulls an interesting quote from Andrej Karpathy:
\n\n\nAndrej Karpathy: Love the word \"comprehension debt\", haven't encountered it so far, it's very accurate. It's so very tempting to just move on when the LLM one-shotted something that seems to work ok.
\n
I was amused to try the new /insights command recently, mentioned here:
\n\nThe new Claude Code command /insights will read your last month\u2019s message history and give you suggestions to improve your workflow.
\n
There is a section of best practices from different sources: Skilling Up
\nYou can also go have a look at some of the previous ones, which are quite interesting as well:\n- Claude Codes\n- Claude Coworks\n- Claude Codes #3
", "image": null, "date_published": "2026-02-08T00:00:00+00:00", "authors": [], "tags": null}, {"id": "https://kplauritzen.dk/2025-12-15-fred-stats-rant/", "url": "https://kplauritzen.dk/2025-12-15-fred-stats-rant/", "title": "Fred", "content_html": "My good friend, Fred, has started blogging on substack recently.
\nFred has MANY skills. He has been DOING things, been poking at the world with every stick he has ever found, changed careers to something more exciting more times than I can easily express.
\nHe could probably express it. One skill he has, maybe the thing he is the best at, is writing.
\nActually, there is at least one thing I'll rank higher than Fred's writing. That is a good Fred rant. And his blog feels like a series of particularly high-quality rants, interspersed with actual, honest-to-god, advice on being a good human.
\nSo I'll drop whatever else I'm doing whenever I see a [FRED] post in my RSS feed, get comfy in my chair, and enjoy the fireworks.
\nYou might need an example:\nHow about this one, opening on the toilet at the therapists office, setting the scene like this
\n\n\nI\u2019ve just suffered through part bowel movement, part catholic exorcism\u2026
\nResulting in one of those near-mythological Fantasy Epic wipes that need more paper than all of Robert Jordan\u2019s Wheel Of Time series.
\nThis is a Sisyphean task straight out of Greek mythology - pushing a brown tar-like with inexplicable-grains-in-it boulder uphill every day until the world runs out of patience and paper.
\n
IS THAT A HOOK OR WHAT?
\nOk, so he is also incredibly crass. That just makes it better.
\nHis most recent post is about productivity.
\nI think you should read the post, but the shortest possible summary is:\n- Suppose your daily produtivity is following a truncated normal distribution, from 0% to 100% productive days. \n- Then you will have a few days in a month that feel very productive, a few that feel not-at-all productive, and most days will be in-between. Those are the Gooch Days.
\nFred points out that you can't rely on the super productive days, they are too rare. Instead, the real hero is the Gooch. This is where progress is made, because this is where the volume is at.
\nI wanted to play around with this intuition a little bit, and since Python is my hammer and everything is a nail, I fired up some matplotlib.
\nGiven a normal distribution with a mean of 50% and a standard deviation of 20%, we can approximate the histogram Fred showed. And if we want to know how much each day contributes to the total productivity, we can plot $x \\times p(x)$, the expected contribution. For me, it helped to think of Expected Value calculations: Imagine you make a bet that will give 100 EUR if you win, and you have a 20% chance to win. The expected value is $100 \\times 0.2 = 20$ EUR.
\nSo now the height on the histogram is not representing the frequency of days with that amount of productivity. Instead it represents how much those days contribute to the overall productivity (of the month/year/whatever).
\nI've plotted both the assumed productivy distribution and the contribution below.
\n
Note the area with a slightly off-putting color, between 20% and 80% productivity, the Gooch Days. All together, they represent 87% of your total productivity! They are basically all that matter.
", "image": null, "date_published": "2025-12-15T00:00:00+00:00", "authors": [], "tags": null}, {"id": "https://kplauritzen.dk/2025-11-25-reading-rss-feeds/", "url": "https://kplauritzen.dk/2025-11-25-reading-rss-feeds/", "title": "Reading RSS feeds", "content_html": "My good friend Zeth just posted on using RSS feeds to reduce distractions and I think that is super interesting! He is always running small experiments on himself to optimize or nudge his behaviour in some way. Here, here is trying out RSS feeds. Go read his post now.
\nNot being easily distracted seems like a core skill today.\nI'm using RSS readers to reduce the slot machine behaviour from clicking on a bookmark hoping for a new exciting post.\nNow there is just one place to go, and I can more easily control my cravings.
\nI'm also using RSS readers as a way of being less reliant on someone else's algorithm deciding what I get to see. Some RSS readers push AI filtering or \"trending topics\", as Zeth mentions. I don't like that!\nAvoiding the algorithm means that curation is only up to you.
\nI'm using FreshRSS self-hosted in a one-node Kubernetes cluster in my home lab.\nI like that it \"gets out of the way\" quickly.\nIt is just a chronological list of posts, with some simple options to see e.g. all posts from a certain feed or category.
\nFull disclosure: I also still have a very bad habit of opening Hacker News whenever I'm stuck on something at work. Do as I say, not as I do-
\nFind good feeds to subscribe to!\nBy far the most important thing, having high-quality stuff to read.\nI'm collecting my recommendations. Mostly people I actually know and then a few other sources I really like.\nIt's easy to add a news feed that pushes 20 news stories every day.\nThat defeats the whole purpose. You will be drowning in noise.
\nPrune your feed\nI have some sources that I have added due to a particularly high-quality post, and if I haven't been impressed with their writing over the next weeks or months, I will remove that feed again.
\nDon't try to read everything\nI think this one is important as well.\nYou don't owe any of these writers your attention.\nWhen I ignore or forget my RSS reader for a week (or a month), and then get back to dozens or hundreds of unread stories, I will usually just mark everything older than some date \"read\".\nMaybe there was some gold hiding there, but there is no way I'm going to find it in the pile of random stuff.\nThe internet is not lacking in content. Trying to read everything is a permanent backlog out of your control. Avoid that!
\nDon't feel obligated to read the post completely\nI practice this mostly for Zvi's notoriously long AI newsletters, but it is also true for anything else.\nThey are very interesting, and I enjoy them a lot, but not enough to read a short novel several times a week.\nSkim, skip, consider asking your AI if there are sections you should focus on given your interests.\nZvi also marks the most interesting sections with a bold heading in the Table of Contents. Thanks!
", "image": null, "date_published": "2025-11-25T00:00:00+00:00", "authors": [], "tags": null}, {"id": "https://kplauritzen.dk/2025-11-22-homelab-metrics/", "url": "https://kplauritzen.dk/2025-11-22-homelab-metrics/", "title": "Monitoring usage metrics on the homelab", "content_html": "I have a very small and underpowered kubernetes cluster at home (see Homelab #1 and Homelab #2), and I'm always concerned that I'm overutilizing its resources when I want to add another service to it.
\nFor example, sometimes my FreshRSS site is not responding. Is that because there is a network problem, or am I running out of memory, or something completely different.
\nOne option could be to run Prometheus and Grafana, scraping metrics from each service at regular intervals. The problem with that is that I'm already running on a low-power system, so spending a significant chunk of the total resources only on MONITORING how many resources I'm using seems like a bad idea.
\nInstead, partly inspired by Markus' love for SQLite, I'm doing a much simpler thing:
\nkubectl top nodes and kubectl top pods every 5 minutes with an entry in crontab. The database is a local file, stored on the same computer running k3s. When I want to look at how my system has been behaving recently, I can copy the whole database to my desktop and start digging into it.
No fuss, minimal overhead, no fancy dashboard. It's weird to do YAGNI on my homelab, because I don't NEED any of it.
\nThe script I use to collect metrics:\n
#!/bin/bash\nset -euo pipefail\n\n# Database location\nDB_PATH="${DB_PATH:-/var/lib/k8s-metrics/metrics.db}"\nTIMESTAMP=$(date -u +"%Y-%m-%d %H:%M:%S")\n\n# Full path to kubectl (k3s)\nKUBECTL="${KUBECTL:-/usr/local/bin/kubectl}"\n\n# Ensure database directory exists\nmkdir -p "$(dirname "$DB_PATH")"\n\n# Initialize database if it doesn't exist\nsqlite3 "$DB_PATH" <<EOF\nCREATE TABLE IF NOT EXISTS node_metrics (\n timestamp TEXT NOT NULL,\n node_name TEXT NOT NULL,\n cpu_cores TEXT NOT NULL,\n cpu_percent TEXT NOT NULL,\n memory_bytes TEXT NOT NULL,\n memory_percent TEXT NOT NULL,\n PRIMARY KEY (timestamp, node_name)\n);\n\nCREATE TABLE IF NOT EXISTS pod_metrics (\n timestamp TEXT NOT NULL,\n namespace TEXT NOT NULL,\n pod_name TEXT NOT NULL,\n cpu_cores TEXT NOT NULL,\n memory_bytes TEXT NOT NULL,\n PRIMARY KEY (timestamp, namespace, pod_name)\n);\n\nCREATE INDEX IF NOT EXISTS idx_node_timestamp ON node_metrics(timestamp);\nCREATE INDEX IF NOT EXISTS idx_pod_timestamp ON pod_metrics(timestamp);\nEOF\n\n# Collect node metrics\n$KUBECTL top nodes --no-headers | while read -r name cpu cpu_pct memory memory_pct; do\n sqlite3 "$DB_PATH" <<EOF\nINSERT INTO node_metrics (timestamp, node_name, cpu_cores, cpu_percent, memory_bytes, memory_percent)\nVALUES ('$TIMESTAMP', '$name', '$cpu', '$cpu_pct', '$memory', '$memory_pct');\nEOF\ndone\n\n# Collect pod metrics\n$KUBECTL top pods -A --no-headers | while read -r namespace name cpu memory; do\n sqlite3 "$DB_PATH" <<EOF\nINSERT INTO pod_metrics (timestamp, namespace, pod_name, cpu_cores, memory_bytes)\nVALUES ('$TIMESTAMP', '$namespace', '$name', '$cpu', '$memory');\nEOF\ndone\n\necho "Metrics collected at $TIMESTAMP"\nThe crontab entry\n
*/5 * * * * KUBECONFIG=/home/kasper/.kube/config DB_PATH=/home/kasper/k8s-metrics/metrics.db /home/kasper/k8s-metrics/collect-metrics.sh >> /home/kasper/k8s-metrics/collect.log 2>&1\nYesterday, Anthropic released their new LLM, Claude 3.7 Sonnet.
\nFor a run-through of the vibes, I always wait for Zvi Mowshowitz's review.
\nHowever, there are a few fun highlights here.
\nBack in the pre-historic era when I was at Uni, I lost a lot of time to Twitch Plays Pokemon.\nI was facinated by the idea of a collective playing a game, and the chaos that ensued. A whole mythos got built up, and it was a lot of fun to follow and occasionally spam in the chat.
\nAnthropic has started using \"How far can Claude get in Pokemon\" as one of their benchmarks, but they have also set up an emulator streaming on Twitch.
\n![\"Claude](\"../images/claude-pokemon.png\")
You can watch ClaudePlaysPokemon and see how far the model can get.
\nThere is an easter egg in the system prompt, where if you ask \"How many R's in Strawberry?\" it will generate a little website that counts the number of R's in the word strawberry.
\nI have pasted the HTML directly below (simultaneously testing how markdown with HTML gets handled by MkDocs)
\n\n\n\n \nI've been really happy with my self-hosted RSS reader FreshRSS .\nI've consolidated a lot of the newsletter spam and occasional checking of blogs with a more considered decision to go to the RSS reader.
\nThis means that I have had less desire to check out whats going on on Mastodon.
\nI just found out that I can get an RSS feed for Mastodon accounts. So, if I wanted to follow @charliemarsh I can just append .rss to the url of his profile webpage. (Note, this has to be his original page).\nSo, Charlies original page is https://hachyderm.io/@charliermarsh and the RSS feed with all his public posts are available at https://hachyderm.io/@charliermarsh.rss.
I have started to follow a few Python people where I'm not aware of any blog to follow instead. Aside from Charlie Marsh, I also now follow:
\n\nI also have an account that I barely use at @KPLauritzen that you can follow.
\nVia Feditips
", "image": null, "date_published": "2025-02-05T00:00:00+00:00", "authors": [], "tags": null}, {"id": "https://kplauritzen.dk/2025-02-04-terraform-destructive-pr/", "url": "https://kplauritzen.dk/2025-02-04-terraform-destructive-pr/", "title": "Warn about destructive Terraform changes in pull requests", "content_html": "When automating infrastructure changes through CI/CD pipelines, it can be VERY scary to merge a pull request that changes something in your infrastructure that you are not very familiar with.
\nSure, you have tested the change in a test environment. Did you try to make a plan against a dev environment too? Have you tested against all the relevant targets of this change?
\nMaybe you are reviewing someone else's infrastructure changes. How can you be sure you have caught if this is actually destroying and recreating all the databases?
\nI've dealt with too much of this anxiety! AUTOMATE IT AWAY!
\nWith some inspiration from my friend, Lasse Hels, I created this bash script for Azure DevOps Pipelines.
\ntfplan, it parses the plan as plaintext and json#!/bin/bash\nset -euo pipefail\n\n# Somehow create your plan as tfplan\nPLAN_TEXT=$(terraform show tfplan)\nPLAN_JSON=$(terraform show -json tfplan)\nHAS_DESTRUCTIVE_CHANGES=$(echo "$PLAN_JSON" | jq -r '.resource_changes[] | select(.change.actions[] | contains("delete"))')\n\n# Conditional alert\nDANGER_MESSAGE=""\nif [ ! -z "$HAS_DESTRUCTIVE_CHANGES" ]; then\n DANGER_MESSAGE="**DANGER! YOU ARE ABOUT TO DESTROY RESOURCES**"\nfi\n\n# Actual comment to be posted\nCODE_BLOCK_FENCE='```'\nCOMMENT=$(cat << EOF\n${DANGER_MESSAGE}\n<details><summary>Click to expand</summary>\n\n${CODE_BLOCK_FENCE}\n${PLAN_TEXT}\n${CODE_BLOCK_FENCE}\n</details>\nEOF\n)\n\n# Set comment status to Active for destructive changes, Resolved otherwise\nCOMMENT_STATUS=2 # Resolved\nif [ ! -z "$HAS_DESTRUCTIVE_CHANGES" ]; then\n COMMENT_STATUS=1 # Active\nfi\n\n# Build payload for ADO API\nJSON_PAYLOAD=$(jq -n \\\n --arg content "$COMMENT" \\\n --arg status "$COMMENT_STATUS" \\\n '{comments: [{content: $content}], status: ($status|tonumber)}'\n)\n\n# Call ADO API to make the comment\ncurl -X POST \\\n "$(SYSTEM_COLLECTIONURI)/$(SYSTEM_TEAMPROJECT)/_apis/git/repositories/$(BUILD_REPOSITORY_NAME)/pullrequests/$(SYSTEM_PULLREQUEST_PULLREQUESTID)/threads?api-version=6.0" \\\n -H "Authorization: Bearer $(SYSTEM_ACCESSTOKEN)" \\\n -H "Content-Type: application/json" \\\n -d "$JSON_PAYLOAD"\nuv run --resolutionI'm distributing a small Python package at work. A small library with some utilities for doing Machine Learning work.\nI'm using uv to manage the dependencies and the build process.
\nPart of my pyproject.toml file looks like this:
\n\nHow do I know that my library will work with both pydantic==2.0 and pydantic==2.10 (The current version at time of writing)?\nI could just require a much smaller band of possible versions, but I want my library to be useful for as many users as possible.\nAnd they might need to use a different version of pydantic for their own projects.
Similarly, I want to make sure my library actually works with the range of allowed Python versions.
\nI run my tests with uv run pytest. This will use the locked dependencies in the uv.lock file to create a virtual environment and run the tests in that environment.
But, I can use the --resolution flag to test my library with different versions of the dependencies.\nAccording to the uv documentation, there are three possible values for the --resolution flag:
I have found that using --resolution lowest is not really that useful, because some transitive dependencies might not specify a version range. Maybe they just require \"numpy\" without specifying a version. In that case, I will be testing my library against numpy==0.0.1 or whatever the lowest version is. That is not really useful. Instead, I use --resolution lowest-direct to test against the lowest version of the direct dependencies and then just select the highest version of the transitive dependencies.
I can also specify the python version to use with the --python flag.
Finally, I can use the --isolated flag to make sure that the tests are run in an isolated virtual environment, not affecting the active venv of my workspace.
Here is the entry in my justfile that runs the tests with different dependency resolutions:
\ntest_dependency_bounds:\n uv run --python 3.10 --resolution lowest-direct --isolated pytest\n uv run --python 3.13 --resolution highest --isolated pytest\nI spent some time tonight setting configuring some services on my home kubernetes cluster. See this post for more details on how I set up the cluster.\nSo far it's been a fun experiment to see if I can avoid anything spontanously catching file. At work there is a full team of experts dedicated to keep our cluster running smoothly. At home, there is... me.
\nToday I managed to get FreshRSS and Atuin Sync running.
\nI've been using Cursor as a guide generating the yaml files and asking questions about how Kubernetes works. I think I am a decent user of Kubernetes clusters, but a rank novice as an operator of a cluster.
\nI want to try to get away from doomscrolling, and being caught in some algorithmically generated news feed. I'll try FreshRSS for a while at least.
\nTo get started I asked Cursor to generate a deployment, giving it a link to the FreshRSS documentation.
\nI had to go back and forth a few times to understand how to get a URL to resolve on my home network. \nThe kubernetes cluster is running on the host tyr, so I can ping that from my home network on tyr.local.
Initially I wanted to host FreshRSS at rss.tyr.local, but I didn't figure out how to do that. Instead I hosted it at tyr.local/rss and then added Middleware to strip the /rss path before sending the traffic to the Service.
---\n# deployment.yaml\n\napiVersion: apps/v1\nkind: Deployment\nmetadata:\nname: freshrss\nnamespace: freshrss\nlabels:\n app: freshrss\nspec:\nreplicas: 1\nselector:\n matchLabels:\n app: freshrss\ntemplate:\n metadata:\n labels:\n app: freshrss\n spec:\n containers:\n - name: freshrss\n image: freshrss/freshrss:latest\n ports:\n - containerPort: 80\n env:\n - name: TZ\n value: "Europe/Copenhagen"\n - name: CRON_MIN\n value: "13,43"\n volumeMounts:\n - name: data\n mountPath: /var/www/FreshRSS/data\n - name: extensions\n mountPath: /var/www/FreshRSS/extensions\n resources:\n requests:\n memory: "128Mi"\n cpu: "100m"\n limits:\n memory: "256Mi"\n cpu: "500m"\n volumes:\n - name: data\n persistentVolumeClaim:\n claimName: freshrss-data\n - name: extensions\n persistentVolumeClaim:\n claimName: freshrss-extensions\n\n---\n# ingress.yaml\n\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\nname: freshrss\nnamespace: freshrss\nannotations:\n traefik.ingress.kubernetes.io/router.entrypoints: web\n traefik.ingress.kubernetes.io/router.middlewares: "freshrss-strip-prefix@kubernetescrd"\n traefik.ingress.kubernetes.io/router.tls: "false"\nspec:\nrules:\n - host: tyr.local\n http:\n paths:\n - path: /rss\n pathType: Prefix\n backend:\n service:\n name: freshrss\n port:\n number: 80\n\n---\n# middleware.yaml\n\napiVersion: traefik.io/v1alpha1\nkind: Middleware\nmetadata:\nname: strip-prefix\nnamespace: freshrss\nspec:\nstripPrefix:\n prefixes:\n - /rss\n forceSlash: true\n\n---\n# namespace.yaml\n\napiVersion: v1\nkind: Namespace\nmetadata:\nname: freshrss\n\n---\n# pvc.yaml\n\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\nname: freshrss-data\nnamespace: freshrss\nspec:\naccessModes:\n - ReadWriteOnce\nresources:\n requests:\n storage: 1Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\nname: freshrss-extensions\nnamespace: freshrss\nspec:\naccessModes:\n - ReadWriteOnce\nresources:\n requests:\n storage: 100Mi\n\n---\n# service.yaml\n\napiVersion: v1\nkind: Service\nmetadata:\nname: freshrss\nnamespace: freshrss\nspec:\nselector:\n app: freshrss\nports:\n - port: 80\n targetPort: 80\ntype: ClusterIP\nAgain, I just asked Cursor to generate the Kubernetes manifests after linking to the self-hosting docs.
\nOne issue I had here, was that I want to commit the manifests to git, but Atuin requires a password to communicate with the local Postgres database.
\napiVersion: v1\nkind: Secret\nmetadata:\n name: atuin-secrets\ntype: Opaque\nstringData:\n ATUIN_DB_USERNAME: atuin\n ATUIN_DB_PASSWORD: seriously-insecure\n ATUIN_HOST: "127.0.0.1"\n ATUIN_PORT: "8888"\n ATUIN_OPEN_REGISTRATION: "true"\n ATUIN_DB_URI: "postgres://atuin:seriously-insecure@postgres/atuin"\nimmutable: true\nSince both services are local and not exposed to the internet, there is probably no harm in commiting something like this.
\nBut I had a discussion about this problem with my smart, very kubernetes savvy friend, Lasse. So I wanted to find some solution to this that did not commit any deadly sins, such as committing a secret in plaintext.
\nAgain, I asked Cursor for suggestions, prompting it to not make any changes to my files, just list out options.
\nHere is the full list of suggestions:
\nUsing Sealed Secrets:
\n# Install sealed-secrets controller\nhelm install sealed-secrets sealed-secrets/sealed-secrets\n# Encrypt your secrets\nkubeseal < atuin/secrets.yaml > atuin/sealed-secrets.yaml\nThis lets you safely commit encrypted secrets that only your cluster can decrypt.
\nUsing External Secrets Operator with a secrets manager:
\napiVersion: external-secrets.io/v1beta1\nkind: ExternalSecret\nmetadata:\nname: atuin-secrets\nnamespace: atuin\nspec:\nrefreshInterval: 1h\nsecretStoreRef:\n name: vault-backend\n kind: ClusterSecretStore\ntarget:\n name: atuin-secrets\ndata:\n - secretKey: ATUIN_DB_PASSWORD\n remoteRef:\n key: atuin/db-password\nThis fetches secrets from services like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.
\nUsing Kustomize with secret generators:\n
\n This generates secrets from local files that aren't committed.\nUsing environment variables with a CI/CD pipeline:\n
\n Your CI/CD system (like GitHub Actions) replaces the variables before applying.\nUsing SOPS (Secrets OPerationS):\n
\n This encrypts secrets with age, PGP, or cloud KMS keys.\nI chose to use Sealed Secrets. It sounded by far the simplest, and I could sort-of understand what was going on:
\nkubeseal on my local machine. It can communicate with the SealedSecret controller running in the cluster. Secret to kubeseal, it encrypts it and I get back a SealedSecret. SealedSecret. It will get unsealed when applied to my cluster (so my services can use it), but the unsealing only happens inside the cluster. My local manifest file is encrypted. Let's say I want to encrypt this Secret\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: atuin-secrets\n namespace: atuin\ntype: Opaque\nstringData:\n ATUIN_DB_USERNAME: atuin\n ATUIN_DB_PASSWORD: "123"\n ATUIN_DB_URI: "postgres://username:123@postgres/atuin" # Match the password here\nI can run kubeseal to encrypt:\n
and I get back
\napiVersion: bitnami.com/v1alpha1\nkind: SealedSecret\nmetadata:\n creationTimestamp: null\n name: atuin-secrets\n namespace: atuin\nspec:\n encryptedData:\n ATUIN_DB_PASSWORD: AgBKfphBarMiNX8CIsvjAXqEtRp/Bq+a4y67k/M6bxMm1w/[TRUNCATED FOR SPACE]\n ATUIN_DB_URI: AgCfm2AisGVBlMrOqPvMWOor0e0UXDruZnWVG3klrfSzbtZfrzYF4x[TRUNCATED FOR SPACE]\n ATUIN_DB_USERNAME: AgAt8yDkKRjmvJtB4ecxOOcuEm1Zcoa8pX1UvtvwAAT4M18PN3JK[TRUNCATED FOR SPACE]\n template:\n metadata:\n creationTimestamp: null\n name: atuin-secrets\n namespace: atuin\n type: Opaque\nPretty cool! \nI have also backed up the Sealed Secrets private key in my 1Password.
\nkubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-master.key\nIf my cluster suddenly catches fire, I can recreate my deployments in a new cluster by adding the key to that cluster\n
kubectl apply -f sealed-secrets-master.key\nkubectl delete pod -n kube-system -l name=sealed-secrets-controller\nHere is the complete manifest
\n# config.yaml\n\napiVersion: v1\nkind: ConfigMap\nmetadata:\nname: atuin-config\nnamespace: atuin\ndata:\nATUIN_HOST: "0.0.0.0"\nATUIN_PORT: "8888"\nATUIN_OPEN_REGISTRATION: "true"\n\n---\n# deployment.yaml\n\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\nname: postgres\nnamespace: atuin\nspec:\nreplicas: 1\nstrategy:\n type: Recreate # Prevent data corruption by ensuring only one pod runs\nselector:\n matchLabels:\n app: postgres\ntemplate:\n metadata:\n labels:\n app: postgres\n spec:\n containers:\n - name: postgresql\n image: postgres:14\n ports:\n - containerPort: 5432\n env:\n - name: POSTGRES_DB\n value: atuin\n - name: POSTGRES_PASSWORD\n valueFrom:\n secretKeyRef:\n name: atuin-secrets\n key: ATUIN_DB_PASSWORD\n - name: POSTGRES_USER\n valueFrom:\n secretKeyRef:\n name: atuin-secrets\n key: ATUIN_DB_USERNAME\n lifecycle:\n preStop:\n exec:\n command:\n [\n "/usr/local/bin/pg_ctl",\n "stop",\n "-D",\n "/var/lib/postgresql/data",\n "-w",\n "-t",\n "60",\n "-m",\n "fast",\n ]\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\n limits:\n cpu: 250m\n memory: 600Mi\n volumeMounts:\n - mountPath: /var/lib/postgresql/data/\n name: database\n volumes:\n - name: database\n persistentVolumeClaim:\n claimName: database\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\nname: atuin\nnamespace: atuin\nspec:\nreplicas: 1\nselector:\n matchLabels:\n app: atuin\ntemplate:\n metadata:\n labels:\n app: atuin\n spec:\n containers:\n - name: atuin\n image: ghcr.io/atuinsh/atuin:v18.4.0 # Using a specific version as recommended\n args:\n - server\n - start\n env:\n - name: ATUIN_DB_URI\n valueFrom:\n secretKeyRef:\n name: atuin-secrets\n key: ATUIN_DB_URI\n - name: ATUIN_HOST\n valueFrom:\n configMapKeyRef:\n name: atuin-config\n key: ATUIN_HOST\n - name: ATUIN_PORT\n valueFrom:\n configMapKeyRef:\n name: atuin-config\n key: ATUIN_PORT\n - name: ATUIN_OPEN_REGISTRATION\n valueFrom:\n configMapKeyRef:\n name: atuin-config\n key: ATUIN_OPEN_REGISTRATION\n ports:\n - containerPort: 8888\n resources:\n limits:\n cpu: 250m\n memory: 1Gi\n requests:\n cpu: 250m\n memory: 1Gi\n volumeMounts:\n - mountPath: /config\n name: atuin-config\n volumes:\n - name: atuin-config\n persistentVolumeClaim:\n claimName: atuin-config\n\n---\n# ingress.yaml\n\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\nname: atuin\nnamespace: atuin\nannotations:\n traefik.ingress.kubernetes.io/router.entrypoints: web\n traefik.ingress.kubernetes.io/router.middlewares: "atuin-strip-prefix@kubernetescrd"\nspec:\nrules:\n - host: tyr.local\n http:\n paths:\n - path: /atuin\n pathType: Prefix\n backend:\n service:\n name: atuin\n port:\n number: 8888\n\n---\n# middleware.yaml\n\napiVersion: traefik.io/v1alpha1\nkind: Middleware\nmetadata:\nname: strip-prefix\nnamespace: atuin\nspec:\nstripPrefix:\n prefixes:\n - /atuin\n forceSlash: true\n\n---\n# namespace.yaml\n\napiVersion: v1\nkind: Namespace\nmetadata:\nname: atuin\n\n---\n# sealed-secrets.yaml\n\n---\napiVersion: bitnami.com/v1alpha1\nkind: SealedSecret\nmetadata:\ncreationTimestamp: null\nname: atuin-secrets\nnamespace: atuin\nspec:\nencryptedData:\n ATUIN_DB_PASSWORD: AgBKfphBarMiNX8CIsvjAXqEtRp/Bq+a4y67k/M6bxMm1w/fJUERNqBKaPWqaABfHR4WEk9ePj4CWcVbHb2xVCviX4zYE4pZ9onMvzRGJa2UUl1qRsJGN/ooMRJux+ztfSXJfRzzZxt1QjBlJOmMxG0XjKu0TdahXnI4BMJ2rrBPPmWx9sr4z8YxG8BU/TL8DiJGiD2DtarQWmqSogueGpsOE/9hdeWvW4E7RNlcd7JJ0Hv/nELlhVIUB9fzGoaioDJO6qodYBWNtt2ckyNp3KwoOKXddwRV5tq1ggPKnZOqlHpDgmTaYAFNPXVGIpMNxzUfs+CU0VdT60hx5e3qMbVD86NrnqmbQ38GYc/A7TDrWImSEPjkweLPSTgK5YuQEHJBGYDy9jNNVTMHwfcXkAZkD8swu8+2Whw6No1D2WO2LwewVdTDOynjVhekGk3UF6B2lqIn9TowkIBbZZ6mYYK4VzXRCRXmo2ZiEqDMQK78ejUHdK5m43cZ9M+BEmE3lKzAmgZt+xons/xcisI63pff31urXWZsFylZvnVUnR/l0cp5jmr8KDnMp1WDPf+UyhSlxVvnfAKRyXIGi6jpMQluXVvx/waX4MdqgJMfyn3cQ6tFH4YiZCX6kdNNWjJp5lYxmhRdqWRznCB1vxuWIfXCc9eUT8Kz0Houmw/S8HR11ApNoxopbalC23wdTa9ZXlJdC4bXElfdC8HHwjTcNezDN9mc+4e+WdaKkbuYZljP\n ATUIN_DB_URI: AgCfm2AisGVBlMrOqPvMWOor0e0UXDruZnWVG3klrfSzbtZfrzYF4x+sY7fVLsfUY3RSRF84m13hIJPBxhiO3pFPAs6e6zm5GH7B+8Iem1ijIXWNVW5oc7h/Kas77k1h+TcJTVyZ4gL52oqzZM3cwAX0UdE/enNrvYWoeTsJ0UMbNw3bKZ9Ll0BPfdirdHT8Ve7jMzaDF+d11difPOhyZ7wgK3ykzOGu9G8LbzJ8IwUYYFK/1DETYU76XC/d79tUOwSYxGwf88/r2zjn9ZFA7rnzzEnV7ECR33fSoRJALZMyHMUOp8cxa1rYGPrBRyHhivdhhUnyRgXqAq/oymQo4+cwBHZFSpmtEqafQ8RpuOr2ymRgrxBGfe4n4eLprzY5EUZpFRhgxonb10YL16vg/oAlWObdYkS17ZayQtsfbHBD2udjljQXrjWNIWlT6fXG8JeJth+kFewr9+2c0Rfh9sQJ+F2otBk5x+dbt5xTKppAsAEHIy9lN8/Gbh+U+woCxgP11x+w/HYX9KXDkGHcOiAteYEI7Cf2Eo1TKD7ICVTVfReETWxAzSpKMabltNuM8fuLj6dHakvkQ6PgS537ShhyGofbLQaWTB8AMpwRCIUZme6EkfZuoO2CBt8gCnL3U6geDhHUB4ZGU4g9wPL/FlIqSPaWhafwbjc+PCyXqpOMNHdXtNc7D7bAsWN1Nri3Gk1D4ae0BDTunG/SgX4rlx6zc8kGgmFtJ/cnX//RO40Om2Yf36bdeb3KgDo4Ia49EZDaH7FlRn1cwUax0Gr3Jz4=\n ATUIN_DB_USERNAME: AgAt8yDkKRjmvJtB4ecxOOcuEm1Zcoa8pX1UvtvwAAT4M18PN3JK+6yOyhHuuTwWtWphlQnAjSWx6Bu8usgIxrw9dhBCRxf4pJIaW2VmszUnn1HOtdEFcU6+40PEZ8vJEqCQz/sQoilhZyH06VYecNZFtUHleFAaEFfSGPtxd73lqpjY62fOI8yoGfd/lmXays5vjSx9kUtUVd71FYEOf7P6x+OWlFWsbQ6FepiHygoCXTiCi9umbherpIHWCMZxELja/mNdVZp2wIO+NytedM47LIy2U0FP3b6quPc1H52OK/9AK9TJf/Ke8vUaRDE6TAqv1K0fT5diD4zwERzpNoHKHhnejKj1FOCm6WVcnPHk17zy9Et+kdB+feKpgbeZlolCSJ+JgNWnM2Y3WaovQI4i4yq3ipqQDI1AgY6hHMj1HGNH8gpFjHRy/+UfPd1f4aDO6hGAbL86O2y18VcqD7gESRJ7XVWikJWpU2hIp2FAEpopoqU1QPWyTGvvC46g+gfTARIphn1EzjKymdc4ICb8Viuy/B1oVuwFaD7y9FnNx3tPP4cSuODiG2u6q0j/UTMkAftGqPZUNu3yfkrJHziKUnGc9kuasgAFJKXL2qJuG4VBxNPwTmp2VnJiBysvUb1JTTYd+2uEu4woGmzVfm/9kjkP1rbRk+hAUj5fyW2Nebds9dgD2gXZ2yGOK/S1G0TXnriSQA==\ntemplate:\n metadata:\n creationTimestamp: null\n name: atuin-secrets\n namespace: atuin\n type: Opaque\n\n\n---\n# services.yaml\n\n---\napiVersion: v1\nkind: Service\nmetadata:\nname: atuin\nnamespace: atuin\nspec:\ntype: ClusterIP\nports:\n - port: 8888\n targetPort: 8888\nselector:\n app: atuin\n---\napiVersion: v1\nkind: Service\nmetadata:\nname: postgres\nnamespace: atuin\nspec:\ntype: ClusterIP\nports:\n - port: 5432\n targetPort: 5432\nselector:\n app: postgres\n\n---\n# storage.yaml\n\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\nname: database\nnamespace: atuin\nspec:\naccessModes:\n - ReadWriteOnce\nresources:\n requests:\n storage: 1Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\nname: atuin-config\nnamespace: atuin\nspec:\naccessModes:\n - ReadWriteOnce\nresources:\n requests:\n storage: 100Mi\nI found an old mini PC (ASUS Mini PC PN30), left in a drawer from when I thought I needed it to run a Plex media server.\nWith a sudden (unexpected) burst of motivation I decided to run a local kubernetes cluster on it.\n(In hindsight, I think I might also have been inspired to try self-hosting an RSS reader by this post. I just got distracted, by deciding to self-host using kubernetes).
\nI asked ChatGPT and Claude for help on how to set up a simple kubernetes setup at home.\nAfter some back-and-forth I landed on installing Debian with no graphical desktop environment, and then installing k3s.\nThe choice of k3s was mainly made to limit the resource requirements. The Mini PC is not exactly beefy, with an underpowered CPU and only 4GB RAM (While trying to confirm this number, I found the listing for this on Amazon and it claims that I can upgrade to 8GB RAM. I might do that at some point).
I downloaded an ISO of Debian 12 and made a bootable usb.\nI connected the Mini PC to a monitor, keyboard and mouse and booted the Debian installer from the usb stick.\nI selected graphical installer and followed the guidance.\nI did not create a root user, instead letting my own user get sudo privileges.\nI did not install a desktop environment.\nI gave it the hostname tyr.\nI made sure to select SSH, to allow access after I unplug the peripherals.
I tried accessing the mini PC over shh from my desktop.
\n\nThis did not work, but using the local IP directly works fine.
\nI really want to use the hostname.local thing, I learned that it is called mDNS, and I need a mDNS service.\nI installed Avahi, both on my desktop and on the Mini PC
k3sNow, to install k3s. Following docs at https://k3s.io/.
After a minute, the kubernetes cluster is running and I can query it from tyr
$ sudo k3s kubectl get node \nNAME STATUS ROLES AGE VERSION\ntyr Ready control-plane,master 15h v1.31.4+k3s1\nNext, I want to access it from my desktop.\nFollowing the k3s guide I copy /etc/rancher/k3s/k3s.yaml from tyr to ~/.kube/config my desktop with scp, and edit the server field to point to IP of tyr.\nI tried a lot get tyr.local to resolve instead of the IP, but as far as I can tell, kubectl is not using the mDNS stuff from above.\nHere is the last chat message (in a long back-and-forth) from o1 on why .local does not work.
\n\nA statically compiled binary often does not use the system's usual NSS (Name Service Switch) mechanisms\u2014like
\n/etc/nsswitch.confandlibnss-mdns\u2014for hostname resolution. Instead, it typically performs \"pure DNS\" lookups.That explains why:
\n\n
\n- ping tyr.local succeeds, because it honors
\nnsswitch.confand uses Avahi/mDNS.- kubectl fails on
\ntyr.local, because it bypasses your local mDNS setup and tries querying a DNS server that doesn't know about.localnames.'
ChatGPT suggest some ways to fix it, but the simplest seemed to be to just plug in the IP.
\nI made sure to go to my router and reserve the local IP address of tyr, so it does not change after a reboot or something.
And finally, I can run the following from my desktop
\n$ kubectl get node \nNAME STATUS ROLES AGE VERSION\ntyr Ready control-plane,master 44h v1.31.4+k3s1\nI have posted (very) few post to my blog over the years.\nRecently, one of the things holding me back from posting is that I can't really build it locally any more.\nI'm using Github Pages, and although it probably very easy to use, I just use it so rarely that I don't really know what is going on.
\nMy brief guide says to run bundle install, but I don't have bundle installed. I also don't know what it is.\nGithub tells me to install Jekyll and Ruby. I don't have either of them.
At work I use Python a lot, and I have created a few docs sites with MkDocs, with Material for MkDocs helping out in making everything pretty. I want to use that tool-stack instead.\nAll the content is markdown anyway, so it should not be too bad.
\nI start by cloning https://github.com/KPLauritzen/kplauritzen.github.io and opening it VSCode.
\nLet's create a justfile to document how to interact with the repo.
\ninstall:\n uv sync --all-extras\n uv run pre-commit install\n\nlint:\n uv run pre-commit run --all-files\nNone of this works yet, there is no Python project, but it is a start.
\nI set up pyproject.toml with uv init and add some packages with uv add pre-commit mkdocs mkdocs-material.
Now I just need the most basic config for MkDocs and we are ready to serve some HTML!
\n\nI can see my site locally with mkdocs serve
It's terrible, but it works! I add that as a command in the justfile\n![\"\"](\"../images/migrate-1.png\")
How little effort can I put in to make this tolerable?
\nAdd a theme to mkdocs.yml
Move index.md to posts_ so we don't start with a 404 error.
That's it, now it actually looks serviceable.\n![\"\"](\"../images/migrate-2.png\")
There is A BUNCH of improvements that could be helpful, but it is too much fun to do. I will save some of that for a rainy day.
\nFor now, I will just try to create a github workflow to publish this.
\nI already have a Github Action called jekyll.yml. Let's delete that and make a new one.
I start by stealing the basic outline from mkdocs-material.\nAfter that, I follow the guide to uv in Github Actions.
\nThis is the result:
\nname: ci \non:\n push:\n branches:\n - master \n - main\npermissions:\n contents: write\njobs:\n deploy:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - name: Configure Git Credentials\n run: |\n git config user.name github-actions[bot]\n git config user.email 41898282+github-actions[bot]@users.noreply.github.com\n - name: Install uv\n uses: astral-sh/setup-uv@v5\n - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV \n - uses: actions/cache@v4\n with:\n key: mkdocs-material-${{ env.cache_id }}\n path: .cache\n restore-keys: |\n mkdocs-material-\n - run: uv run mkdocs gh-deploy --force\nEDIT: After publishing I had some problems with my custom domain, kplauritzen.dk. Every time I ran mkdocs gh-deploy it wanted to deploy to kplauritzen.github.io instead.
I think the solution is to create a CNAME file in _posts/ as that will get picked up during the build.\nSee the docs.
TL;DR: Edit your ~/.databrickscfg file to create a profile for your service principals.
I have a CI/CD pipeline that interacts with a Databricks workspace through the Databricks CLI.\nI usually develop the pipeline locally, testing it against a sandbox Databricks workspace, authenticated as myself.
\nBut when I deploy the pipeline to the CI/CD environment, it runs as a service principal, first against a dev workspace, then against a prod workspace.
\nThere can be some issues that only appear when running as a service principal, like permissions errors or workspace configurations. And the feedback loop is too slow: I have to commit, push, wait for the pipeline to run, check the logs, and repeat.
\nI want to test the pipeline locally, authenticated as a service principal, to catch these issues earlier.
\nReading about the one million ways to authenticate to an Azure Databricks workspace is enough to give me a headache (Seriously, there are too many options).\nI have previously used environment variables to authenticate as a service principal, the various secrets in an .env file, and commenting and un-commenting as needed.\nIt is a mess, and I'm guaranteed to forget to switch back to my user account at some point.
Instead, I can use databricks profiles to store the different authentication configurations.\nIn ~/.databrickscfg, I can create a profile for each service principal, and switch between them with the --profile flag.
Here is an example of a ~/.databrickscfg file with two Service principal profiles:
[DEFAULT]\nhost = <SOME_HOST>\ntoken = <SOME_TOKEN>\n\n[project-prod-sp]\nhost = \nazure_client_id = \nazure_client_secret = \nazure_tenant_id = \n\n[project-dev-sp]\n<same setup as above>\nOf course, you should replace the placeholders with the actual values.
\nTo test what workspace and user your profile is using, you can try the following command:
\n\nThis will also show you where the authentication is coming from (because, as I mentioned above, there are too many ways to authenticate).
\nFinally, you can run your pipeline locally, using the --profile flag to specify that you want to use the service principal profile:
--profile flagIf you still want to use environment variables, you can set the DATABRICKS_CONFIG_PROFILE variable to the profile name you want to use, e.g.:
TL;DR: Use pytest-docker to create a test fixture that starts a Kafka container.
At work, we need to consume and produce messages to some queue. And one of the tools available already is Kafka.
\nBefore integrating with the existing Kafka cluster, I want to test my client code. I want to ensure that it can consume and produce messages correctly.
\nI have an existing BaseQueueService class like this:
class BaseQueueService(ABC):\n @abstractmethod\n def publish(self, message: str) -> None:\n pass\n\n @abstractmethod\n def consume(self) -> str | None:\n pass\nwith existing implementations for Azure Service Bus and an InMemoryQueue for testing business logic.
\nSo I want to create a KafkaQueueService class that implements this interface. And I want to test it, but I don't have a Kafka cluster available.
I can use pytest-docker to create a test fixture that starts a Kafka container. This way, I can test my KafkaQueueService class without needing a Kafka cluster.
This is how I did it:
\nA docker-compose.yml file to start a Kafka container:
services:\n zookeeper:\n image: 'confluentinc/cp-zookeeper:latest'\n environment:\n ZOOKEEPER_CLIENT_PORT: 2181\n ZOOKEEPER_TICK_TIME: 2000\n ports:\n - "2181:2181"\n\n kafka:\n image: 'confluentinc/cp-kafka:latest'\n depends_on:\n - zookeeper\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092\n KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT\n ports:\n - "9092:9092"\n expose:\n - "29092"\n\n init-kafka:\n image: 'confluentinc/cp-kafka:latest'\n depends_on:\n - kafka\n entrypoint: [ '/bin/sh', '-c' ]\n command: |\n "\n # blocks until kafka is reachable\n kafka-topics --bootstrap-server kafka:29092 --list\n\n echo -e 'Creating kafka topics'\n kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic testtopic --replication-factor 1 --partitions 1\n kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic input_test_topic --replication-factor 1 --partitions 1\n kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic output_test_topic --replication-factor 1 --partitions 1\n\n echo -e 'Successfully created the following topics:'\n kafka-topics --bootstrap-server kafka:29092 --list\n "\nA conftest.py file to create a test fixture that starts the Kafka container:
def check_kafka_ready(required_topics, host="localhost", port=9092):\n from confluent_kafka import KafkaException\n from confluent_kafka.admin import AdminClient\n\n try:\n admin = AdminClient({"bootstrap.servers": f"{host}:{port}"})\n topics = admin.list_topics(timeout=5)\n # Check if all required topics are present\n if all(topic in topics.topics for topic in required_topics):\n return True\n else:\n return False\n except KafkaException:\n return False\n\n\n@pytest.fixture(scope="session")\ndef kafka_url(docker_services):\n """Start kafka service and return the url."""\n port = docker_services.port_for("kafka", 9092)\n required_topics = ["testtopic", "input_test_topic", "output_test_topic"]\n docker_services.wait_until_responsive(\n check=lambda: check_kafka_ready(port=port, required_topics=required_topics),\n timeout=30.0,\n pause=0.1,\n )\n return f"localhost:{port}"\nAnd finally, a test file to test the KafkaQueueService class:
@pytest.mark.kafka\ndef test_kafka_queue_can_publish_and_consume(kafka_url):\n kafka_queue_service = KafkaQueueService(\n broker=kafka_url,\n topic="testtopic",\n group_id="testgroup",\n )\n clear_messages_from_queue(kafka_queue_service)\n\n unique_message = "hello" + str(uuid.uuid4())\n kafka_queue_service.publish(unique_message)\n\n received_message = kafka_queue_service.consume()\n assert received_message == unique_message\nNow I can test my KafkaQueueService class without needing a Kafka cluster. This even works on my CI/CD pipeline in Azure DevOps.
NOTE: The docker-services fixture starts ALL the docker services in the docker-compose.yml file.
KafkaQueueServiceThis passes the test above (and a few other tests I wrote):
\nfrom confluent_kafka import Consumer, KafkaError, Producer\n\nclass KafkaQueueService(BaseQueueService):\n def __init__(self, broker: str, topic: str, group_id: str):\n # Configuration for the producer and consumer\n self.topic = topic\n self.producer: Producer = Producer({"bootstrap.servers": broker})\n self.consumer: Consumer = Consumer(\n {\n "bootstrap.servers": broker,\n "group.id": group_id,\n "auto.offset.reset": "earliest",\n "enable.partition.eof": "true",\n }\n )\n self.consumer.subscribe([self.topic])\n\n def publish(self, message: str) -> None:\n """Publish a message to the Kafka topic."""\n logger.debug(f"Publishing message to topic {self.topic}: {message}")\n\n self.producer.produce(self.topic, message.encode("utf-8"))\n self.producer.flush()\n\n def consume(self) -> str | None:\n """Consume a single message from the Kafka topic."""\n logger.debug(f"Consuming message from topic {self.topic}")\n\n # Get the next message\n message = self.consumer.poll(timeout=20)\n if message is None:\n logger.debug("Consumer poll timeout")\n return None\n # No new message\n if message.error() is not None and message.error().code() == KafkaError._PARTITION_EOF:\n logger.debug("No new messages in topic")\n return None\n # Check for errors\n if message.error() is not None:\n raise Exception(f"Consumer error: {message.error()}")\n self.consumer.commit(message, asynchronous=False)\n return message.value().decode("utf-8")\n\n def __repr__(self) -> str:\n return f"{self.__class__.__name__}(topic={self.topic})"\nTL;DR: Create a Linux VM in the cloud, then create a docker context for it with
then build your image with
\n\nAt work, I'm using an M3 Macbook. It's a great machine, but it's not perfect.\nOne issue is that I can't always build Docker images target to linux/amd64 on it.
Recently, I had an issue where I needed to package a Python application in Docker, and one of the dependencies was pytorch.\nI suspect that is where my issue was coming from.
Building the image on Mac works fine when running it on the same machine, but when I try to run it on a Linux machine, it fails with the following error:
\n\nThis indicated that the Python binary was built for the wrong architecture. Luckily, you can specify the target architecture using\nthe --platform flag when building the image.
Unfortunately, this didn't work for me. I suspect that the pytorch dependency was causing the issue. I got the following error:
To solve this issue, I decided to build the image on a remote x86_64 Linux VM. This way, I can ensure that the image is built for the correct architecture.
\nI used an Azure Virtual Machine with an Ubuntu 24.04 image. I enabled \"Auto-shutdown\" at midnight every day to save costs.
\nAfter ssh-ing into the VM, I installed docker and ensured the user was added to the docker group.
\ncurl -fsSL https://get.docker.com -o get-docker.sh\nsudo sh get-docker.sh\nsudo usermod -aG docker azureuser\nCheck that the docker daemon is running:
\n\nNow, back on my local machine, I created a docker context for the remote VM:
\n\nNow, I can build the image using the context:
\n\nI can also enable the context for all future commands:
\n", "image": null, "date_published": "2024-10-28T00:00:00+00:00", "authors": [], "tags": null}, {"id": "https://kplauritzen.dk/2023-02-26-azure-docker-cache/", "url": "https://kplauritzen.dk/2023-02-26-azure-docker-cache/", "title": "Caching Docker images in Azure DevOps Pipelines", "content_html": "TL;DR: Go to the bottom of the post to see the full Pipeline template.
In the Data Science team at DFDS, we are using Azure DevOps Pipelines to build and deploy our models.\nWe are using Docker containers to package our models, and we are using Azure Pipelines for our CI/CD.
\nFor most projects we will build the docker images in:
\nStep 1 usually happens more than once, as issues with a PR will often require multiple iterations of reviews and fixes.\nFor this reason, it is important that the build time is as short as possible. Long feedback loops are not good for productivity.
\nSo the solution is to cache the docker images between builds. Azure Pipelines even has a Cache task that claims to help with caching docker builds.\nBut the commands listed on that documentation page have never worked for me.
\nMy brilliant friend Morten Hels came up with a solution that works.\nI'm taking the liberty of writing it down here, but he is the one who deserves the credit.
\nInstead of using docker save and docker load for (attempting to) make cached docker layers available, we use docker buildx to build the image from, and save to, a cache.
The commend to run is:
\ndocker buildx create --name builder --driver docker-container --use #1\ndocker buildx build \\ \n --cache-from=type=local,src=docker_cache \\ #2\n --cache-to=type=local,dest=docker_cache,mode=max \\ #3\n --file Dockerfile \\ \n --output=type=docker,name=myimage \\ #4\n .\n--cache-from and --cache-to options available. I'm using the docker-container driver, but there are other options available. This one is just the easiest to set up, both locally and in a pipeline.Here is a complete pipeline template that you can use in your own pipelines.
\nparameters:\n - name: docker_image_name\n type: string\n displayName: 'The name of the Docker image to build. Example: klaur-testing.'\n - name: additional_docker_build_args\n type: string\n default: ''\n displayName: 'Additional arguments to pass to the docker build command. Example: --build-arg SOME_ARG=some_value.'\n - name: dockerfile_path\n type: string\n default: 'Dockerfile'\n displayName: 'The path to the Dockerfile to use. Example: Dockerfile.'\n - name: docker_build_context\n type: string\n default: '.'\n displayName: 'The path to the directory to use as the build context. Example: .'\n\nsteps:\n - task: Cache@2\n displayName: Cache Docker layers\n inputs:\n key: '"docker" | "$(Agent.OS)" | "${{ parameters.docker_image_name }}" | ${{ parameters.dockerfile_path }}'\n restoreKeys: |\n "docker" | "$(Agent.OS)" | "${{ parameters.docker_image_name }}"\n path: $(Pipeline.Workspace)/docker_cache\n\n - script: |\n docker buildx create --name builder --driver docker-container --use\n docker buildx build \\\n --cache-from=type=local,src=$(Pipeline.Workspace)/docker_cache \\\n --cache-to=type=local,dest=$(Pipeline.Workspace)/docker_cache,mode=max \\\n --file ${{ parameters.dockerfile_path }} \\\n --output=type=docker,name=${{ parameters.docker_image_name }} \\\n ${{ parameters.additional_docker_build_args }} ${{ parameters.docker_build_context }}\n displayName: Build Docker image\n env:\n DOCKER_BUILDKIT: 1\nIf the above yaml is saved in a templates.yaml file, you can use it in your pipeline like this:
jobs:\n - job: BuildDockerImage\n steps:\n - template: templates.yaml\n parameters:\n docker_image_name: 'my-image'\n additional_docker_build_args: '--build-arg SOME_ARG=some_value'\n dockerfile_path: 'Dockerfile'\n docker_build_context: '.'\ndocker buildx.I'm working in a team of data scientists, and most of us don't have a \"proper\" software background. Most here have some sort of natural sciences education and have picked up machine learning and software development along the way.\nThis means that we don't have the same software craftmanship foundation to build from when our ML models need to grow, scale, and change.
\nThere is a lot of ways to improve in this area, but a simple one to implement for a whole team in one go is to require pre-commit installed in all projects. This is a tool that lets you define a set of checks that are performed on your code every time you make a commit in git (you are using git, right?).
Make (or copy from below) a file called .pre-commit-config.yaml and place it in the root of your repository.\nThen
Every time you git commit the hooks you have defined in .pre-commit-config.yaml will be run on the changed files.
If for some reason you want to run the hooks on all files (for instance in your CI/CD) pipeline, you can do
\n\n- repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v3.2.0\n hooks:\n - id: end-of-file-fixer\n - id: trailing-whitespace\n- repo: https://github.com/pycqa/isort\n rev: 5.8.0\n hooks:\n - id: isort\n name: isort\nThe two first hooks fixes small whitespace mistakes. Each file should end with just a newline, and there should be no whitespace at the end of a line.
\nisort sorts your import statements. It is a minor thing, but it will group imports into 3 groups:
There is some setup needed to make it compatible with black. See Full setup for details.
- repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v3.2.0\n hooks:\n - id: check-ast\n - id: check-json\n - id: check-yaml\n - id: debug-statements\n - id: detect-aws-credentials\n args: [--allow-missing-credentials]\n - id: detect-private-key\n - id: check-merge-conflict\n - id: check-added-large-files\n args: ['--maxkb=3000']\nHere is a bunch of hooks that will
\nCheck if your Python code is valid (avoiding those SyntaxErrors that sometimes crop up)
Check that json and yaml files can be parsed
\nCheck that you don't have any leftover breakpoint() statements from a debugging session.
Check that you haven't accidentally committed secrets.
\nCheck that you haven't committed an unresolved merge conflict, like leaving
\n\nin the file.
\nCheck that you haven't committed an unusally large file. If you actually need large files inside your repo, use git-lfs.
\nnbstripout is very useful if you commit a lot of Jupyter Notebooks to your repo. The output cells are saved in the file, so if you are outputting some large plots, each notebook can become quite big.\nIf your notebooks are not just one-off explorations, but you come back to them more than once, this will make the PR diffs much easier to read.
If that is NOT the case, maybe you don't want or need this one.
\n- repo: https://github.com/psf/black\n rev: 21.7b0\n hooks:\n - id: black\n- repo: https://gitlab.com/pycqa/flake8\n rev: 3.7.9\n hooks:\n - id: flake8\n additional_dependencies:\n - flake8-unused-arguments\nblack is a code autoformatter. It has opinions on what is good style and bad, and I mostly agree with those opinions. The very cool thing about black is that it does not just find instances where you are not following the style, it can automatically fix your code to follow the style.
flake8 is a linter. It can check more kinds style errors, but it will not fix anything. It can only complain. This is mostly fine, because it is often trivial to fix the issues that flake8 raises.
Both of these tools needs some config to work as desired. See Full setup for details.
\n- repo: https://github.com/pre-commit/mirrors-mypy\n rev: v0.782\n hooks:\n - id: mypy\n args: [--ignore-missing-imports]\nYou can optionally do static typing in Python now.\nmypy is a tool to run static analysis on your python files and it will complain if you are inputting or return types that don't match your typehints.
If you just want to copy my setup, add these three files to the root of your repo:
\nrepos:\n- repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v3.2.0\n hooks:\n - id: check-ast\n - id: check-json\n - id: check-yaml\n - id: debug-statements\n - id: detect-aws-credentials\n args: [--allow-missing-credentials]\n - id: detect-private-key\n - id: check-merge-conflict\n - id: check-added-large-files\n args: ['--maxkb=3000']\n- repo: https://github.com/pre-commit/mirrors-mypy\n rev: v0.782\n hooks:\n - id: mypy\n args: [--ignore-missing-imports]\n- repo: https://github.com/pycqa/isort\n rev: 5.8.0\n hooks:\n - id: isort\n name: isort\n- repo: https://github.com/psf/black\n rev: 21.7b0\n hooks:\n - id: black\n- repo: https://gitlab.com/pycqa/flake8\n rev: 3.7.9\n hooks:\n - id: flake8\n additional_dependencies:\n - flake8-unused-arguments\n- repo: https://github.com/kynan/nbstripout\n rev: 0.5.0\n hooks:\n - id: nbstripout\n[tool.black]\nline-length = 100\ninclude = '\\.pyi?$'\nexclude = '''\n/(\n \\.git\n | \\.hg\n | \\.mypy_cache\n | \\.tox\n | \\.venv\n | _build\n | buck-out\n | build\n | dist\n)/\n'''\n\n[tool.isort]\nprofile = "black"\nline_length = 100\n[flake8]\nignore = E203, E266, E501, W503\nmax-line-length = 100\nmax-complexity = 18\nselect = B,C,E,F,W,T4,B9,U100\nunused-arguments-ignore-abstract-functions = True\nflake8-unused-arguments.TL;DR: Implement __array__(), __len__() and __getitem__() methods on your dataclass.\nSee the final section for a working example.
I have gotten increasingly interested in python typehints,\nand in a recent project I'm creating a lot of custom types to create interfaces for different modules in my application.\nI usually try to keep the types as standardlib python types, but the dataclass can be pretty neat.
Here is an example of a simple custom dataclass
\n\nIf I want a simple way to convert this to a numpy array, I run into a few stumbling blocks:
import numpy as np\n\np = Point2D(x=0.2, y=3.0)\narr = np.array(p)\nprint(arr, type(arr), arr.dtype)\n# Point2D(x=0.2, y=3.0) <class 'numpy.ndarray'> object\nI don't get the values from Point2D, I just get an array with the object inside.\nHowever, we can implement an __array__ method on Point2D that will allow numpy to produce an array with the correct dtype.
Now we get a much more sensible result when converting
\np = Point2D(x=0.2, y=3.0)\narr = np.array(p)\nprint(arr, type(arr), arr.dtype)\n# [0.2 3. ] <class 'numpy.ndarray'> float64\nThe trouble comes when we want to make a new custom type that inherits from Point2D.
__array__ methodLet's make a simple extension of Point2D to 3 dimensions
If we try to convert this into a numpy array, we run into trouble
\np = Point3D(x=0.2, y=3.0, z=-1.0)\narr = np.array(p)\nprint(arr, type(arr), arr.dtype)\n# [0.2 3. ] <class 'numpy.ndarray'> float64\nWe are missing the new z dimension!
\nOne fix is to make a new __array__ method.
@dataclass\nclass Point3D(Point2D):\n ...\n def __array__(self):\n return np.array([self.x, self.y, self.z])\nThat will definitely work, but it breaks the DRY principle.\nInstead, we can make use of dataclasses.astuple
from dataclasses import astuple\n\n@dataclass\nclass Point2D:\n x: float\n y: float\n def __array__(self):\n return np.array(astuple(self))\n\n@dataclass\nclass Point3D(Point2D):\n z: float\n\np = Point3D(x=0.2, y=3.0, z=-1.0)\narr = np.array(p)\nprint(arr, type(arr), arr.dtype)\n# [ 0.2 3. -1. ] <class 'numpy.ndarray'> float64\nLess repetition and less chance of mistakes. Nice.
\nOur next issue is when dealing with more than one instance of these custom classes at a time.
\nIf I have a few Points, I might want a 2D np.array with all the values. The naive approach would be to do
p1 = Point3D(1, 2, 3)\np2 = Point3D(4, 5, 6)\nlist_of_points = [p1, p2] \narr = np.array(list_of_points)\nprint(arr, type(arr), arr.dtype, arr.shape)\n# [Point3D(x=1, y=2, z=3) Point3D(x=4, y=5, z=6)] <class 'numpy.ndarray'> object (2,)\nNot only do I not get what I expected, I even get a bunch of warnings from numpy that this is a no-go
\n<input>:3: FutureWarning: The input object of type 'Point3D' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'Point3D', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.\n<input>:3: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\nWe already know we can get a numpy array from a single instance, so we can get around this hurdle with a simple list comprehension
\narr = np.array([np.array(p) for p in list_of_points])\nprint(arr, type(arr), arr.dtype, arr.shape)\n# [[1 2 3]\n# [4 5 6]] <class 'numpy.ndarray'> int32 (2, 3)\nThat works, but it feels more like a workaround than a real solution. Should I really have to remember to do this nested conversion every time I want to get my data in a 2D matrix?
\nNo, if I just implement two additional methods on the base class, I don't have to think about this any more.
\n__len__ and __getitem__from dataclasses import dataclass, astuple\nimport numpy as np\n\n@dataclass\nclass Point2D:\n x: float\n y: float\n\n def __array__(self):\n return np.array(astuple(self))\n\n def __len__(self):\n return astuple(self).__len__()\n\n def __getitem__(self, item):\n return astuple(self).__getitem__(item)\n\n@dataclass\nclass Point3D(Point2D):\n z: float\n\np1 = Point3D(1, 2, 3)\np2 = Point3D(4, 5, 6)\nlist_of_points = [p1, p2] \narr = np.array(list_of_points)\nprint(arr, type(arr), arr.dtype, arr.shape)\n# [[1 2 3]\n# [4 5 6]] <class 'numpy.ndarray'> int32 (2, 3)\nWe are again abusing dataclass.astuple to let us access each class variable programatically, in order.
To be honest, I don't really understand why __array__ does not work for lists of custom dataclasses,\nbut __len__ and __getitem__ does.\nIf numpy is looping through each element one at a time to add it to an array,\nwe might run into some performance issues at some point.
But, for now, this looks fairly clean for my taste and it is very practical.
", "image": null, "date_published": "2021-08-11T00:00:00+00:00", "authors": [], "tags": null}, {"id": "https://kplauritzen.dk/2019-02-27-thistextdoesnotexist/", "url": "https://kplauritzen.dk/2019-02-27-thistextdoesnotexist/", "title": "This text does not exist", "content_html": "TL;DR: I stole a thing.
I stole samples from the recent OpenAI language model, generating text without human input. This continues the excellent work of many others.
\nMany thanks to Peter Ericson for helping me bodge some javascript together for this project. I have absolutely no idea what I'm doing, so anything that remotely works is entirely Peters fault.
\nSee the thing here and we also bought thistextdoesnotexist.com because that seemed reasonable at the time.
", "image": null, "date_published": "2019-02-27T00:00:00+00:00", "authors": [], "tags": null}, {"id": "https://kplauritzen.dk/2018-06-13-worldcup/", "url": "https://kplauritzen.dk/2018-06-13-worldcup/", "title": "Predicting the 2018 FIFA World Cup", "content_html": "TL;DR: Check the predictions at the bottom of the page, and see the code on GitHub.
The FIFA World Cup 2018 is starting tomorrow in Russia.\nI joined a competition at work to predict the final score of all the games of the tournament, but since I don't know anything about football (and do know something about machine learning), I thought it would be fun to build a simple model to do the predictions.
\nI want to guess the final score of all matches. There is partial credit for just guessing the match winner, so if it is not possible to guess the final score I want to err towards guessing the winner.
\nI see two main challenges:
\nI have to predict how the national teams will perform. Each team usually plays less than 10 games a year, so there is not a lot of data to go on.
\nI have to predict scores instead of just outcomes. So this is not a classification problem, but instead a regression problem. I'm not so confident in that class of machine learning problems, so this will be a fun exercise.
\nI will be using the team ratings from the game FIFA 18 as my training features. They are available for both international and club games and they are updated every week.\nFor each team there are ratings for attacking ('ATT'), midfield ('MID'), defending ('DEF') and overall ('OVR').\nMY hypothesis is that if I find a pattern in how many goals a 85 ATT teams scores against a 74 DEF team, this will hold for both clubs and international teams. In this way I will be able to train on many games from for example premier league.
\nAdditionally, I will include the implied odds from bookmaker odds for each match outcome. I hope this will make it more likely to predict the correct winner as this also gives some points in my office tournament.
\nI found FIFA team ratings at FIFA index. This was not available as a download, so I had to scrape the ratings.
\nThe final scores of matches for several European leagues are available at Football-data.co.uk. Here the data is available as a .csv file, so this couldn't be easier. It even includes bookmaker odds!
The odds from the world cup games was downloaded and scraped from Odds Checker.
\nThe fixtures for the world cup was downloaded from Fixture Download.
\nI wanted to use this as an opportunity to use luigi for handling the data dependencies. This is definitely overkill for a simple project like this, but it was still a good testing grounds.\nTo illustrate, here is a portion of the final dependency graph for creating the training dataset.
\n![\"luigi\"](\"../images/luigi.png\")
You can check out the code at my GitHub.
\nTo summarize the training data creation:
\nFor a certain league (e.g. Premier League) I download the weekly team ratings pages from FIFA Index, using urllib.
I extract the ratings for each team with beautifulsoup4.
I download the results of all games in the league from football-data. This dataset includes bookmaker odds. I use the Bet365 odds.
\nI merge the ratings for home and away teams onto the results, using pandas. I merge on team name and date.
To avoid having to account for home field advantage (which won't be a thing at the world cup - except for Russia), I add all matches AGAIN with home and away team reversed. This should make home and away teams symmetric.
\nRepeat steps 1-5 for other leagues if desired. It could be beneficial to include lower tier leagues to have examples of ratings like the lower tier national teams.
\nI've not really explored the possibilities very carefully here. I've trained a Ridge regression in scikit-learn with home AND away goals as the targets.\nIt works the best of all the methods I tried with a mean absolute error of about 0.9. So on average my predictions are about 1 goal off the correct answer.
The data gathering setup is very similar to the process described above for club games.
\nOne issue is that the predictions are not integers but floats. How do you interpret a prediction of 1.43 against 1.18? Should that be rounded to 1-1? How about 0.6 against 1.4? Is it still a 1-1 if I predict one team to have 0.8 goals advantage?
\nI think there is a smart answer to this, maybe using the bookmaker odds to decide if this should be a draw or if one team winning is more likely.
\nI haven't figured this out, and I would love some more input. Please reach out if you have an idea.
\nNo more beating around the bush. Here are my final predictions:
\n| team_home | goals_home | goals_away | team_away |\n|--------------|------------|------------|--------------|\n| Russia | 2 | 1 | Saudi Arabia |\n| Egypt | 0 | 2 | Uruguay |\n| Morocco | 1 | 1 | Iran |\n| Portugal | 1 | 2 | Spain |\n| France | 2 | 1 | Australia |\n| Argentina | 2 | 1 | Iceland |\n| Peru | 1 | 1 | Denmark |\n| Croatia | 2 | 1 | Nigeria |\n| Costa Rica | 1 | 1 | Serbia |\n| Germany | 2 | 1 | Mexico |\n| Brazil | 2 | 1 | Switzerland |\n| Sweden | 1 | 1 | South Korea |\n| Belgium | 2 | 1 | Panama |\n| Tunisia | 1 | 2 | England |\n| Colombia | 2 | 1 | Japan |\n| Poland | 1 | 1 | Senegal |\n| Russia | 1 | 1 | Egypt |\n| Portugal | 2 | 1 | Morocco |\n| Uruguay | 2 | 0 | Saudi Arabia |\n| Iran | 1 | 2 | Spain |\n| Denmark | 2 | 1 | Australia |\n| France | 2 | 1 | Peru |\n| Argentina | 2 | 1 | Croatia |\n| Brazil | 2 | 0 | Costa Rica |\n| Nigeria | 1 | 1 | Iceland |\n| Serbia | 1 | 1 | Switzerland |\n| Belgium | 2 | 1 | Tunisia |\n| South Korea | 1 | 2 | Mexico |\n| Germany | 2 | 1 | Sweden |\n| England | 2 | 1 | Panama |\n| Japan | 1 | 1 | Senegal |\n| Poland | 1 | 1 | Colombia |\n| Uruguay | 1 | 1 | Russia |\n| Saudi Arabia | 1 | 2 | Egypt |\n| Iran | 1 | 2 | Portugal |\n| Spain | 2 | 1 | Morocco |\n| Denmark | 1 | 2 | France |\n| Australia | 1 | 1 | Peru |\n| Nigeria | 1 | 2 | Argentina |\n| Iceland | 1 | 2 | Croatia |\n| Mexico | 1 | 1 | Sweden |\n| South Korea | 1 | 2 | Germany |\n| Serbia | 1 | 2 | Brazil |\n| Switzerland | 1 | 1 | Costa Rica |\n| Japan | 1 | 1 | Poland |\n| Senegal | 1 | 1 | Colombia |\n| Panama | 1 | 1 | Tunisia |\n| England | 1 | 1 | Belgium |\nThere is a very nice paper by Andreas Groll et al., doing the same thing but better and with much more rigor.
\nAnd this great tweet in response to the paper:
\n\n\nYour first mistake was using fifa rankings...they are an absolute horrible indicator of a countrys actual ability
\n
Marko Bilal (@markobilal) on Twitter
\nOf course, the ever interesting FiveThirtyEight have also tried their hand at predicting the world cup. They also publish their methodology and their data. How can you not love them?
", "image": null, "date_published": "2018-06-13T00:00:00+00:00", "authors": [], "tags": null}]}