I spend all day at work exploring the inside of the k8s sausage factory so I’m inured to the horrors and can fix basically anything that breaks. The way k8s handles ingress and service discovery makes it absolutely worth it to me. The fact that I can create an HTTPProxy and have external-dns automagically expose it via DNS is really nice. I never have to worry about port conflicts, and I can upgrade my shit whenever with no (or minimal) downtime, which is nice for smart home stuff. Most of what I run tends to be singleton statefulsets or single-leader deployments managed with leases, and I only do horizontal for minimal HA, not at all for perf. If something gives me more trouble running in HA than it does in singleton mode then it’s being run as a singleton.
k8s is a complex system with priorities that diverge from what is ideal for usage at home, but it can be really nice. There are certain things that just get their own VM (Home Assistant is a big one) because they don’t containerize/k8serize well though.
Yup, same here. being able to skip all the networking and DNS hassle and have it automated for you is so nice.
Having databases fully managed with cnpg is AMAZING
I just have renovate set to auto update my argocd, so everything just runs itself with zero issues. Only the occasional stateful container that has breaking changes in a minor version.
If something OOMs or crashes, it all just self heals, I never need to worry about it. I don’t have any HPAs (nor cluster scaling obv), though I do have some HA stuff set up just to reduce restart times and help keep the databases happy.
The main issue with Kubernetes is that a lot of self-hosted software makes bad design decisions that actively make kubernetes harder, eg sqlite instead of postgres and secrets stored in large config files. The other big issue is that documentation only supports docker compose and not kubernetes 90% of the time so you have to know how to write yaml and read documentation.
Moving my hass from a statefulset to kubevirt sounds tempting. Did you have better reliability/ergonomics? I have been looking into moving my Hass automation to NodeRed, so that I can GitOps it all, since NodeRed supports git syncing.
I spend all day at work exploring the inside of the k8s sausage factory so I’m inured to the horrors and can fix basically anything that breaks. The way k8s handles ingress and service discovery makes it absolutely worth it to me. The fact that I can create an
HTTPProxyand have external-dns automagically expose it via DNS is really nice. I never have to worry about port conflicts, and I can upgrade my shit whenever with no (or minimal) downtime, which is nice for smart home stuff. Most of what I run tends to be singleton statefulsets or single-leader deployments managed with leases, and I only do horizontal for minimal HA, not at all for perf. If something gives me more trouble running in HA than it does in singleton mode then it’s being run as a singleton.k8s is a complex system with priorities that diverge from what is ideal for usage at home, but it can be really nice. There are certain things that just get their own VM (Home Assistant is a big one) because they don’t containerize/k8serize well though.
Yup, same here. being able to skip all the networking and DNS hassle and have it automated for you is so nice.
Having databases fully managed with cnpg is AMAZING
I just have renovate set to auto update my argocd, so everything just runs itself with zero issues. Only the occasional stateful container that has breaking changes in a minor version.
If something OOMs or crashes, it all just self heals, I never need to worry about it. I don’t have any HPAs (nor cluster scaling obv), though I do have some HA stuff set up just to reduce restart times and help keep the databases happy.
The main issue with Kubernetes is that a lot of self-hosted software makes bad design decisions that actively make kubernetes harder, eg sqlite instead of postgres and secrets stored in large config files. The other big issue is that documentation only supports docker compose and not kubernetes 90% of the time so you have to know how to write yaml and read documentation.
Moving my hass from a statefulset to kubevirt sounds tempting. Did you have better reliability/ergonomics? I have been looking into moving my Hass automation to NodeRed, so that I can GitOps it all, since NodeRed supports git syncing.