No public path to production — private GKE, SSO access, and GitOps

The control plane and the database were treated as one access problem, not two. It would have been easy to scope this as a Kubernetes project and leave Cloud SQL for later, and that framing is exactly how platforms end up with one hardened surface and one forgotten one. The two are the same problem wearing different clothes: an administrative interface that stayed public because closing it would have inconvenienced somebody. So both were closed in the same piece of work. The GKE control plane's external endpoint was disabled and administrative access moved to its private endpoint, which is Google's own recommendation for IP-based control-plane access. Cloud SQL was configured for private-IP connectivity and then had its public IP removed outright, leaving it reachable only from the connected VPC and the private paths approved into it. What came out the other side is a boundary that is consistent rather than partial — private nodes, a private control plane, a private-IP-only database, and no direct public administrative route into any of it. Consistency matters more than it sounds here, because a boundary with one exception in it is not a boundary, it is a habit of documenting where the exception is.

NetBird was chosen over Connect Gateway because the requirement was wider than kubectl. GKE Connect Gateway was the obvious candidate and would have solved the Kubernetes half cleanly — authenticated kubectl access to a private cluster with no VPN and no bastion. The reason it was not chosen is that the team's real operational need did not stop at the Kubernetes API. Engineers periodically needed direct, controlled connectivity to Cloud SQL to investigate data-level problems, and Connect Gateway is built to reach registered Kubernetes clusters; it does not provide a general private network path to other resources sitting in the VPC. Choosing it would have meant standing up one access mechanism for the cluster and a second, different one for the database — two enrollment flows, two policy models, two sets of assumptions about who has what, and two things to review whenever someone joins or leaves. NetBird covered both from a single encrypted route into the VPC: Kubernetes administration, controlled database debugging, and any other private service explicitly approved later. The general principle is worth naming, because it comes up constantly in platform decisions: the narrower tool is often the better tool right up until the second requirement arrives, and the cost of a second access mechanism is paid every single time access is granted, audited, or revoked.

SSO decided who could connect, and policy decided what they could reach. NetBird was integrated with the organization's existing SSO provider, so engineers authenticate with the identity they already have rather than a VPN credential issued separately and distributed by hand. That distinction is the whole point of doing it this way. Shared or separately managed VPN credentials drift immediately — they outlive the people they were issued to, they get copied into password managers and chat threads, and offboarding depends on somebody remembering a system that is not connected to anything else. Tying the VPN to SSO means access follows the identity: revoke the account and the private route closes with it, no separate action required. Beyond authentication, NetBird groups and access policies were designed around specific destinations rather than granting every peer a view of the whole VPC. Reaching the private GKE endpoint and reaching Cloud SQL are separate grants, given to approved users and devices, because being on the VPN should establish that you are who you say you are — not that everything inside the network is now yours to talk to.

The result is an IAM-plus-network model, where neither half is sufficient alone. It is worth being precise about what the private endpoint actually bought, because "we made it private" is often where the explanation stops. None of the existing authorization was removed or weakened: Google Cloud IAM and Kubernetes RBAC still independently authorize every action against the API server, and database authentication still applies after a network connection to Cloud SQL is established. What changed is that they are no longer the first line — they are the second. An engineer now authenticates through SSO, NetBird verifies the user and admits the authorized device, NetBird policy determines which private resources that device may reach, and only then do IAM and RBAC decide what the request is permitted to do. For Kubernetes access specifically, this means valid Google Cloud credentials are no longer sufficient on their own. A leaked service account key or a compromised session, which previously would have been usable from anywhere on the internet, now also requires an authorized private route to a control plane that no longer answers the public internet at all. Two boundaries, failing independently, is a materially different proposition from one boundary doing all the work.

The existing pipeline moved behind the boundary before anything was redesigned. Before Argo CD entered the picture, releases were deployed with Helm from GitHub Actions, and the moment the public control-plane endpoint was disabled that workflow could no longer reach the Kubernetes API. The tempting move at that point is to redesign the delivery model and cut over in one step — which turns a network change and an architectural change into a single event where any failure is ambiguous. Instead the deployment job was updated to join the NetBird network, giving the hosted runner a temporary authorized path to the private endpoint, after which it authenticated to Google Cloud and ran the same Helm deployment it always had. Nothing about the release process changed, and public cluster access was never reopened to keep it working. This was explicitly a bridge rather than a destination: a push-based pipeline still needs cluster credentials and direct network connectivity to the API server, and the fact that the connectivity was now private and short-lived improved the situation without resolving it. But it meant the team crossed the network boundary while their delivery process still worked exactly as they knew it, and took on the GitOps migration afterwards as a separate, reversible decision.

Argo CD removed direct cluster access from routine deployments entirely. Argo CD was deployed inside GKE and configured against the application's existing Helm definitions — the charts the team had already written and tested were retained rather than replaced, so the migration was a change in who applies the manifests, not a rewrite of how the application is described. GitHub Actions kept doing continuous integration: validating changes, building the application, publishing the release artifact, and recording the desired version in Git. From there the flow inverts. Argo CD, running inside the cluster, notices the difference between what Git declares and what is actually running, renders the Helm configuration, and reconciles the change from the inside — pulling rather than being pushed to. Because the reconciliation happens within the cluster, a GitHub-hosted runner no longer needs to join the VPN or hold Kubernetes credentials to ship a release. The two properties this produces are the ones that make GitOps worth the migration: Git becomes the auditable source of truth for what is deployed, so the answer to "what changed and who approved it" is a commit history rather than a pipeline log; and drift becomes visible and correctable, because anything that alters live resources out of band is reported as a difference against the declared state and reconciled according to the configured sync policy. NetBird stayed in place as the administrative path for engineers and for the exceptional operations that genuinely need a human at the cluster — but the routine path, the one that runs many times a week, stopped requiring privileged access at all.