This is Part 1 of a two part series.
As part of the SRE team at Harness, the goal is to help increase stability and reliability for our org while keeping in mind the need to scale as we grow. Here I’ll explain where we were with our platform, how we changed some things, for the better, in our move from AWS to GCP, and how we think we can help improve things here at Harness.
A Brief History of the ZeroNorth Platform
ZeroNorth was acquired by Harness in November 2021 and I joined six months before to help bring an aging, poorly constructed monolith of a platform into the now. Running EKS version 1.14 with no autoscaling, no service mesh, manually applied manifests/resources, insufficient monitoring and alerting. Saying it was in bad shape is putting it lightly.
The plan we formulated was to spin up a new cluster on the latest version of EKS, by establishing IaC with Terraform, adding in ArgoCD, Istio, and the other services we saw as standard for EKS clusters, e.g. Autoscaler, ExternalDNS, Prometheus Stack, Grafana, Kiali, and Jaeger.
The result was a completely overhauled platform that allowed our devs to push and, if necessary, roll back releases on their own with monitoring and alerting tools that enabled us to respond to issues without being notified first by customers.
My Time at Harness So Far
Harness is heavily invested in both GCP and Kubernetes and this was my, as well as my co-workers, first foray into GCP. While getting situated in our new environment, we quickly noticed areas where we could lend our expertise in helping scale Harness to the next level.
The current platform, concerning infrastructure, is perfectly cromulent and has served Harness well over time. However, I don’t believe it is sufficient to take us to the next level.
Architecting a Better Platform
I took on the idea of presenting a PoC for a multi-regional platform utilizing a feature from each, Istio and GCP. The intent is to have multiple clusters in different regions, and potentially globally, sharing traffic internally as a single istio service mesh with ingress being managed by GCP’s multicluster ingress feature. The hypothetical result is a platform that exists across regions and if a cluster failed or a region disappeared, failover would be automatic and platform outages greatly reduced.
Istio Multicluster
Using the Istio service mesh in your Kubernetes cluster has inherent benefits and when you incorporate its ability to connect multiple, geographically dispersed clusters in a single service mesh you set yourself up with the ability to create highly available services.
Istio, in a multicluster mesh configuration, makes use of an east/west load balancer between all clusters in its mesh. The gateway resource attached to this load balancer uses TLS certs for authentication. Clusters communicate with each other over these east/west load balancers by making API calls to the other clusters. This requires each cluster to have the kube context of all other clusters loaded into a secret via the istioctl x create-remote-secret command.
GKE Multicluster Ingress
GKE’s multicluster ingress creates a globally available, layer 7 http/s load balancer with an Anycast VIP. It uses a single IP address, performs proximity-based routing, monitors the clusters in its fleet for health, and handles failover automatically. Latency is reduced via the Anycast VIP which is created in every PoP (point-of-presence) and routed accordingly as pictured below.
GKE Multicluster Ingress Limitations
Multi-cluster ingress has some requirements and limitations as well as default quotas to keep in mind. Below are the ones of concern.
You can only deploy one instance of Multi Cluster Ingress per project.
You must turn off mTLS on your application because mTLS is not supported for L7 external load balancers. Note that this does not interfere with mTLS inside the istio mesh but would prevent mTLS with external clients.
SNI is not supported by GCLB.
Node Auto-Provisioning
The final part of this equation is utilizing Node Auto Provisionsing(NaP) to automatically create and scale node groups on our behalf as needed as well as right-sizing node types based on resource requests and limits. While spinning up our new clusters we noticed NaP which allowed us to skip making node groups via IaC and simply let GCP spin them up for us based on the requests of what’s scheduled in the cluster. Further, by adding nodeSelector to our resources we’re able to get things scheduled onto pre-emptive or spot nodes. With tolerations, NaP will automatically taint nodes for us. This means we can provision nodes as needed via YAML.
Conclusion
With the above components, I believe we can achieve a platform that scales on-demand, is highly available, and, fault-tolerant. With multi-mesh and ingress, we can also add on other clusters globally as needed which thanks to the Anycast routing, can reduce latency to customers.
In the next post, I’ll detail my PoC along with code, gotchas, and issues I’ve experienced.