Multicluster Part Deux

This is part two of this series. Check part one for some context on what’s happening here.

The Github repo containing files discussed or shown in this post can be found here.

Assumptions and Requirements

Before beginning, I’ll go over some requirements and I’m also going to make some assumptions about you, the reader, and the environment you’re working from.

Assumptions

I’m going to first assume that you’ve got a base level of knowledge and experience with GKE, Istio, and Kubernetes in general.

Requirements

You’ll need two GKE clusters for this, preferably in different regions. In this blog post, I’m using clusters in us-east1 and us-central1. These clusters must be running Kubernetes 1.21+ and will need the below-listed items completed before proceeding.

• HTTP Load Balancing, Workload Identity, and Node Auto Provisioning need to be enabled on the clusters. These can be enabled via the GCP UI or IaC options.
• I used private GKE clusters for this project and as such there are some firewall changes to make as mentioned here in the For private GKE clusters box.
  • I have Control plane global access enabled on my clusters which seems to render the section on Multicluster communication moot in my case. If you experience issues you should follow those instructions to see if they remedy them.
• You’ll also need to install istioctl which is available here and also via Homebrew.

Setting Up Multicluster Istio Service Mesh

Requirements

First, create an istio-system namespace in both clusters so our istio resources will have a place to live, kubectl create ns istio-system.

Multicluster Istio has its own requirements before installing/configuring multicluster. The most important part of this is the certs. Ideally, you have access to a CA so you can issue intermediate certs for each of your clusters so workload traffic can be authenticated.

Since this is a PoC I’m just using the Istio plugin certs made with the scripts in their repo.

Simply follow the directions to create the kube context remote secrets and get the certs installed.

Installing Istio

I find the easiest way to install and manage istio is via the operator. Per the Istio documentation it’s not the preferred way to install/manage Istio but it’s not a deprecated method of installation and in my experience, if something seems off with Istio it’s an easy thing to simply kick the istio operator pod and let it check, and possibly correct, all associated resources when it comes up. I also find the operator easier to use when upgrading istio.

I’m using the multi-primary install here so both of my clusters reside in the same VPC but not subnet or region. I’ve prepared Istio CRDs for each regional cluster. Both have the necessary settings detailed in the multicluster documentation along with a lot of other settings which are detailed in the operator options and global mesh options documentation.

So, starting with the cluster in us-central1 and the corresponding CRD we’ll install the istio operator with istioctl operator init -f istio-operator-1-14-1.yaml -r 1-14-1. I’m using the -r flag here to tag this install with the revision 1-14-1. I use revision tags as it makes upgrading istio easier. I’ll cover upgrades in another post.

The installation will take a couple of minutes to settle. It will install all the needed things and it will provision an internal layer-4 network load balancer for the istio eastwest-gateway. I chose internal because well, this should be an internal-only load balancer in this instance.

If you need an external load balancer here because your clusters are in different VPCs with no direct route between them for whatever reason, you’d need to remove the service annotation here and I’d recommend you at least use an authorization policy to restrict ingress. However, with properly configured route tables, firewall/sec group rules, or even a VPN you can likely make internal load balancers talk across different VPCs or even other cloud providers.

Lastly, we need to apply the cross-network-gateway in the istio-system namespace. This exposes all services, (*.local), on the east-west gateway in both clusters. This gateway file exists in both directories in the repo and needs to be applied to both clusters.

Now repeat the same operator init in the us-east1 cluster using the istio operator CRD for us-east1 and make sure you create the gateway too. If you’ve followed everything thus far, the remote kube contexts, the certs, and applied the CRDs in the right clusters you should now have both clusters using the same service mesh. You can verify if everything is working correctly by following these steps.

Setting up Multicluster Ingress

With our multicluster service mesh up and running we’ll now install our multicluster ingress. First, we need a global IP address. You can create this with the following gcloud command; gcloud compute addresses create your-cool-ip-name --global. We’ll only refer to the IP address in code by the name you gave it but if you prefer to know the IPv4 address you can run gcloud compute addresses list to see it.

Now you’ll follow the instructions to register your clusters to a fleet and specify a config cluster

Let’s start by creating our ingress. Edit the annotation on line 8 in the multi-cluster-ingress file with the name of the global IP you created earlier. You can also edit the TLS secret name on line 17 if you want a TLS cert associated with the lb, LetsEncrypt is a good option for this. Now you can kubectl apply this file and in a couple of minutes, the load balancer will be up.

Here’s where we stray from the documentation to make this work specifically with our istio multicluster mesh. GKE multicluster ingress assumes you’ll make a multicluster service for every service in your config cluster that you want to be associated with this load balancer. Istio typically handles all ingress traffic through its istio-ingressgateway service and pod(s). To funnel traffic from the load balancer into these pods we need to create a multi-cluster-service that points to them and it needs to look almost exactly like the existing istio-ingressgateway service as seen here. You can kubectl apply this file as is.

Lastly, we’ll want to apply the ingress-frontendconfig.yaml and ingress-backendconfig.yaml to both clusters. The former redirects all inbound traffic on the load balancer to port 443 and the latter establishes node group health checks on our load balancer.

Verifying External Traffic Hits Both Clusters

In each regional directory is a verify.yaml file which, when applied, creates all the necessary pieces for you to test both the ingress and the multi-cluster service mesh. Apply the manifest to both clusters, open your IP in a browser and you should see the Nginx page which will give you the pod name where the traffic was routed and if you tick the radio box to refresh the page constantly you’ll see that the pod name changes because your requests are being routed to both clusters over the eastwest-gateway.

Consider checking out Locality Load Balancing for handling service issues in your mesh.

Routing External Traffic to a Specific Cluster

Since I run a Prometheus stack in my cluster I also run Alertmanager. Alertmanager operates in what’s called a gossip cluster meaning that all Alertmanager instances are (should be) in communication with each other so that alerts aren’t duplicated and that HA instances are aware if one of them stops responding.

Because of GCPs multi-cluster ingress feature of always routing requests to the geographically closest cluster this creates issues with alertmanager. The resolution here is to give each alertmanager instance it’s own DNS record and then incorporate destination rule subsets. This will still utilize the east-west gateway but allows them to be reached statically.

Istio CRD Options Explained

Here I’ll discuss some of the more opaque settings in the istio CRD files. The majority of settings can be grokked via the documentation and the files are almost identical save for the geographical specific settings however some are specific to GKE, issues I’ve encountered, or what I consider best practice for Istio.

I’ll start with the below snippet. Here I’m setting a 100% sample rate for traces and pointing it to my Jaeger installation. I think tracing is an important tool in Kubernetes clusters and when using Istio, creating a stack of Jaeger, Kiali, Grafana, and Prometheus can help you debug issues in your mesh, with your services, visualize metrics and data and also help you monitor and alert on them.

tracing:
  sampling: 100 # Default is 1%
    zipkin:
      address: "jaeger-collector:9411"

Now’s a good time to link information on federating Prometheus in a multicluster setup.

The below is sort of self-explanatory thanks to the comment but what isn’t clear is the telemetry resource I mention and how it relates to this.

extensionProviders:
  # This was required along with an istio telemetry resource to see stackdriver logs in GCP
  - name: stackdriver
    stackdriver:
      maxTagLength: 256

I included the telemetry file in the repo. It’s worth noting that out of the box, Istio will work with Prometheus but this file is needed if you want logs/metrics to go to stackdriver in GCP. We needed to specify this so we could test out alerting and SLOs in GCP.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  # no selector specified, applies to all workloads
  metrics:
  - providers:
    - name: stackdriver
    - name: prometheus

One difference in the files is an annotation on the istio-ingressgateway in us-central1.

cloud.google.com/neg: '{"exposed_ports": {"443": {"name": "us-central1"}}}'

The above differs from its us-east1 counterpart and does so because I’ve set us-east1 as the config cluster. As such, us-central1 needs its own NEG (Network Endpoint Group) which is how ingress delineates this as a separate cluster for node-group health checking. I’m only exposing 443 here since our redirect makes 80 useless and I’ve given it a self-explanatory name.

Finally, you’ll notice that all of the Istio resources, save for the operator, have a corresponding pdb, hpa, nodeSelector, toleration, and pod antiAffinity configured. I put everything on preemptive nodes for cost savings and I ensure that these pods are always available via the aforementioned configurations.

Conclusion

So there you have it, multicluster service mesh with multicluster ingress in GKE. Now that I’ve written it all out it doesn’t seem that difficult to do but when I started looking into istio in GKE in a single cluster with an ingress there wasn’t much documentation on how to make it all work that wasn’t just “use Anthos”. I spent a few weeks trying and failing to get the right mix of things to work so I hope this helps you down whatever reliability path you’re on.

If you have questions about this post, DM me on Twitter.