Part Two: Multi-Region Apache Cassandra on Azure Kubernetes Service with the K8ssandra Operator

system · May 11, 2022, 3:00pm

Originally published at: Part Two: Multi-Region Apache Cassandra on Azure Kubernetes Service with the K8ssandra Operator - K8ssandra, Apache Cassandra® on Kubernetes

This is the second post in a two-part series that demonstrates how the K8ssandra Operator simplifies deploying and managing a multi-region Apache Cassandra^® cluster within Azure Kubernetes Service (AKS). To learn how to set up your AKS clusters before deploying the K8ssandra Operator, check Part One.

Before installing the new K8ssandra Operator, let’s take a quick check of what we’ve accomplished so far:

Deployed four AKS clusters, each in a different geographic region
Configured each AKS cluster with non-overlapping network spaces
Provisioned a Virtual Network Gateway within each cluster’s virtual network
Configured VPN connections between each deployed gateway
Configured our local environment for access to each AKS cluster

Long story short, we’re now ready to deploy the K8ssandra Operator and build a K8ssandraCluster—the easy part!

What is a K8ssandraCluster?

A K8ssandraCluster is the new Kubernetes custom resource that the K8ssandra Operator primarily watches and reconciles. It is the definition of your K8ssandra deployment and can specify many different configuration options from the basic elements we’ll demonstrate here to the most complex Apache Cassandra configuration options. To learn more about the K8ssandraCluster check out the K8ssandra documentation.

Install cert-manager

Cert-manager has become the de facto tool for managing certificates within the Kubernetes ecosystem. Cass Operator and K8ssandra Operator have a dependency on Cert-manager, and before deploying K8ssandra Operator it must be deployed within the cluster.

We will use the cert-manager Helm Charts to easily install the tool within each of our clusters. First, we need to add the cert-manager Helm repository:

% helm repo add jetstack https://charts.jetstack.io
% helm repo update

Now, let’s start installing on each cluster.

% kubectx k8ssandra-dev-multi-test-1
Switched to context "k8ssandra-dev-multi-test-1".
% helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Thu Jan 27 15:28:59 2022
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.7.0 has been deployed successfully!

% kubectx k8ssandra-dev-multi-test-2
Switched to context "k8ssandra-dev-multi-test-2".
% helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Thu Jan 27 15:28:59 2022
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.7.0 has been deployed successfully!

% kubectx k8ssandra-dev-multi-test-3
Switched to context "k8ssandra-dev-multi-test-3".
% helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Thu Jan 27 15:28:59 2022
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.7.0 has been deployed successfully!

% kubectx k8ssandra-dev-multi-test-4
Switched to context "k8ssandra-dev-multi-test-4".
% helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Thu Jan 27 15:28:59 2022
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.7.0 has been deployed successfully!

Install K8ssandra Operator

We’re now ready to deploy the K8ssandra Operator. Before we start installing, let’s take a look at our deployment design:

K8ssandra Operator must be deployed to each Kubernetes cluster that it intends to manage resources within. One of the operators must be configured to act as the “Control Plane” while all others should be configured as “Data Planes”. The “Control Plane” serves as the management point for the cluster and it’s within that cluster that the K8ssandraCluster resource must be deployed. To learn more details about the operator, take a look at our architectural documentation.

We will use Helm to install the operator, so let’s add the K8ssandra repository.

% helm repo add k8ssandra https://helm.k8ssandra.io/stable
% helm repo update

Next, we install the control plane.

% kubectx k8ssandra-dev-multi-test-1
Switched to context "k8ssandra-dev-multi-test-1".
% helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --create-namespace

Then we install the data planes.

% kubectx k8ssandra-dev-multi-test-2
Switched to context "k8ssandra-dev-multi-test-2".
% helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --create-namespace --set controlPlane=false

% kubectx k8ssandra-dev-multi-test-3
Switched to context "k8ssandra-dev-multi-test-3".
% helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --create-namespace --set controlPlane=false

% kubectx k8ssandra-dev-multi-test-4
Switched to context "k8ssandra-dev-multi-test-4".
% helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --create-namespace --set controlPlane=false

Note that we are installing the operator in its default namespace scoped mode and into the k8ssandra-operator namespace. It’s within that namespace that we’ll find all of our resources deployed.

We now have the operator installed in each cluster. We can verify this by checking out the deployments within each cluster, for example:

% kubectl -n k8ssandra-operator get deployments
NAME                           	READY   UP-TO-DATE   AVAILABLE   AGE
k8ssandra-operator             	1/1 	1        	1       	11d
k8ssandra-operator-cass-operator   1/1 	1        	1       	11d

You’ll see here that the Helm Chart deployed both the K8ssandra Operator and its dependency, Cass Operator.

Create ClientConfig resources

K8ssandra Operator uses another Kubernetes custom resource called a ClientConfig to securely communicate between clusters over which it is expected to make deployments of the various K8ssandra components. These resources are also used to identify which Kubernetes cluster a particular Cassandra data center should be deployed to.

The K8ssandra Operator project provides a utility to generate these resources, given an existing Kubernetes context configuration. This makes it very easy to enable remote clusters within K8ssandra.

These resources represent each remote data plane cluster and are provided within the control plane cluster. So in our example here, we’ll need to create three ClientConfig resources for each of our clusters: k8ssandra-dev-test-multi-2, k8ssandra-dev-test-multi-3, and k8ssandra-dev-test-multi-4.

Let’s start with cluster 2 as an example. The utility provided by the project is called create-clientconfig.sh and can be downloaded from GitHub. In this example, we’ll download it using wget, however you could also clone the full repository or download the file via browser as well.

% wget https://raw.githubusercontent.com/k8ssandra/k8ssandra-operator/main/scripts/create-clientconfig.sh
--2022-02-26 15:47:56--  https://raw.githubusercontent.com/k8ssandra/k8ssandra-operator/main/scripts/create-clientconfig.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6503 (6.4K) [text/plain]
Saving to: ‘create-clientconfig.sh’
create-clientconfig.sh                                      	100%[=====================================================================================================================================================>]   6.35K  --.-KB/s	in 0s
2022-02-26 15:47:56 (20.8 MB/s) - ‘create-clientconfig.sh’ saved [6503/6503]
% chmod +x create-clientconfig.sh

You can check for info on how to use the script through it’s built-in help output.

% ./create-clientconfig.sh --help
Syntax: create-client-config.sh [options]
Options:
  --src-context <ctx>         	The context for the source cluster that contains the service account.
                              	This or the src-kubeconfig option must be set.
  --src-kubeconfig <cfg>      	The kubeconfig for the source cluster that contains the service account.
                              	This or the src-context option must be set.
  --dest-context <ctx>        	The context for the cluster where the ClientConfig will be created.
                              	Defaults to the current context of the kubeconfig used.
  --dest-kubeconfig <cfg>     	The kubeconfig for the cluster where the ClientConfig will be created.
                              	Defaults to /Users/k8ssandra-demo/.kube/config.
  --namespace <ns>            	The namespace in which the service account exists and
                              	where the ClientConfig will be created.
  --serviceaccount <name>     	The name of the service account from which the ClientConfig will be created.
                              	Defaults to k8ssandra-operator.
  --output-dir <path>         	The directory where generated artifacts are written.
                              	If not specified a temp directory is created.
  --in-cluster-kubeconfig <path>  Should be set when using kind cluster. This is the kubeconfig that has the
                              	internal IP address of the API server.
  --help                      	Displays this help message.

In our scenario, we’ll use the src-context and dest-context. The src-context represents the remote context and the dest-context represents the context where the ClientConfig should be deployed. This relies on these contexts being locally available within your kube config.

First, we create the ClientConfig for data plane cluster k8ssandra-dev-multi-test-2.

% ./create-clientconfig.sh --src-context k8ssandra-dev-multi-test-2 --dest-context k8ssandra-dev-mulit-test-1 --namespace k8ssandra-operator
Creating /var/folders/sv/h5nqzlxd7pj6w_lk1c74qplw0000gn/T/tmp.zJtAAXqh/kubeconfig
Creating secret k8ssandra-dev-multi-test-2-config
secret "k8ssandra-dev-multi-test-2-config" deleted
secret/k8ssandra-dev-multi-test-2-config created
Creating ClientConfig /var/folders/sv/h5nqzlxd7pj6w_lk1c74qplw0000gn/T/tmp.zJtAAXqh/k8ssandra-dev-multi-test-2.yaml
clientconfig.config.k8ssandra.io/k8ssandra-dev-mulit-test-2 created

Then we repeat for clusters 3 and 4.

% ./create-clientconfig.sh --src-context k8ssandra-dev-multi-test-3 --dest-context k8ssandra-dev-mulit-test-1 --namespace k8ssandra-operator
Creating /var/folders/sv/h5nqzlxd7pj6w_lk1c74qplw0000gn/T/tmp.zJtAAXqh/kubeconfig
Creating secret k8ssandra-dev-multi-test-3-config
secret "k8ssandra-dev-multi-test-3-config" deleted
secret/k8ssandra-dev-multi-test-3-config created
Creating ClientConfig /var/folders/sv/h5nqzlxd7pj6w_lk1c74qplw0000gn/T/tmp.zJtAAXqh/k8ssandra-dev-multi-test-3.yaml
clientconfig.config.k8ssandra.io/k8ssandra-dev-mulit-test-3 created

% ./create-clientconfig.sh --src-context k8ssandra-dev-multi-test-4 --dest-context k8ssandra-dev-mulit-test-1 --namespace k8ssandra-operator
Creating /var/folders/sv/h5nqzlxd7pj6w_lk1c74qplw0000gn/T/tmp.zJtAAXqh/kubeconfig
Creating secret k8ssandra-dev-multi-test-4-config
secret "k8ssandra-dev-multi-test-4-config" deleted
secret/k8ssandra-dev-multi-test-4-config created
Creating ClientConfig /var/folders/sv/h5nqzlxd7pj6w_lk1c74qplw0000gn/T/tmp.zJtAAXqh/k8ssandra-dev-multi-test-2.yaml
clientconfig.config.k8ssandra.io/k8ssandra-dev-mulit-test-4 created

All systems go!

At this point we are now ready to deploy a K8ssandraCluster. Let’s look back at what we’ve done so far:

Installed cert-manager within each cluster
Installed K8ssandraOperator within each cluster—either in control plane or data plane, mode depending on the cluster
Created and installed ClientConfig resources into the control plane cluster representing each data plane cluster

All that’s left to do now is create and deploy a K8ssandraCluster! Here’s the example we’ll be deploying.

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: demo
spec:
  Cassandra:
    serverVersion: 4.0.1
    storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: managed-premium
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
    config:
      jvmOptions:
        heapSize: 512M
    networking:
      hostNetwork: true
    datacenters:
      - metadata:
           name: dc1
        k8sContext: k8ssandra-dev-multi-test-2
        size: 3
        stargate:
          size: 1
          heapSize: 256M
      - metadata:
           name: dc2
        k8sContext: k8ssandra-dev-multi-test-3
        size: 3
        stargate:
          size: 1
          heapSize: 256M
      - metadata:
           name: dc3
        k8sContext: k8ssandra-dev-multi-test-4
        size: 3
        stargate:
          size: 1
          heapSize: 256M

You can learn more about the complete spec and options available within a K8ssandraCluster from the K8ssandra documentation. Let’s take a look at our small example.

Our example will only be deploying Cassandra and Stargate. Again, this isn’t a production example, in a complete deployment we’d want to see the other supporting components of K8ssandra being configured and deployed: Medusa and Reaper. It’s also worth noting that we’re deploying with very minimal system resource requirements, very small disk allocation, and minimal memory requests.

Here are a few of the key configuration properties we’re using:

cassandra.serverVersion: "4.0.1" specifies the version of Cassandra to be deployed
storageConfig.cassandraDataVolumeClaimSpec.storageClassName: managed-premium specifies the Kubernetes storage class to be used, this particular class is relevant to Azure and will differ depending on the Kubernetes platform used
datacenters.metadata[0].name: dc1 and datacenters.metadata[0].k8sContext: k8ssandra-dev-multi-test-2 specify that we want a Cassandra data center named dc1 deployed to the Kubernetes cluster at context k8ssandra-dev-multi-test-2. This is the key piece of configuration that binds our deployment to the ClientConfig resources that we generated in the previous steps.

Let’s deploy that resource to the control plane cluster (saved to k8ssandra-cluster-example.yaml) and see what we get.

% kubectx k8ssandra-dev-multi-test-1
Switched to context "k8ssandra-dev-multi-test-1".
% kubectl apply -n k8ssandra-operator -f k8ssandra-cluster-example.yaml

It’s alive!

Once the K8ssandraCluster resource has been deployed to the control plane cluster, the operator will begin its work, reconciling the requested deployment and distributing work out to the data planes as necessary. Let’s check how the resources look once they’re deployed.

Switch over to one of the data plane clusters and take a look at the pods that will be deployed.

% kubectx k8ssandra-dev-multi-test-3
Switched to context "k8ssandra-dev-multi-test-3".
% kubectl -n k8ssandra-operator get pods
NAME                                                	READY   STATUS	RESTARTS   AGE
demo-dc2-default-stargate-deployment-69945b7bd7-5qx6f   1/1 	Running   0      	11d
demo-dc2-default-sts-0                              	2/2 	Running   0      	11d
demo-dc2-default-sts-1                              	2/2 	Running   0      	11d
demo-dc2-default-sts-2                              	2/2 	Running   0      	11d
k8ssandra-operator-58cbc55989-hpkjp                 	1/1 	Running   0      	11d
k8ssandra-operator-cass-operator-fb69d4547-5ngs9    	1/1 	Running   2      	11d

Here you’ll see the following:

k8ssandra-operator-58cbc55989-hpkjp is K8ssandra Operator
k8ssandra-operator-cass-operator-fb69d4547-5ngs9 is Cass Operator
demo-dc2-default-stargate-deployment-69945b7bd7-5qx6f is Stargate
demo-dc2-default-sts-0 -1 and -2 are the Cassandra pods (remember we asked for data centers of size 3 in our spec)

If you peek at the other clusters you’ll find similarities there as well, with slightly different naming conventions, indicating the data center that those resources are a part of.

You can also interrogate status from the K8ssandraCluster resource itself in the control plane cluster. This is handy because it provides a consolidated single point of information about the broader deployment. There is more detail on this in the K8ssandra documentation, but let’s take a quick look at that.

First, switch over to the control plane cluster.

% kubectx k8ssandra-dev-multi-test-1
Switched to context "k8ssandra-dev-multi-test-1".

Then we’ll describe the K8ssandraCluster resource and look at its status (we’ll be omitting portions of the output here because it can be quite large).

% kubectl describe k8ssandracluster demo -n k8ssandra-operator
Name:     	demo
Namespace:	k8ssandra-operator
Labels:   	<none>
Annotations:  k8ssandra.io/initial-system-replication: {"datacenters":["dc1","dc2","dc3"],"replicationFactor":3}
API Version:  k8ssandra.io/v1alpha1
Kind:     	K8ssandraCluster
...
Status:
...
  Datacenters:
	dc1:
  	Cassandra:
    	Cassandra Operator Progress:  Ready
    	Conditions:
...
      	Message:
      	Reason:
      	Status:              	True
      	Type:                	Valid
      	Last Transition Time:	2022-02-15T19:56:26Z
      	Message:
      	Reason:
      	Status:              	True
      	Type:                	Initialized
      	Last Transition Time:	2022-02-15T19:56:26Z
      	Message:
      	Reason:
      	Status:              	True
      	Type:                	Ready
    	Last Server Node Started:  2022-02-15T19:54:54Z
    	Node Statuses:
      	demo-dc1-default-sts-0:
        		Host ID:  31c944f4-b093-4f83-b2db-81a815ed42d2
      	demo-dc1-default-sts-1:
        		Host ID:  4a352e22-a8a4-48be-a92f-e5bbe2383c82
      	demo-dc1-default-sts-2:
        		Host ID:      	d74b1881-8138-4bc6-a279-9566336b406a
...
  	Stargate:
    	Available Replicas:  1
    	Conditions:
      	Last Transition Time:  2022-02-15T20:09:11Z
      	Status:            	True
      	Type:              	Ready
    	Deployment Refs:
      		demo-dc1-default-stargate-deployment
    	Progress:          	Running
    	Ready Replicas:    	1
    	Ready Replicas Ratio:  1/1
    	Replicas:          	1
    	Service Ref:       	demo-dc1-stargate-service
    	Updated Replicas:  	1
...

The trimmed output above shows the core status elements for our first data center, dc1. The same status information is available for each data center in the cluster.

From this, we can see a few key pieces of information:

The Conditions shown indicate to us that the Cassandra data center is Valid, Initialized, and Ready
We have three nodes in our datacenter as requested in our spec: demo-dc1-default-sts-0, -1, and -2
The Stargate deployment is Ready and all of the requested nodes have been deployed, Ready Replicas Ratio: 1/1

Finally, to test that our Cassandra cluster is healthy and ready to be used, let’s exec into one of the containers and use nodetool to check on the status of the cluster from the Cassandra perspective.

Let’s switch over to one of the data plane clusters.

% kubectx k8ssandra-dev-multi-test-2
Switched to context "k8ssandra-dev-multi-test-2".

Retrieve and store the super-user username and password.

% CASS_USERNAME=$(kubectl get secret demo-superuser -n k8ssandra-operator -o=jsonpath='{.data.username}' | base64 --decode)
% CASS_PASSWORD=$(kubectl get secret demo-superuser -n k8ssandra-operator -o=jsonpath='{.data.password}' | base64 --decode)

Now run the nodetool status command.

% kubectl exec --stdin --tty demo-dc1-default-sts-0 -n k8ssandra-operator -c cassandra -- nodetool -u $CASS_USERNAME -pw $CASS_PASSWORD status Datacenter: dc1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.2.1.81 186.46 KiB 16 100.0% d74b1881-8138-4bc6-a279-9566336b406a default UN 10.2.0.4 181.68 KiB 16 100.0% 4a352e22-a8a4-48be-a92f-e5bbe2383c82 default UN 10.2.0.226 182.95 KiB 16 100.0% 31c944f4-b093-4f83-b2db-81a815ed42d2 default

Datacenter: dc2

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
– Address Load Tokens Owns (effective) Host ID Rack
UN 10.3.0.4 156 KiB 16 100.0% 0466962e-ffa6-4f67-9c12-c6963c209b73 default
UN 10.3.0.115 157.19 KiB 16 100.0% 89ff35bc-b27b-4f6a-9deb-fc73b4b63133 default
UN 10.3.1.81 157.91 KiB 16 100.0% ba38385f-72f7-4802-8dc3-5e5f4efeece9 default

Datacenter: dc3

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
– Address Load Tokens Owns (effective) Host ID Rack
UN 10.4.0.115 154.87 KiB 16 100.0% 5a6efa6c-aebe-4c0d-a1dc-ecd5be9360e4 default
UN 10.4.0.226 152.53 KiB 16 100.0% b4fb1cd9-58e6-4c0f-945b-0f3da83dc0b4 default
UN 10.4.0.4 153.15 KiB 16 100.0% 6b160ae5-f2b2-4efd-a3d3-bed7000e3cb0 default

All of our expected data centers seem to be of the correct size and in the Up/Normal state – a fully functional multi-datacenter Cassandra deployment distributed across geographic regions within Azure Kubernetes Service!

Wrapping up

Let’s take a quick look back at what we accomplished in this two-part series:

Deployed a multi-region series of clusters within Azure Kubernetes Service
Established connectivity between those clusters using Azure Virtual Network Gateways
Deployed a multi-datacenter Apache Cassandra cluster across those AKS clusters using K8ssandra Operator

In the end, we established that we had a fully functional cluster that’s now ready to be developed against. But there’s much more to explore. There are other topics to consider, like ingress configuration to allow access to the cluster or backup/restore strategies using Medusa, as you continue to explore what the K8ssandra project can help you accomplish.

You can learn more about those topics and others related to K8ssandra Operator from the K8ssandra documentation. Lastly, let us know what you think in the K8ssandra Discord or ask us your questions in the K8ssandra Forum.

Topic		Replies	Views
Part One: Multi-Region Apache Cassandra on Azure Kubernetes Service with the K8ssandra Operator	0	150	April 13, 2022
Deploying to Multiple Kubernetes Clusters with the K8ssandra Operator	0	267	December 22, 2021
The Best Methods for Growing K8ssandra Clusters in a Kubernetes Environment with Multiple Regions Welcome and Announcements	0	23	September 11, 2024
Multi-Region Cassandra on EKS with K8ssandra and Kubefed	1	146	July 30, 2023
Deploying a Sample Application with Cassandra in Kubernetes	0	133	January 27, 2022

K8ssandra