nodeAffinity does not schedule pods on the required workers

K8ssandra help
1

Posting on behalf of another user:

I have added node affinity as below in CassandraDatacenter object yaml and deployed, I can that this is being added to cassandradatacenter as expected but is not replicating to cassandra statefulset and pods are not scheduling on the required nodes - can you please assist us if I am doing something wrong?

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: dc1
spec:
  clusterName: cluster1
  managementApiAuth:
    insecure: {}
  size: 3
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: standard-ssd
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node_pool
            operator: In
            values:
            - pool-n2-s64-1
  serverType: cassandra
2

First, let me point out that k8ssandra does not currently (as of v1.1.0) support configuring node affinity. It is possible though if you are working directly with cass-operator.

This example won’t work because there is no affinity property in the CassandraDatacenter spec. What version of cass-operator is being used? I ask because there were some relevant changes introduced in 1.6.0.

Prior to 1.6.0 you could use the nodeSelector property. That accepts key/value pairs that are assigned directly to the nodeSelector property in the pod template spec in the StatefulSet.

The nodeAffinityLabels was added to the CassandraDatacenter spec in 1.6.0. These labels will be added just like in the example above using the In operator. The labels will be applied to all racks. You can also specify the labels on a per rack basis.

Here is a simple example to illustrate:

cassandra:
  datacenters:
    - name: dc1
      size: 3
      racks:
        - name: r1
          affinityLabels:
            topology.kubernetes.io/zone: us-east1-b
        - name: r2
          affinityLabels:
            topology.kubernetes.io/zone: us-east1-a
        - name: r3
          affinityLabels:
            topology.kubernetes.io/zone: us-east1-c

Hope That helps!

Cheers

John

3

Hi @jsanda

Thanks for the response - we are using cass-operator version 1.6.0.
nodeSelector is working fine and this property is being assigned to pod template spec in statefulset but affinity is not being assigned to statefulset. However, as mentioned above its being added to cassandradatacenter object. Not sure, why it’s not replicating in statefulset.

4

Hey @Phani

The CassandraDatacenter spec does not have an affinity property. Does kubectl get cassandradatacenter dc1 -o yaml show the affinity property. I would not expect to see it.

5

Hi @jsanda,

I have attached kubectl get cassandradatacenter dc1 -o yaml output in the following ticket

https://community.datastax.com/questions/11304/cass-operator-nodeaffinity-is-not-working.html

Please let me know if you need more info.

6

I was incorrect earlier in thinking the nodeAffinity property would not be persisted. I thought that there was validation checks in place to prevent it.

The reason it does not modify the StatefulSet remains the same though. The CassandraDatacenter spec does not define a nodeAffinity property even though you can add it. cass-operator ignores it. It will take actions on the fields it knows about, more precisely, those declared in the CRD. You need to declare node affinity labels as I describe above.

Cheers

John