Another K8ssandra operator reconciliation error

Philip_Boyarchenko · June 27, 2023, 11:00am

I have this issue in a few clusters both on 1.5.2 and 1.7.0. Upgrade didn’t help

2023-06-27T10:57:06.482Z        INFO    Auth is disabled: skipping Reaper user secret reconciliation    {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra"}
2023-06-27T10:57:06.483Z        INFO    Reconciling Medusa user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra"}
2023-06-27T10:57:06.483Z        INFO    Medusa user secrets successfully reconciled     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "MedusaCassandraUserSecretRef": {"name":"cassandra-medusa"}}
2023-06-27T10:57:06.483Z        INFO    Reconciling replicated secrets  {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra"}
2023-06-27T10:57:06.483Z        INFO    new metrics endpoint is available, so MCAC can be disabled      {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "serverVersion": "4.0.7"}
2023-06-27T10:57:06.483Z        INFO    Initial token computation could not be performed or is not required in this cluster     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "error": "cannot compute initial tokens: at least one DC has num_tokens >= 16"}
2023-06-27T10:57:06.483Z        INFO    Medusa reconcile for dc1 on namespace default   {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.483Z        INFO    Medusa is enabled       {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.484Z        INFO    Reconciling Medusa configMap on namespace : default     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.484Z        INFO    Medusa ConfigMap successfully reconciled        {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.484Z        INFO    Couldn't find medusa-restore init container     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.484Z        INFO    Couldn't find medusa container  {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.484Z        INFO    Vector Agent ConfigMap successfully reconciled  {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.487Z        INFO    Reconciling seeds       {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:06.487Z        INFO    Updating datacenter     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-27T10:57:16.510Z        ERROR   Failed to update datacenter     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": "", "error": "Internal error occurred: failed calling webhook \"vcassandradatacenter.kb.io\": failed to call webhook: Post \"https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=10s\": context deadline exceeded"}
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcileDatacenters
        /workspace/controllers/k8ssandra/datacenters.go:170
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcile
        /workspace/controllers/k8ssandra/k8ssandracluster_controller.go:143
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).Reconcile
        /workspace/controllers/k8ssandra/k8ssandracluster_controller.go:91
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234
2023-06-27T10:57:16.510Z        DEBUG   events  Internal error occurred: failed calling webhook "vcassandradatacenter.kb.io": failed to call webhook: Post "https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=10s": context deadline exceeded {"type": "Warning", "object": {"kind":"K8ssandraCluster","namespace":"default","name":"cassandra","uid":"34397e08-17a4-4da9-a201-b93e2a8c6a76","apiVersion":"k8ssandra.io/v1alpha1","resourceVersion":"214548399"}, "reason": "Reconcile Error"}
2023-06-27T10:57:16.536Z        INFO    updated k8ssandracluster status {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "K8ssandraCluster": "default/cassandra"}
2023-06-27T10:57:16.536Z        ERROR   Reconciler error        {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "112798e6-c3b1-4069-9190-27696ab6cd79", "error": "Internal error occurred: failed calling webhook \"vcassandradatacenter.kb.io\": failed to call webhook: Post \"https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=10s\": context deadline exceeded"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234

I get this in output

kubectl get k8c
NAME        ERROR
cassandra   Internal error occurred: failed calling webhook "vcassandradatacenter.kb.io": failed to call webhook: Post "https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=10s": context deadline exceeded

And I can’t update neither cluster definition not medusa schedule. Please help.

Philip_Boyarchenko · June 28, 2023, 9:39am

Cluster config is quite simple. It worked on one cluster and failing on 3 others.

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: cassandra
spec:
  auth: false
  cassandra:
    serverVersion: "4.0.7"
    storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: standard-rwo
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    config:
      jvmOptions:
        heap_initial_size: 4G
        heap_max_size: 4G
    networking:
      hostNetwork: true
    datacenters:
    - metadata:
        name: dc1
      size: 3
      racks:
      - name: rack1
        nodeAffinityLabels:
          topology.kubernetes.io/zone: us-central1-a
      - name: rack2
        nodeAffinityLabels:
          topology.kubernetes.io/zone: us-central1-b
      - name: rack3
        nodeAffinityLabels:
          topology.kubernetes.io/zone: us-central1-c
      tolerations:
      - key: "app"
        operator: "Equal"
        value: "cassandra"
        effect: "NoSchedule"

alexander · June 28, 2023, 10:17am

Hi,

this isn’t related to the K8ssandraCluster manifest you’re applying but rather to the k8ssandra-operator installation. Currently it looks like there’s an issue with the validation webhook from cass-operator which takes too long to answer back.
You should check if cass-operator is up and running first, and check for errors in its logs.
Then you can check the deployed validation webhooks with kubectl get validatingwebhookconfigurations.
It should display at least these 3:

NAME                                                  WEBHOOKS   AGE
cass-operator-validating-webhook-configuration        1          19m
cert-manager-webhook                                  1          23m
k8ssandra-operator-validating-webhook-configuration   2          19m

Describe the cass-operator one to see if everything looks good:

kubectl describe validatingwebhookconfigurations cass-operator-validating-webhook-configuration

Then you can try to modify the timeout (although 10s is already a lot):

kubectl edit validatingwebhookconfigurations cass-operator-validating-webhook-configuration

Change the timeoutSeconds value at the end to your liking.

If it still times out with 30s for example, you may want to check network policies.
It’s possible that they could be blocking the calls from the API server to the webhook.

Philip_Boyarchenko · June 28, 2023, 10:54am

kubectl get validatingwebhookconfigurations

NAME                                                                WEBHOOKS   AGE
cert-manager-webhook                                                1          124d
k8ssandra-operator-cass-operator-validating-webhook-configuration   1          124d
k8ssandra-operator-validating-webhook-configuration                 2          124d

Not sure if k8ssandra-operator-cass-operator-validating-webhook-configuration is identical to cass-operator-validating-webhook-configuration.
kubectl describe validatingwebhookconfigurations k8ssandra-operator-cass-operator-validating-webhook-configuration

Name:         k8ssandra-operator-cass-operator-validating-webhook-configuration
Namespace:
Labels:       app.kubernetes.io/instance=k8ssandra-operator
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=cass-operator
              app.kubernetes.io/part-of=k8ssandra-k8ssandra-operator-k8ssandra-operator
              helm.sh/chart=cass-operator-0.41.0
Annotations:  cert-manager.io/inject-ca-from: k8ssandra-operator/k8ssandra-operator-cass-operator-serving-cert
              meta.helm.sh/release-name: k8ssandra-operator
              meta.helm.sh/release-namespace: k8ssandra-operator
API Version:  admissionregistration.k8s.io/v1
Kind:         ValidatingWebhookConfiguration
Metadata:
  Creation Timestamp:  2023-02-23T16:19:11Z
  Generation:          5
  Resource Version:    214544865
  UID:                 676ac861-5566-4731-b27e-5586a9b10082
Webhooks:
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURYakNDQWthZ0F3SUJBZ0lSQU83b0xORGdiQUFINTRXaUdESENGVzB3RFFZSktvWklodmNOQVFFTEJRQXcKQURBZUZ3MHlNekEyTWpjeE1EVXhNek5hRncweU16QTVNalV4TURVeE16TmFNQUF3Z2dFaU1BMEdDU3FHU0liMwpEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUM2S3QxRG5OOVorUmdYUVNJOEJvb3hLZW9KalcwbUNSQUx1cFovCmJmUzJtWWIyVVJZdUFsU0NRVS9VaW1ETXFPZ3QrM1d3L2YzWitsY1piWnRzUnUxa01tb0ErTW9pZ01xOWFkbGoKalQ0YklxNWlXSkxaaXhjdSttTEhNY25MWHRYUittMjVybVhydUtybk5uQWNVOTNxKzhNS3hHS0FDZldJVjFMSAp5SjVJSDZwbSthY3pWK0Yydm9uclJIVXU1TGMwQVVZenN3UGVJakszNnVZeTcvT3krQkRxUUxCUlYzMEpRb3FkCnhncms1RlVoVm9QdEtBdlhMUkZvc2lkVzVhSkZLZ2IzcTBrRTNKaU51SXRDNGtZNkxET01wTktVZTFWV1VtcXoKM2J3S2JPZVd4T1pheFV1dXJpOUJ1ZUZRcFFIVzR3VjYveERVUjNYNXFFaDM4dHZmQWdNQkFBR2pnZEl3Z2M4dwpEZ1lEVlIwUEFRSC9CQVFEQWdXZ01Bd0dBMVVkRXdFQi93UUNNQUF3Z2E0R0ExVWRFUUVCL3dTQm96Q0JvSUpICmF6aHpjMkZ1WkhKaExXOXdaWEpoZEc5eUxXTmhjM010YjNCbGNtRjBiM0l0ZDJWaWFHOXZheTF6WlhKMmFXTmwKTG1zNGMzTmhibVJ5WVMxdmNHVnlZWFJ2Y2k1emRtT0NWV3M0YzNOaGJtUnlZUzF2Y0dWeVlYUnZjaTFqWVhOegpMVzl3WlhKaGRHOXlMWGRsWW1odmIyc3RjMlZ5ZG1salpTNXJPSE56WVc1a2NtRXRiM0JsY21GMGIzSXVjM1pqCkxtTnNkWE4wWlhJdWJHOWpZV3d3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUJtYmNjRTdOanErRnptanY2QVUKbGw0a0dKK3ROZVBNdEVneWdRNHUxQS93OGpJa2pJbEU5S3ZET3F2R1FJR1dKSDYyOTd4VkFLdmJmL2JkejhRSwo1OFEvNHJzS0MyZlJkUGcwMzFuNlpuek1VSDZMRG45WFdpRU1CbzRGbXRSQUIwSHpreVdrMlpVbkNXSGEzUTVOCkNMRVd6dVBpajdXK3laN2RFN01wTmR6bU0ycEl6SWNmZVEyNkNjblNEdTVlb0JERVFHclltLzZUQTFNWGdDWDAKbzJEVXRaUVVkNnA2bVFMdnRNTnZlckJkVnlhMnhweVFxVmYwbzBxVURnZzlxUUszQ002Rk1vMWZzZ0tHbi9ldgpmQ3VvVm5UeW4vNldpZFVqY3ZOMmtTZjhRRUZFcHQ3L1lvMW9Vd1VwZ3VVVE9YTXNZVU5XODRpUlFMVVRuQTlGCmYwaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    Service:
      Name:        k8ssandra-operator-cass-operator-webhook-service
      Namespace:   k8ssandra-operator
      Path:        /validate-cassandra-datastax-com-v1beta1-cassandradatacenter
      Port:        443
  Failure Policy:  Fail
  Match Policy:    Equivalent
  Name:            vcassandradatacenter.kb.io
  Namespace Selector:
  Object Selector:
  Rules:
    API Groups:
      cassandra.datastax.com
    API Versions:
      v1beta1
    Operations:
      CREATE
      UPDATE
    Resources:
      cassandradatacenters
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
Events:             <none>

Then testing but still see 10s timeout

# kubectl edit k8c cassandra
error: k8ssandraclusters.k8ssandra.io "cassandra" could not be patched: Internal error occurred: failed calling webhook "vk8ssandracluster.kb.io": failed to call webhook: Post "https://k8ssandra-operator-webhook-service.k8ssandra-operator.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s": context deadline exceeded

Philip_Boyarchenko · June 29, 2023, 11:24am

Thank you @alexander for taking a look. I verified there are no network policies

# kubectl get NetworkPolicy --all-namespaces
No resources found

k8ssandra-operator-8b6656dc9-9dpb8 sends these logs every 20 mins or so. same as I pasted above

2023-06-29T11:11:49.636Z        INFO    Auth is disabled: skipping Reaper user secret reconciliation    {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra"}
2023-06-29T11:11:49.636Z        INFO    Reconciling Medusa user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra"}
2023-06-29T11:11:49.636Z        INFO    Medusa user secrets successfully reconciled     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "MedusaCassandraUserSecretRef": {"name":"cassandra-medusa"}}
2023-06-29T11:11:49.636Z        INFO    Reconciling replicated secrets  {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra"}
2023-06-29T11:11:49.636Z        INFO    new metrics endpoint is available, so MCAC can be disabled      {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "serverVersion": "4.0.7"}
2023-06-29T11:11:49.636Z        INFO    Initial token computation could not be performed or is not required in this cluster     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "error": "cannot compute initial tokens: at least one DC has num_tokens >= 16"}
2023-06-29T11:11:49.637Z        INFO    Medusa reconcile for dc1 on namespace default   {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.637Z        INFO    Medusa is enabled       {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.637Z        INFO    Reconciling Medusa configMap on namespace : default     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.638Z        INFO    Medusa ConfigMap successfully reconciled        {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.638Z        INFO    Couldn't find medusa-restore init container     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.638Z        INFO    Couldn't find medusa container  {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.638Z        INFO    Vector Agent ConfigMap successfully reconciled  {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.642Z        INFO    Reconciling seeds       {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:11:49.642Z        INFO    Updating datacenter     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": ""}
2023-06-29T11:12:19.670Z        ERROR   Failed to update datacenter     {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra", "CassandraDatacenter": "default/dc1", "K8SContext": "", "error": "Internal error occurred: failed calling webhook \"vcassandradatacenter.kb.io\": failed to call webhook: Post \"https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=30s\": context deadline exceeded"}
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcileDatacenters
        /workspace/controllers/k8ssandra/datacenters.go:170
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcile
        /workspace/controllers/k8ssandra/k8ssandracluster_controller.go:143
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).Reconcile
        /workspace/controllers/k8ssandra/k8ssandracluster_controller.go:91
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234
2023-06-29T11:12:19.671Z        DEBUG   events  Internal error occurred: failed calling webhook "vcassandradatacenter.kb.io": failed to call webhook: Post "https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=30s": context deadline exceeded {"type": "Warning", "object": {"kind":"K8ssandraCluster","namespace":"default","name":"cassandra","uid":"34397e08-17a4-4da9-a201-b93e2a8c6a76","apiVersion":"k8ssandra.io/v1alpha1","resourceVersion":"216331008"}, "reason": "Reconcile Error"}
2023-06-29T11:12:19.690Z        INFO    updated k8ssandracluster status {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "K8ssandraCluster": "default/cassandra"}
2023-06-29T11:12:19.691Z        ERROR   Reconciler error        {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cassandra","namespace":"default"}, "namespace": "default", "name": "cassandra", "reconcileID": "cbf644f2-ea0d-4ed3-b28b-6a2a060d496c", "error": "Internal error occurred: failed calling webhook \"vcassandradatacenter.kb.io\": failed to call webhook: Post \"https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=30s\": context deadline exceeded"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234

k8ssandra-operator-cass-operator-6f4886b7b6-7sg57 logs are fine without errors.
I updated both timeouts in k8ssandra-operator-validating-webhook-configuration to 30 sec and now request fails in 30 sec…
I tested URL availability from default namespace

curl -v https://k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter
*   Trying 10.6.32.98:443...
* Connected to k8ssandra-operator-cass-operator-webhook-service.k8ssandra-operator.svc (10.6.32.98) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self-signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self-signed certificate

It’s reachable

alexander · June 29, 2023, 1:03pm

if it fails with both 10s and 30s, it makes me think of a firewall issue. Maybe the firewall is blocking requests from the Kube api server (which should be on the control plane) to the cass-operator pod.

I don’t see any other valid reason for what you’re seeing but network shenanigans.

Philip_Boyarchenko · June 29, 2023, 1:28pm

Thank you very much for pointing this out. The solution was to add firewall rule in GKE to allow port 443 and 9443 for internal subnets. I was under impression google cloud doesn’t block traffic from control plane but I was wrong.

Topic		Replies	Views
Medusa (bad) credentials errors K8ssandra help	0	780	September 15, 2021
Standalone Medusa crashing K8ssandra help	2	247	September 15, 2023
K8ssandra-operator v1.1 release	0	121	May 17, 2022
K8ssandra operator reconciliation error K8ssandra help	2	428	June 26, 2023
Welcome to the K8ssandra Community Welcome and Announcements	8	1355	April 11, 2024

K8ssandra

Another K8ssandra operator reconciliation error

Related topics