Unable to create a cluster in GKE

danmatulescu · January 10, 2024, 2:12pm

Hello,

I am trying to install k8ssandra in a GKE cluster, version 1.26.6.
The environment I have doesn’t have access to the helm repo or docker hub, so what I did was to pull all the required images into the private registry and locally downloaded all the required helm charts for k8ssandra-operator v1.11

Operator pods are starting fine:

k8ssandra-operator-7c49d4796d-rh4r7 1/1 Running 0 175m
k8ssandra-operator-cass-operator-c79846f7-dtc4q 1/1 Running 0 175m

2024-01-10T11:12:39.895Z 2024-01-10T11:12:39.896Z I0110 11:12:40.948817 2024-01-10T11:12:42.306Z 2024-01-10T11:12:44.718Z 2024-01-10T11:12:44.719Z 2024-01-10T11:12:44.719Z 2024-01-10T11:12:44.719Z 2024-01-10T11:12:44.719Z 2024-01-10T11:12:44.725Z 2024-01-10T11:12:44.725Z 2024-01-10T11:12:44.725Z 2024-01-10T11:12:44.726Z 2024-01-10T11:12:44.726Z 2024-01-10T11:12:44.726Z 2024-01-10T11:12:44.726Z 2024-01-10T11:12:44.726Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.727Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.728Z 2024-01-10T11:12:44.832Z 2024-01-10T11:12:44.832Z 2024-01-10T11:12:44.832Z 2024-01-10T11:12:44.832Z 2024-01-10T11:12:44.834Z 2024-01-10T11:12:44.834Z 2024-01-10T11:12:44.835Z 2024-01-10T11:12:44.835Z 2024-01-10T11:12:44.835Z 2024-01-10T11:12:44.835Z INFO setup watch namespace configured {“namespace”: “k8ssandra-operator”}
INFO setup watch namespace configured {“namespace”: “k8ssandra-operator”}
1 request.go:690] Waited for 1.027297832s due to client-side throttling, not priority and fairness, request: GET:https://240.12.20.1:443/apis/networking.istio.io/v1beta1?timeout=32s
INFO controller-runtime.metrics Metrics server is starting to listen {“addr”: “:8080”}
DEBUG Finished initializing 0 client configs
INFO controller-runtime.builder Registering a mutating webhook {“GVK”: “k8ssandra.io/v1alpha1, Kind=K8ssandraCluster”, “path”: “/mutate-k8ssandra-io-v1alpha1-k8ssandracluster”}
INFO controller-runtime.webhook Registering webhook {“path”: “/mutate-k8ssandra-io-v1alpha1-k8ssandracluster”}
INFO controller-runtime.builder Registering a validating webhook {“GVK”: “k8ssandra.io/v1alpha1, Kind=K8ssandraCluster”, “path”: “/validate-k8ssandra-io-v1alpha1-k8ssandracluster”}
INFO controller-runtime.webhook Registering webhook {“path”: “/validate-k8ssandra-io-v1alpha1-k8ssandracluster”}
INFO controller-runtime.builder skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn’t called {“GVK”: “medusa.k8ssandra.io/v1alpha1, Kind=MedusaBackupSchedule”}
INFO controller-runtime.builder Registering a validating webhook {“GVK”: “medusa.k8ssandra.io/v1alpha1, Kind=MedusaBackupSchedule”, “path”: “/validate-medusa-k8ssandra-io-v1alpha1-medusabackupschedule”}
INFO controller-runtime.webhook Registering webhook {“path”: “/validate-medusa-k8ssandra-io-v1alpha1-medusabackupschedule”}
INFO controller-runtime.webhook Registering webhook {“path”: “/mutate-v1-pod-secrets-inject”}
INFO setup starting manager
INFO Starting server {“path”: “/metrics”, “kind”: “metrics”, “addr”: “[::]:8080”}
INFO Starting server {“kind”: “health probe”, “addr”: “[::]:8081”}
INFO controller-runtime.webhook.webhooks Starting webhook server
INFO Starting EventSource {“controller”: “replicatedsecret”, “controllerGroup”: “replication.k8ssandra.io”, “controllerKind”: “ReplicatedSecret”, “source”: “kind source: *v1alpha1.ReplicatedSecret”}
INFO Starting EventSource {“controller”: “replicatedsecret”, “controllerGroup”: “replication.k8ssandra.io”, “controllerKind”: “ReplicatedSecret”, “source”: “kind source: *v1.Secret”}
INFO Starting Controller {“controller”: “replicatedsecret”, “controllerGroup”: “replication.k8ssandra.io”, “controllerKind”: “ReplicatedSecret”}
INFO Starting EventSource {“controller”: “stargate”, “controllerGroup”: “stargate.k8ssandra.io”, “controllerKind”: “Stargate”, “source”: “kind source: *v1alpha1.Stargate”}
INFO Starting EventSource {“controller”: “stargate”, “controllerGroup”: “stargate.k8ssandra.io”, “controllerKind”: “Stargate”, “source”: “kind source: *v1.Deployment”}
INFO Starting EventSource {“controller”: “stargate”, “controllerGroup”: “stargate.k8ssandra.io”, “controllerKind”: “Stargate”, “source”: “kind source: *v1.Service”}
INFO Starting EventSource {“controller”: “k8ssandratask”, “controllerGroup”: “control.k8ssandra.io”, “controllerKind”: “K8ssandraTask”, “source”: “kind source: *v1alpha1.K8ssandraTask”}
INFO Starting EventSource {“controller”: “medusatask”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaTask”, “source”: “kind source: *v1alpha1.MedusaTask”}
INFO Starting Controller {“controller”: “medusatask”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaTask”}
INFO Starting Controller {“controller”: “stargate”, “controllerGroup”: “stargate.k8ssandra.io”, “controllerKind”: “Stargate”}
INFO Starting EventSource {“controller”: “k8ssandracluster”, “controllerGroup”: “k8ssandra.io”, “controllerKind”: “K8ssandraCluster”, “source”: “kind source: *v1alpha1.K8ssandraCluster”}
INFO Starting EventSource {“controller”: “k8ssandracluster”, “controllerGroup”: “k8ssandra.io”, “controllerKind”: “K8ssandraCluster”, “source”: “kind source: *v1beta1.CassandraDatacenter”}
INFO Starting EventSource {“controller”: “k8ssandracluster”, “controllerGroup”: “k8ssandra.io”, “controllerKind”: “K8ssandraCluster”, “source”: “kind source: *v1alpha1.Stargate”}
INFO Starting EventSource {“controller”: “k8ssandracluster”, “controllerGroup”: “k8ssandra.io”, “controllerKind”: “K8ssandraCluster”, “source”: “kind source: *v1alpha1.Reaper”}
INFO Starting EventSource {“controller”: “k8ssandracluster”, “controllerGroup”: “k8ssandra.io”, “controllerKind”: “K8ssandraCluster”, “source”: “kind source: *v1.ConfigMap”}
INFO Starting Controller {“controller”: “k8ssandracluster”, “controllerGroup”: “k8ssandra.io”, “controllerKind”: “K8ssandraCluster”}
INFO Starting EventSource {“controller”: “k8ssandratask”, “controllerGroup”: “control.k8ssandra.io”, “controllerKind”: “K8ssandraTask”, “source”: “kind source: *v1alpha1.CassandraTask”}
INFO Starting Controller {“controller”: “k8ssandratask”, “controllerGroup”: “control.k8ssandra.io”, “controllerKind”: “K8ssandraTask”}
INFO controller-runtime.certwatcher Updated current TLS certificate
INFO Starting EventSource {“controller”: “reaper”, “controllerGroup”: “reaper.k8ssandra.io”, “controllerKind”: “Reaper”, “source”: “kind source: *v1alpha1.Reaper”}
INFO Starting EventSource {“controller”: “medusarestorejob”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaRestoreJob”, “source”: “kind source: *v1alpha1.MedusaRestoreJob”}
INFO Starting EventSource {“controller”: “reaper”, “controllerGroup”: “reaper.k8ssandra.io”, “controllerKind”: “Reaper”, “source”: “kind source: *v1.Deployment”}
INFO Starting Controller {“controller”: “medusarestorejob”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaRestoreJob”}
INFO Starting EventSource {“controller”: “medusabackupjob”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaBackupJob”, “source”: “kind source: *v1alpha1.MedusaBackupJob”}
INFO Starting Controller {“controller”: “medusabackupjob”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaBackupJob”}
INFO Starting EventSource {“controller”: “reaper”, “controllerGroup”: “reaper.k8ssandra.io”, “controllerKind”: “Reaper”, “source”: “kind source: *v1.Service”}
INFO Starting Controller {“controller”: “reaper”, “controllerGroup”: “reaper.k8ssandra.io”, “controllerKind”: “Reaper”}
INFO Starting EventSource {“controller”: “medusabackupschedule”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaBackupSchedule”, “source”: “kind source: *v1alpha1.MedusaBackupSchedule”}
INFO Starting Controller {“controller”: “medusabackupschedule”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaBackupSchedule”}
INFO Starting EventSource {“controller”: “clientconfig”, “controllerGroup”: “config.k8ssandra.io”, “controllerKind”: “ClientConfig”, “source”: “kind source: *v1beta1.ClientConfig”}
INFO Starting EventSource {“controller”: “clientconfig”, “controllerGroup”: “config.k8ssandra.io”, “controllerKind”: “ClientConfig”, “source”: “kind source: *v1.Secret”}
INFO Starting Controller {“controller”: “clientconfig”, “controllerGroup”: “config.k8ssandra.io”, “controllerKind”: “ClientConfig”}
INFO controller-runtime.webhook Serving webhook server {“host”: “”, “port”: 9443}
INFO controller-runtime.certwatcher Starting certificate watcher
INFO Starting workers {“controller”: “medusabackupschedule”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaBackupSchedule”, “worker count”: 1}
INFO Starting workers {“controller”: “medusatask”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaTask”, “worker count”: 1}
INFO Starting workers {“controller”: “reaper”, “controllerGroup”: “reaper.k8ssandra.io”, “controllerKind”: “Reaper”, “worker count”: 1}
INFO Starting workers {“controller”: “stargate”, “controllerGroup”: “stargate.k8ssandra.io”, “controllerKind”: “Stargate”, “worker count”: 1}
INFO Starting workers {“controller”: “medusabackupjob”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaBackupJob”, “worker count”: 1}
INFO Starting workers {“controller”: “replicatedsecret”, “controllerGroup”: “replication.k8ssandra.io”, “controllerKind”: “ReplicatedSecret”, “worker count”: 1}
INFO Starting workers {“controller”: “medusarestorejob”, “controllerGroup”: “medusa.k8ssandra.io”, “controllerKind”: “MedusaRestoreJob”, “worker count”: 1}
INFO Starting workers {“controller”: “k8ssandratask”, “controllerGroup”: “control.k8ssandra.io”, “controllerKind”: “K8ssandraTask”, “worker count”: 1}
INFO Starting workers {“controller”: “clientconfig”, “controllerGroup”: “config.k8ssandra.io”, “controllerKind”: “ClientConfig”, “worker count”: 1}
INFO Starting workers {“controller”: “k8ssandracluster”, “controllerGroup”: “k8ssandra.io”, “controllerKind”: “K8ssandraCluster”, “worker count”: 1}

Now, when I try to create a demo cluster, I get the below error:

Error from server (InternalError): error when creating “…/k8ssandra-demo.yaml”: Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.k8ssandra-operator.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded

Can you please advise on what may be missing?

Thanks,
Dan

markhley · January 25, 2024, 9:25pm

Dan - I have exact same issue. We are able to deploy same configuration in Azure AKS, but we get the same error message in our GCP GKE environment

alexander · January 26, 2024, 7:51am

Hi,

it looks like the validation webhook isn’t responding within due time, which could be related to firewall rules or networking in place.
I see in the initial post that istio is running in the cluster, which may have an impact here.
Also check that the webhook is running and responsive by trying to reach it directly using curl. Check the logs of the k8ssandra-operator pod to see if there’s any problem reported there.

Rest assured that k8ssandra-operator works fine in GKE, this is where we test it most of the time.

markhley · January 26, 2024, 8:01pm

Here are the errors that show up in Operator log

2024-01-26T18:00:43.217Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307163”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:43.247Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307833”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:43.279Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307833”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:43.318Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307833”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:43.383Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307833”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:43.484Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307833”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:43.664Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307833”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:44.007Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307833”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:44.668Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307841”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:45.970Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307841”}, “reason”: “Reconcile Error”}
2024-01-26T18:00:48.550Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: no endpoints available for service “k8ssandra-operator-webhook-service” {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307865”}, “reason”: “Reconcile Error”}
2024-01-26T18:01:03.703Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428307892”}, “reason”: “Reconcile Error”}
2024-01-26T18:01:23.970Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428308088”}, “reason”: “Reconcile Error”}
2024-01-26T18:01:54.475Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428308336”}, “reason”: “Reconcile Error”}
2024-01-26T18:02:45.462Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428308688”}, “reason”: “Reconcile Error”}
2024-01-26T18:04:17.408Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428309279”}, “reason”: “Reconcile Error”}
2024-01-26T18:07:11.273Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428310345”}, “reason”: “Reconcile Error”}
2024-01-26T18:12:48.976Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: dial tcp 10.32.6.21:9443: i/o timeout {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428312380”}, “reason”: “Reconcile Error”}
2024-01-26T18:23:54.359Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428316347”}, “reason”: “Reconcile Error”}
2024-01-26T18:40:44.386Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428324124”}, “reason”: “Reconcile Error”}
2024-01-26T18:57:34.410Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: dial tcp 10.32.6.21:9443: i/o timeout {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428335942”}, “reason”: “Reconcile Error”}
2024-01-26T19:14:24.433Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428347729”}, “reason”: “Reconcile Error”}
2024-01-26T19:31:14.455Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428359559”}, “reason”: “Reconcile Error”}
2024-01-26T19:48:04.479Z DEBUG events Internal error occurred: failed calling webhook “vk8ssandracluster.kb.io”: failed to call webhook: Post “https://k8ssandra-operator-webhook-service.xxxxx.svc:443/validate-k8ssandra-io-v1alpha1-k8ssandracluster?timeout=10s”: context deadline exceeded {“type”: “Warning”, “object”: {“kind”:“K8ssandraCluster”,“namespace”:“xxxxx”,“name”:“cassandra”,“uid”:“f549ef0d-7ec6-47c0-9e32-67df06f0224e”,“apiVersion”:“k8ssandra.io/v1alpha1",“resourceVersion”:"428371377”}, “reason”: “Reconcile Error”}

markhley · January 26, 2024, 8:44pm

Dan - is your GKE cluster setup as private? There seem to be some general issues with some webhooks and private GKE clusters

markhley · January 26, 2024, 10:12pm

Dan - issue is definitely regarding GCP GKE private cluster and I have solution. By default, the private GKE cluster has firewall rules to enable 443, but the target port for the K8ssandra webhook is 9443. You have to add firewall rule to account for this to allow access to control plane. Here is article that talks about this and steps to add the firewall rule - Creating a private cluster | Google Kubernetes Engine (GKE) | Google Cloud - My cluster is building now. Let me know if this works for you

Gongort · February 1, 2024, 6:54pm

Thanks @markhley. It helped me.

danmatulescu · February 7, 2024, 10:24am

Hello @markhley, in my case I didn’t see those errors in the k8ssandra-operator log.
@alexander I suspect the issue on my side is somewhere else, I have istio and other addons running on the cluster. Webhooks are running and responding to curl. I was able to overcome the issue by removing the ValidatingWebhookConfiguration.