Pods Start Successfully, but Stargate Fails to Start in k8ssandra

Jennifer · December 7, 2023, 1:36pm

I’m encountering an issue with k8ssandra, specifically when starting Stargate pods. The pods start successfully, but it seems that Stargate fails to initialize properly. This problem occurs in both a single-cluster setup and a multi-cluster setup.

**kubectl get all -n k8ssandra-operator**
NAME                                                        READY   STATUS             RESTARTS         AGE
pod/demo-dc1-default-stargate-deployment-6bf79bfc45-67knw   0/1     CrashLoopBackOff   13 (4m43s ago)   36m
pod/demo-dc1-default-sts-0                                  2/2     Running            0                33m
pod/demo-dc1-default-sts-1                                  2/2     Running            0                33m
pod/demo-dc1-default-sts-2                                  2/2     Running            0                33m
pod/k8ssandra-operator-74c6764fc7-ngm5z                     1/1     Running            0                93m
pod/k8ssandra-operator-cass-operator-847875b4f6-mjhkd       1/1     Running            0                93m

NAME                                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
service/demo-dc1-additional-seed-service                   ClusterIP   None            <none>        <none>                                         40m
service/demo-dc1-all-pods-service                          ClusterIP   None            <none>        9042/TCP,8080/TCP,9103/TCP,9000/TCP            40m
service/demo-dc1-service                                   ClusterIP   None            <none>        9042/TCP,9142/TCP,8080/TCP,9103/TCP,9000/TCP   40m
service/demo-seed-service                                  ClusterIP   None            <none>        <none>                                         40m
service/k8ssandra-operator-cass-operator-webhook-service   ClusterIP   10.96.108.135   <none>        443/TCP                                        93m
service/k8ssandra-operator-webhook-service                 ClusterIP   10.96.219.134   <none>        443/TCP                                        93m

NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-dc1-default-stargate-deployment   0/1     1            0           36m
deployment.apps/k8ssandra-operator                     1/1     1            1           93m
deployment.apps/k8ssandra-operator-cass-operator       1/1     1            1           93m

NAME                                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-dc1-default-stargate-deployment-6bf79bfc45   1         1         0       36m
replicaset.apps/k8ssandra-operator-74c6764fc7                     1         1         1       93m
replicaset.apps/k8ssandra-operator-cass-operator-847875b4f6       1         1         1       93m

NAME                                    READY   AGE
statefulset.apps/demo-dc1-default-sts   3/3     40m

Problem Description:
After successfully starting the pods, when checking the logs of the pod, I receive the following error message:

**kubectl logs demo-dc1-default-sts-0 -n k8ssandra-operator -c server-system-logger**
:
:
INFO  [GossipTasks:1] 2023-12-07 10:33:39,928 Gossiper.java:1087 - FatClient /172.18.0.6:7000 has been silent for 30000ms, removing from gossip
WARN  [epollEventLoopGroup-5-3] 2023-12-07 10:33:39,942 K8SeedProvider41x.java:47 - Seed provider couldn't lookup host demo-dc1-additional-seed-service
WARN  [epollEventLoopGroup-5-3] 2023-12-07 10:33:42,109 K8SeedProvider41x.java:47 - Seed provider couldn't lookup host demo-dc1-additional-seed-service
WARN  [epollEventLoopGroup-5-3] 2023-12-07 10:33:44,250 K8SeedProvider41x.java:47 - Seed provider couldn't lookup host demo-dc1-additional-seed-service
INFO  [Messaging-EventLoop-3-4] 2023-12-07 10:33:46,378 InboundConnectionInitiator.java:529 - /172.18.0.5:7000(/172.18.0.5:46708)->/172.18.0.4:7000-SMALL_MESSAGES-c6fd7fa6 messaging connection established, version = 12, framing = CRC, encryption = unencrypted
WARN  [epollEventLoopGroup-5-3] 2023-12-07 10:33:46,381 K8SeedProvider41x.java:47 - Seed provider couldn't lookup host demo-dc1-additional-seed-service
INFO  [Messaging-EventLoop-3-5] 2023-12-07 10:33:46,395 InboundConnectionInitiator.java:529 - /172.18.0.5:7000(/172.18.0.5:46716)->/172.18.0.4:7000-LARGE_MESSAGES-de5fcc13 messaging connection established, version = 12, framing = CRC, encryption = unencrypted
INFO  [Messaging-EventLoop-3-4] 2023-12-07 10:33:46,397 OutboundConnection.java:1153 - /172.18.0.4:7000(/172.18.0.4:50206)->/172.18.0.5:7000-SMALL_MESSAGES-eccfb8bf successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO  [Messaging-EventLoop-3-5] 2023-12-07 10:33:46,419 OutboundConnection.java:1153 - /172.18.0.4:7000(/172.18.0.4:50216)->/172.18.0.5:7000-LARGE_MESSAGES-bc4d7404 successfully connected, version = 12, framing = CRC, encryption = unencrypted
WARN  [epollEventLoopGroup-5-3] 2023-12-07 10:33:48,574 K8SeedProvider41x.java:47 - Seed provider couldn't lookup host demo-dc1-additional-seed-service
WARN  [epollEventLoopGroup-5-3] 2023-12-07 10:33:49,824 K8SeedProvider41x.java:47 - Seed provider couldn't lookup host demo-dc1-additional-seed-service
INFO  [epollEventLoopGroup-5-3] 2023-12-07 10:33:49,921 Keyspace.java:381 - Creating replication strategy none params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc1=1}}
INFO  [epollEventLoopGroup-5-3] 2023-12-07 10:33:49,923 NetworkTopologyStrategy.java:93 - Configured datacenter replicas are dc1:rf(1)
WARN  [epollEventLoopGroup-5-3] 2023-12-07 10:33:55,858 K8SeedProvider41x.java:47 - Seed provider couldn't lookup host demo-dc1-additional-seed-service
INFO  [epollEventLoopGroup-5-3] 2023-12-07 10:33:55,993 Keyspace.java:381 - Creating replication strategy none params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc1=1}}
INFO  [epollEventLoopGroup-5-3] 2023-12-07 10:33:55,993 NetworkTopologyStrategy.java:93 - Configured datacenter replicas are dc1:rf(1)
INFO  [Messaging-EventLoop-3-2] 2023-12-07 10:34:10,009 NoSpamLogger.java:105 - /172.18.0.4:7000->/172.18.0.6:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.18.0.6:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
        at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
        at io.netty.channel.unix.Socket.finishConnect(Socket.java:251)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)
:
:

Steps Taken So Far:
I tested it with different cassandra serverVersion:
serverVersion: “4.0.3”
serverVersion: “4.0.7”
serverVersion: “4.1.0”
serverVersion: “4.1.1”
serverVersion: “4.1.2”
serverVersion: “4.1.3”
I always get the same error message.

My Yaml-Manifest File mydemo.yaml:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: demo
  namespace: k8ssandra-operator
spec:
  cassandra:
    **serverVersion: "4.0.3"**     # I tested with "4.0.3" , "4.0.7", "4.1.0", "4.1.1", "4.1.2", "4.1.3"
    storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: standard
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 25Gi
    config:
      jvmOptions:
        heapSize: 512M
    networking:
      hostNetwork: true
    datacenters:
      - metadata:
          name: dc1
        size: 3
        stargate:
          size: 1
          heapSize: 512M

Ubuntu Server Version:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04

kubectl version -short
Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

docker info
Client: Docker Engine - Community
Version: 24.0.7
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

My goal is to create a multi-cluster setup in k8ssandra, and I’m facing this issue, which also occurs in a single-cluster scenario.

I would greatly appreciate your support and ideas on how to resolve this problem. If additional information is needed, I’m available to provide it.

Thank you in advance!

alexander · December 7, 2023, 1:58pm

Hi Jennifer, could you provide the output of:

kubectl describe pod/demo-dc1-default-stargate-deployment-6bf79bfc45-67knw -n k8ssandra-operator
kubectl logs demo-dc1-default-stargate-deployment-6bf79bfc45-67knw -n k8ssandra-operator --previous

Also which version of k8ssandra-operator are you running?

Jennifer · December 7, 2023, 2:22pm

Thank you for your assistance. Here is the information you requested:

Here the describe of stargate

kubectl describe pod/demo-dc1-default-stargate-deployment-6bf79bfc45-59l5m -n k8ssandra-operator
Name:             demo-dc1-default-stargate-deployment-6bf79bfc45-59l5m
Namespace:        k8ssandra-operator
Priority:         0
Service Account:  default
Node:             k8ssandra-0-worker4/172.18.0.5
Start Time:       Thu, 07 Dec 2023 15:02:42 +0100
Labels:           app.kubernetes.io/component=stargate
                  app.kubernetes.io/name=k8ssandra-operator
                  app.kubernetes.io/part-of=k8ssandra
                  k8ssandra.io/cluster-name=demo
                  k8ssandra.io/cluster-namespace=k8ssandra-operator
                  k8ssandra.io/stargate=demo-dc1-stargate
                  k8ssandra.io/stargate-deployment=demo-dc1-default-stargate-deployment
                  pod-template-hash=6bf79bfc45
Annotations:      <none>
Status:           Running
IP:               172.18.0.5
IPs:
  IP:           172.18.0.5
Controlled By:  ReplicaSet/demo-dc1-default-stargate-deployment-6bf79bfc45
Containers:
  demo-dc1-default-stargate-deployment:
    Container ID:   containerd://e9cf2e78359f912869fe4806e9406f9f0a8a89b25a33c57678c3beae035e4f7e
    Image:          docker.io/stargateio/stargate-4_0:v1.0.67
    Image ID:       docker.io/stargateio/stargate-4_0@sha256:eec28880951dcc45e6925e4ff3d6d0538b124f0607cf77d660bea64ce230b1e3
    Ports:          8080/TCP, 8081/TCP, 8082/TCP, 8084/TCP, 8085/TCP, 8090/TCP, 9042/TCP, 8609/TCP, 7000/TCP, 7001/TCP
    Host Ports:     8080/TCP, 8081/TCP, 8082/TCP, 8084/TCP, 8085/TCP, 8090/TCP, 9042/TCP, 8609/TCP, 7000/TCP, 7001/TCP
    State:          Running
      Started:      Thu, 07 Dec 2023 15:06:13 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Thu, 07 Dec 2023 15:04:30 +0100
      Finished:     Thu, 07 Dec 2023 15:06:13 +0100
    Ready:          False
    Restart Count:  3
    Limits:
      cpu:     1
      memory:  2048M
    Requests:
      cpu:      200m
      memory:   1024M
    Liveness:   http-get http://:health/checker/liveness delay=30s timeout=10s period=10s #success=1 #failure=5
    Readiness:  http-get http://:health/checker/readiness delay=30s timeout=10s period=10s #success=1 #failure=5
    Environment:
      LISTEN:                  (v1:status.podIP)
      JAVA_OPTS:              -XX:+CrashOnOutOfMemoryError -Xms512000000 -Xmx512000000 -Dstargate.unsafe.cassandra_config_path=/config/cassandra.yaml -Dstargate.cql.config_path=/config/stargate-cql.yaml
      CLUSTER_NAME:           demo
      CLUSTER_VERSION:        4.0
      SEED:                   demo-seed-service.k8ssandra-operator.svc
      DATACENTER_NAME:        dc1
      RACK_NAME:              default
      DISABLE_BUNDLES_WATCH:  true
      ENABLE_AUTH:            true
    Mounts:
      /config from cassandra-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xkrzk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  cassandra-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      demo-dc1-cassandra-config
    Optional:  false
  kube-api-access-xkrzk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  3m38s                 default-scheduler  Successfully assigned k8ssandra-operator/demo-dc1-default-stargate-deployment-6bf79bfc45-59l5m to k8ssandra-0-worker4
  Normal   Killing    2m18s                 kubelet            Container demo-dc1-default-stargate-deployment failed liveness probe, will be restarted
  Normal   Pulled     111s (x3 over 3m37s)  kubelet            Container image "docker.io/stargateio/stargate-4_0:v1.0.67" already present on machine
  Normal   Created    111s (x3 over 3m37s)  kubelet            Created container demo-dc1-default-stargate-deployment
  Normal   Started    110s (x3 over 3m37s)  kubelet            Started container demo-dc1-default-stargate-deployment
  Warning  Unhealthy  68s (x8 over 2m58s)   kubelet            Readiness probe failed: Get "http://172.18.0.5:8084/checker/readiness": dial tcp 172.18.0.5:8084: connect: connection refused
  Warning  Unhealthy  68s (x7 over 2m58s)   kubelet            Liveness probe failed: Get "http://172.18.0.5:8084/checker/liveness": dial tcp 172.18.0.5:8084: connect: connection refused

Jennifer · December 7, 2023, 2:24pm

my logs look like this

kubectl logs demo-dc1-default-stargate-deployment-6bf79bfc45-59l5m -n k8ssandra-operator --previous
Using environment for config
Running java -server -XX:+CrashOnOutOfMemoryError -Xms512000000 -Xmx512000000 -Dstargate.unsafe.cassandra_config_path=/config/cassandra.yaml -Dstargate.cql.config_path=/config/stargate-cql.yaml -Dstargate.libdir=./stargate-lib -Djava.awt.headless=true -jar ./stargate-lib/stargate-starter-1.0.67.jar --cluster-name demo --cluster-version 4.0 --cluster-seed demo-seed-service.k8ssandra-operator.svc --listen 172.18.0.5 --dc dc1 --rack default --enable-auth --disable-bundles-watch
JAR DIR: ./stargate-lib
Loading persistence backend persistence-cassandra-4.0-1.0.67.jar
Installing bundle persistence-cassandra-4.0-1.0.67.jar
Installing bundle animal-sniffer-annotations-1.9.jar
Installing bundle asm-7.1.jar
Installing bundle asm-analysis-7.1.jar
Installing bundle asm-tree-7.1.jar
Installing bundle auth-api-1.0.67.jar
Installing bundle auth-jwt-service-1.0.67.jar
Installing bundle auth-table-based-service-1.0.67.jar
Installing bundle authnz-1.0.67.jar
Installing bundle commons-beanutils-1.9.4.jar
Installing bundle commons-collections-3.2.2.jar
Installing bundle commons-digester-2.1.jar
Installing bundle commons-logging-1.2.jar
Installing bundle commons-validator-1.7.jar
Installing bundle config-store-api-1.0.67.jar
Installing bundle config-store-yaml-1.0.67.jar
Installing bundle core-1.0.67.jar
Installing bundle cql-1.0.67.jar
Installing bundle graphqlapi-1.0.67.jar
Installing bundle grpc-1.0.67.jar
Installing bundle health-checker-1.0.67.jar
Installing bundle org.apache.felix.scr-2.1.20.jar
Installing bundle org.apache.felix.scr.ds-annotations-1.2.10.jar
Installing bundle org.apache.felix.scr.generator-1.18.4.jar
Installing bundle org.osgi.compendium-4.2.0.jar
Installing bundle org.osgi.util.function-1.1.0.jar
Installing bundle org.osgi.util.promise-1.1.1.jar
Installing bundle persistence-api-1.0.67.jar
Installing bundle rate-limiting-global-1.0.67.jar
Installing bundle restapi-1.0.67.jar
Starting bundle io.stargate.db.cassandra_4_0
INFO  [main] 2023-12-07 14:06:16,966 BaseActivator.java:97 - Starting persistence-cassandra-4.0 ...
Starting bundle null
Starting bundle org.objectweb.asm
Starting bundle org.objectweb.asm.tree.analysis
Starting bundle org.objectweb.asm.tree
Starting bundle io.stargate.auth.api
INFO  [main] 2023-12-07 14:06:17,466 BaseActivator.java:97 - Starting authApiServer ...
Starting bundle io.stargate.auth.jwt
Starting bundle io.stargate.auth.table
INFO  [main] 2023-12-07 14:06:17,664 BaseActivator.java:97 - Starting authnTableBasedService and authzTableBasedServie ...
Starting bundle io.stargate.auth
Starting bundle org.apache.commons.commons-beanutils
Starting bundle org.apache.commons.collections
Starting bundle org.apache.commons.digester
Starting bundle org.apache.commons.logging
Starting bundle org.apache.commons.commons-validator
Starting bundle io.stargate.config.store.api
Starting bundle io.stargate.config.store.yaml
INFO  [main] 2023-12-07 14:06:17,780 BaseActivator.java:97 - Starting Config Store YAML ...
Starting bundle io.stargate.core
INFO  [main] 2023-12-07 14:06:17,783 BaseActivator.java:97 - Starting core services ...
INFO  [main] 2023-12-07 14:06:17,865 BaseActivator.java:178 - Registering core services as io.stargate.core.metrics.api.Metrics
INFO  [main] 2023-12-07 14:06:17,874 ConfigStoreActivator.java:57 - Creating Config Store YAML for config file location: /etc/stargate/stargate-config.yaml 
INFO  [main] 2023-12-07 14:06:18,197 BaseActivator.java:178 - Registering Config Store YAML as io.stargate.config.store.api.ConfigStore
INFO  [main] 2023-12-07 14:06:18,198 BaseActivator.java:185 - Started Config Store YAML
INFO  [main] 2023-12-07 14:06:20,369 AbstractCassandraPersistence.java:100 - Initializing Apache Cassandra
INFO  [main] 2023-12-07 14:06:20,847 DatabaseDescriptor.java:416 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  [main] 2023-12-07 14:06:20,856 DatabaseDescriptor.java:477 - Global memtable on-heap threshold is enabled at 117MB
INFO  [main] 2023-12-07 14:06:20,857 DatabaseDescriptor.java:481 - Global memtable off-heap threshold is enabled at 117MB
WARN  [main] 2023-12-07 14:06:20,863 DatabaseDescriptor.java:628 - Only 5.999GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  [main] 2023-12-07 14:06:20,959 GossipingPropertyFileSnitch.java:67 - Unable to load cassandra-topology.properties; compatibility mode disabled
INFO  [main] 2023-12-07 14:06:21,176 JMXServerUtils.java:271 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO  [main] 2023-12-07 14:06:21,186 CassandraDaemon.java:625 - Hostname: k8ssandra-0-worker4:7000:7001
INFO  [main] 2023-12-07 14:06:21,187 CassandraDaemon.java:632 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_342
INFO  [main] 2023-12-07 14:06:21,188 CassandraDaemon.java:633 - Heap size: 470.000MiB/470.000MiB
INFO  [main] 2023-12-07 14:06:21,189 CassandraDaemon.java:638 - Code Cache Non-heap memory: init = 2555904(2496K) used = 6399360(6249K) committed = 6422528(6272K) max = 251658240(245760K)
INFO  [main] 2023-12-07 14:06:21,190 CassandraDaemon.java:638 - Metaspace Non-heap memory: init = 0(0K) used = 30455504(29741K) committed = 31457280(30720K) max = -1(-1K)
INFO  [main] 2023-12-07 14:06:21,190 CassandraDaemon.java:638 - Compressed Class Space Non-heap memory: init = 0(0K) used = 3600080(3515K) committed = 3932160(3840K) max = 1073741824(1048576K)
INFO  [main] 2023-12-07 14:06:21,239 CassandraDaemon.java:638 - PS Eden Space Heap memory: init = 128974848(125952K) used = 124191336(121280K) committed = 128974848(125952K) max = 128974848(125952K)
INFO  [main] 2023-12-07 14:06:21,240 CassandraDaemon.java:638 - PS Survivor Space Heap memory: init = 20971520(20480K) used = 20614512(20131K) committed = 20971520(20480K) max = 20971520(20480K)
INFO  [main] 2023-12-07 14:06:21,241 CassandraDaemon.java:638 - PS Old Gen Heap memory: init = 342884352(334848K) used = 19590248(19131K) committed = 342884352(334848K) max = 342884352(334848K)
INFO  [main] 2023-12-07 14:06:21,242 CassandraDaemon.java:640 - Classpath: ./stargate-lib/stargate-starter-1.0.67.jar
INFO  [main] 2023-12-07 14:06:21,243 CassandraDaemon.java:642 - JVM Arguments: [-XX:+CrashOnOutOfMemoryError, -Xms512000000, -Xmx512000000, -Dstargate.unsafe.cassandra_config_path=/config/cassandra.yaml, -Dstargate.cql.config_path=/config/stargate-cql.yaml, -Dstargate.libdir=./stargate-lib, -Djava.awt.headless=true]
WARN  [main] 2023-12-07 14:06:21,543 NativeLibrary.java:201 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK.
INFO  [main] 2023-12-07 14:06:21,741 MonotonicClock.java:202 - Scheduling approximate time conversion task with an interval of 10000 milliseconds
INFO  [main] 2023-12-07 14:06:21,744 MonotonicClock.java:338 - Scheduling approximate time-check task with a precision of 2 milliseconds
WARN  [main] 2023-12-07 14:06:21,746 StartupChecks.java:143 - jemalloc shared library could not be preloaded to speed up memory allocations
INFO  [main] 2023-12-07 14:06:21,760 StartupChecks.java:194 - JMX is enabled to receive remote connections on port: 7199
INFO  [main] 2023-12-07 14:06:21,765 SigarLibrary.java:44 - Initializing SIGAR library
INFO  [main] 2023-12-07 14:06:21,840 SigarLibrary.java:57 - Could not initialize SIGAR library org.hyperic.sigar.Sigar.getFileSystemListNative()[Lorg/hyperic/sigar/FileSystem; 
INFO  [main] 2023-12-07 14:06:21,841 SigarLibrary.java:185 - Sigar could not be initialized, test for checking degraded mode omitted.
WARN  [main] 2023-12-07 14:06:21,842 StartupChecks.java:329 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
INFO  [main] 2023-12-07 14:06:22,982 Keyspace.java:386 - Creating replication strategy system params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}}
INFO  [main] 2023-12-07 14:06:23,087 ColumnFamilyStore.java:385 - Initializing system.IndexInfo
INFO  [main] 2023-12-07 14:06:25,748 ColumnFamilyStore.java:385 - Initializing system.batches
INFO  [main] 2023-12-07 14:06:25,775 ColumnFamilyStore.java:385 - Initializing system.paxos
INFO  [main] 2023-12-07 14:06:25,849 ColumnFamilyStore.java:385 - Initializing system.local
INFO  [main] 2023-12-07 14:06:25,868 ColumnFamilyStore.java:385 - Initializing system.peers_v2
INFO  [main] 2023-12-07 14:06:25,887 ColumnFamilyStore.java:385 - Initializing system.peers
INFO  [main] 2023-12-07 14:06:25,959 ColumnFamilyStore.java:385 - Initializing system.peer_events_v2
INFO  [main] 2023-12-07 14:06:25,982 ColumnFamilyStore.java:385 - Initializing system.peer_events
INFO  [main] 2023-12-07 14:06:26,051 ColumnFamilyStore.java:385 - Initializing system.compaction_history
INFO  [main] 2023-12-07 14:06:26,080 ColumnFamilyStore.java:385 - Initializing system.sstable_activity
INFO  [main] 2023-12-07 14:06:26,105 ColumnFamilyStore.java:385 - Initializing system.size_estimates
INFO  [main] 2023-12-07 14:06:26,151 ColumnFamilyStore.java:385 - Initializing system.table_estimates
INFO  [main] 2023-12-07 14:06:26,170 ColumnFamilyStore.java:385 - Initializing system.available_ranges_v2
INFO  [main] 2023-12-07 14:06:26,189 ColumnFamilyStore.java:385 - Initializing system.available_ranges
INFO  [main] 2023-12-07 14:06:26,263 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges_v2
INFO  [main] 2023-12-07 14:06:26,285 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges
INFO  [main] 2023-12-07 14:06:26,360 ColumnFamilyStore.java:385 - Initializing system.view_builds_in_progress
INFO  [main] 2023-12-07 14:06:26,383 ColumnFamilyStore.java:385 - Initializing system.built_views
INFO  [main] 2023-12-07 14:06:26,441 ColumnFamilyStore.java:385 - Initializing system.prepared_statements
INFO  [main] 2023-12-07 14:06:26,472 ColumnFamilyStore.java:385 - Initializing system.repairs
INFO  [main] 2023-12-07 14:06:26,542 QueryProcessor.java:115 - Initialized prepared statement caches with 10 MB
INFO  [main] 2023-12-07 14:06:26,552 ClientState.java:101 - Using io.stargate.db.cassandra.impl.StargateQueryHandler as query handler for native protocol queries (as requested with -Dcassandra.custom_query_handler_class)
INFO  [main] 2023-12-07 14:06:26,765 Keyspace.java:386 - Creating replication strategy system_schema params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}}
INFO  [main] 2023-12-07 14:06:26,767 ColumnFamilyStore.java:385 - Initializing system_schema.keyspaces
INFO  [main] 2023-12-07 14:06:26,850 ColumnFamilyStore.java:385 - Initializing system_schema.tables
INFO  [main] 2023-12-07 14:06:26,880 ColumnFamilyStore.java:385 - Initializing system_schema.columns
INFO  [main] 2023-12-07 14:06:26,953 ColumnFamilyStore.java:385 - Initializing system_schema.triggers
INFO  [main] 2023-12-07 14:06:27,049 ColumnFamilyStore.java:385 - Initializing system_schema.dropped_columns
INFO  [main] 2023-12-07 14:06:27,084 ColumnFamilyStore.java:385 - Initializing system_schema.views
INFO  [main] 2023-12-07 14:06:27,152 ColumnFamilyStore.java:385 - Initializing system_schema.types
INFO  [main] 2023-12-07 14:06:27,190 ColumnFamilyStore.java:385 - Initializing system_schema.functions
INFO  [main] 2023-12-07 14:06:27,226 ColumnFamilyStore.java:385 - Initializing system_schema.aggregates
INFO  [main] 2023-12-07 14:06:27,268 ColumnFamilyStore.java:385 - Initializing system_schema.indexes
INFO  [main] 2023-12-07 14:06:27,436 SystemKeyspaceMigrator40.java:76 - system.peers_v2 table was empty, migrating legacy system.peers, if this fails you should fix the issue and then truncate system.peers_v2 to have it try again.
INFO  [main] 2023-12-07 14:06:27,448 SystemKeyspaceMigrator40.java:100 - Migrating rows from legacy system.peers to system.peers_v2
INFO  [main] 2023-12-07 14:06:27,453 SystemKeyspaceMigrator40.java:119 - Migrated 0 rows from legacy system.peers to system.peers_v2
INFO  [main] 2023-12-07 14:06:27,454 SystemKeyspaceMigrator40.java:129 - system.peer_events_v2 table was empty, migrating legacy system.peer_events to system.peer_events_v2
INFO  [main] 2023-12-07 14:06:27,457 SystemKeyspaceMigrator40.java:152 - Migrated 0 rows from legacy system.peer_events to system.peer_events_v2
INFO  [main] 2023-12-07 14:06:27,458 SystemKeyspaceMigrator40.java:162 - system.transferred_ranges_v2 table was empty, migrating legacy system.transferred_ranges to system.transferred_ranges_v2
INFO  [main] 2023-12-07 14:06:27,461 SystemKeyspaceMigrator40.java:190 - Migrated 0 rows from legacy system.transferred_ranges to system.transferred_ranges_v2
INFO  [main] 2023-12-07 14:06:27,462 SystemKeyspaceMigrator40.java:200 - system.available_ranges_v2 table was empty, migrating legacy system.available_ranges to system.available_ranges_v2
INFO  [main] 2023-12-07 14:06:27,466 SystemKeyspaceMigrator40.java:226 - Migrated 0 rows from legacy system.available_ranges to system.available_ranges_v2
INFO  [main] 2023-12-07 14:06:27,467 StorageService.java:838 - Populating token metadata from system tables
INFO  [main] 2023-12-07 14:06:27,485 StorageService.java:832 - Token metadata: 
INFO  [main] 2023-12-07 14:06:27,566 CacheService.java:103 - Initializing key cache with capacity of 23 MBs.
INFO  [main] 2023-12-07 14:06:27,580 CacheService.java:125 - Initializing row cache with capacity of 0 MBs
INFO  [main] 2023-12-07 14:06:27,582 CacheService.java:154 - Initializing counter cache with capacity of 11 MBs
INFO  [main] 2023-12-07 14:06:27,583 CacheService.java:165 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  [main] 2023-12-07 14:06:27,664 CommitLog.java:168 - No commitlog files found; skipping replay
INFO  [main] 2023-12-07 14:06:27,664 StorageService.java:838 - Populating token metadata from system tables
INFO  [main] 2023-12-07 14:06:27,667 StorageService.java:832 - Token metadata: 
INFO  [main] 2023-12-07 14:06:28,246 ColumnFamilyStore.java:2252 - Truncating system.size_estimates
INFO  [main] 2023-12-07 14:06:28,348 ColumnFamilyStore.java:2289 - Truncating system.size_estimates with truncatedAt=1701957988270
INFO  [main] 2023-12-07 14:06:28,403 ColumnFamilyStore.java:878 - Enqueuing flush of local: 2.157KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2023-12-07 14:06:28,544 Memtable.java:469 - Writing Memtable-local@984586412(0.415KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2023-12-07 14:06:28,549 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big-Data.db (0.210KiB) for commitlog position CommitLogPosition(segmentId=1701957981666, position=31193)
INFO  [MemtableFlushWriter:2] 2023-12-07 14:06:28,675 LogTransaction.java:240 - Unfinished transaction log, deleting /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_d0b87d60-9509-11ee-9c1e-01fe56a2a549.log 
INFO  [main] 2023-12-07 14:06:28,698 ColumnFamilyStore.java:2316 - Truncate of system.size_estimates is complete
INFO  [main] 2023-12-07 14:06:28,699 ColumnFamilyStore.java:2252 - Truncating system.table_estimates
INFO  [main] 2023-12-07 14:06:28,701 ColumnFamilyStore.java:2289 - Truncating system.table_estimates with truncatedAt=1701957988701
INFO  [main] 2023-12-07 14:06:28,703 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.575KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2023-12-07 14:06:28,715 Memtable.java:469 - Writing Memtable-local@150787849(0.091KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2023-12-07 14:06:28,719 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big-Data.db (0.063KiB) for commitlog position CommitLogPosition(segmentId=1701957981666, position=31304)
INFO  [MemtableFlushWriter:2] 2023-12-07 14:06:28,749 LogTransaction.java:240 - Unfinished transaction log, deleting /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_d0e61d10-9509-11ee-9c1e-01fe56a2a549.log 
INFO  [main] 2023-12-07 14:06:28,753 ColumnFamilyStore.java:2316 - Truncate of system.table_estimates is complete
INFO  [main] 2023-12-07 14:06:28,759 QueryProcessor.java:167 - Preloaded 0 prepared statements
INFO  [main] 2023-12-07 14:06:28,760 StorageService.java:736 - Cassandra version: 4.0.4
INFO  [main] 2023-12-07 14:06:28,761 StorageService.java:737 - CQL version: 3.4.5
INFO  [main] 2023-12-07 14:06:28,761 StorageService.java:738 - Native protocol supported versions: 3/v3, 4/v4, 5/v5, 6/v6-beta (default: 5/v5)
INFO  [main] 2023-12-07 14:06:28,801 IndexSummaryManager.java:84 - Initializing index summary manager with a memory pool size of 23 MB and a resize interval of 60 minutes
INFO  [main] 2023-12-07 14:06:28,803 StorageService.java:755 - Loading persisted ring state
INFO  [main] 2023-12-07 14:06:28,803 StorageService.java:838 - Populating token metadata from system tables
INFO  [main] 2023-12-07 14:06:28,853 BufferPools.java:49 - Global buffer pool limit is 117.000MiB for chunk-cache and 29.000MiB for networking
INFO  [main] 2023-12-07 14:06:29,083 InboundConnectionInitiator.java:127 - Listening on address: (/172.18.0.5:7000), nic: eth0, encryption: unencrypted
WARN  [main] 2023-12-07 14:06:29,249 SystemKeyspace.java:1130 - No host ID found, created a10106fc-a860-4532-88a7-357c9dfd21ea (Note: This should happen exactly once per node).
INFO  [Messaging-EventLoop-3-4] 2023-12-07 14:06:30,012 InboundConnectionInitiator.java:464 - /172.18.0.3:7000(/172.18.0.3:46628)->/172.18.0.5:7000-URGENT_MESSAGES-37f8ac65 messaging connection established, version = 12, framing = LZ4, encryption = unencrypted
INFO  [Messaging-EventLoop-3-3] 2023-12-07 14:06:30,012 OutboundConnection.java:1150 - /172.18.0.5:7000(/172.18.0.5:41368)->demo-seed-service.k8ssandra-operator.svc/172.18.0.2:7000-URGENT_MESSAGES-847e4eca successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO  [Messaging-EventLoop-3-2] 2023-12-07 14:06:30,012 InboundConnectionInitiator.java:464 - /172.18.0.2:7000(/172.18.0.2:53414)->/172.18.0.5:7000-URGENT_MESSAGES-560289f8 messaging connection established, version = 12, framing = LZ4, encryption = unencrypted
INFO  [Messaging-EventLoop-3-2] 2023-12-07 14:06:30,014 OutboundConnection.java:1150 - /172.18.0.5:7000(/172.18.0.5:56728)->demo-seed-service.k8ssandra-operator.svc/172.18.0.3:7000-URGENT_MESSAGES-6f00a794 successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO  [Messaging-EventLoop-3-3] 2023-12-07 14:06:30,041 InboundConnectionInitiator.java:464 - /172.18.0.6:7000(/172.18.0.6:41798)->/172.18.0.5:7000-URGENT_MESSAGES-3efe3ce6 messaging connection established, version = 12, framing = LZ4, encryption = unencrypted
INFO  [Messaging-EventLoop-3-1] 2023-12-07 14:06:30,041 OutboundConnection.java:1150 - /172.18.0.5:7000(/172.18.0.5:47328)->demo-seed-service.k8ssandra-operator.svc/172.18.0.6:7000-URGENT_MESSAGES-b726d545 successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO  [main] 2023-12-07 14:06:30,509 StorageService.java:960 - Starting up server gossip
INFO  [main] 2023-12-07 14:06:30,532 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.583KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2023-12-07 14:06:30,553 Memtable.java:469 - Writing Memtable-local@1344816810(0.079KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2023-12-07 14:06:30,555 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1701957981666, position=31468)
INFO  [MemtableFlushWriter:1] 2023-12-07 14:06:30,598 LogTransaction.java:240 - Unfinished transaction log, deleting /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_d1fd0b50-9509-11ee-9c1e-01fe56a2a549.log 
INFO  [MigrationStage:1] 2023-12-07 14:06:30,665 ColumnFamilyStore.java:878 - Enqueuing flush of columns: 167.017KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2023-12-07 14:06:30,682 Memtable.java:469 - Writing Memtable-columns@104544358(35.013KiB serialized bytes, 232 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2023-12-07 14:06:30,750 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb-1-big-Data.db (17.447KiB) for commitlog position CommitLogPosition(segmentId=1701957981666, position=35326)
INFO  [MemtableFlushWriter:2] 2023-12-07 14:06:31,708 LogTransaction.java:240 - Unfinished transaction log, deleting /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb_txn_flush_d2a224f0-9509-11ee-9c1e-01fe56a2a549.log 
INFO  [MigrationStage:1] 2023-12-07 14:06:32,165 Keyspace.java:386 - Creating replication strategy system_auth params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}
INFO  [MigrationStage:1] 2023-12-07 14:06:32,167 ColumnFamilyStore.java:385 - Initializing system_auth.network_permissions
INFO  [MigrationStage:1] 2023-12-07 14:06:32,249 ColumnFamilyStore.java:385 - Initializing system_auth.resource_role_permissons_index
INFO  [MigrationStage:1] 2023-12-07 14:06:32,340 ColumnFamilyStore.java:385 - Initializing system_auth.role_members
INFO  [MigrationStage:1] 2023-12-07 14:06:32,383 ColumnFamilyStore.java:385 - Initializing system_auth.role_permissions
INFO  [MigrationStage:1] 2023-12-07 14:06:32,568 ColumnFamilyStore.java:385 - Initializing system_auth.roles
INFO  [main] 2023-12-07 14:06:32,769 AuthCache.java:215 - (Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000)
INFO  [main] 2023-12-07 14:06:32,785 StorageService.java:817 - Not joining ring as requested. Use JMX (StorageService->joinRing()) to initiate ring joining
INFO  [main] 2023-12-07 14:06:32,786 Gossiper.java:2214 - Waiting for gossip to settle...
INFO  [main] 2023-12-07 14:06:40,796 Gossiper.java:2245 - No gossip backlog; proceeding
INFO  [main] 2023-12-07 14:06:40,835 CassandraDaemon.java:697 - Not starting native transport as requested. Use JMX (StorageService->startNativeTransport()) or nodetool (enablebinary) to start it
INFO  [OptionalTasks:1] 2023-12-07 14:06:42,763 CassandraRoleManager.java:370 - Setup task failed with error, rescheduling
INFO  [OptionalTasks:1] 2023-12-07 14:06:52,764 CassandraRoleManager.java:370 - Setup task failed with error, rescheduling
INFO  [GossipStage:1] 2023-12-07 14:07:14,919 Gossiper.java:1364 - Node /172.18.0.2:7000 is now part of the cluster
INFO  [GossipStage:1] 2023-12-07 14:07:14,965 TokenMetadata.java:511 - Updating topology for /172.18.0.2:7000
INFO  [GossipStage:1] 2023-12-07 14:07:14,967 TokenMetadata.java:511 - Updating topology for /172.18.0.2:7000
INFO  [GossipStage:1] 2023-12-07 14:07:14,974 Gossiper.java:1364 - Node /172.18.0.6:7000 is now part of the cluster
INFO  [GossipStage:1] 2023-12-07 14:07:15,070 TokenMetadata.java:511 - Updating topology for /172.18.0.6:7000
INFO  [GossipStage:1] 2023-12-07 14:07:15,072 TokenMetadata.java:511 - Updating topology for /172.18.0.6:7000
INFO  [GossipStage:1] 2023-12-07 14:07:15,080 Gossiper.java:1364 - Node /172.18.0.3:7000 is now part of the cluster
INFO  [GossipStage:1] 2023-12-07 14:07:15,150 TokenMetadata.java:511 - Updating topology for /172.18.0.3:7000
INFO  [GossipStage:1] 2023-12-07 14:07:15,157 TokenMetadata.java:511 - Updating topology for /172.18.0.3:7000
INFO  [GossipStage:1] 2023-12-07 14:07:15,163 Gossiper.java:1312 - InetAddress /172.18.0.2:7000 is now UP
INFO  [GossipStage:1] 2023-12-07 14:07:15,164 Gossiper.java:1312 - InetAddress /172.18.0.6:7000 is now UP
INFO  [GossipStage:1] 2023-12-07 14:07:15,165 Gossiper.java:1312 - InetAddress /172.18.0.3:7000 is now UP
WARN  [GossipTasks:1] 2023-12-07 14:07:15,775 FailureDetector.java:319 - Not marking nodes down due to local pause of 49181112781ns > 5000000000ns
INFO  [Messaging-EventLoop-3-1] 2023-12-07 14:07:22,827 OutboundConnection.java:1150 - /172.18.0.5:7000(/172.18.0.5:51448)->demo-seed-service.k8ssandra-operator.svc/172.18.0.2:7000-SMALL_MESSAGES-211eeb20 successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO  [Messaging-EventLoop-3-1] 2023-12-07 14:07:22,840 InboundConnectionInitiator.java:464 - /172.18.0.2:7000(/172.18.0.2:36168)->/172.18.0.5:7000-SMALL_MESSAGES-ecbc6595 messaging connection established, version = 12, framing = LZ4, encryption = unencrypted
INFO  [StorageServiceShutdownHook] 2023-12-07 14:07:32,972 HintsService.java:222 - Paused hints dispatch
WARN  [StorageServiceShutdownHook] 2023-12-07 14:07:32,974 Gossiper.java:2032 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2023-12-07 14:07:32,976 MessagingService.java:441 - Waiting for messaging service to quiesce
INFO  [Messaging-EventLoop-3-3] 2023-12-07 14:07:32,999 OutboundConnection.java:1150 - /172.18.0.5:7000(/172.18.0.5:39894)->demo-seed-service.k8ssandra-operator.svc/172.18.0.6:7000-SMALL_MESSAGES-f139d511 successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO  [Messaging-EventLoop-3-1] 2023-12-07 14:07:32,999 OutboundConnection.java:1150 - /172.18.0.5:7000(/172.18.0.5:58018)->demo-seed-service.k8ssandra-operator.svc/172.18.0.3:7000-LARGE_MESSAGES-1d122a45 successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO  [StorageServiceShutdownHook] 2023-12-07 14:07:35,078 ColumnFamilyStore.java:878 - Enqueuing flush of peers_v2: 14.211KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [StorageServiceShutdownHook] 2023-12-07 14:07:35,091 ColumnFamilyStore.java:878 - Enqueuing flush of peers: 13.520KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [StorageServiceShutdownHook] 2023-12-07 14:07:35,147 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.485KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:3] 2023-12-07 14:07:35,163 Memtable.java:469 - Writing Memtable-peers_v2@179318514(2.281KiB serialized bytes, 39 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
INFO  [PerDiskMemtableFlushWriter_0:3] 2023-12-07 14:07:35,165 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/peers_v2-c4325fbb8e5e3bafbd070f9250ed818e/nb-1-big-Data.db (1.490KiB) for commitlog position CommitLogPosition(segmentId=1701957981666, position=43267)
INFO  [PerDiskMemtableFlushWriter_0:4] 2023-12-07 14:07:35,167 Memtable.java:469 - Writing Memtable-peers@139859851(2.211KiB serialized bytes, 39 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
INFO  [PerDiskMemtableFlushWriter_0:4] 2023-12-07 14:07:35,168 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/peers-37f71aca7dc2383ba70672528af04d4f/nb-1-big-Data.db (1.458KiB) for commitlog position CommitLogPosition(segmentId=1701957981666, position=43267)
INFO  [PerDiskMemtableFlushWriter_0:3] 2023-12-07 14:07:35,258 Memtable.java:469 - Writing Memtable-local@74961364(0.060KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
INFO  [PerDiskMemtableFlushWriter_0:3] 2023-12-07 14:07:35,259 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.05909498809842391144/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db (0.043KiB) for commitlog position CommitLogPosition(segmentId=1701957981666, position=43267)
INFO  [StorageServiceShutdownHook] 2023-12-07 14:07:35,344 HintsService.java:222 - Paused hints dispatch

This version of k8ssandra-operator are running:

kubectl get deployment k8ssandra-operator-cass-operator -n k8ssandra-operator -o jsonpath='{.spec.template.spec.containers[0].image}'
docker.io/k8ssandra/cass-operator:v1.18.2

alexander · December 7, 2023, 2:34pm

It looks like the liveness probe is currently too sensitive for the speed at which it’s starting up.
You can extend the periodSeconds explicitly in the manifest: K8ssandra-operator CRDs v1.8 | K8ssandra, Apache Cassandra on Kubernetes

I’d say this is probably due to a lack of resources on the node/pod.