Can't find reaper pod/service after k8ssandra deployment

K8ssandra help
1

Greetings,
I was following this guide to install k8ssandra operator and cluster, but after deployment, I can’t find reaper pod or service. I tried this on a k8s cluster deployed in docker-desktop.

But based on description in Quickstart for Site Reliability Engineers | K8ssandra, Apache Cassandra on Kubernetes, “Reaper is deployed as part of the K8ssandra Operator”.

What else I need to do to get reaper pod and service ready?

Thanks!

2

Hi,

I apologize for the incomplete documentation.
In order to deploy Reaper, you’ll need to add the corresponding reaper section to your K8ssandraCluster definition:

reaper:
    heapSize: 512Mi
    autoScheduling:
      enabled: false
cassandra:
  ...
  ...

You can check the CRD documentation here to discover all the settings or the reaper field.

3

Thanks for the quick reply @alexander , I 've changed my yaml file to below format, but still did not see reaper or medusa pod after deployment, could you help check if there is anything wrong? Thanks!

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: demo
spec:
  cassandra:
    serverVersion: "4.0.1"
    datacenters:
      - metadata:
          name: dc1
        size: 3
        storageConfig:
          cassandraDataVolumeClaimSpec:
            storageClassName: standard
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 5Gi
        config:
          jvmOptions:
            heapSize: 512M
        stargate:
          size: 1
          heapSize: 256M
  reaper:
    heapSize: 512Mi
    autoScheduling:
      enabled: false
  medusa:
    containerImage:
      registry: xxxx
      tag: 0.12.2
    storageProperties:
      storageProvider: s3_compatible
      storageSecretRef:
        name: medusa-bucket-key
      bucketName: cassandra-backups
      prefix: test
      host: xxxx
      port: 443
4

Do you see errors in the logs of the k8ssandra-operator pod?

5

Hi @alexander , please see below:

  • from cassandra container:
    INFO [nioEventLoopGroup-3-14] 2022-10-26 07:29:50,936 UnixSocketCQLAccess.java:88 - Cannot create Driver CQLSession as the driver socket has not been created. This should resolve once Cassandra has started and created the socket at /tmp/cassandra.sock
    INFO [nioEventLoopGroup-2-1] 2022-10-26 07:29:50,937 Cli.java:617 - address=/10.222.56.50:52852 url=/api/v0/probes/readiness status=500 Internal Server Error
    INFO [nioEventLoopGroup-2-2] 2022-10-26 07:29:55,937 Cli.java:617 - address=/10.222.56.50:52862 url=/api/v0/probes/liveness status=200 OK
    INFO [pool-2-thread-1] 2022-10-26 07:29:57,777 UnixSocketCQLAccess.java:88 - Cannot create Driver CQLSession as the driver socket has not been created. This should resolve once Cassandra has started and created the socket at /tmp/cassandra.sock
    INFO [pool-2-thread-1] 2022-10-26 07:29:57,929 LifecycleResources.java:218 - Started Cassandra

  • from server-system-logger
    WARN [main] 2022-10-26 07:55:29,789 NativeLibrary.java:201 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK.
    ERROR [main] 2022-10-26 07:55:29,794 CassandraDaemon.java:909 - Exception encountered during startup
    java.lang.ExceptionInInitializerError: null
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:260)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
    Caused by: java.lang.IllegalStateException: Failed to mkdirs /opt/cassandra/data/data; unable to start server
    at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:1422)
    at org.apache.cassandra.db.commitlog.CommitLog.(CommitLog.java:94)
    at org.apache.cassandra.db.commitlog.CommitLog.construct(CommitLog.java:77)
    at org.apache.cassandra.db.commitlog.CommitLog.(CommitLog.java:64)
    … 3 common frames omitted

6

Looks like a storage access issue.
Was the kubernetes cluster created with Kind?

I’d recommend using the script we provide in our codebase to create the kind cluster in Docker Desktop, as shown here.

As long as Cassandra won’t be able to start, you won’t see the other components being deployed.

7

Thanks @alexander , I 'll try again.

8

It’s working now, thanks @alexander for the help!