We are using FluxCD to deploy k8ssandra CRDs as a Helm chart. I updated Cassandra to 4.1.10 in the K8ssandraCluster manifest. Cluster is up and running, and Authentication is enabled, but nodetool doesn’t work. I get this error when I try to run nodetool -u xxx -pw yyy nodetool: Failed to connect to '127.0.0.1:7199' - IOException: '/opt/cassandra/conf/cassandra-jaas.config (No such file or directory). It was working before the update. I checked the file /opt/cassandra/conf/cassandra-jaas.config, but it doesn’t exist. I tried to add it manually, but it didn’t work. Also, when I run cqlsh -u xxx -p yyy, it’s working with the WARNING message like this;
Warning: Cannot create directory at `/opt/cassandra/.cassandra`. Command history will not be saved. Please check what was the environment property CQL_HISTORY set to.
Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.
Let me share my k8ssandra-operator and k8ssandraCluster manifest files.
Let me share my k8ssandra-operator and k8ssandraCluster manifest files.
K8ssandra-operator.yaml
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: k8ssandra
namespace: default
spec:
interval: 1m0s
url: https://helm.k8ssandra.io/stable
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: k8ssandra-operator
namespace: default
spec:
chart:
spec:
chart: k8ssandra-operator
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: k8ssandra
namespace: default
version: 1.23.1
interval: 1m0s
k8ssandraCluster.yaml
k8ssandraCluster.yaml
apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
name: cassandra
namespace: default
annotations:
kustomize.toolkit.fluxcd.io/prune: true
spec:
auth: true
reaper:
containerImage:
name: cassandra-reaper
registry: docker.io
repository: thelastpickle
tag: 3.8.0
initContainerImage:
name: cassandra-reaper
registry: docker.io
repository: thelastpickle
tag: 3.8.0
autoScheduling:
enabled: false
telemetry:
prometheus:
enabled: true
cassandra:
telemetry:
prometheus:
enabled: true
mcac:
metricFilters:
- "deny:org.apache.cassandra.metrics.Table"
serverVersion: "4.1.10"
resources:
requests:
memory: "13Gi"
cpu: "1"
limits:
memory: "13Gi"
datacenters:
- metadata:
name: dc1
size: 3
racks:
- name: me-south-1a
nodeAffinityLabels:
topology.kubernetes.io/zone: me-south-1a
- name: me-south-1b
nodeAffinityLabels:
topology.kubernetes.io/zone: me-south-1b
- name: me-south-1c
nodeAffinityLabels:
topology.kubernetes.io/zone: me-south-1c
storageConfig:
cassandraDataVolumeClaimSpec:
storageClassName: gp3-retain
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
config:
cassandraYaml:
batch_size_fail_threshold: 5000KiB
batch_size_warn_threshold: 1000KiB
jvmOptions:
gc: ZGC
heapSize: 3072M
medusa:
storageProperties:
storageProvider: s3
storageSecretRef:
name: s3-bucket-key
bucketName: demo-k8ssandra-backup-store1
prefix: demo
region: eu-central-1
# Whether or not to use SSL to connect to the storage backend
secure: true
# Maximum backup age that the purge process should observe.
# 0 equals unlimited
maxBackupAge: 15
# File size in bytes over which cloud specific cli tools are used for transfer.
#Defaults to 100 MB.
multiPartUploadThreshold: 109951162777600
k8ssandra-operator version:
$ k get deployment k8ssandra-operator -ojsonpath="{.spec.template.spec.containers[0].image}"
cr.k8ssandra.io/k8ssandra/k8ssandra-operator:v1.23.1
cass-operator version:
$ k get deployment k8ssandra-operator-cass-operator -ojsonpath="{.spec.template.spec.containers[0].image}"
cr.k8ssandra.io/k8ssandra/cass-operator:v1.24.1
cass-operator CRD version:
$ k get crd cassandradatacenters.cassandra.datastax.com -o json | jq '.metadata.annotations'
{
"controller-gen.kubebuilder.io/version": "v0.16.4"
}
I would expect the operator(1.23.1) to create this missed file “cassandra-jass.config”
$ winpty kubectl exec -it cassandra-dc1-me-south-1a-sts-0 -c cassandra -- bash -c "ls -l opt/cassandra/conf/"
total 56
-rwxr-x--- 1 cassandra cassandra 13843 Sep 16 00:37 cassandra-env.sh
-rwxr-x--- 1 cassandra cassandra 24 Sep 16 00:37 cassandra-rackdc.properties
-rw-r----- 1 cassandra cassandra 4223 Sep 16 00:37 cassandra.yaml
-rw-r--r-- 1 cassandra cassandra 2134 Sep 16 00:37 jvm11-clients.options
-rwxr-x--- 1 cassandra cassandra 940 Sep 16 00:37 jvm11-server.options
-rwxr-x--- 1 cassandra cassandra 0 Sep 16 00:37 jvm17-server.options
-rw-r--r-- 1 cassandra cassandra 1323 Sep 16 00:37 jvm-clients.options
-rwxr-x--- 1 cassandra cassandra 261 Sep 16 00:37 jvm-server.options
-rw-r--r-- 1 cassandra cassandra 1195 Sep 16 00:37 logback-tools.xml
-rw-r--r-- 1 cassandra cassandra 5098 Sep 16 00:37 logback.xml
I also checked the logs using grep “jass” in the operator, cass-operator, flux pods, cassandra pod… to catch any logs regarding this missed file, but I couldn’t find anything.
Is there something I’ve missed that’s required? How can I fix this issue?
Also, there is no information about a compatible matrix between the operator and Cassandra in the official documents. How can I make sure which operator version is compatible with the Cassandra version that I want to update?
I need your support asap. Thanks.
