Hello, after installing “cass-operator” v1.13.0 using k8sandra/k8sandra’s helm chart, the “cassandra” cluster in v4.0.0(cass-management-api:4.0.0-v0.1.37 docker image) has been configured and in use. Recently, I am interested in CassandraTask CRD, so I tested it to try the replaceNode job, but the reconciler error is being returned from the operator log.
Cassandra is installed, but the “lastRollingRestart” field does not exist in Status field when viewed with the ‘kubectl describe cassdc dc1’ command. The situation is the same even if run the rolling restart statefulset. If i look at the cass-operator repository code and look at the “CheckRollingRestart” function in “cass-operator/pkg/reconciliation/reconciliation_racks.go”, i can see the code below, but does it depend on the value of lastRollingRestart in this code? I would appreciate it if you let me know if I am using the Cassandra Task CRD incorrectly.
cutoff := &dc.Status.LastRollingRestart
for _, pod := range rc.dcPods {
podStartTime := pod.GetCreationTimestamp()
if podStartTime.Before(cutoff) {
here’s my cassandraTask yaml, and cass-operator logs.
apiVersion: control.k8ssandra.io/v1alpha1
kind: CassandraTask
metadata:
name: k8ssandra-dc1-rack1-sts-4-replace
spec:
datacenter:
name: dc1
namespace: ellie-test-v1
jobs:
- name: replace-run
command: replacenode
args:
pod_name: k8ssandra-dc1-rack1-sts-4
1.667809762391828e+09 ERROR Reconciler error {"controller": "cassandratask", "controllerGroup": "control.k8ssandra.io", "controllerKind": "CassandraTask", "cassandraTask": {"name":"k8ssandra-dc1-rack1-sts-4-replace","namespace":"ellie-test-v1"}, "namespace": "ellie-test-v1", "name": "k8ssandra-dc1-rack1-sts-4-replace", "reconcileID": "b51122dd-c536-4dd6-ba2a-fbf379693851", "error": "CassandraDatacenter.cassandra.datastax.com \"dc1\" is invalid: status.lastRollingRestart: Invalid value: \"null\": status.lastRollingRestart in body must be of type string: \"null\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234
and here’s my cassdc dc1 from ‘kubectl describe cassdc dc1’
Name: dc1
Namespace: ellie-test-v1
Labels: app.kubernetes.io/instance=k8ssandra
app.kubernetes.io/name=k8ssandra
graph-name=test-v1
group-name=ellie
Annotations: reaper.cassandra-reaper.io/instance: k8ssandra-reaper
API Version: cassandra.datastax.com/v1beta1
Kind: CassandraDatacenter
Metadata:
Creation Timestamp: 2022-11-07T06:36:14Z
Finalizers:
finalizer.cassandra.datastax.com
Generation: 3
Managed Fields:
API Version: cassandra.datastax.com/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:reaper.cassandra-reaper.io/instance:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:graph-name:
f:group-name:
f:spec:
.:
f:clusterName:
f:config:
.:
f:cassandra-yaml:
.:
f:allocate_tokens_for_local_replication_factor:
f:authenticator:
f:authorizer:
f:credentials_update_interval_in_ms:
f:credentials_validity_in_ms:
f:num_tokens:
f:permissions_update_interval_in_ms:
f:permissions_validity_in_ms:
f:role_manager:
f:roles_update_interval_in_ms:
f:roles_validity_in_ms:
f:jvm-server-options:
.:
f:heap_size_young_generation:
f:initial_heap_size:
f:max_heap_size:
f:jvm11-server-options:
.:
f:additional-jvm-opts:
f:conc_gc_threads:
f:g1r_set_updating_pause_time_percent:
f:garbage_collector:
f:initiating_heap_occupancy_percent:
f:max_gc_pause_millis:
f:parallel_gc_threads:
f:configBuilderImage:
f:dockerImageRunsAsCassandra:
f:managementApiAuth:
.:
f:insecure:
f:podTemplateSpec:
.:
f:metadata:
.:
f:annotations:
.:
f:vault.hashicorp.com/agent-image:
f:vault.hashicorp.com/agent-init-first:
f:vault.hashicorp.com/agent-inject:
f:vault.hashicorp.com/agent-inject-secret-config:
f:vault.hashicorp.com/agent-inject-template-config:
f:vault.hashicorp.com/agent-pre-populate-only:
f:vault.hashicorp.com/role:
f:vault.hashicorp.com/tls-skip-verify:
f:labels:
.:
f:app:
f:spec:
.:
f:volumes:
f:racks:
f:resources:
.:
f:limits:
.:
f:memory:
f:requests:
.:
f:memory:
f:serverType:
f:serverVersion:
f:serviceAccount:
f:size:
f:storageConfig:
.:
f:additionalVolumes:
f:cassandraDataVolumeClaimSpec:
.:
f:accessModes:
f:resources:
.:
f:requests:
.:
f:storage:
f:storageClassName:
f:systemLoggerImage:
Manager: Kubernetes Java Client
Operation: Update
Time: 2022-11-07T06:36:14Z
API Version: cassandra.datastax.com/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
f:spec:
f:additionalServiceConfig:
.:
f:additionalSeedService:
f:allpodsService:
f:dcService:
f:nodePortService:
f:seedService:
f:configBuilderResources:
f:podTemplateSpec:
f:spec:
f:containers:
f:resources:
f:limits:
f:cpu:
f:requests:
f:cpu:
f:systemLoggerResources:
f:status:
.:
f:cassandraOperatorProgress:
f:conditions:
f:lastServerNodeStarted:
f:nodeReplacements:
f:nodeStatuses:
.:
f:k8ssandra-dc1-rack1-sts-0:
.:
f:hostID:
f:k8ssandra-dc1-rack1-sts-1:
.:
f:hostID:
f:k8ssandra-dc1-rack1-sts-2:
.:
f:hostID:
f:k8ssandra-dc1-rack1-sts-3:
.:
f:hostID:
f:k8ssandra-dc1-rack1-sts-4:
.:
f:hostID:
f:observedGeneration:
f:quietPeriod:
f:superUserUpserted:
f:usersUpserted:
Manager: manager
Operation: Update
Time: 2022-11-07T08:07:11Z
API Version: cassandra.datastax.com/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:podTemplateSpec:
f:spec:
f:initContainers:
f:serverImage:
f:status:
f:lastRollingRestart:
Manager: kubectl-edit
Operation: Update
Time: 2022-11-07T08:31:15Z
Resource Version: 1136079099
Self Link: /apis/cassandra.datastax.com/v1beta1/namespaces/ellie-test-v1/cassandradatacenters/dc1
UID: d76d7e50-8443-4d32-91b7-612eb9fdf557
Spec:
...(ommission)
Status:
Cassandra Operator Progress: Ready
Conditions:
Last Transition Time: 2022-11-07T07:59:55Z
Message:
Reason:
Status: True
Type: Healthy
Last Transition Time: 2022-11-07T06:44:38Z
Message:
Reason:
Status: False
Type: Stopped
Last Transition Time: 2022-11-07T06:44:38Z
Message:
Reason:
Status: False
Type: ReplacingNodes
Last Transition Time: 2022-11-07T08:07:05Z
Message:
Reason:
Status: False
Type: Updating
Last Transition Time: 2022-11-07T06:44:38Z
Message:
Reason:
Status: False
Type: RollingRestart
Last Transition Time: 2022-11-07T06:44:38Z
Message:
Reason:
Status: False
Type: Resuming
Last Transition Time: 2022-11-07T06:44:38Z
Message:
Reason:
Status: False
Type: ScalingDown
Last Transition Time: 2022-11-07T06:44:38Z
Message:
Reason:
Status: True
Type: Valid
Last Transition Time: 2022-11-07T06:44:39Z
Message:
Reason:
Status: True
Type: Initialized
Last Transition Time: 2022-11-07T06:44:39Z
Message:
Reason:
Status: True
Type: Ready
Last Server Node Started: 2022-11-07T08:02:30Z
Node Statuses:
k8ssandra-dc1-rack1-sts-0:
Host ID: 70c0d9(...ommision)27bc7295fb
k8ssandra-dc1-rack1-sts-1:
Host ID: 25fc276d-3(...ommision)141
k8ssandra-dc1-rack1-sts-2:
Host ID: 3dba(...ommision)ea25c5
k8ssandra-dc1-rack1-sts-3:
Host ID: 00ba9c(...ommision)d6663
k8ssandra-dc1-rack1-sts-4:
Host ID: 6cc0b(...ommision)3
Observed Generation: 3
Quiet Period: 2022-11-07T08:07:16Z
Super User Upserted: 2022-11-07T08:07:11Z
Users Upserted: 2022-11-07T08:07:11Z
Events: <none>
I look forward to hearing from you. Thank you.