Medusa backup job created with no status - no logs in medusa container

K8ssandra help
1

Problem: I am trying to create medusa backup by running below command : kubectl apply -n k8ssandra-operator -f ‘medusa-backup-full.yaml’. This creates the medusabackup job in MedusaBackupJobs folder, but I never get the status in the medusabackup.

I checked the medusa container log and couldn’t find any logs for the backup job snapshot creation or upload, the only logs that I could see in the medusa is as below => “starting server. Listening on port 50051.”

Below is the logs:
MEDUSA_MODE = GRPC
sleeping for 0 sec
Starting Medusa gRPC service
WARNING:root:The CQL_USERNAME environment variable is deprecated and has been replaced by the MEDUSA_CQL_USERNAME variable
WARNING:root:The CQL_PASSWORD environment variable is deprecated and has been replaced by the MEDUSA_CQL_PASSWORD variable
WARNING:root:The CQL_USERNAME environment variable is deprecated and has been replaced by the MEDUSA_CQL_USERNAME variable
WARNING:root:The CQL_PASSWORD environment variable is deprecated and has been replaced by the MEDUSA_CQL_PASSWORD variable
INFO:root:Init service
[2023-10-13 00:34:22,039] INFO: Init service
DEBUG:root:Loading storage_provider: s3_compatible
[2023-10-13 00:34:22,040] DEBUG: Loading storage_provider: s3_compatible
DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
[2023-10-13 00:34:22,040] DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway
[2023-10-13 00:34:22,041] DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
[2023-10-13 00:34:22,042] DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
[2023-10-13 00:34:22,043] DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
[2023-10-13 00:34:22,043] DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
[2023-10-13 00:34:22,043] DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section
[2023-10-13 00:34:22,044] DEBUG: Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
[2023-10-13 00:34:22,045] DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
DEBUG:botocore.hooks:Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section
[2023-10-13 00:34:22,046] DEBUG: Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
[2023-10-13 00:34:22,046] DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section
[2023-10-13 00:34:22,046] DEBUG: Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section
DEBUG:root:Setting AWS credentials file to /etc/medusa-secrets/credentials
[2023-10-13 00:34:22,046] DEBUG: Setting AWS credentials file to /etc/medusa-secrets/credentials
DEBUG:botocore.session:Setting config variable for credentials_file to ‘/etc/medusa-secrets/credentials’
[2023-10-13 00:34:22,046] DEBUG: Setting config variable for credentials_file to ‘/etc/medusa-secrets/credentials’
DEBUG:botocore.utils:IMDS ENDPOINT: http://169.254.xxx.xxx/
[2023-10-13 00:34:22,054] DEBUG: IMDS ENDPOINT: http://169.254.xxx.xxx/
DEBUG:botocore.credentials:Looking for credentials via: env
[2023-10-13 00:34:22,056] DEBUG: Looking for credentials via: env
DEBUG:botocore.credentials:Looking for credentials via: assume-role
[2023-10-13 00:34:22,057] DEBUG: Looking for credentials via: assume-role
DEBUG:botocore.credentials:Looking for credentials via: assume-role-with-web-identity
[2023-10-13 00:34:22,057] DEBUG: Looking for credentials via: assume-role-with-web-identity
DEBUG:botocore.credentials:Looking for credentials via: sso
[2023-10-13 00:34:22,057] DEBUG: Looking for credentials via: sso
DEBUG:botocore.credentials:Looking for credentials via: shared-credentials-file
[2023-10-13 00:34:22,057] DEBUG: Looking for credentials via: shared-credentials-file
INFO:botocore.credentials:Found credentials in shared credentials file: /etc/medusa-secrets/credentials
[2023-10-13 00:34:22,058] INFO: Found credentials in shared credentials file: /etc/medusa-secrets/credentials
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): minio-di-dev.xxx.cloud:443
[2023-10-13 00:34:22,061] DEBUG: Starting new HTTPS connection (1): minio-di-dev.xxx.cloud:443
DEBUG:urllib3.connectionpool:https://minio-di-dev.xxx.cloud:443 “HEAD /di-cassandra-backups HTTP/1.1” 200 0
[2023-10-13 00:34:22,092] DEBUG: https://minio-di-dev.xxx.cloud:443 “HEAD /di-cassandra-backups HTTP/1.1” 200 0
INFO:root:Starting server. Listening on port 50051.
[2023-10-13 00:34:22,094] INFO: Starting server. Listening on port 50051.

Can I check anything else to debug this issue. Thanks in Advance.

2

anyone please help
@alexander @ErickRamirez

3

Hi,

could you share with us the following elements:

  • The version of k8ssandra-operator and Medusa you’re currently running
  • The K8ssandraCluster manifest
  • The MedusaBackupJob manifest
  • Check the logs of the k8ssandra-operator container for any error message relating to the backup operation

Thanks

4
  • The version of k8ssandra-operator and Medusa you’re currently running
    K8sssandra-operator version : v1.2.0
    NAME : k8ssandra-operator
    NAMESPACE : k8ssandra-operator
    REVISION : 1
    UPDATED : 2023-06-21 15:37:26.530954 +1000 +1000
    STATUS : deployed
    CHART : k8ssandra-operator-0.38.0
    APP VERSION : 1.2.0
    Medusa version : medusa:0.13.4 → this version is for medusa container
    Medusa operator : v0.4.0
  • The K8ssandraCluster manifest:
    apiVersion: k8ssandra.io/v1alpha1
    kind: K8ssandraCluster
    metadata:
    name: cass
    spec:
    cassandra:
    serverVersion: “4.0.4”
    datacenters:
    - metadata:
    name: dc1
    image:
    registry: private-xx.xx.xxx.cloud/docker-repository
    jmxInitContainerImage:
    registry: private-xx.xx.xxx.cloud/docker-repository
    size: 4
    storageConfig:
    cassandraDataVolumeClaimSpec:
    storageClassName: xxxcloud-vol-storage
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 80Gi
    config:
    jvmOptions:
    heapSize: 4Gi
    stargate:
    containerImage:
    registry: private-xx.xx.xxx.cloud/docker-repository
    tag: v1.0.63
    size: 1
    heapSize: 300M
    resources:
    limits:
    memory: 20Gi
    cpu: 8
    requests:
    memory: 10Gi
    cpu: 4
    reaper:
    autoScheduling:
    enabled: false
    containerImage:
    name: cassandra-reaper
    registry: private-xx.xx.xxx.cloud/docker-repository
    repository: thelastpickle
    tag: 3.2.0
    deploymentMode: PER_DC
    heapSize: 2Gi
    initContainerImage:
    name: cassandra-reaper
    registry: private-xx.xx.xxx.cloud/docker-repository
    repository: thelastpickle
    tag: 3.2.0
    keyspace: reaper_db
    medusa:
    containerImage:
    registry: private-xx.xx.xxx.cloud/docker-repository
    tag: 0.13.4
    storageProperties:
    bucketName: di-cassandra-backups
    concurrentTransfers: 1
    host: minio-xx-xx.xx-corp.cloud
    port: 443
    maxBackupAge: 1
    maxBackupCount: 2
    multiPartUploadThreshold: 10000000000000000
    prefix: dc1
    storageProvider: s3_compatible
    storageSecretRef:
    name: di-cass-backups
  • The MedusaBackupJob manifest:
    apiVersion: medusa.k8ssandra.io/v1alpha1
    kind: MedusaBackupJob
    metadata:
    annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
    {“apiVersion”:“medusa.k8ssandra.io/v1alpha1",“kind”:“MedusaBackupJob”,“metadata”:{“annotations”:{},“name”:“backupfull-oct16-2023”,“namespace”:“k8ssandra-operator”},“spec”:{“backupType”:“full”,“cassandraDatacenter”:"dc1”}}
    creationTimestamp: “2023-10-16T23:11:08Z”
    generation: 1
    managedFields:

  • apiVersion: medusa.k8ssandra.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
    f:metadata:
    f:annotations:
    .: {}
    f:kubectl.kubernetes.io/last-applied-configuration: {}
    f:spec:
    .: {}
    f:backupType: {}
    f:cassandraDatacenter: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: “2023-10-16T23:11:08Z”

  • apiVersion: medusa.k8ssandra.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
    f:metadata:
    f:ownerReferences:
    .: {}
    k:{“uid”:“6d1b456a-49da-4b43-bf3b-7e2832e8d9a2”}: {}
    manager: manager
    operation: Update
    time: “2023-10-16T23:11:08Z”
    name: backupfull-oct16-2023
    namespace: k8ssandra-operator
    ownerReferences:

  • apiVersion: cassandra.datastax.com/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CassandraDatacenter
    name: dc1
    uid: 6d1b456a-49da-4b43-bf3b-7e2832e8d9a2
    resourceVersion: “449137244”
    uid: effcdd89-e32d-461b-8ae7-f8df88d8c738
    spec:
    backupType: full
    cassandraDatacenter: dc1

  • Check the logs of the k8ssandra-operator container :
    found one error :

Backups have not been started yet {“reconciler group”: “medusa.k8ssandra.io”, “reconciler kind”: “MedusaBackupJob”, “name”: “backupfull-oct16-2023”, “namespace”: “k8ssandra-operator”, “medusabackupjob”: “k8ssandra-operator/backupfull-oct16-2023”}

1.6976541951570272e+09 ERROR controller.medusabackupjob medusa is not deployed {“reconciler group”: “medusa.k8ssandra.io”, “reconciler kind”: “MedusaBackupJob”, “name”: “backupfull-oct16-2023”, “namespace”: “k8ssandra-operator”, “medusabackupjob”: “k8ssandra-operator/backupfull-oct16-2023”, “CassandraDatacenter”: {“namespace”: “k8ssandra-operator”, “name”: “dc1”}, “error”: “the backup sidecar was not found”}

  • Medusa container is running fine, I tried running backup from medusa cmd and I was able to do it. but when I try using medusabackupjob it does not work.

@alexander FYI:
@ErickRamirez