Standalone Medusa crashing

K8ssandra help
1

We have a cluster where the older CRDs for k8ssandra have been installed (and we are running an older cluster with the old setup), and now we have also installed the CRDs for the k8ssandra-operator (testing a new setup to migrate to).

[root@c6420-5j8s2s2-obs-ad ~]# kubectl get crd | grep ssandracassandrabackups.cassandra.k8ssandra.io 2021-01-29T12:08:29Z cassandradatacenters.cassandra.datastax.com 2021-01-29T12:08:31Z cassandrarestores.cassandra.k8ssandra.io 2021-01-29T12:08:30Z clientconfigs.config.k8ssandra.io 2023-08-24T13:14:44Z
k8ssandraclusters.k8ssandra.io 2023-08-24T13:14:44Z
k8ssandratasks.control.k8ssandra.io 2023-08-24T13:14:45Z
medusabackupjobs.medusa.k8ssandra.io 2023-08-24T13:14:45Z
medusabackups.medusa.k8ssandra.io 2023-08-24T13:14:45Z
medusabackupschedules.medusa.k8ssandra.io 2023-08-24T13:14:45Z
medusarestorejobs.medusa.k8ssandra.io 2023-08-24T13:14:45Z
medusatasks.medusa.k8ssandra.io 2023-08-24T13:14:45Z
reapers.reaper.cassandra-reaper.io 2021-01-29T12:08:31Z
reapers.reaper.k8ssandra.io 2023-08-24T13:14:45Z
replicatedsecrets.replication.k8ssandra.io 2023-08-24T13:14:45Z
stargates.stargate.k8ssandra.io 2023-08-24T13:14:45Z

I tried to install the control-plane in a namespace (v1.8.1) but my medusa-standalone pod doesn’t start up. Therefore the cluster does not come up. I have a k8ssandra-operator and a cass-operator-controller-manager pod that are running. The weird thing about the medusa-standalone pod is the error, its not finding python packages?

[louiseo@pc5053 ~/Documents/gitlab.met.no/tjenester/oda]$ kubectl -n oda-2023-dev logs k8ssandra-oda-medusa-standalone-69f6ff4476-98pqj
MEDUSA_MODE = GRPC
sleeping for 0 sec
Starting Medusa gRPC service
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/cassandra/medusa/service/grpc/server.py", line 27, in <module>
    import grpc
ModuleNotFoundError: No module named 'grpc'

Its yaml looks fine?

[louiseo@pc5053 ~/Documents/gitlab.met.no/tjenester/oda]$ kubectl -n oda-2023-dev describe deployment k8ssandra-oda-medusa-standalone
Name:                   k8ssandra-oda-medusa-standalone
Namespace:              oda-2023-dev
CreationTimestamp:      Thu, 07 Sep 2023 22:52:58 +0200
Labels:                 k8ssandra.io/cleaned-up-by=k8ssandracluster-controller
                        k8ssandra.io/cluster-name=k8ssandra
                        k8ssandra.io/cluster-namespace=oda-2023-dev
Annotations:            deployment.kubernetes.io/revision: 1
                        k8ssandra.io/resource-hash: Lerb0TCjMQIPRhFsbnLMJ6ljameC8FSKEqdwmW+5IuM=
Selector:               app=k8ssandra-oda-medusa-standalone
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=k8ssandra-oda-medusa-standalone
  Containers:
   medusa:
    Image:      docker.io/k8ssandra/medusa:0.15.0
    Port:       50051/TCP
    Host Port:  0/TCP
    Limits:
      memory:  8Gi
    Requests:
      cpu:      100m
      memory:   100Mi
    Liveness:   exec [/bin/grpc_health_probe --addr=:50051] delay=10s timeout=1s period=10s #success=1 #failure=10
    Readiness:  exec [/bin/grpc_health_probe --addr=:50051] delay=10s timeout=1s period=10s #success=1 #failure=10
    Environment:
      MEDUSA_MODE:                  GRPC
      MEDUSA_TMP_DIR:               /var/lib/cassandra
      POD_NAME:                      (v1:metadata.name)
      CQL_USERNAME:                 <set to the key 'username' in secret 'k8ssandra-medusa'>  Optional: false
      CQL_PASSWORD:                 <set to the key 'password' in secret 'k8ssandra-medusa'>  Optional: false
      MEDUSA_RESOLVE_IP_ADDRESSES:  False
    Mounts:
      /etc/cassandra from server-config (rw)
      /etc/medusa from k8ssandra-medusa (rw)
      /etc/medusa-secrets from s3-secret (rw)
      /etc/podinfo from podinfo (rw)
      /var/lib/cassandra from server-data (rw)
  Volumes:
   server-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   server-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   medusa-backups:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   k8ssandra-medusa:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      k8ssandra-medusa
    Optional:  false
   s3-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  s3-secret
    Optional:    false
   podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  <none>
NewReplicaSet:   k8ssandra-oda-medusa-standalone-69f6ff4476 (1/1 replicas created)
2

Hi,

I was unable to replicate this behavior, so I was wondering, could you output the YAML of the Pod for standalone Medusa? Just do kubectl get k8ssandra-oda-medusa-standalone and output that one. I can’t think of any other reason than the startup entrypoint somehow being overridden.

  • Micke
3

Hmm, interesting that you say it could be that the entrypoint is overridden. What does your yaml look like in a functioning deployment?

[louiseo@pc5053 ~/Documents/gitlab.met.no/tjenester/oda-2023(dev)]$ kubectl -n oda-2023-dev get deployments.apps k8ssandra-oda-medusa-standalone  -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    k8ssandra.io/resource-hash: Lerb0TCjMQIPRhFsbnLMJ6ljameC8FSKEqdwmW+5IuM=
  creationTimestamp: "2023-09-14T12:35:11Z"
  generation: 1
  labels:
    k8ssandra.io/cleaned-up-by: k8ssandracluster-controller
    k8ssandra.io/cluster-name: k8ssandra
    k8ssandra.io/cluster-namespace: oda-2023-dev
  name: k8ssandra-oda-medusa-standalone
  namespace: oda-2023-dev
  resourceVersion: "2616931329"
  uid: ddbaeed6-b49f-4996-b4fb-19565e811550
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: k8ssandra-oda-medusa-standalone
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: k8ssandra-oda-medusa-standalone
    spec:
      containers:
      - env:
        - name: MEDUSA_MODE
          value: GRPC
        - name: MEDUSA_TMP_DIR
          value: /var/lib/cassandra
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: CQL_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: k8ssandra-medusa
        - name: CQL_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: k8ssandra-medusa
        - name: MEDUSA_RESOLVE_IP_ADDRESSES
          value: "False"
        image: docker.io/k8ssandra/medusa:0.15.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - --addr=:50051
          failureThreshold: 10
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: medusa
        ports:
        - containerPort: 50051
          name: grpc
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - --addr=:50051
          failureThreshold: 10
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 8Gi
          requests:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/cassandra
          name: server-config
        - mountPath: /var/lib/cassandra
          name: server-data
        - mountPath: /etc/medusa
          name: k8ssandra-medusa
        - mountPath: /etc/podinfo
          name: podinfo
        - mountPath: /etc/medusa-secrets
          name: s3-secret
      dnsPolicy: ClusterFirst
      hostname: k8ssandra-oda-medusa-standalone
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: server-config
      - emptyDir: {}
        name: server-data
      - emptyDir: {}
        name: medusa-backups
      - configMap:
          defaultMode: 420
          name: k8ssandra-medusa
        name: k8ssandra-medusa
      - name: s3-secret
        secret:
          defaultMode: 420
          secretName: s3-secret
      - downwardAPI:
          defaultMode: 420
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels
            path: labels
        name: podinfo
status:
  conditions:
  - lastTransitionTime: "2023-09-14T12:35:11Z"
    lastUpdateTime: "2023-09-14T12:35:11Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2023-09-14T12:45:12Z"
    lastUpdateTime: "2023-09-14T12:45:12Z"
    message: ReplicaSet "k8ssandra-oda-medusa-standalone-69f6ff4476" has timed out
      progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1