Hello and thanks to anyone if you know anything about this issue
I created a simple medusabackupschedule and it seems to at least populate the bucket with objects
the problem is that the medusabackupjobs is stuck “in progress” for a really long time, the database is only 40Mb so I don’t see a reason for this
As a result of status being stuck the “medusabackups” object is not created so we can not use it for restore.
apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaBackupSchedule
metadata:
name: medusa-backup-schedule
namespace: sre
spec:
backupSpec:
backupType: differential
cassandraDatacenter: us-east-dc
cronSchedule: "*/15 * * * *"
disabled: false
the object looks like this:
apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaBackupJob
metadata:
creationTimestamp: "2023-12-03T10:00:27Z"
generation: 1
name: medusa-backup-schedule-1701597627
namespace: sre
ownerReferences:
- apiVersion: cassandra.datastax.com/v1beta1
blockOwnerDeletion: true
controller: true
kind: CassandraDatacenter
name: us-east-dc
uid: 86a92fd3-e6a0-4dc6-a583-23fdef315317
resourceVersion: "16799984"
uid: d8cd0fdc-33cb-43aa-a75b-aa0e5401de60
spec:
backupType: differential
cassandraDatacenter: us-east-dc
status:
inProgress:
- cassandra-us-east-dc-r1-sts-0
- cassandra-us-east-dc-r1-sts-2
- cassandra-us-east-dc-r1-sts-1
startTime: "2023-12-03T10:00:42Z"
logs from k8ssandra-operator
2023-12-03T11:35:56.804Z INFO connecting to backup sidecar {"controller": "medusabackupjob", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaBackupJob", "MedusaBackupJob": {"name":"medusa-backup-schedule-1701597627","namespace":"sre"}, "namespace": "sre", "name": "medusa-backup-schedule-1701597627", "reconcileID": "fda6f0a4-19de-4db5-a52f-f6cb54163b39", "medusabackupjob": "sre/medusa-backup-schedule-1701597627", "Pod": "cassandra-us-east-dc-r1-sts-1", "Address": "10.40.175.241:50051"}
2023-12-03T11:35:57.180Z INFO MedusaBackupJob is still being processed {"controller": "medusabackupjob", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaBackupJob", "MedusaBackupJob": {"name":"medusa-backup-schedule-1701597627","namespace":"sre"}, "namespace": "sre", "name": "medusa-backup-schedule-1701597627", "reconcileID": "fda6f0a4-19de-4db5-a52f-f6cb54163b39", "medusabackupjob": "sre/medusa-backup-schedule-1701597627", "Backup": {"namespace": "sre", "name": "medusa-backup-schedule-1701597627"}}
logs from medusa sidecar
[2023-12-03 11:37:32,043] DEBUG: [Storage] Getting object cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701598500/meta/manifest.json
DEBUG:root:Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701598500/meta/manifest.json last modification time is 2023-12-03 10:15:31+00:00
[2023-12-03 11:37:32,055] DEBUG: Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701598500/meta/manifest.json last modification time is 2023-12-03 10:15:31+00:00
DEBUG:root:Disconnecting from S3...
[2023-12-03 11:37:32,055] DEBUG: Disconnecting from S3...
DEBUG:root:Loading storage_provider: s3
[2023-12-03 11:37:33,269] DEBUG: Loading storage_provider: s3
DEBUG:root:Setting AWS credentials file to /etc/medusa-secrets/credentials
[2023-12-03 11:37:33,277] DEBUG: Setting AWS credentials file to /etc/medusa-secrets/credentials
INFO:root:Using credentials CensoredCredentials(access_key_id=A..G, secret_access_key=*****, region=us-east-1)
[2023-12-03 11:37:33,482] INFO: Using credentials CensoredCredentials(access_key_id=A..G, secret_access_key=*****, region=us-east-1)
INFO:root:Connecting to s3 with args {}
[2023-12-03 11:37:33,483] INFO: Connecting to s3 with args {}
DEBUG:root:Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/schema.cql was not found in cache.
[2023-12-03 11:37:33,490] DEBUG: Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/schema.cql was not found in cache.
DEBUG:root:[Storage] Getting object cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/schema.cql
[2023-12-03 11:37:33,490] DEBUG: [Storage] Getting object cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/schema.cql
DEBUG:root:Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/schema.cql last modification time is 2023-12-03 10:01:20+00:00
[2023-12-03 11:37:33,592] DEBUG: Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/schema.cql last modification time is 2023-12-03 10:01:20+00:00
DEBUG:root:Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/manifest.json was not found in cache.
[2023-12-03 11:37:33,592] DEBUG: Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/manifest.json was not found in cache.
DEBUG:root:[Storage] Getting object cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/manifest.json
[2023-12-03 11:37:33,592] DEBUG: [Storage] Getting object cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/manifest.json
DEBUG:root:Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/manifest.json last modification time is 2023-12-03 10:02:05+00:00
[2023-12-03 11:37:33,605] DEBUG: Blob cassandra/cassandra-us-east-dc-r1-sts-2/medusa-backup-schedule-1701597627/meta/manifest.json last modification time is 2023-12-03 10:02:05+00:00
DEBUG:root:Disconnecting from S3...
[2023-12-03 11:37:33,605] DEBUG: Disconnecting from S3...
medusa configuration is:
medusa:
containerImage:
repository: ***
registry: ***
tag: 0.16.3-119430
storageProperties:
storageProvider: s3
bucketName: <bucket-name>
storageSecretRef:
name: medusa-bucket-key
maxBackupCount: 10
region: us-east-1
we are using an image created from the original cassandra-medusa code , but removed the credentials of aws user ,and instead use a role with access to the bucket the role has the permissions as listed in medusa docs,
the removed lines in cassandra-medusa code in file :
medusa/storage/s3_base_storage.py
aws_access_key_id=self.credentials.access_key_id,
aws_secret_access_key=self.credentials.secret_access_key,
The IAM role
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetLifecycleConfiguration",
"s3:GetBucketTagging",
"s3:GetInventoryConfiguration",
"s3:PutAccelerateConfiguration",
"s3:DeleteObjectVersion",
"s3:GetObjectVersionTagging",
"s3:ListBucketVersions",
"s3:RestoreObject",
"s3:ListBucket",
"s3:GetAccelerateConfiguration",
"s3:GetBucketPolicy",
"s3:ReplicateObject",
"s3:GetObjectVersionTorrent",
"s3:GetObjectAcl",
"s3:GetEncryptionConfiguration",
"s3:GetBucketObjectLockConfiguration",
"s3:AbortMultipartUpload",
"s3:GetObjectVersionAcl",
"s3:GetObjectTagging",
"s3:GetMetricsConfiguration",
"s3:DeleteObject",
"s3:GetBucketPublicAccessBlock",
"s3:GetBucketPolicyStatus",
"s3:ListBucketMultipartUploads",
"s3:GetObjectRetention",
"s3:GetBucketWebsite",
"s3:PutReplicationConfiguration",
"s3:PutObjectLegalHold",
"s3:GetBucketVersioning",
"s3:GetBucketAcl",
"s3:GetObjectLegalHold",
"s3:GetReplicationConfiguration",
"s3:ListMultipartUploadParts",
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectTorrent",
"s3:PutObjectRetention",
"s3:GetAnalyticsConfiguration",
"s3:PutBucketObjectLockConfiguration",
"s3:GetObjectVersionForReplication",
"s3:GetBucketLocation",
"s3:ReplicateDelete",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::<bucket-name>",
"arn:aws:s3:::<bucket-name>/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:GetAccountPublicAccessBlock",
"s3:HeadBucket"
],
"Resource": "*"
}
]
}
the role is only attached to the cassandra sts and not to medusa standalone