K8ssandra medusa with self hosted S3

Andrea_Cucciarre · May 8, 2023, 7:48am

I have run some test on k8ssandra medusa, about the connection to self hosted S3 via https.
I have tested that it fails to connect to self hosted S3 via https when certificate is self-signed and SSL verify is set to false, for that I have raised the jira with proposed fix:

github.com/thelastpickle/cassandra-medusa

K8ssandra medusa container can't connect to s3 storage with self-signed certificate

opened 08:40AM - 21 Dec 22 UTC

handrea2009

K8ssandra medusa containers fails to start cause it can't connect to s3 compatib…le storage with self-signed certificate. The medusa container log the following errors (I got the same whether secure = false or secure = true in medusa.ini) `[2022-12-21 06:55:58,111] DEBUG: Starting new HTTPS connection (1): XXXXXX.com:443 Traceback (most recent call last): File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn conn.connect() File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/connection.py", line 426, in connect tls_in_tls=tls_in_tls, File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket sock, context, tls_in_tls, server_hostname=server_hostname File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/usr/lib/python3.6/ssl.py", line 817, in __init__ self.do_handshake() File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake self._sslobj.do_handshake() File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/cassandra/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/home/cassandra/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='XXXXXX.com', port=443): Max retries exceeded with url: /backup-medusa (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/cassandra/medusa/service/grpc/server.py", line 158, in <module> medusa_pb2_grpc.add_MedusaServicer_to_server(MedusaService(config), server) File "/home/cassandra/medusa/service/grpc/server.py", line 44, in __init__ self.storage = Storage(config=self.config.storage) File "/home/cassandra/medusa/storage/__init__.py", line 72, in __init__ self.storage_driver = self._connect_storage() File "/home/cassandra/medusa/storage/__init__.py", line 88, in _connect_storage s3_storage = S3BaseStorage(self._config) File "/home/cassandra/medusa/storage/abstract_storage.py", line 40, in __init__ self.bucket = self.driver.get_container(container_name=config.bucket_name) File "/home/cassandra/.local/lib/python3.6/site-packages/libcloud/storage/drivers/s3.py", line 357, in get_container method='HEAD') File "/home/cassandra/.local/lib/python3.6/site-packages/libcloud/common/base.py", line 623, in request headers=headers, stream=stream) File "/home/cassandra/.local/lib/python3.6/site-packages/libcloud/http.py", line 232, in request verify=self.verification File "/home/cassandra/.local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/home/cassandra/.local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/home/cassandra/.local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='XXXXX.com', port=443): Max retries exceeded with url: /backup-medusa (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))`

Now, I have test the connection to self hosted S3 via https when certificate is signed by CA and SSL verify is set to true, it works.
However, the following comment in github seems to say that it works only with http:

“Note: By default, MinIO and other self hosted S3 compatible storage systems can only be used in unsecured (non SSL) mode with Medusa due to limitations in Apache Libcloud. To enable SSL access to self hosted S3 compatible storage systems, you will need to set the environment variable SSL_CERT_FILE to the path of a valid certificate file containing trusted CA certificates of your S3 service. In order to get cluster wide commands working properly, you will need to set this in the /etc/default/cassandra-medusa file on all nodes running Medusa containing…”

github.com

thelastpickle/cassandra-medusa/blob/master/docs/minio_setup.md

MinIO Storage setup (and other S3 compatible backends)
======================================================

### Create a bucket

Create a new storage bucket in MinIO that will be used to store the backups.

### Configure Medusa

Copy your MinIO access credentials in a file called `medusa-minio-credentials` using the following format:

```
[default]
aws_access_key_id = <access_key_id>
aws_secret_access_key = <secret_access_key>
```

Place this file on all Apache Cassandra™ nodes running medusa under `/etc/medusa` and set the rights appropriately so that only users running Medusa can read/modify it.
Set the `key_file` value in the `[storage]` section of `/etc/medusa/medusa.ini` to the credentials file:

This file has been truncated. show original

Could you please clarify how k8ssandra medusa should be configured when connecting to self hosted S3 compatible storage?

Topic		Replies	Views
Medusa doesn't seem to honor secure: "false" K8ssandra help	0	310	December 16, 2022
Injecting certificate authorities to medusa container K8ssandra help	0	268	May 17, 2023
Medusa (bad) credentials errors K8ssandra help	0	774	September 15, 2021
Medusa Restore doesn't work K8ssandra help	3	37	September 3, 2024
Medusa 0.16 was released	0	40	September 21, 2023

K8ssandra

K8ssandra medusa with self hosted S3

Related topics