Overview
Cass Operator 1.7.0 is a big release in large part because it is the first release since migrating the project to the k8ssandra GitHub org. Because there are some significant changes and because of some issues we hit, I want to go over what is involved with upgrading
Note: This info is only relevant if you are directly running cass-operator. It does not apply if you are running k8ssandra.
StatefulSet Update Bug
1.7.0 has a bug that makes it unusable when upgrading - Updating Statefulsets is broken when upgrading to 1.7.0 · Issue #103 · k8ssandra/cass-operator · GitHub.
cass-operator attempts to change the value of an immutable property of the StatefulSets. This causes StatefulSet updates to fail. Your Cassandra pods will continue to run, any changes in your CassandraDatacenter that would normally propagate to the underlying StatefulSets will not be applied. For example, if you try to change the heap size, it won’t happen. I want to reiterate that this will be a problem for existing CassandraDatacenters installed with versions of cass-operator prior to 1.7.0.
This will not be an issue for new CassandraDatacenters created with cass-operator 1.7.0.
Cass Operator 1.7.1
Cass Operator 1.7.1 has already been released with a fix for Updating Statefulsets is broken when upgrading to 1.7.0 · Issue #103 · k8ssandra/cass-operator · GitHub.
To add to the confusion, it was reported today that the cass-operation image tag was wrong in the 1.7.1 manifests. See cass-operator image tag is not updated to v1.7.1 in cass-operator-manifests.yaml in 1.7.1 release · Issue #106 · k8ssandra/cass-operator · GitHub for details. This issue has seen been resolved.
Upgrading
Here is a quick summary of upgrade steps with more details to follow:
- Upgrade to cass-operator 1.6.0
- If you are running Cassandra 3.11.6, upgrade to 3.11.7 or later.
- Upgrade to cass-operator 1.7.0
Non-root images
Prior to 1.7.0 the default images ran Cassandra as root. Starting in 1.7.0, the default images run Cassandra with a non-root user.
What do I mean by default images? When you specify the Cassandra version to use by setting the serverVersion property cass-operator picks a default image based on the Cassandra version. All of those default images now run as a non-root user.
If you are setting the serverImage property and using a non-default image that runs as root, you will need to set dockerImageRunsAsCassandra: false. Without this the Cassandra container will fail to start due to some file system permission errors.
Cassandra Versions
Cass Operator supports Cassandra 3.11.6 through 4.0 RC1 (as of this writing). The default image/version mapping for 3.11.6 was removed in 1.7.0. If you try to create/update a CassadraDatacenter with serverVersion: "3.11.6", it is going to fail.
The best way to deal with this is by upgrading Cassandra to 3.11.7 or later prior to upgrading Cass Operator.
Version Upgrade Path
I have been testing from versions other than 1.6.0. I ran into some problems that made the upgrades more complicated. Because of this the safest and easiest path is to first upgrade to 1.6.0 and then upgrade to 1.7.1.
Wrap Up
If you have questions don’t hesitate to reach out on the forum or on Discord!
Cheers
John