Hi all, I’m in the process of switching over our production Cassandra cluster to k8ssandra but in order to do this I need to be able to observe the cluster and set up alerts etc in Grafana.
After an install of k8ssandra 1.3.0 and Cassandra 4.0.0 (on EKS with Helm) with Prometheus and Grafana, data flows into Prometheus but the logs quickly fill up with out-of-order timestamp see:
Not too bad, as we can still view the data in Grafana but after a couple of days MCAC timestamps starts to lag behind (up to 4 hours) which results in Prometheus dropping the messages and an empty Cassandra Grafana dashboards:
“Error on ingesting samples that are too old or are too far into the future” num_dropped=61638
See Remove timestamp from the metrics · Issue #43 · datastax/metric-collector-for-apache-cassandra · GitHub for more details about this.
Are there any known workarounds to this as I can’t deploy to production without being able to monitor my cluster?