Hi.
My first post here, so hope it’s the right place.
We are running several “smaller” 5-node Cassandra clusters (3.11.4). Each cluster has it’s own Reaper instance. Still running Reaper 2.2.x or 2.3.x.
I just had a production Reaper crash with OOM error. It’s running in Azure on a Standard_DS11_v2 (2 vCPU’s and 14GB RAM). Heap is configured as default value of 2GB.
I struggle to find official documentation, so what are the recommendations? I assume raising it from 2 to 3GB or even a bit more would be OK?
Also, why is there no configuration property for heap size? I have to modify it in the /usr/local/bin/cassandra-reaper file itself. Perhaps this is “normal” ?
Thanks in advance for any quick help on this.
Have a nice day.
Raising the heap to 3GB or 4GB is fine. 2GB works for most but it depends on how many clusters and/or total nodes are managed. Is it possible you have lots of tables in the cluster (above 200)? I’m just speculating on whether it might be the cause of the OOM.
And yes, you’ll need to modify cassandra-reaper directly. If you want, you’re welcome to submit a PR so that it’s configurable in a properties file. Cheers!
Thanks for the quick reply @ErickRamirezAU . I changed it to 3GB.
And yes, we do have a lot of tables! I can tell you exactly how many - 704 (!) - as this is monitored via Grafana.
And don’t ask … But we have migrated from an Oracle world a few years ago, so not exactly following best practices on designing for Cassandra. We have one keyspace per customer environment, so you can say 1 keyspace = 1 Oracle instance. Not exactly what is recommended. It works, but has limitations. For example that we can’t really have much more than 15 keyspaces per cluster. Then performance suffers.
Perhaps Cassandra 4 improves this a bit with Java 11 - and even better with Java 17 (hopefully soon). We have plans for upgrading in the near future.
Thanks again.