Filter out new metrics

K8ssandra help
1

Running k8ssandra-operator 1.12, cassandra 4.3.1. Can you please clarify the format to filter out some metrics? For example, this is what I have in K8ssandraCluster spec:

cassandra:
    telemetry:
      mcac:
        enabled: false
      cassandra:
        relabels:

        - sourceLabels:
            - __name__
          regex: "jvm.*"
          action: "drop"

        - source_labels: ["table"]
          separator: "@"
          regex: ".+"
          action: "drop"

First - remove all metrics with names matching jvm*.
Second - remove all metrics that have a table label.

This doesn’t seem to work, the set of metrics is still the same.
Thanks.

2

Hi,

The JVM metrics are not filtered, since this method does not apply to the JVM metrics (the JVM metrics are just proxied). For tables, I could not replicate your behavior. I created the following K8ssandraCluster:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: demo
spec:
  cassandra:
    serverVersion: "4.1.4"
    storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: standard
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
    config:
      jvmOptions:
        heapSize: 512M
    networking:
      hostNetwork: true
    datacenters:
      - metadata:
          name: dc1
        size: 1
        telemetry:
          cassandra:
            relabels:
              - sourceLabels: ["table"]
                regex: ".+"
                action: "drop"
          mcac:
            enabled: false

And then tried to fetch table metrics:

➜  k8ssandra-operator git:(main) ✗ kubectl exec -it pod/demo-dc1-default-sts-0 -- bash
Defaulted container "cassandra" out of: cassandra, server-system-logger, server-config-init-base (init), server-config-init (init)
cassandra@kind-worker5:/$ curl localhost:9000/metrics | grep \"table
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3568k  100 3568k    0     0  14.5M      0 --:--:-- --:--:-- --:--:-- 14.5M
cassandra@kind-worker5:/$

But all of them were filtered.

3

I had “telemetry” under cassandra, I moved it into datacenters as your example shows. I also upgraded to server version 4.1.4.

  cassandra:
...
    datacenters:
      - metadata:
        name:  ...
        size: ...
        telemetry:
          cassandra:
            relabels:
              - sourceLabels: ["table"]
                regex: ".+"
                action: "drop"
          mcac:
            enabled: false

I still see many metrics that have the table label:
kubectl exec -it -n k8ssandra-operator …-sts-0 – curl localhost:9000/metrics | grep "table
…
org_apache_cassandra_metrics_table_all_memtables_live_data_size{host=“da750756-57b5-47be-a592-d6444ce776ea”,…, table=“table_estimates”,should_drop=“false”,} 0.0
org_apache_cassandra_metrics_table_all_memtables_on_heap_data_size{host=“da750756-57b5-47be-a592-d6444ce776ea”,… table=“tables”,should_drop=“false”,} 0.0
org_apache_cassandra_metrics_table_all_memtables_on_heap_data_size{host=“da750756-57b5-47be-a592-d6444ce776ea”,… table=“table_estimates”,should_drop=“false”,} 0.0

4

Where are relabels saved - some specific pod? If so, what’s the file name?

I tried adding telemetry in two places:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
..
spec:
..
  cassandra:
    telemetry:
      cassandra:
        relabels:
          - sourceLabels: ["table"]
            regex: ".+"
            action: "drop"
      mcac:
        enabled: false
      prometheus:
        enabled: false
...
    datacenters:
      - metadata:
..
        telemetry:
          cassandra:
            relabels:
              - sourceLabels: ["table"]
                regex: ".+"
                action: "drop"
          mcac:
            enabled: false
          prometheus:
            enabled: false

I’m still getting metrics with table=", e.g.

org_apache_cassandra_metrics_table_all_memtables_on_heap_data_size{host="938efab7-10ca-4907-b29c-6c6c1b4619ac",instance="10.78.37.98",cluster="test-cassandra",datacenter="dc1",rack="us-west-2a",pod_name="test-cassandra-dc1-us-west-2a-sts-0",node_name="ip-10-78-37-172.us-west-2.compute.internal",keyspace="system",table="table_estimates",should_drop="false",} 268514.0

I don’t know if it matters, but

kubectl describe K8ssandraCluster -A

produces

  Error:                      telemetry spec was invalid for this cluster - is Prometheus installed if you have requested it
    Events:
      Type     Reason           Age                     From                         Message
      ----     ------           ----                    ----                         -------
      Warning  Reconcile Error  4m18s (x19 over 9m48s)  k8ssandracluster-controller  telemetry spec was invalid for this cluster - is Prometheus installed if you have requested it

Thanks.

Thanks.

5

I figured it out, it was a user error, filtering does work. Thanks.