Hi,
We use Cassandra DB on k8s hosted on Azure infrastructure, our set-up is 1 DC with 3 nodes each node with 14 core CPU request and limit. We have recently encountered an issue where the pending tasks have shot up from ~10 to ~50k in 30 seconds on only pod causing the Cassandra pod restart. The other 2 pods had steady pending tasks and were processing normal. Please could you suggest any pointers on why such a sudden spike in pending tasks.Below are the last events from Node(VM)
Warning TaskHung 42m kernel-monitor INFO: task jbd2/sda1-8:216 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task systemd-journal:264 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task python3:2211 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task python3:2467 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task containerd:3110 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task containerd:4852 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task containerd-shim:4895 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task defender:8662 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor INFO: task svlogd:7395 blocked for more than 120 seconds.
Warning TaskHung 42m kernel-monitor (combined from similar events): INFO: task grafana-agent:2842286 blocked for more than 120 seconds
