Good afternoon!
In the process of testing a solution based on the multi-cluster version of k8ssandra, some problems arose, with which I really hope for help.
4 clusters are used (clusters k8s-10 and k8s-20 are located in city A, and clusters k8s-30 and k8s-40 are located in city B)
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-10 Ready control-plane 4d3h v1.24.3 172.16.131.65 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-11 Ready <none> 4d3h v1.24.3 172.16.131.66 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-12 Ready <none> 4d3h v1.24.3 172.16.131.72 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-13 Ready <none> 4d3h v1.24.3 172.16.131.73 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-20 Ready control-plane 4d3h v1.24.3 172.16.131.61 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-21 Ready <none> 4d3h v1.24.3 172.16.131.62 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-22 Ready <none> 4d3h v1.24.3 172.16.131.63 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-23 Ready <none> 4d3h v1.24.3 172.16.131.64 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-30 Ready control-plane 24h v1.24.3 172.17.5.219 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-31 Ready <none> 24h v1.24.3 172.17.5.220 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-32 Ready <none> 24h v1.24.3 172.17.5.221 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-33 Ready <none> 24h v1.24.3 172.17.5.222 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-40 Ready control-plane 24h v1.24.3 172.17.5.215 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-41 Ready <none> 24h v1.24.3 172.17.5.216 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-42 Ready <none> 24h v1.24.3 172.17.5.217 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
k8s-43 Ready <none> 24h v1.24.3 172.17.5.218 <none> CentOS Stream 8 4.18.0-408.el8.x86_64 containerd://1.2.6
Calico is used as CNI, and use pv-local with this config:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-local-node-1
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
local:
path: /mnt/local-storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-41
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-local-node-2
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
local:
path: /mnt/local-storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-42
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-local-node-3
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
local:
path: /mnt/local-storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-43
The first problem arose when using the script to create the userconfig. As a result, a secret was created in the control-plane without a token and a certificate.
To solve the problem, a secret has been manually created in each data-plane cluster, and certificates and tokens have already been added to the control-plane.
However, after applying k8cm1.yml (the cluster names are corrected for the conditions) we see this picture:
NAME READY STATUS RESTARTS AGE
pod/demo-dc1-default-sts-0 0/2 Init:0/2 0 5h42m
pod/demo-dc1-default-sts-1 0/2 Init:0/2 0 5h42m
pod/demo-dc1-default-sts-2 0/2 Pending 0 5h42m
describe pod/demo-dc1-default-sts-0 :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 32m (x26 over 5h38m) kubelet Unable to attach or mount volumes: unmounted volumes=[server-data], unattached volumes=[server-logs server-data encryption-cred-storage server-config kube-api-access-m7vjx]: timed out waiting for the condition
Warning FailedMount 11m (x41 over 5h34m) kubelet Unable to attach or mount volumes: unmounted volumes=[server-data], unattached volumes=[server-config kube-api-access-m7vjx server-logs server-data encryption-cred-storage]: timed out waiting for the condition
Warning FailedMount 7m23s (x173 over 5h43m) kubelet MountVolume.NewMounter initialization failed for volume "pv-local-node-1" : path "/mnt/local-storage" does not exist
Warning FailedMount 2m36s (x27 over 5h25m) kubelet Unable to attach or mount volumes: unmounted volumes=[server-data], unattached volumes=[encryption-cred-storage server-config kube-api-access-m7vjx server-logs server-data]: timed out waiting for the condition
describe pod/demo-dc1-default-sts-1 :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 33m (x17 over 5h33m) kubelet Unable to attach or mount volumes: unmounted volumes=[server-data], unattached volumes=[kube-api-access-vn7v7 server-logs server-data encryption-cred-storage server-config]: timed out waiting for the condition
Warning FailedMount 29m (x47 over 5h37m) kubelet Unable to attach or mount volumes: unmounted volumes=[server-data], unattached volumes=[server-config kube-api-access-vn7v7 server-logs server-data encryption-cred-storage]: timed out waiting for the condition
Warning FailedMount 13m (x34 over 5h42m) kubelet Unable to attach or mount volumes: unmounted volumes=[server-data], unattached volumes=[server-data encryption-cred-storage server-config kube-api-access-vn7v7 server-logs]: timed out waiting for the condition
Warning FailedMount 8m58s (x24 over 5h5m) kubelet Unable to attach or mount volumes: unmounted volumes=[server-data], unattached volumes=[server-logs server-data encryption-cred-storage server-config kube-api-access-vn7v7]: timed out waiting for the condition
Warning FailedMount 2m36s (x176 over 5h44m) kubelet MountVolume.NewMounter initialization failed for volume "pv-local-node-2" : path "/mnt/local-storage" does not exist
describe pod/demo-dc1-default-sts-2 :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 68s (x61 over 5h1m) default-scheduler 0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 2 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't match pod anti-affinity rules, 3 node(s) had volume node affinity conflict, 4 Insufficient cpu. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.