K8ssandra Forum

Local installation on VM: reaper and stargate are stuck

Hi dwettlaufer,

I started k8ssandra without stargate an hour ago and left for a walk. Curiously I can see now the reaper restarting although no stargate is running. Sorry, I hope I wasn’t misleading you. I have the system-server logs. They are quite long.
My cluster is built via kubeadm:

excerpt from vagrantfile:

# kubelet requires swap off
    swapoff -a
    echo "keep swap off after reboot"
    #sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  
    # Get the IP address that VirtualBox has given this VM
    IPADDR=`ifconfig enp0s8 | grep inet | grep -v inet6 | awk '/inet /{print$2}'`
    echo This VM has IP address $IPADDR

    # Set up Kubernetes
    NODENAME=$(hostname -s)
    kubeadm config images pull
    #Note: Pod and Service Cidrs may not overlap
    kubeadm init --apiserver-cert-extra-sans=$IPADDR  --node-name $NODENAME --pod-network-cidr=10.96.0.0/16 --service-cidr=10.97.0.0/16
    # Set up admin creds for the vagrant user
    echo Copying credentials to /home/vagrant...
    sudo --user=vagrant mkdir -p /home/vagrant/.kube
    cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
    chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config 

If you like, I can send you the complete vagrant file. Maybe there is an error.

reaper error:

ERROR  [2021-06-10 16:21:13,002] [main] i.c.ReaperApplication - Storage is not ready yet, trying again to connect shortly... 
org.cognitor.cassandra.migration.MigrationException: Error during migration of script 026_concurrent_repairs.cql while executing 'ALTER TABLE repair_run ADD replicas frozen<map<text, text>>;'
	at org.cognitor.cassandra.migration.Database.execute(Database.java:269)
	at java.util.Collections$SingletonList.forEach(Collections.java:4824)
	at org.cognitor.cassandra.migration.MigrationTask.migrate(MigrationTask.java:68)
	at io.cassandrareaper.storage.CassandraStorage.migrate(CassandraStorage.java:362)
	at io.cassandrareaper.storage.CassandraStorage.initializeCassandraSchema(CassandraStorage.java:293)
	at io.cassandrareaper.storage.CassandraStorage.initializeAndUpgradeSchema(CassandraStorage.java:250)
	at io.cassandrareaper.storage.CassandraStorage.<init>(CassandraStorage.java:238)
	at io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:480)
	at io.cassandrareaper.ReaperApplication.tryInitializeStorage(ReaperApplication.java:303)
	at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:181)
	at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:98)
	at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:93)
	at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:117)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Invalid column name replicas because it conflicts with an existing column
	at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
	at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
	at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293)
	at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58)
	at org.cognitor.cassandra.migration.Database.executeStatement(Database.java:277)
	at org.cognitor.cassandra.migration.Database.execute(Database.java:261)
	... 15 common frames omitted
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Invalid column name replicas because it conflicts with an existing column
	at com.datastax.driver.core.Responses$Error.asException(Responses.java:181)
	at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215)
	at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:235)
	at com.datastax.driver.core.RequestHandler.access$2600(RequestHandler.java:61)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:1011)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:814)
	at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1287)
	at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1205)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

system log (1/5):

tail: can't open '/var/log/cassandra/system.log': No such file or directory
tail: /var/log/cassandra/system.log has appeared; following end of new file
INFO  [main] 2021-06-10 14:48:17,628 SystemDistributedReplicationInterceptor.java:73 - Using override for distributed system keyspaces: {dc1=1, class=NetworkTopologyStrategy}
INFO  [main] 2021-06-10 14:48:30,789 YamlConfigurationLoader.java:92 - Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  [main] 2021-06-10 14:48:44,089 Config.java:537 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=640; batch_size_warn_threshold_in_kb=64; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=10.96.223.162; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=<REDACTED>; cluster_name=k8ssandra; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=3600000; credentials_validity_in_ms=3600000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@f001896; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_materialized_views=true; enable_sasi_indexes=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=<REDACTED>; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.96.223.162; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_flush_in_batches_legacy=true; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=-2147483648; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=3600000; permissions_validity_in_ms=3600000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; repair_session_max_tree_depth=18; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=3600000; roles_validity_in_ms=3600000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.K8SeedProvider{seeds=k8ssandra-seed-service}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_on_duplicate_row_detection=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@13f17eb4; trickle_fsync=true; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  [main] 2021-06-10 14:48:44,175 DatabaseDescriptor.java:381 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  [main] 2021-06-10 14:48:44,177 DatabaseDescriptor.java:439 - Global memtable on-heap threshold is enabled at 125MB
INFO  [main] 2021-06-10 14:48:44,180 DatabaseDescriptor.java:443 - Global memtable off-heap threshold is enabled at 125MB
WARN  [main] 2021-06-10 14:48:52,475 DatabaseDescriptor.java:579 - Only 31.508GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  [main] 2021-06-10 14:48:53,982 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO  [main] 2021-06-10 14:48:53,986 DatabaseDescriptor.java:775 - Back-pressure is disabled with strategy null.
INFO  [main] 2021-06-10 14:49:03,278 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
INFO  [main] 2021-06-10 14:49:09,407 JMXServerUtils.java:253 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO  [main] 2021-06-10 14:49:10,187 CassandraDaemon.java:490 - Hostname: k8ssandra-dc1-default-sts-0
INFO  [main] 2021-06-10 14:49:10,190 CassandraDaemon.java:497 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_282
INFO  [main] 2021-06-10 14:49:10,273 CassandraDaemon.java:498 - Heap size: 500.000MiB/500.000MiB
INFO  [main] 2021-06-10 14:49:10,277 CassandraDaemon.java:503 - Code Cache Non-heap memory: init = 2555904(2496K) used = 6381568(6232K) committed = 6422528(6272K) max = 251658240(245760K)
INFO  [main] 2021-06-10 14:49:10,280 CassandraDaemon.java:503 - Metaspace Non-heap memory: init = 0(0K) used = 25389856(24794K) committed = 26083328(25472K) max = -1(-1K)
INFO  [main] 2021-06-10 14:49:10,283 CassandraDaemon.java:503 - Compressed Class Space Non-heap memory: init = 0(0K) used = 3421712(3341K) committed = 3538944(3456K) max = 1073741824(1048576K)
INFO  [main] 2021-06-10 14:49:10,286 CassandraDaemon.java:503 - G1 Eden Space Heap memory: init = 220200960(215040K) used = 209715200(204800K) committed = 220200960(215040K) max = -1(-1K)
INFO  [main] 2021-06-10 14:49:10,288 CassandraDaemon.java:503 - G1 Survivor Space Heap memory: init = 0(0K) used = 15728640(15360K) committed = 15728640(15360K) max = -1(-1K)
INFO  [main] 2021-06-10 14:49:10,377 CassandraDaemon.java:503 - G1 Old Gen Heap memory: init = 304087040(296960K) used = 0(0K) committed = 304087040(296960K) max = 524288000(512000K)
INFO  [main] 2021-06-10 14:49:10,381 CassandraDaemon.java:505 - Classpath: /opt/cassandra/conf:/opt/cassandra/build/classes/main:/opt/cassandra/build/classes/thrift:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.6.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-3.11.10.jar:/opt/cassandra/lib/apache-cassandra-thrift-3.11.10.jar:/opt/cassandra/lib/asm-5.0.4.jar:/opt/cassandra/lib/caffeine-2.2.6.jar:/opt/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.1.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/compress-lzf-0.8.4.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/cassandra/lib/disruptor-3.0.1.jar:/opt/cassandra/lib/ecj-4.4.2.jar:/opt/cassandra/lib/guava-18.0.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.5.4.jar:/opt/cassandra/lib/jackson-annotations-2.9.10.jar:/opt/cassandra/lib/jackson-core-2.9.10.jar:/opt/cassandra/lib/jackson-databind-2.9.10.4.jar:/opt/cassandra/lib/jamm-0.3.0.jar:/opt/cassandra/lib/javax.inject.jar:/opt/cassandra/lib/jbcrypt-0.3m.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/cassandra/lib/jctools-core-1.2.1.jar:/opt/cassandra/lib/jflex-1.6.0.jar:/opt/cassandra/lib/jna-4.2.2.jar:/opt/cassandra/lib/joda-time-2.4.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/jstackjunit-0.0.1.jar:/opt/cassandra/lib/libthrift-0.9.2.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/cassandra/lib/logback-classic-1.1.3.jar:/opt/cassandra/lib/logback-core-1.1.3.jar:/opt/cassandra/lib/lz4-1.3.0.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/netty-all-4.0.44.Final.jar:/opt/cassandra/lib/ohc-core-0.4.4.jar:/opt/cassandra/lib/ohc-core-j8-0.4.4.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/slf4j-api-1.7.7.jar:/opt/cassandra/lib/snakeyaml-1.11.jar:/opt/cassandra/lib/snappy-java-1.1.1.7.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/thrift-server-0.3.7.jar:/opt/cassandra/lib/jsr223/*/*.jar::/opt/cassandra/lib/jamm-0.3.0.jar:/opt/metrics-collector/lib/datastax-mcac-agent.jar:/opt/management-api/datastax-mgmtapi-agent-0.1.0-SNAPSHOT.jar
INFO  [main] 2021-06-10 14:49:10,387 CassandraDaemon.java:507 - JVM Arguments: [-Xloggc:/opt/cassandra/logs/gc.log, -XX:+AlwaysPreTouch, -Dcassandra.disable_auth_caches_remote_configuration=false, -Dcassandra.force_default_indexing_page_size=false, -Dcassandra.join_ring=true, -Dcassandra.load_ring_state=true, -Dcassandra.write_survey=false, -ea, -XX:G1RSetUpdatingPauseTimePercent=5, -XX:+HeapDumpOnOutOfMemoryError, -Xmn200M, -Xms500M, -Djava.net.preferIPv4Stack=true, -XX:MaxGCPauseMillis=500, -Xmx500M, -Xss256k, -XX:+PerfDisableSharedMem, -XX:+ResizeTLAB, -XX:StringTableSize=1000003, -XX:ThreadPriorityPolicy=42, -XX:+UseThreadPriorities, -XX:+UseTLAB, -XX:+UseG1GC, -Dcom.sun.management.jmxremote.authenticate=false, -Dcassandra.jmx.local.port=7199, -Dcassandra.system_distributed_replication_dc_names=dc1, -Dcassandra.system_distributed_replication_per_dc=1, -XX:CompileCommandFile=/opt/cassandra/conf/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.remote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.authenticate=true, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -javaagent:/opt/metrics-collector/lib/datastax-mcac-agent.jar, -javaagent:/opt/management-api/datastax-mgmtapi-agent-0.1.0-SNAPSHOT.jar, -Dcassandra.libjemalloc=/usr/local/lib/libjemalloc.so, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra.server_process, -Dcassandra.skip_default_role_setup=true, -Ddb.unix_socket_file=/tmp/cassandra.sock]
INFO  [ScheduledTasks:1] 2021-06-10 14:49:12,891 TokenMetadata.java:517 - Updating topology for all endpoints that have changed
WARN  [main] 2021-06-10 14:49:14,001 NativeLibrary.java:189 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  [main] 2021-06-10 14:49:14,075 StartupChecks.java:140 - jemalloc seems to be preloaded from /usr/local/lib/libjemalloc.so
INFO  [main] 2021-06-10 14:49:14,077 StartupChecks.java:176 - JMX is enabled to receive remote connections on port: 7199
INFO  [main] 2021-06-10 14:49:14,174 SigarLibrary.java:44 - Initializing SIGAR library
INFO  [main] 2021-06-10 14:49:14,672 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN  [main] 2021-06-10 14:49:14,787 StartupChecks.java:332 - Directory /opt/cassandra/data/data doesn't exist
WARN  [main] 2021-06-10 14:49:14,883 StartupChecks.java:332 - Directory /opt/cassandra/data/commitlog doesn't exist
WARN  [main] 2021-06-10 14:49:14,889 StartupChecks.java:332 - Directory /opt/cassandra/data/saved_caches doesn't exist
WARN  [main] 2021-06-10 14:49:14,894 StartupChecks.java:332 - Directory /opt/cassandra/data/hints doesn't exist
INFO  [main] 2021-06-10 14:49:18,376 QueryProcessor.java:116 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO  [main] 2021-06-10 14:49:34,305 ColumnFamilyStore.java:421 - Initializing system.IndexInfo
INFO  [main] 2021-06-10 14:49:51,889 ColumnFamilyStore.java:421 - Initializing system.batches
INFO  [main] 2021-06-10 14:49:52,094 ColumnFamilyStore.java:421 - Initializing system.paxos
INFO  [main] 2021-06-10 14:49:52,480 ColumnFamilyStore.java:421 - Initializing system.local
INFO  [main] 2021-06-10 14:49:52,873 ColumnFamilyStore.java:421 - Initializing system.peers
INFO  [main] 2021-06-10 14:49:53,095 ColumnFamilyStore.java:421 - Initializing system.peer_events
INFO  [main] 2021-06-10 14:49:53,388 ColumnFamilyStore.java:421 - Initializing system.range_xfers
INFO  [main] 2021-06-10 14:49:53,596 ColumnFamilyStore.java:421 - Initializing system.compaction_history
INFO  [main] 2021-06-10 14:49:53,793 ColumnFamilyStore.java:421 - Initializing system.sstable_activity
INFO  [main] 2021-06-10 14:49:54,085 ColumnFamilyStore.java:421 - Initializing system.size_estimates
INFO  [main] 2021-06-10 14:49:54,478 ColumnFamilyStore.java:421 - Initializing system.available_ranges
INFO  [main] 2021-06-10 14:49:54,688 ColumnFamilyStore.java:421 - Initializing system.transferred_ranges
INFO  [main] 2021-06-10 14:49:55,094 ColumnFamilyStore.java:421 - Initializing system.views_builds_in_progress
INFO  [main] 2021-06-10 14:49:55,291 ColumnFamilyStore.java:421 - Initializing system.built_views
INFO  [main] 2021-06-10 14:49:55,477 ColumnFamilyStore.java:421 - Initializing system.hints
INFO  [main] 2021-06-10 14:49:55,676 ColumnFamilyStore.java:421 - Initializing system.batchlog
INFO  [main] 2021-06-10 14:49:55,789 ColumnFamilyStore.java:421 - Initializing system.prepared_statements
INFO  [main] 2021-06-10 14:49:55,989 ColumnFamilyStore.java:421 - Initializing system.schema_keyspaces
INFO  [main] 2021-06-10 14:49:56,277 ColumnFamilyStore.java:421 - Initializing system.schema_columnfamilies
INFO  [main] 2021-06-10 14:49:56,401 ColumnFamilyStore.java:421 - Initializing system.schema_columns
INFO  [main] 2021-06-10 14:49:56,587 ColumnFamilyStore.java:421 - Initializing system.schema_triggers
INFO  [main] 2021-06-10 14:49:56,772 ColumnFamilyStore.java:421 - Initializing system.schema_usertypes
INFO  [main] 2021-06-10 14:49:56,875 ColumnFamilyStore.java:421 - Initializing system.schema_functions
INFO  [main] 2021-06-10 14:49:56,987 ColumnFamilyStore.java:421 - Initializing system.schema_aggregates
INFO  [main] 2021-06-10 14:49:57,095 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  [main] 2021-06-10 14:50:00,820 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  [main] 2021-06-10 14:50:01,498 ColumnFamilyStore.java:421 - Initializing system_schema.keyspaces
INFO  [main] 2021-06-10 14:50:01,609 ColumnFamilyStore.java:421 - Initializing system_schema.tables
INFO  [main] 2021-06-10 14:50:01,712 ColumnFamilyStore.java:421 - Initializing system_schema.columns
INFO  [main] 2021-06-10 14:50:01,875 ColumnFamilyStore.java:421 - Initializing system_schema.triggers
INFO  [main] 2021-06-10 14:50:01,908 ColumnFamilyStore.java:421 - Initializing system_schema.dropped_columns
INFO  [main] 2021-06-10 14:50:02,078 ColumnFamilyStore.java:421 - Initializing system_schema.views
INFO  [main] 2021-06-10 14:50:02,185 ColumnFamilyStore.java:421 - Initializing system_schema.types
INFO  [main] 2021-06-10 14:50:02,300 ColumnFamilyStore.java:421 - Initializing system_schema.functions
INFO  [main] 2021-06-10 14:50:02,676 ColumnFamilyStore.java:421 - Initializing system_schema.aggregates
INFO  [main] 2021-06-10 14:50:03,482 ColumnFamilyStore.java:421 - Initializing system_schema.indexes
INFO  [main] 2021-06-10 14:50:03,686 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,377 CacheService.java:100 - Initializing key cache with capacity of 25 MBs.
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,504 CacheService.java:122 - Initializing row cache with capacity of 0 MBs
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,574 CacheService.java:151 - Initializing counter cache with capacity of 12 MBs
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,579 CacheService.java:162 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  [CompactionExecutor:2] 2021-06-10 14:50:32,477 BufferPool.java:234 - Global buffer pool is enabled, when pool is exhausted (max is 125.000MiB) it will allocate on heap
INFO  [main] 2021-06-10 14:50:32,588 StorageService.java:674 - Populating token metadata from system tables
INFO  [main] 2021-06-10 14:50:34,685 StorageService.java:681 - Token metadata: 
INFO  [main] 2021-06-10 14:50:35,496 AutoSavingCache.java:174 - Completed loading (16 ms; 7 keys) KeyCache cache
INFO  [main] 2021-06-10 14:50:35,983 CommitLog.java:142 - No commitlog files found; skipping replay
INFO  [main] 2021-06-10 14:50:35,985 StorageService.java:674 - Populating token metadata from system tables
INFO  [main] 2021-06-10 14:50:36,292 StorageService.java:681 - Token metadata: 
INFO  [main] 2021-06-10 14:50:38,076 QueryProcessor.java:163 - Preloaded 0 prepared statements
INFO  [main] 2021-06-10 14:50:38,079 StorageService.java:692 - Cassandra version: 3.11.10
INFO  [main] 2021-06-10 14:50:38,080 StorageService.java:693 - Thrift API version: 20.1.0
INFO  [main] 2021-06-10 14:50:38,081 StorageService.java:694 - CQL supported versions: 3.4.4 (default: 3.4.4)
INFO  [main] 2021-06-10 14:50:38,083 StorageService.java:696 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
INFO  [main] 2021-06-10 14:50:38,629 IndexSummaryManager.java:87 - Initializing index summary manager with a memory pool size of 25 MB and a resize interval of 60 minutes
INFO  [main] 2021-06-10 14:50:38,904 MessagingService.java:750 - Starting Messaging Service on /10.96.223.162:7000 (eth0)
WARN  [main] 2021-06-10 14:50:39,095 SystemKeyspace.java:1130 - No host ID found, created eabcfb38-2228-44ec-8c7b-93b242279d72 (Note: This should happen exactly once per node).
INFO  [main] 2021-06-10 14:50:39,580 StorageService.java:624 - Unable to gossip with any peers but continuing anyway since node is in its own seed list
INFO  [main] 2021-06-10 14:50:39,905 StorageService.java:778 - Loading persisted ring state
INFO  [main] 2021-06-10 14:50:39,978 StorageService.java:907 - Starting up server gossip
INFO  [main] 2021-06-10 14:50:40,876 StorageService.java:975 - This node will not auto bootstrap because it is configured to be a seed node.
INFO  [main] 2021-06-10 14:50:41,095 BootStrapper.java:228 - Generated random tokens. tokens are [-2720173960227806934, 102874866486492279, -4043982151340349156, 8244495832140069825, -1108465967161584836, 5942357438635438243, -467336415558042237, 1689209749270397008, 6381568128261402116, -2538864389039541517, -732318877749129874, -7173720524077589707, -3837441790427577103, -58882038721969026, 8882268106477515464, -4510725406762032445, -7898469585652770192, -4631963988473228168, -2848816379959625476, -7362712832172262545, -5357693675350828761, -6110606762275587868, 904314765635942665, 5727008329433058148, -1554942015363340855, 4129114088865379810, 4656700048853215175, -2989234313431351789, -5467878456903754095, -740902785517551758, -4487906613427686223, 8371883300483420623, 382867047690396460, 1750501122173236481, -8685274006463258154, 5372712800945850355, -4600338466605065745, 5658378368966419299, 6441370883011461095, -967219028969739783, -8337124429096790277, -1859601574685955241, 6687167831751678920, 2171905509024198139, -5585979489286692090, 6274878285314680107, 7624865871433379925, 880335130366642824, -8473564868351722182, -141559402171305774, 8220901731693314598, -138447571085077616, -3671798617091430729, 5642300732458716332, -2238874622771450172, 2876555101554945298, 3706598558013698997, 4457675295140754765, -1977839050525883974, 4175368693430266909, -128010568429873543, -2103355519034368905, 4320514844027026896, -5057367066109970672, 80336036951358335, 6584879507240867781, 3808583371298503027, -7824833842984032136, -8408196114459541807, 3284928508360030557, 8441623824600301376, 7107119675809714507, -2768441923994914984, -3735774942149951345, -342005671562172105, 3951592779773933980, -3235240486328875262, -3896705566029666535, -5856207900264740301, 8337580518843435910, 2780917456058250673, 547798387480683735, 4963200284496230548, 1825096305951920567, 2578799508729804753, 1043835247920206909, -4666967122352469408, -692039916642395232, -8274772749630303580, -6447871849145316099, 4824513595759466127, -8515586515461068823, -5034033638382926554, 6542254966018078131, -523208371785866111, 3771750913322430505, 861448076577012102, -2553638739952967378, 7745344673238668546, 8000221491293056553, 2178143364119841542, 5340639379970678628, -1543770693152228107, 6396161193825553590, 9160231641906364922, -2843346665511641101, -3161883770380139293, -7806878864668727128, -1307340504950970775, -6416162654882853241, 5289809337164366639, -1174637516862948941, 5468193695845260227, 8260203434485692904, 1419338517745040290, 3855533980430031412, 4305282655498518056, 767575782029702231, -3653914842458168900, -8358323637294468261, -1584581986741205724, -6267382002853187332, 346944225045050001, 8428948672874497918, -6089118574983287692, -8790222595951503448, -4035739798978873078, -2667347931646649067, 282375473359124677, -150193288030501462, -4034482821404725776, 6113788495339697532, -3870068038914272972, 2080827577697796037, -6108759893617485870, 1156958707386235718, -4199151660858425356, 4167882616054591133, 1919651933762841770, 8061698680029506978, 5570127152174483051, -278946020829229096, 278076755317572463, -2099131739343285631, 5578545507549117194, -7655529923613660322, 4146564756735628141, 4253115386175649404, 2947932023480270018, -1755391776835909381, 5342473882602336500, 4157431744662092323, 1428276055348912421, 9133536251314650084, 3446106452941502637, -6841897430520896364, 1368743060233702724, 142193719608626737, -4039432526734506341, 7214407687648534416, 7255505897419018440, -1678832576332318340, -7683832674382327967, 3166170212967654810, -8650489416838712972, -8952601124494202231, 4832439991944083190, -3014063363913879208, 7790747877572150753, 4148534590873191816, -751083783949871995, 3026465849446086370, -504420836186608754, 862312491277969930, -8432577716159657485, 4348492866234143247, -1168574790986442183, -5769816747938831732, -4650300158065157338, -214139698983102275, 4623806110024816682, -3947935989588515868, -8931840578235330834, -6971134187899709721, 2487500177411704461, 383536855041592709, -5459049128357104589, 9165933209033157261, -2442326241834061425, -3801470649109977760, 310269603940986931, 3846153270684104389, -1188200506247645121, -6385021775135553677, 4284679422238949169, -4968325071044213359, -4880922413759111552, 2508833201732811809, 380723652361017458, -3966243250412640287, -6533806598808054359, -7984278691181410277, -8277422941749931948, 3620300925802532702, -3553839472957143743, 7039055400242562661, -2185240214656989714, 6395598474867981573, -2073180429352142122, 7656152996344165975, -3413835646413200815, 237311564505948681, 7931218247653052475, 8631984142881623801, -8725730873158108544, 6635724599783382667, 1765214887436696998, 2064064943931901772, 2235166633279918827, -6955002867723862506, -4221378199867331992, 8127215598317123656, -2322019507995431523, 8489971704471542280, -7654408074661683719, -1897700783861729580, 5628495415851229860, -2161858969046376450, 753495512864632251, -4841190928860763647, -8741724990342687119, 6531693883231696397, -7645833104774880609, -4209248482112847164, 7434372056341849944, 3267361412349435533, -5161977761639589864, 2566606314122197782, 1954545205736765739, 2033163874454040278, 641725531815595496, 7212079875414051435, 2713738175243038450, -8231616384085036047, -5023499857894808540, -5043366042469199108, 3471392664475279120, -3816451212777619729, -2557190001345523494, 2603753179157627932, -8657112904275592260, -3654657765516859349, 2527687924554227766, 6232776685634937669, 8662234453385971879, 7538514584360120465]
INFO  [MigrationStage:1] 2021-06-10 14:50:46,498 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO  [MigrationStage:1] 2021-06-10 14:50:46,576 ColumnFamilyStore.java:421 - Initializing system_traces.events
INFO  [MigrationStage:1] 2021-06-10 14:50:46,787 ColumnFamilyStore.java:421 - Initializing system_traces.sessions
INFO  [MigrationStage:1] 2021-06-10 14:50:47,076 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized

Sorry, tried to split the server log. It is far too large. Is there a section you are interested in most?

Regarding the k8ssandra values: Yes, I only touched the values file. I changed the RAM requirements for cassandra and stargate and adapted the lifeness/readiness probes:

last version:

cassandra:
  version: "3.11.10"
  cassandraLibDirVolume:
    storageClass: local-path
    size: 5Gi
  allowMultipleNodesPerWorker: true
  heap:
    size: 500M
    newGenSize: 200M
  resources:
    requests:
      cpu: 1000m
      memory: 2Gi
    limits:
      cpu: 1000m
      memory: 2Gi
  datacenters:
  - name: dc1
    size: 1
    racks:
    - name: default 
kube-prometheus-stack:
  grafana:
    adminUser: admin
    adminPassword: admin123 
stargate:
  enabled: true
  replicas: 0
  heapMB: 1024
  cpuReqMillicores: 200
  cpuLimMillicores: 1000
  livenessInitialDelaySeconds: 60
  readinessInitialDelaySeconds: 60

For your convenience, here is the complete vagrant file:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "ubuntu/focal64"

  # Disable automatic box update checking. If you disable this, then
  # boxes will only be checked for updates when the user runs
  # `vagrant box outdated`. This is not recommended.
  # config.vm.box_check_update = false

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine. In the example below,
  # accessing "localhost:8080" will access port 80 on the guest machine.
  # NOTE: This will enable public access to the opened port
  # config.vm.network "forwarded_port", guest: 80, host: 8080

  #kubectl proxy
  config.vm.network "forwarded_port", guest: 8001, host: 8001
  #calico network provider
  config.vm.network "forwarded_port", guest: 179, host: 179
  #etcd server client api
  config.vm.network "forwarded_port", guest: 2379, host: 2379
  #etcd server client api
  config.vm.network "forwarded_port", guest: 2380, host: 2380
  #kubernetes api server
  config.vm.network "forwarded_port", guest: 6443, host: 6443
  #kubelet api
  config.vm.network "forwarded_port", guest: 10250, host: 10250
  #kube-control-manager
  config.vm.network "forwarded_port", guest: 10252, host: 10252
  #kube-scheduler
  config.vm.network "forwarded_port", guest: 10251, host: 10251

  #cassandra graphql interface
  config.vm.network "forwarded_port", guest: 8080, host: 8080
  #cassandra rest authorization service for generating token
  config.vm.network "forwarded_port", guest: 8081, host: 8081
  #cassandra rest interface
  config.vm.network "forwarded_port", guest: 8082, host: 8082
  #cassandra stargate rest interface
  config.vm.network "forwarded_port", guest: 8084, host: 8084
  #cassandra cql service
  config.vm.network "forwarded_port", guest: 9042, host: 9042
  #default inter-node communication port that Cassandra and Stargate use to pass communications.
  config.vm.network "forwarded_port", guest: 7000, host: 7000
  #grafana
  config.vm.network "forwarded_port", guest: 3000, host: 3000




  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine and only allow access
  # via 127.0.0.1 to disable public access
  # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  #config.vm.network "private_network", ip: "192.168.33.10"
  config.vm.network "private_network", type: "dhcp"


  # Create a public network, which generally matched to bridged network.
  # Bridged networks make the machine appear as another physical device on
  # your network.
  # config.vm.network "public_network"

  # Share an additional folder to the guest VM. The first argument is
  # the path on the host to the actual folder. The second argument is
  # the path on the guest to mount the folder. And the optional third
  # argument is a set of non-required options.
  # config.vm.synced_folder "../data", "/vagrant_data"

  # Provider-specific configuration so you can fine-tune various
  # backing providers for Vagrant. These expose provider-specific options.
  # Example for VirtualBox:
  #
  # config.vm.provider "virtualbox" do |vb|
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #
  #   # Customize the amount of memory on the VM:
  #   vb.memory = "1024"
  # end
  config.vm.provider "virtualbox" do |vb|
    vb.memory = 16384
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
	  vb.cpus = 8
  end
  

  # View the documentation for the provider you are using for more
  # information on available options.

  # Enable provisioning with a shell script. Additional provisioners such as
  # Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
  # documentation for more information about their specific syntax and use.
  config.vm.provision "shell", inline: <<-SHELL  
    apt-get update
    apt-get install -y  curl dos2unix

    #update dns nameserver
    #sed -i -e 's/#DNS=/DNS=8.8.8.8/' /etc/systemd/resolved.conf
  	#service systemd-resolved restart

    #install docker
    apt-get update	
    apt-get install -y apt-transport-https ca-certificates curl software-properties-common net-tools
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
    echo "add docker repo"
    add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
    echo "install docker"
    apt-get update && apt-get install -y docker-ce
    usermod -a -G docker vagrant

    #add docker daemon.json to
    # 1. set docker cgroup driver to systemd : docker and kubelet both need the same cgroup driver systemd
    # kubeadm init by default uses systemd, hence only docker needs to be adapted
    # see https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/
    # see https://kubernetes.io/docs/setup/production-environment/container-runtimes/
    cp /vagrant/docker-daemon.json /etc/docker/daemon.json
    
    systemctl restart docker

    #grafana stargate needs this
    echo 'vm.max_map_count=1048575' >> /etc/sysctl.conf
    sysctl -p

    #install kubectl
    sudo apt-get update && sudo apt-get install -y apt-transport-https
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    touch /etc/apt/sources.list.d/kubernetes.list 
    echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
    apt-get update
    apt-get install -y kubelet kubeadm kubectl
    apt-mark hold kubelet kubeadm kubectl

    # kubelet requires swap off
    swapoff -a
    echo "keep swap off after reboot"
    #sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  
    # Get the IP address that VirtualBox has given this VM
    IPADDR=`ifconfig enp0s8 | grep inet | grep -v inet6 | awk '/inet /{print$2}'`
    echo This VM has IP address $IPADDR

    # Set up Kubernetes
    NODENAME=$(hostname -s)
    kubeadm config images pull
    #Note: Pod and Service Cidrs may not overlap
    kubeadm init --apiserver-cert-extra-sans=$IPADDR  --node-name $NODENAME --pod-network-cidr=10.96.0.0/16 --service-cidr=10.97.0.0/16
    # Set up admin creds for the vagrant user
    echo Copying credentials to /home/vagrant...
    sudo --user=vagrant mkdir -p /home/vagrant/.kube
    cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
    chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config 

    
    #setup tools
    apt-get install -y maven
    echo "set java home"
    echo "JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64" >> /etc/environment

    #install helm3
    echo "install helm3"
    curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
    chmod 700 get_helm.sh
    ./get_helm.sh
  SHELL

  config.vm.provision "shell", privileged:false, inline: <<-SHELL
    #allow scheduling of pods on master node
    kubectl taint nodes --all node-role.kubernetes.io/master-
	  cd /home/vagrant

    #create k8 resources
    kubectl create -f /vagrant/resources
    
    #add k8ssandra helm repos
    #see https://docs.k8ssandra.io/install/local/#configure-the-k8ssandra-helm-repository
    helm repo add k8ssandra https://helm.k8ssandra.io/stable
    helm repo add traefik https://helm.traefik.io/traefik
    helm repo update



   SHELL
end


1 Like

Primarily whatever lines up with the time that you added stargate. Another option is to upload a gist of the log

I’ll give k8ssandra a fresh start. The log is currently so large, because I let it run so long. Maybe, I can shorten it a bit. Today, I probably cannot do it anymore.

It doesn’t look like I’m able to recreate the problem.

After making the following changes I ssh’d into the VM. The first change was because on mac docker is already listening on port 3000 and the second was because /vagrant/resources didn’t exist.

╰─$ diff Vagrantfile Vagrantfile.orig
58c58
<   config.vm.network "forwarded_port", guest: 3000, host: 3001
---
>   config.vm.network "forwarded_port", guest: 3000, host: 3000
187d186
<
189,193c188,189
<     # kubectl create -f /vagrant/resources
<     wget https://docs.projectcalico.org/manifests/calico.yaml
<     kubectl apply -f calico.yaml
<
<
---
>     kubectl create -f /vagrant/resources
>
203a200
>

And then ran

$ helm repo update

# values.yaml is the file provided above
$ helm install -f values.yaml k8ssandra k8ssandra/k8ssandra

# Realized I was missing the storageclass so added here
$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

# Changed stargate replicas to 1 and ran after everything was up and settled
$ helm upgrade k8ssandra k8ssandra/k8ssandra -f values.yaml

Everything has been up and running for a few minutes now

vagrant@ubuntu-focal:~$ kubectl get po
NAME                                                READY   STATUS      RESTARTS   AGE
k8ssandra-cass-operator-65fd67b6d8-lb9t7            1/1     Running     0          21m
k8ssandra-crd-upgrader-job-k8ssandra-rjfnd          0/1     Completed   0          11m
k8ssandra-dc1-default-sts-0                         2/2     Running     0          20m
k8ssandra-dc1-stargate-74787d79c7-jmd6v             1/1     Running     0          11m
k8ssandra-grafana-584dfb486f-xfk9r                  2/2     Running     0          21m
k8ssandra-kube-prometheus-operator-85695ffb-zj6wd   1/1     Running     0          21m
k8ssandra-reaper-7ffb485bb8-k7vsc                   1/1     Running     0          15m
k8ssandra-reaper-operator-b67dc8cdf-bbjcv           1/1     Running     0          21m
prometheus-k8ssandra-kube-prometheus-prometheus-0   2/2     Running     1          21m

Hi Doug,
you perfectly anticipated the missing resources. Your result means bad news for me. I am going to face a hard time figuring out the root cause. Nevertheless, I would like to thank all of you for your fast and valuable help. If am able to find out the problem, I’ll let you know.
Thanks guys!