Local installation on VM: reaper and stargate are stuck

K8ssandra help
21

Hi dwettlaufer,

I started k8ssandra without stargate an hour ago and left for a walk. Curiously I can see now the reaper restarting although no stargate is running. Sorry, I hope I wasn’t misleading you. I have the system-server logs. They are quite long.
My cluster is built via kubeadm:

excerpt from vagrantfile:

# kubelet requires swap off
    swapoff -a
    echo "keep swap off after reboot"
    #sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  
    # Get the IP address that VirtualBox has given this VM
    IPADDR=`ifconfig enp0s8 | grep inet | grep -v inet6 | awk '/inet /{print$2}'`
    echo This VM has IP address $IPADDR

    # Set up Kubernetes
    NODENAME=$(hostname -s)
    kubeadm config images pull
    #Note: Pod and Service Cidrs may not overlap
    kubeadm init --apiserver-cert-extra-sans=$IPADDR  --node-name $NODENAME --pod-network-cidr=10.96.0.0/16 --service-cidr=10.97.0.0/16
    # Set up admin creds for the vagrant user
    echo Copying credentials to /home/vagrant...
    sudo --user=vagrant mkdir -p /home/vagrant/.kube
    cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
    chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config

If you like, I can send you the complete vagrant file. Maybe there is an error.

reaper error:

ERROR  [2021-06-10 16:21:13,002] [main] i.c.ReaperApplication - Storage is not ready yet, trying again to connect shortly... 
org.cognitor.cassandra.migration.MigrationException: Error during migration of script 026_concurrent_repairs.cql while executing 'ALTER TABLE repair_run ADD replicas frozen<map<text, text>>;'
	at org.cognitor.cassandra.migration.Database.execute(Database.java:269)
	at java.util.Collections$SingletonList.forEach(Collections.java:4824)
	at org.cognitor.cassandra.migration.MigrationTask.migrate(MigrationTask.java:68)
	at io.cassandrareaper.storage.CassandraStorage.migrate(CassandraStorage.java:362)
	at io.cassandrareaper.storage.CassandraStorage.initializeCassandraSchema(CassandraStorage.java:293)
	at io.cassandrareaper.storage.CassandraStorage.initializeAndUpgradeSchema(CassandraStorage.java:250)
	at io.cassandrareaper.storage.CassandraStorage.<init>(CassandraStorage.java:238)
	at io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:480)
	at io.cassandrareaper.ReaperApplication.tryInitializeStorage(ReaperApplication.java:303)
	at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:181)
	at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:98)
	at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:93)
	at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:117)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Invalid column name replicas because it conflicts with an existing column
	at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
	at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
	at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293)
	at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58)
	at org.cognitor.cassandra.migration.Database.executeStatement(Database.java:277)
	at org.cognitor.cassandra.migration.Database.execute(Database.java:261)
	... 15 common frames omitted
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Invalid column name replicas because it conflicts with an existing column
	at com.datastax.driver.core.Responses$Error.asException(Responses.java:181)
	at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215)
	at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:235)
	at com.datastax.driver.core.RequestHandler.access$2600(RequestHandler.java:61)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:1011)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:814)
	at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1287)
	at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1205)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
22

system log (1/5):

tail: can't open '/var/log/cassandra/system.log': No such file or directory
tail: /var/log/cassandra/system.log has appeared; following end of new file
INFO  [main] 2021-06-10 14:48:17,628 SystemDistributedReplicationInterceptor.java:73 - Using override for distributed system keyspaces: {dc1=1, class=NetworkTopologyStrategy}
INFO  [main] 2021-06-10 14:48:30,789 YamlConfigurationLoader.java:92 - Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  [main] 2021-06-10 14:48:44,089 Config.java:537 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=640; batch_size_warn_threshold_in_kb=64; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=10.96.223.162; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=<REDACTED>; cluster_name=k8ssandra; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=3600000; credentials_validity_in_ms=3600000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@f001896; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_materialized_views=true; enable_sasi_indexes=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=<REDACTED>; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.96.223.162; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_flush_in_batches_legacy=true; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=-2147483648; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=3600000; permissions_validity_in_ms=3600000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; repair_session_max_tree_depth=18; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=3600000; roles_validity_in_ms=3600000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.K8SeedProvider{seeds=k8ssandra-seed-service}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_on_duplicate_row_detection=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@13f17eb4; trickle_fsync=true; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  [main] 2021-06-10 14:48:44,175 DatabaseDescriptor.java:381 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  [main] 2021-06-10 14:48:44,177 DatabaseDescriptor.java:439 - Global memtable on-heap threshold is enabled at 125MB
INFO  [main] 2021-06-10 14:48:44,180 DatabaseDescriptor.java:443 - Global memtable off-heap threshold is enabled at 125MB
WARN  [main] 2021-06-10 14:48:52,475 DatabaseDescriptor.java:579 - Only 31.508GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  [main] 2021-06-10 14:48:53,982 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO  [main] 2021-06-10 14:48:53,986 DatabaseDescriptor.java:775 - Back-pressure is disabled with strategy null.
INFO  [main] 2021-06-10 14:49:03,278 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
INFO  [main] 2021-06-10 14:49:09,407 JMXServerUtils.java:253 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO  [main] 2021-06-10 14:49:10,187 CassandraDaemon.java:490 - Hostname: k8ssandra-dc1-default-sts-0
INFO  [main] 2021-06-10 14:49:10,190 CassandraDaemon.java:497 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_282
INFO  [main] 2021-06-10 14:49:10,273 CassandraDaemon.java:498 - Heap size: 500.000MiB/500.000MiB
INFO  [main] 2021-06-10 14:49:10,277 CassandraDaemon.java:503 - Code Cache Non-heap memory: init = 2555904(2496K) used = 6381568(6232K) committed = 6422528(6272K) max = 251658240(245760K)
INFO  [main] 2021-06-10 14:49:10,280 CassandraDaemon.java:503 - Metaspace Non-heap memory: init = 0(0K) used = 25389856(24794K) committed = 26083328(25472K) max = -1(-1K)
INFO  [main] 2021-06-10 14:49:10,283 CassandraDaemon.java:503 - Compressed Class Space Non-heap memory: init = 0(0K) used = 3421712(3341K) committed = 3538944(3456K) max = 1073741824(1048576K)
INFO  [main] 2021-06-10 14:49:10,286 CassandraDaemon.java:503 - G1 Eden Space Heap memory: init = 220200960(215040K) used = 209715200(204800K) committed = 220200960(215040K) max = -1(-1K)
INFO  [main] 2021-06-10 14:49:10,288 CassandraDaemon.java:503 - G1 Survivor Space Heap memory: init = 0(0K) used = 15728640(15360K) committed = 15728640(15360K) max = -1(-1K)
INFO  [main] 2021-06-10 14:49:10,377 CassandraDaemon.java:503 - G1 Old Gen Heap memory: init = 304087040(296960K) used = 0(0K) committed = 304087040(296960K) max = 524288000(512000K)
INFO  [main] 2021-06-10 14:49:10,381 CassandraDaemon.java:505 - Classpath: /opt/cassandra/conf:/opt/cassandra/build/classes/main:/opt/cassandra/build/classes/thrift:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.6.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-3.11.10.jar:/opt/cassandra/lib/apache-cassandra-thrift-3.11.10.jar:/opt/cassandra/lib/asm-5.0.4.jar:/opt/cassandra/lib/caffeine-2.2.6.jar:/opt/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.1.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/compress-lzf-0.8.4.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/cassandra/lib/disruptor-3.0.1.jar:/opt/cassandra/lib/ecj-4.4.2.jar:/opt/cassandra/lib/guava-18.0.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.5.4.jar:/opt/cassandra/lib/jackson-annotations-2.9.10.jar:/opt/cassandra/lib/jackson-core-2.9.10.jar:/opt/cassandra/lib/jackson-databind-2.9.10.4.jar:/opt/cassandra/lib/jamm-0.3.0.jar:/opt/cassandra/lib/javax.inject.jar:/opt/cassandra/lib/jbcrypt-0.3m.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/cassandra/lib/jctools-core-1.2.1.jar:/opt/cassandra/lib/jflex-1.6.0.jar:/opt/cassandra/lib/jna-4.2.2.jar:/opt/cassandra/lib/joda-time-2.4.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/jstackjunit-0.0.1.jar:/opt/cassandra/lib/libthrift-0.9.2.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/cassandra/lib/logback-classic-1.1.3.jar:/opt/cassandra/lib/logback-core-1.1.3.jar:/opt/cassandra/lib/lz4-1.3.0.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/netty-all-4.0.44.Final.jar:/opt/cassandra/lib/ohc-core-0.4.4.jar:/opt/cassandra/lib/ohc-core-j8-0.4.4.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/slf4j-api-1.7.7.jar:/opt/cassandra/lib/snakeyaml-1.11.jar:/opt/cassandra/lib/snappy-java-1.1.1.7.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/thrift-server-0.3.7.jar:/opt/cassandra/lib/jsr223/*/*.jar::/opt/cassandra/lib/jamm-0.3.0.jar:/opt/metrics-collector/lib/datastax-mcac-agent.jar:/opt/management-api/datastax-mgmtapi-agent-0.1.0-SNAPSHOT.jar
INFO  [main] 2021-06-10 14:49:10,387 CassandraDaemon.java:507 - JVM Arguments: [-Xloggc:/opt/cassandra/logs/gc.log, -XX:+AlwaysPreTouch, -Dcassandra.disable_auth_caches_remote_configuration=false, -Dcassandra.force_default_indexing_page_size=false, -Dcassandra.join_ring=true, -Dcassandra.load_ring_state=true, -Dcassandra.write_survey=false, -ea, -XX:G1RSetUpdatingPauseTimePercent=5, -XX:+HeapDumpOnOutOfMemoryError, -Xmn200M, -Xms500M, -Djava.net.preferIPv4Stack=true, -XX:MaxGCPauseMillis=500, -Xmx500M, -Xss256k, -XX:+PerfDisableSharedMem, -XX:+ResizeTLAB, -XX:StringTableSize=1000003, -XX:ThreadPriorityPolicy=42, -XX:+UseThreadPriorities, -XX:+UseTLAB, -XX:+UseG1GC, -Dcom.sun.management.jmxremote.authenticate=false, -Dcassandra.jmx.local.port=7199, -Dcassandra.system_distributed_replication_dc_names=dc1, -Dcassandra.system_distributed_replication_per_dc=1, -XX:CompileCommandFile=/opt/cassandra/conf/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.remote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.authenticate=true, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -javaagent:/opt/metrics-collector/lib/datastax-mcac-agent.jar, -javaagent:/opt/management-api/datastax-mgmtapi-agent-0.1.0-SNAPSHOT.jar, -Dcassandra.libjemalloc=/usr/local/lib/libjemalloc.so, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra.server_process, -Dcassandra.skip_default_role_setup=true, -Ddb.unix_socket_file=/tmp/cassandra.sock]
INFO  [ScheduledTasks:1] 2021-06-10 14:49:12,891 TokenMetadata.java:517 - Updating topology for all endpoints that have changed
WARN  [main] 2021-06-10 14:49:14,001 NativeLibrary.java:189 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  [main] 2021-06-10 14:49:14,075 StartupChecks.java:140 - jemalloc seems to be preloaded from /usr/local/lib/libjemalloc.so
INFO  [main] 2021-06-10 14:49:14,077 StartupChecks.java:176 - JMX is enabled to receive remote connections on port: 7199
INFO  [main] 2021-06-10 14:49:14,174 SigarLibrary.java:44 - Initializing SIGAR library
INFO  [main] 2021-06-10 14:49:14,672 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN  [main] 2021-06-10 14:49:14,787 StartupChecks.java:332 - Directory /opt/cassandra/data/data doesn't exist
WARN  [main] 2021-06-10 14:49:14,883 StartupChecks.java:332 - Directory /opt/cassandra/data/commitlog doesn't exist
WARN  [main] 2021-06-10 14:49:14,889 StartupChecks.java:332 - Directory /opt/cassandra/data/saved_caches doesn't exist
WARN  [main] 2021-06-10 14:49:14,894 StartupChecks.java:332 - Directory /opt/cassandra/data/hints doesn't exist
INFO  [main] 2021-06-10 14:49:18,376 QueryProcessor.java:116 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO  [main] 2021-06-10 14:49:34,305 ColumnFamilyStore.java:421 - Initializing system.IndexInfo
INFO  [main] 2021-06-10 14:49:51,889 ColumnFamilyStore.java:421 - Initializing system.batches
INFO  [main] 2021-06-10 14:49:52,094 ColumnFamilyStore.java:421 - Initializing system.paxos
INFO  [main] 2021-06-10 14:49:52,480 ColumnFamilyStore.java:421 - Initializing system.local
INFO  [main] 2021-06-10 14:49:52,873 ColumnFamilyStore.java:421 - Initializing system.peers
INFO  [main] 2021-06-10 14:49:53,095 ColumnFamilyStore.java:421 - Initializing system.peer_events
INFO  [main] 2021-06-10 14:49:53,388 ColumnFamilyStore.java:421 - Initializing system.range_xfers
INFO  [main] 2021-06-10 14:49:53,596 ColumnFamilyStore.java:421 - Initializing system.compaction_history
INFO  [main] 2021-06-10 14:49:53,793 ColumnFamilyStore.java:421 - Initializing system.sstable_activity
INFO  [main] 2021-06-10 14:49:54,085 ColumnFamilyStore.java:421 - Initializing system.size_estimates
INFO  [main] 2021-06-10 14:49:54,478 ColumnFamilyStore.java:421 - Initializing system.available_ranges
INFO  [main] 2021-06-10 14:49:54,688 ColumnFamilyStore.java:421 - Initializing system.transferred_ranges
INFO  [main] 2021-06-10 14:49:55,094 ColumnFamilyStore.java:421 - Initializing system.views_builds_in_progress
INFO  [main] 2021-06-10 14:49:55,291 ColumnFamilyStore.java:421 - Initializing system.built_views
INFO  [main] 2021-06-10 14:49:55,477 ColumnFamilyStore.java:421 - Initializing system.hints
INFO  [main] 2021-06-10 14:49:55,676 ColumnFamilyStore.java:421 - Initializing system.batchlog
INFO  [main] 2021-06-10 14:49:55,789 ColumnFamilyStore.java:421 - Initializing system.prepared_statements
INFO  [main] 2021-06-10 14:49:55,989 ColumnFamilyStore.java:421 - Initializing system.schema_keyspaces
INFO  [main] 2021-06-10 14:49:56,277 ColumnFamilyStore.java:421 - Initializing system.schema_columnfamilies
INFO  [main] 2021-06-10 14:49:56,401 ColumnFamilyStore.java:421 - Initializing system.schema_columns
INFO  [main] 2021-06-10 14:49:56,587 ColumnFamilyStore.java:421 - Initializing system.schema_triggers
INFO  [main] 2021-06-10 14:49:56,772 ColumnFamilyStore.java:421 - Initializing system.schema_usertypes
INFO  [main] 2021-06-10 14:49:56,875 ColumnFamilyStore.java:421 - Initializing system.schema_functions
INFO  [main] 2021-06-10 14:49:56,987 ColumnFamilyStore.java:421 - Initializing system.schema_aggregates
INFO  [main] 2021-06-10 14:49:57,095 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  [main] 2021-06-10 14:50:00,820 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  [main] 2021-06-10 14:50:01,498 ColumnFamilyStore.java:421 - Initializing system_schema.keyspaces
INFO  [main] 2021-06-10 14:50:01,609 ColumnFamilyStore.java:421 - Initializing system_schema.tables
INFO  [main] 2021-06-10 14:50:01,712 ColumnFamilyStore.java:421 - Initializing system_schema.columns
INFO  [main] 2021-06-10 14:50:01,875 ColumnFamilyStore.java:421 - Initializing system_schema.triggers
INFO  [main] 2021-06-10 14:50:01,908 ColumnFamilyStore.java:421 - Initializing system_schema.dropped_columns
INFO  [main] 2021-06-10 14:50:02,078 ColumnFamilyStore.java:421 - Initializing system_schema.views
INFO  [main] 2021-06-10 14:50:02,185 ColumnFamilyStore.java:421 - Initializing system_schema.types
INFO  [main] 2021-06-10 14:50:02,300 ColumnFamilyStore.java:421 - Initializing system_schema.functions
INFO  [main] 2021-06-10 14:50:02,676 ColumnFamilyStore.java:421 - Initializing system_schema.aggregates
INFO  [main] 2021-06-10 14:50:03,482 ColumnFamilyStore.java:421 - Initializing system_schema.indexes
INFO  [main] 2021-06-10 14:50:03,686 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,377 CacheService.java:100 - Initializing key cache with capacity of 25 MBs.
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,504 CacheService.java:122 - Initializing row cache with capacity of 0 MBs
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,574 CacheService.java:151 - Initializing counter cache with capacity of 12 MBs
INFO  [MemtableFlushWriter:1] 2021-06-10 14:50:29,579 CacheService.java:162 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  [CompactionExecutor:2] 2021-06-10 14:50:32,477 BufferPool.java:234 - Global buffer pool is enabled, when pool is exhausted (max is 125.000MiB) it will allocate on heap
INFO  [main] 2021-06-10 14:50:32,588 StorageService.java:674 - Populating token metadata from system tables
INFO  [main] 2021-06-10 14:50:34,685 StorageService.java:681 - Token metadata: 
INFO  [main] 2021-06-10 14:50:35,496 AutoSavingCache.java:174 - Completed loading (16 ms; 7 keys) KeyCache cache
INFO  [main] 2021-06-10 14:50:35,983 CommitLog.java:142 - No commitlog files found; skipping replay
INFO  [main] 2021-06-10 14:50:35,985 StorageService.java:674 - Populating token metadata from system tables
INFO  [main] 2021-06-10 14:50:36,292 StorageService.java:681 - Token metadata: 
INFO  [main] 2021-06-10 14:50:38,076 QueryProcessor.java:163 - Preloaded 0 prepared statements
INFO  [main] 2021-06-10 14:50:38,079 StorageService.java:692 - Cassandra version: 3.11.10
INFO  [main] 2021-06-10 14:50:38,080 StorageService.java:693 - Thrift API version: 20.1.0
INFO  [main] 2021-06-10 14:50:38,081 StorageService.java:694 - CQL supported versions: 3.4.4 (default: 3.4.4)
INFO  [main] 2021-06-10 14:50:38,083 StorageService.java:696 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
INFO  [main] 2021-06-10 14:50:38,629 IndexSummaryManager.java:87 - Initializing index summary manager with a memory pool size of 25 MB and a resize interval of 60 minutes
INFO  [main] 2021-06-10 14:50:38,904 MessagingService.java:750 - Starting Messaging Service on /10.96.223.162:7000 (eth0)
WARN  [main] 2021-06-10 14:50:39,095 SystemKeyspace.java:1130 - No host ID found, created eabcfb38-2228-44ec-8c7b-93b242279d72 (Note: This should happen exactly once per node).
INFO  [main] 2021-06-10 14:50:39,580 StorageService.java:624 - Unable to gossip with any peers but continuing anyway since node is in its own seed list
INFO  [main] 2021-06-10 14:50:39,905 StorageService.java:778 - Loading persisted ring state
INFO  [main] 2021-06-10 14:50:39,978 StorageService.java:907 - Starting up server gossip
INFO  [main] 2021-06-10 14:50:40,876 StorageService.java:975 - This node will not auto bootstrap because it is configured to be a seed node.
INFO  [main] 2021-06-10 14:50:41,095 BootStrapper.java:228 - Generated random tokens. tokens are [-2720173960227806934, 102874866486492279, -4043982151340349156, 8244495832140069825, -1108465967161584836, 5942357438635438243, -467336415558042237, 1689209749270397008, 6381568128261402116, -2538864389039541517, -732318877749129874, -7173720524077589707, -3837441790427577103, -58882038721969026, 8882268106477515464, -4510725406762032445, -7898469585652770192, -4631963988473228168, -2848816379959625476, -7362712832172262545, -5357693675350828761, -6110606762275587868, 904314765635942665, 5727008329433058148, -1554942015363340855, 4129114088865379810, 4656700048853215175, -2989234313431351789, -5467878456903754095, -740902785517551758, -4487906613427686223, 8371883300483420623, 382867047690396460, 1750501122173236481, -8685274006463258154, 5372712800945850355, -4600338466605065745, 5658378368966419299, 6441370883011461095, -967219028969739783, -8337124429096790277, -1859601574685955241, 6687167831751678920, 2171905509024198139, -5585979489286692090, 6274878285314680107, 7624865871433379925, 880335130366642824, -8473564868351722182, -141559402171305774, 8220901731693314598, -138447571085077616, -3671798617091430729, 5642300732458716332, -2238874622771450172, 2876555101554945298, 3706598558013698997, 4457675295140754765, -1977839050525883974, 4175368693430266909, -128010568429873543, -2103355519034368905, 4320514844027026896, -5057367066109970672, 80336036951358335, 6584879507240867781, 3808583371298503027, -7824833842984032136, -8408196114459541807, 3284928508360030557, 8441623824600301376, 7107119675809714507, -2768441923994914984, -3735774942149951345, -342005671562172105, 3951592779773933980, -3235240486328875262, -3896705566029666535, -5856207900264740301, 8337580518843435910, 2780917456058250673, 547798387480683735, 4963200284496230548, 1825096305951920567, 2578799508729804753, 1043835247920206909, -4666967122352469408, -692039916642395232, -8274772749630303580, -6447871849145316099, 4824513595759466127, -8515586515461068823, -5034033638382926554, 6542254966018078131, -523208371785866111, 3771750913322430505, 861448076577012102, -2553638739952967378, 7745344673238668546, 8000221491293056553, 2178143364119841542, 5340639379970678628, -1543770693152228107, 6396161193825553590, 9160231641906364922, -2843346665511641101, -3161883770380139293, -7806878864668727128, -1307340504950970775, -6416162654882853241, 5289809337164366639, -1174637516862948941, 5468193695845260227, 8260203434485692904, 1419338517745040290, 3855533980430031412, 4305282655498518056, 767575782029702231, -3653914842458168900, -8358323637294468261, -1584581986741205724, -6267382002853187332, 346944225045050001, 8428948672874497918, -6089118574983287692, -8790222595951503448, -4035739798978873078, -2667347931646649067, 282375473359124677, -150193288030501462, -4034482821404725776, 6113788495339697532, -3870068038914272972, 2080827577697796037, -6108759893617485870, 1156958707386235718, -4199151660858425356, 4167882616054591133, 1919651933762841770, 8061698680029506978, 5570127152174483051, -278946020829229096, 278076755317572463, -2099131739343285631, 5578545507549117194, -7655529923613660322, 4146564756735628141, 4253115386175649404, 2947932023480270018, -1755391776835909381, 5342473882602336500, 4157431744662092323, 1428276055348912421, 9133536251314650084, 3446106452941502637, -6841897430520896364, 1368743060233702724, 142193719608626737, -4039432526734506341, 7214407687648534416, 7255505897419018440, -1678832576332318340, -7683832674382327967, 3166170212967654810, -8650489416838712972, -8952601124494202231, 4832439991944083190, -3014063363913879208, 7790747877572150753, 4148534590873191816, -751083783949871995, 3026465849446086370, -504420836186608754, 862312491277969930, -8432577716159657485, 4348492866234143247, -1168574790986442183, -5769816747938831732, -4650300158065157338, -214139698983102275, 4623806110024816682, -3947935989588515868, -8931840578235330834, -6971134187899709721, 2487500177411704461, 383536855041592709, -5459049128357104589, 9165933209033157261, -2442326241834061425, -3801470649109977760, 310269603940986931, 3846153270684104389, -1188200506247645121, -6385021775135553677, 4284679422238949169, -4968325071044213359, -4880922413759111552, 2508833201732811809, 380723652361017458, -3966243250412640287, -6533806598808054359, -7984278691181410277, -8277422941749931948, 3620300925802532702, -3553839472957143743, 7039055400242562661, -2185240214656989714, 6395598474867981573, -2073180429352142122, 7656152996344165975, -3413835646413200815, 237311564505948681, 7931218247653052475, 8631984142881623801, -8725730873158108544, 6635724599783382667, 1765214887436696998, 2064064943931901772, 2235166633279918827, -6955002867723862506, -4221378199867331992, 8127215598317123656, -2322019507995431523, 8489971704471542280, -7654408074661683719, -1897700783861729580, 5628495415851229860, -2161858969046376450, 753495512864632251, -4841190928860763647, -8741724990342687119, 6531693883231696397, -7645833104774880609, -4209248482112847164, 7434372056341849944, 3267361412349435533, -5161977761639589864, 2566606314122197782, 1954545205736765739, 2033163874454040278, 641725531815595496, 7212079875414051435, 2713738175243038450, -8231616384085036047, -5023499857894808540, -5043366042469199108, 3471392664475279120, -3816451212777619729, -2557190001345523494, 2603753179157627932, -8657112904275592260, -3654657765516859349, 2527687924554227766, 6232776685634937669, 8662234453385971879, 7538514584360120465]
INFO  [MigrationStage:1] 2021-06-10 14:50:46,498 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO  [MigrationStage:1] 2021-06-10 14:50:46,576 ColumnFamilyStore.java:421 - Initializing system_traces.events
INFO  [MigrationStage:1] 2021-06-10 14:50:46,787 ColumnFamilyStore.java:421 - Initializing system_traces.sessions
INFO  [MigrationStage:1] 2021-06-10 14:50:47,076 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
23

Sorry, tried to split the server log. It is far too large. Is there a section you are interested in most?

24

Regarding the k8ssandra values: Yes, I only touched the values file. I changed the RAM requirements for cassandra and stargate and adapted the lifeness/readiness probes:

last version:

cassandra:
  version: "3.11.10"
  cassandraLibDirVolume:
    storageClass: local-path
    size: 5Gi
  allowMultipleNodesPerWorker: true
  heap:
    size: 500M
    newGenSize: 200M
  resources:
    requests:
      cpu: 1000m
      memory: 2Gi
    limits:
      cpu: 1000m
      memory: 2Gi
  datacenters:
  - name: dc1
    size: 1
    racks:
    - name: default 
kube-prometheus-stack:
  grafana:
    adminUser: admin
    adminPassword: admin123 
stargate:
  enabled: true
  replicas: 0
  heapMB: 1024
  cpuReqMillicores: 200
  cpuLimMillicores: 1000
  livenessInitialDelaySeconds: 60
  readinessInitialDelaySeconds: 60
25

For your convenience, here is the complete vagrant file:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "ubuntu/focal64"

  # Disable automatic box update checking. If you disable this, then
  # boxes will only be checked for updates when the user runs
  # `vagrant box outdated`. This is not recommended.
  # config.vm.box_check_update = false

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine. In the example below,
  # accessing "localhost:8080" will access port 80 on the guest machine.
  # NOTE: This will enable public access to the opened port
  # config.vm.network "forwarded_port", guest: 80, host: 8080

  #kubectl proxy
  config.vm.network "forwarded_port", guest: 8001, host: 8001
  #calico network provider
  config.vm.network "forwarded_port", guest: 179, host: 179
  #etcd server client api
  config.vm.network "forwarded_port", guest: 2379, host: 2379
  #etcd server client api
  config.vm.network "forwarded_port", guest: 2380, host: 2380
  #kubernetes api server
  config.vm.network "forwarded_port", guest: 6443, host: 6443
  #kubelet api
  config.vm.network "forwarded_port", guest: 10250, host: 10250
  #kube-control-manager
  config.vm.network "forwarded_port", guest: 10252, host: 10252
  #kube-scheduler
  config.vm.network "forwarded_port", guest: 10251, host: 10251

  #cassandra graphql interface
  config.vm.network "forwarded_port", guest: 8080, host: 8080
  #cassandra rest authorization service for generating token
  config.vm.network "forwarded_port", guest: 8081, host: 8081
  #cassandra rest interface
  config.vm.network "forwarded_port", guest: 8082, host: 8082
  #cassandra stargate rest interface
  config.vm.network "forwarded_port", guest: 8084, host: 8084
  #cassandra cql service
  config.vm.network "forwarded_port", guest: 9042, host: 9042
  #default inter-node communication port that Cassandra and Stargate use to pass communications.
  config.vm.network "forwarded_port", guest: 7000, host: 7000
  #grafana
  config.vm.network "forwarded_port", guest: 3000, host: 3000




  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine and only allow access
  # via 127.0.0.1 to disable public access
  # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  #config.vm.network "private_network", ip: "192.168.33.10"
  config.vm.network "private_network", type: "dhcp"


  # Create a public network, which generally matched to bridged network.
  # Bridged networks make the machine appear as another physical device on
  # your network.
  # config.vm.network "public_network"

  # Share an additional folder to the guest VM. The first argument is
  # the path on the host to the actual folder. The second argument is
  # the path on the guest to mount the folder. And the optional third
  # argument is a set of non-required options.
  # config.vm.synced_folder "../data", "/vagrant_data"

  # Provider-specific configuration so you can fine-tune various
  # backing providers for Vagrant. These expose provider-specific options.
  # Example for VirtualBox:
  #
  # config.vm.provider "virtualbox" do |vb|
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #
  #   # Customize the amount of memory on the VM:
  #   vb.memory = "1024"
  # end
  config.vm.provider "virtualbox" do |vb|
    vb.memory = 16384
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
	  vb.cpus = 8
  end
  

  # View the documentation for the provider you are using for more
  # information on available options.

  # Enable provisioning with a shell script. Additional provisioners such as
  # Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
  # documentation for more information about their specific syntax and use.
  config.vm.provision "shell", inline: <<-SHELL  
    apt-get update
    apt-get install -y  curl dos2unix

    #update dns nameserver
    #sed -i -e 's/#DNS=/DNS=8.8.8.8/' /etc/systemd/resolved.conf
  	#service systemd-resolved restart

    #install docker
    apt-get update	
    apt-get install -y apt-transport-https ca-certificates curl software-properties-common net-tools
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
    echo "add docker repo"
    add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
    echo "install docker"
    apt-get update && apt-get install -y docker-ce
    usermod -a -G docker vagrant

    #add docker daemon.json to
    # 1. set docker cgroup driver to systemd : docker and kubelet both need the same cgroup driver systemd
    # kubeadm init by default uses systemd, hence only docker needs to be adapted
    # see https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/
    # see https://kubernetes.io/docs/setup/production-environment/container-runtimes/
    cp /vagrant/docker-daemon.json /etc/docker/daemon.json
    
    systemctl restart docker

    #grafana stargate needs this
    echo 'vm.max_map_count=1048575' >> /etc/sysctl.conf
    sysctl -p

    #install kubectl
    sudo apt-get update && sudo apt-get install -y apt-transport-https
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    touch /etc/apt/sources.list.d/kubernetes.list 
    echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
    apt-get update
    apt-get install -y kubelet kubeadm kubectl
    apt-mark hold kubelet kubeadm kubectl

    # kubelet requires swap off
    swapoff -a
    echo "keep swap off after reboot"
    #sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  
    # Get the IP address that VirtualBox has given this VM
    IPADDR=`ifconfig enp0s8 | grep inet | grep -v inet6 | awk '/inet /{print$2}'`
    echo This VM has IP address $IPADDR

    # Set up Kubernetes
    NODENAME=$(hostname -s)
    kubeadm config images pull
    #Note: Pod and Service Cidrs may not overlap
    kubeadm init --apiserver-cert-extra-sans=$IPADDR  --node-name $NODENAME --pod-network-cidr=10.96.0.0/16 --service-cidr=10.97.0.0/16
    # Set up admin creds for the vagrant user
    echo Copying credentials to /home/vagrant...
    sudo --user=vagrant mkdir -p /home/vagrant/.kube
    cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
    chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config 

    
    #setup tools
    apt-get install -y maven
    echo "set java home"
    echo "JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64" >> /etc/environment

    #install helm3
    echo "install helm3"
    curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
    chmod 700 get_helm.sh
    ./get_helm.sh
  SHELL

  config.vm.provision "shell", privileged:false, inline: <<-SHELL
    #allow scheduling of pods on master node
    kubectl taint nodes --all node-role.kubernetes.io/master-
	  cd /home/vagrant

    #create k8 resources
    kubectl create -f /vagrant/resources
    
    #add k8ssandra helm repos
    #see https://docs.k8ssandra.io/install/local/#configure-the-k8ssandra-helm-repository
    helm repo add k8ssandra https://helm.k8ssandra.io/stable
    helm repo add traefik https://helm.traefik.io/traefik
    helm repo update



   SHELL
end
1 Like
26

Primarily whatever lines up with the time that you added stargate. Another option is to upload a gist of the log

27

I’ll give k8ssandra a fresh start. The log is currently so large, because I let it run so long. Maybe, I can shorten it a bit. Today, I probably cannot do it anymore.

28

It doesn’t look like I’m able to recreate the problem.

After making the following changes I ssh’d into the VM. The first change was because on mac docker is already listening on port 3000 and the second was because /vagrant/resources didn’t exist.

╰─$ diff Vagrantfile Vagrantfile.orig
58c58
<   config.vm.network "forwarded_port", guest: 3000, host: 3001
---
>   config.vm.network "forwarded_port", guest: 3000, host: 3000
187d186
<
189,193c188,189
<     # kubectl create -f /vagrant/resources
<     wget https://docs.projectcalico.org/manifests/calico.yaml
<     kubectl apply -f calico.yaml
<
<
---
>     kubectl create -f /vagrant/resources
>
203a200
>

And then ran

$ helm repo update

# values.yaml is the file provided above
$ helm install -f values.yaml k8ssandra k8ssandra/k8ssandra

# Realized I was missing the storageclass so added here
$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

# Changed stargate replicas to 1 and ran after everything was up and settled
$ helm upgrade k8ssandra k8ssandra/k8ssandra -f values.yaml

Everything has been up and running for a few minutes now

vagrant@ubuntu-focal:~$ kubectl get po
NAME                                                READY   STATUS      RESTARTS   AGE
k8ssandra-cass-operator-65fd67b6d8-lb9t7            1/1     Running     0          21m
k8ssandra-crd-upgrader-job-k8ssandra-rjfnd          0/1     Completed   0          11m
k8ssandra-dc1-default-sts-0                         2/2     Running     0          20m
k8ssandra-dc1-stargate-74787d79c7-jmd6v             1/1     Running     0          11m
k8ssandra-grafana-584dfb486f-xfk9r                  2/2     Running     0          21m
k8ssandra-kube-prometheus-operator-85695ffb-zj6wd   1/1     Running     0          21m
k8ssandra-reaper-7ffb485bb8-k7vsc                   1/1     Running     0          15m
k8ssandra-reaper-operator-b67dc8cdf-bbjcv           1/1     Running     0          21m
prometheus-k8ssandra-kube-prometheus-prometheus-0   2/2     Running     1          21m
29

Hi Doug,
you perfectly anticipated the missing resources. Your result means bad news for me. I am going to face a hard time figuring out the root cause. Nevertheless, I would like to thank all of you for your fast and valuable help. If am able to find out the problem, I’ll let you know.
Thanks guys!

30

No problem, Hanno. Happy to help.

31

Hi guys,

just wanted to let you know, also in case somebody else is facing the same problem as me:
Bottom line is, Doug was right. My laptop is just too slow to start a VM and all the processes in time so that the liveness and readyness parameters must be increased. Thats all. In the end, I gave not less than 600s for both redyness and liveness and now it it startgin just as expected.
Thanks again for your help, and I am sure I will get back to you in near future when I start to explore cassandra. :slight_smile:

1 Like