Deploy CockroachDB in kubernetes cluster
Deploy Kubernetes cluster
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node0 Ready <none> 115d v1.19.2
node1 Ready <none> 115d v1.19.2
node2 Ready <none> 115d v1.19.2
node3 Ready master 115d v1.19.2
Start CockroachDB cluster
Start the CockroachDB nodes with a configuration file that has been customized for performance:
$ curl -O https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/performance/cockroachdb-statefulset-insecure.yaml
$ kubectl create -f cockroachdb-statefulset-insecure.yaml
service/cockroachdb-public created
service/cockroachdb created
poddisruptionbudget.policy/cockroachdb-budget created
statefulset.apps/cockroachdb created
Confirm that three pods are Running successfully. Note that they will not be considered Ready until after the cluster has been initialized:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroachdb-0 0/1 Running 0 2m
cockroachdb-1 0/1 Running 0 2m
cockroachdb-2 0/1 Running 0 2m
Confirm that the persistent volumes and corresponding claims were created successfully for all three pods:
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-cockroachdb-0 Bound pvc-ac210538-2a20-4d99-9b5d-c5a03d733b4d 1Ti RWO px-repl1 35s
datadir-cockroachdb-1 Bound pvc-54f24a59-a063-46bb-9645-d4292eb483a3 1Ti RWO px-repl1 25s
datadir-cockroachdb-2 Bound pvc-0ebd7fa7-dc36-4c77-ab9d-003d10680f56 1Ti RWO px-repl1 15s
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-0ebd7fa7-dc36-4c77-ab9d-003d10680f56 1Ti RWO Delete Bound default/datadir-cockroachdb-2 px-repl1 17s
pvc-54f24a59-a063-46bb-9645-d4292eb483a3 1Ti RWO Delete Bound default/datadir-cockroachdb-1 px-repl1 27s
pvc-ac210538-2a20-4d99-9b5d-c5a03d733b4d 1Ti RWO Delete Bound default/datadir-cockroachdb-0 px-repl1 27s
Before initialize the CockroachDB cluster, check the network(use ping command) to make sure each host can communicate with each other. Otherwise, you may run into connection failure between cluster nodes.
In my case, I want to run the cluster over the private network which doesn’t have DNS setup. So, I just add the private IP addresses and Kubernetes qualified hostnames and CockroachDB node names in /etc/hosts file on each host as below.
10.0.0.1 ip-10-10-0-1 cockroachdb-0.cockroachdb
10.0.0.2 ip-10-10-0-2 cockroachdb-1.cockroachdb
10.0.0.3 ip-10-10-0-3 cockroachdb-2.cockroachdb
Use the provided cluster-init.yaml file to perform a one-time initialization that joins the CockroachDB nodes into a single cluster:
$ kubectl create \
-f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cluster-init.yaml
Confirm that cluster initialization has completed successfully. The job should be considered successful and the Kubernetes pods should soon be considered Ready:
$ kubectl get job cluster-init
NAME COMPLETIONS DURATION AGE
cluster-init 1/1 7s 27s
kubectl get pods
NAME READY STATUS RESTARTS AGE
cluster-init-cqf8l 0/1 Completed 0 56s
cockroachdb-0 1/1 Running 0 7m51s
cockroachdb-1 1/1 Running 0 7m51s
cockroachdb-2 1/1 Running 0 7m51s
Use the built-in SQL client
$ kubectl run cockroachdb -it --image=cockroachdb/cockroach:v22.1.7 --rm --restart=Never -- sql --insecure --host=cockroachdb-public
root@cockroachdb-public:26257/defaultdb> show databases;
database_name | owner | primary_region | regions | survival_goal
----------------+-------+----------------+---------+----------------
defaultdb | root | NULL | {} | NULL
postgres | root | NULL | {} | NULL
system | node | NULL | {} | NULL
(3 rows)
Time: 5ms total (execution 4ms / network 1ms)
Run TPC-C workload on the CockroachDB cluster
$ kubectl run cockroachdb -it --image=cockroachdb/cockroach:v22.1.7 --rm --restart=Never -- bash
[root@cockroachdb cockroach]# cockroach node ls --insecure --host=cockroachdb-public
id
------
1
2
3
(3 rows)
[root@cockroachdb cockroach]# cockroach workload init tpcc "postgresql://root@cockroachdb-public:26257?sslmode=disable" --warehouses 10 --drop
I220922 19:48:15.543311 1 workload/workloadsql/dataload.go:146 [-] 1 imported warehouse (0s, 10 rows)
I220922 19:48:15.580634 1 workload/workloadsql/dataload.go:146 [-] 2 imported district (0s, 100 rows)
I220922 19:48:20.098986 1 workload/workloadsql/dataload.go:146 [-] 3 imported customer (5s, 300000 rows)
I220922 19:48:21.601978 1 workload/workloadsql/dataload.go:146 [-] 4 imported history (2s, 300000 rows)
I220922 19:48:24.317881 1 workload/workloadsql/dataload.go:146 [-] 5 imported order (3s, 300000 rows)
I220922 19:48:24.540100 1 workload/workloadsql/dataload.go:146 [-] 6 imported new_order (0s, 90000 rows)
I220922 19:48:24.998829 1 workload/workloadsql/dataload.go:146 [-] 7 imported item (0s, 100000 rows)
I220922 19:48:34.737491 1 workload/workloadsql/dataload.go:146 [-] 8 imported stock (10s, 1000000 rows)
I220922 19:48:49.030366 1 workload/workloadsql/dataload.go:146 [-] 9 imported order_line (14s, 3001222 rows)
[root@cockroachdb cockroach]# cockroach sql --insecure --database tpcc --host=cockroachdb-public:26257 -e 'show tables'
schema_name | table_name | type | owner | estimated_row_count | locality
--------------+------------+-------+-------+---------------------+-----------
public | customer | table | root | 300000 | NULL
public | district | table | root | 100 | NULL
public | history | table | root | 300000 | NULL
public | item | table | root | 100000 | NULL
public | new_order | table | root | 90000 | NULL
public | order | table | root | 300000 | NULL
public | order_line | table | root | 3001222 | NULL
public | stock | table | root | 1000000 | NULL
public | warehouse | table | root | 10 | NULL
(9 rows)
[root@cockroachdb cockroach]# cockroach workload run tpcc --warehouses=10 --ramp=10s --duration=20s postgres://root@cockroachdb-public:26257?sslmode=disable
I220922 19:52:41.633625 1 workload/cli/run.go:414 [-] 1 creating load generator...
Initializing 20 connections...
Initializing 0 idle connections...
Initializing 100 workers and preparing statements...
I220922 19:52:51.646000 1 workload/cli/run.go:445 [-] 2 creating load generator... done (took 10.012371773s)
_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms)
20.0s 153.0 119.0% 38.2 44.0 48.2 48.2 50.3 52.4