Home Deploy CockroachDB in kubernetes cluster
Post
Cancel

Deploy CockroachDB in kubernetes cluster

Deploy Kubernetes cluster

1
2
3
4
5
6
$ kubectl get nodes
NAME            STATUS   ROLES    AGE    VERSION
node0   Ready    <none>   115d   v1.19.2
node1   Ready    <none>   115d   v1.19.2
node2   Ready    <none>   115d   v1.19.2
node3   Ready    master   115d   v1.19.2

Start CockroachDB cluster

Start the CockroachDB nodes with a configuration file that has been customized for performance:

1
2
3
4
5
6
7
$ curl -O https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/performance/cockroachdb-statefulset-insecure.yaml

$ kubectl create -f cockroachdb-statefulset-insecure.yaml
service/cockroachdb-public created
service/cockroachdb created
poddisruptionbudget.policy/cockroachdb-budget created
statefulset.apps/cockroachdb created

Confirm that three pods are Running successfully. Note that they will not be considered Ready until after the cluster has been initialized:

1
2
3
4
5
$ kubectl get pods
NAME            READY     STATUS    RESTARTS   AGE
cockroachdb-0   0/1       Running   0          2m
cockroachdb-1   0/1       Running   0          2m
cockroachdb-2   0/1       Running   0          2m

Confirm that the persistent volumes and corresponding claims were created successfully for all three pods:

1
2
3
4
5
6
7
8
9
10
11
$ kubectl get pvc
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-cockroachdb-0   Bound    pvc-ac210538-2a20-4d99-9b5d-c5a03d733b4d   1Ti        RWO            px-repl1       35s
datadir-cockroachdb-1   Bound    pvc-54f24a59-a063-46bb-9645-d4292eb483a3   1Ti        RWO            px-repl1       25s
datadir-cockroachdb-2   Bound    pvc-0ebd7fa7-dc36-4c77-ab9d-003d10680f56   1Ti        RWO            px-repl1       15s

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
pvc-0ebd7fa7-dc36-4c77-ab9d-003d10680f56   1Ti        RWO            Delete           Bound    default/datadir-cockroachdb-2   px-repl1                17s
pvc-54f24a59-a063-46bb-9645-d4292eb483a3   1Ti        RWO            Delete           Bound    default/datadir-cockroachdb-1   px-repl1                27s
pvc-ac210538-2a20-4d99-9b5d-c5a03d733b4d   1Ti        RWO            Delete           Bound    default/datadir-cockroachdb-0   px-repl1                27s

Before initialize the CockroachDB cluster, check the network(use ping command) to make sure each host can communicate with each other. Otherwise, you may run into connection failure between cluster nodes.

In my case, I want to run the cluster over the private network which doesn’t have DNS setup. So, I just add the private IP addresses and Kubernetes qualified hostnames and CockroachDB node names in /etc/hosts file on each host as below.

1
2
3
10.0.0.1 ip-10-10-0-1 cockroachdb-0.cockroachdb
10.0.0.2 ip-10-10-0-2 cockroachdb-1.cockroachdb
10.0.0.3 ip-10-10-0-3 cockroachdb-2.cockroachdb

Use the provided cluster-init.yaml file to perform a one-time initialization that joins the CockroachDB nodes into a single cluster:

1
2
$ kubectl create \
-f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cluster-init.yaml

Confirm that cluster initialization has completed successfully. The job should be considered successful and the Kubernetes pods should soon be considered Ready:

1
2
3
4
5
6
7
8
9
$ kubectl get job cluster-init
NAME           COMPLETIONS   DURATION   AGE
cluster-init   1/1           7s         27s
kubectl get pods
NAME                 READY   STATUS      RESTARTS   AGE
cluster-init-cqf8l   0/1     Completed   0          56s
cockroachdb-0        1/1     Running     0          7m51s
cockroachdb-1        1/1     Running     0          7m51s
cockroachdb-2        1/1     Running     0          7m51s

Use the built-in SQL client

1
2
3
4
5
6
7
8
9
10
11
12
$ kubectl run cockroachdb -it --image=cockroachdb/cockroach:v22.1.7 --rm --restart=Never -- sql --insecure --host=cockroachdb-public

root@cockroachdb-public:26257/defaultdb> show databases;
  database_name | owner | primary_region | regions | survival_goal
----------------+-------+----------------+---------+----------------
  defaultdb     | root  | NULL           | {}      | NULL
  postgres      | root  | NULL           | {}      | NULL
  system        | node  | NULL           | {}      | NULL
(3 rows)


Time: 5ms total (execution 4ms / network 1ms)

Run TPC-C workload on the CockroachDB cluster

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
$ kubectl run cockroachdb -it --image=cockroachdb/cockroach:v22.1.7 --rm --restart=Never -- bash
[root@cockroachdb cockroach]# cockroach node ls --insecure --host=cockroachdb-public
  id
------
   1
   2
   3
(3 rows)

[root@cockroachdb cockroach]# cockroach workload init tpcc "postgresql://root@cockroachdb-public:26257?sslmode=disable" --warehouses 10 --drop
I220922 19:48:15.543311 1 workload/workloadsql/dataload.go:146  [-] 1  imported warehouse (0s, 10 rows)
I220922 19:48:15.580634 1 workload/workloadsql/dataload.go:146  [-] 2  imported district (0s, 100 rows)
I220922 19:48:20.098986 1 workload/workloadsql/dataload.go:146  [-] 3  imported customer (5s, 300000 rows)
I220922 19:48:21.601978 1 workload/workloadsql/dataload.go:146  [-] 4  imported history (2s, 300000 rows)
I220922 19:48:24.317881 1 workload/workloadsql/dataload.go:146  [-] 5  imported order (3s, 300000 rows)
I220922 19:48:24.540100 1 workload/workloadsql/dataload.go:146  [-] 6  imported new_order (0s, 90000 rows)
I220922 19:48:24.998829 1 workload/workloadsql/dataload.go:146  [-] 7  imported item (0s, 100000 rows)
I220922 19:48:34.737491 1 workload/workloadsql/dataload.go:146  [-] 8  imported stock (10s, 1000000 rows)
I220922 19:48:49.030366 1 workload/workloadsql/dataload.go:146  [-] 9  imported order_line (14s, 3001222 rows)

[root@cockroachdb cockroach]# cockroach sql --insecure --database tpcc --host=cockroachdb-public:26257 -e 'show tables'
  schema_name | table_name | type  | owner | estimated_row_count | locality
--------------+------------+-------+-------+---------------------+-----------
  public      | customer   | table | root  |              300000 | NULL
  public      | district   | table | root  |                 100 | NULL
  public      | history    | table | root  |              300000 | NULL
  public      | item       | table | root  |              100000 | NULL
  public      | new_order  | table | root  |               90000 | NULL
  public      | order      | table | root  |              300000 | NULL
  public      | order_line | table | root  |             3001222 | NULL
  public      | stock      | table | root  |             1000000 | NULL
  public      | warehouse  | table | root  |                  10 | NULL
(9 rows)

[root@cockroachdb cockroach]# cockroach workload run tpcc --warehouses=10 --ramp=10s --duration=20s postgres://root@cockroachdb-public:26257?sslmode=disable
I220922 19:52:41.633625 1 workload/cli/run.go:414  [-] 1  creating load generator...
Initializing 20 connections...
Initializing 0 idle connections...
Initializing 100 workers and preparing statements...
I220922 19:52:51.646000 1 workload/cli/run.go:445  [-] 2  creating load generator... done (took 10.012371773s)

_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms)
   20.0s      153.0 119.0%     38.2     44.0     48.2     48.2     50.3     52.4

Reference

This post is licensed under CC BY 4.0 by the author.

Compile Linux kernel on CentOS

CockroachDB performance benchmarking

Comments powered by Disqus.