Home CockroachDB TPC-C performance benchmarking
Post
Cancel

CockroachDB TPC-C performance benchmarking

Provision VMs

Create 4 VM instances, 3 for CockroachDB nodes and 1 for the TPC-C workload.

On each VM, disable firewall and enable ntp service.

1
2
3
$ systemctl stop firewalld
$ systemctl disable firewalld
$ systemctl status firewalld
1
2
$ systemctl start ntpd
$ systemctl enable ntpd

On each VM, check the open file limit and modify it to the recommended value here. Otherwise, you would run into the “too many open files” issue which is explained in troubleshooting section below.

Start CockroachDB cluster

  1. SSH to the first VM where you want to run a CockroachDB node.

  2. Download the CockroachDB archive for Linux, extract the binary, and copy it into the PATH

    1
    
    $ curl https://binaries.cockroachdb.com/cockroach-v22.1.6.linux-amd64.tgz | tar -xz && sudo cp -i cockroach-v22.1.6.linux-amd64/cockroach /usr/local/bin/
    
  3. Run the cockroach start command

    1
    
    $ cockroach start --store=/mnt/cockroanchdb_mnt1 --insecure --advertise-addr=host1 --join=host1,host2,host3 --cache=.25 -max-sql-memory=.25 --background
    
  4. Repeat steps 1 - 3 for the other 2 VMs for CockroachDB nodes. Each time, be sure to adjust the –advertise-addr flag.

    You can see the following log message after you start the CockroachDB node.

    1
    2
    3
    4
    5
    
    $ vim /mnt/cockroanchdb_mnt1/logs/cockroach.log
    I220903 04:07:36.890467 557 server/init.go:418 ⋮ [n?] 162  ‹host3:26257› is itself waiting for init, will retry
    I220903 04:07:37.890838 557 server/init.go:418 ⋮ [n?] 163  ‹host1:26257› is itself waiting for init, will retry
    I220903 04:07:38.890544 557 server/init.go:418 ⋮ [n?] 164  ‹host3:26257› is itself waiting for init, will retry
    <omitted..>
    
  5. On any of the VMs with the cockroach binary, run the one-time cockroach init command to join the first nodes into a cluster

    1
    
    $ cockroach init --insecure --host=host1
    

    You can see the following message after the cluster initialization.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    
    $ vim /mnt/cockroanchdb_mnt1/logs/cockroach.log
    I220903 04:07:39.902527 128 server/init.go:285 ⋮ [n?] 165  joined cluster ‹9fb42bb4-b41b-4246-af2a-5dd8bdf47e65› through join rpc
    I220903 04:07:39.902589 128 server/init.go:286 ⋮ [n?] 166  received node ID: 2
    I220903 04:07:39.902626 128 server/init.go:287 ⋮ [n?] 167  received cluster version: 22.1
    I220903 04:07:39.904004 2170 1@server/server.go:1307 ⋮ [n2] 169  connecting to gossip network to verify cluster ID ‹"9fb42bb4-b41b-4246-af2a-5dd8bdf47e65"›
    W220903 04:07:39.903941 128 2@gossip/pkg/gossip/gossip.go:1533 ⋮ [n2] 168  no incoming or outgoing connections
    I220903 04:07:39.904540 128 gossip/pkg/gossip/gossip.go:401 ⋮ [n2] 170  NodeDescriptor set to ‹node_id:2 address:<network_field:"tcp" address_field:"host2:26257" > attrs:<> locality:<> ServerVersion:<major_val:22 minor_val:1 patch:0 internal:0 > build_tag:"v22.1.6" started_at:1662178059904515958 cluster_name:"" sql_address:<network_field:"tcp" address_field:"host2:26257" > http_address:<network_field:"tcp" address_field:"host2:8080" >›
    I220903 04:07:39.907116 128 server/node.go:469 ⋮ [n2] 171  initialized store s2
    I220903 04:07:39.907205 128 kv/kvserver/pkg/kv/kvserver/stores.go:250 ⋮ [n2] 172  read 0 node addresses from persistent storage
    I220903 04:07:39.907311 2124 gossip/pkg/gossip/client.go:124 ⋮ [n2] 173  started gossip client to n0 (‹host1:26257›)
    I220903 04:07:39.908083 2170 1@server/server.go:1310 ⋮ [n2] 174  node connected via gossip
    I220903 04:07:39.908784 128 kv/kvserver/pkg/kv/kvserver/stores.go:269 ⋮ [n2] 175  wrote 1 node addresses to persistent storage
    I220903 04:07:39.908873 128 server/node.go:548 ⋮ [n2] 176  started with engine type ‹2›
    <omitted..>
    I220903 04:08:21.689124 5124 kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:864 ⋮ [n2,s2,r32/2:{-}] 295  applying INITIAL snapshot 07fda1af from (n1,s1):1 at applied index 17
    I220903 04:08:21.699036 5124 kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:875 ⋮ [n2,s2,r32/2:‹/Table/3{6-7}›] 296  applied INITIAL snapshot 07fda1af from (n1,s1):1 at applied index 17 (total=10ms data=681 B ingestion=6@8ms)
    <omitted..>
    
  6. Check the CockroachDB data

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    
    $ df -h  | egrep "Filesystem|cockroachdb"
    Filesystem                      Size  Used Avail Use% Mounted on
    /dev/testvg/testvol             1007G  1.5G  955G   1% /mnt/cockroachdb_mnt1
    
    $ ls -la /mnt/cockroachdb_mnt1
    total 433208
    drwxr-xr-x. 5 root root     4096 Sep  3 04:14 .
    drwxr-xr-x. 3 root root       30 Sep  3 03:55 ..
    -rw-r-----  1 root root   227938 Sep  3 04:07 000002.log
    -rw-r-----  1 root root   250497 Sep  3 04:07 000004.log
    -rw-r-----  1 root root   806894 Sep  3 04:07 000005.log
    -rw-r-----  1 root root  1712878 Sep  3 04:07 000006.log
    -rw-r-----  1 root root  3683977 Sep  3 04:08 000007.log
    -rw-r-----  1 root root 13349647 Sep  3 04:16 000011.log
    -rw-r-----  1 root root   613678 Sep  3 04:14 000012.sst
    drwxr-x---  2 root root     4096 Sep  3 04:05 auxiliary
    -rw-r-----  1 root root       19 Sep  3 04:05 cockroach.advertise-addr
    -rw-r-----  1 root root       19 Sep  3 04:05 cockroach.advertise-sql-addr
    -rw-r-----  1 root root       18 Sep  3 04:05 cockroach.http-addr
    -rw-r-----  1 root root       10 Sep  3 04:05 cockroach.listen-addr
    -rw-r-----  1 root root       10 Sep  3 04:05 cockroach.sql-addr
    drwxr-xr-x  3 root root     4096 Sep  3 04:05 cockroach-temp3812830763
    -rw-r-----  1 root root       16 Sep  3 04:05 CURRENT
    -rw-r-----  1 root root        0 Sep  3 04:05 LOCK
    drwxr-x---  5 root root     4096 Sep  3 04:14 logs
    -rw-r-----  1 root root      199 Sep  3 04:14 MANIFEST-000001
    -rw-r-----  1 root root        0 Sep  3 04:07 marker.format-version.000005.006
    -rw-r-----  1 root root        0 Sep  3 04:05 marker.manifest.000001.MANIFEST-000001
    -rw-r-----  1 root root     2337 Sep  3 04:05 OPTIONS-000003
    -rw-r-----  1 root root        8 Sep  3 04:07 STORAGE_MIN_VERSION
    -rw-r-----  1 root root       47 Sep  3 04:05 temp-dirs-record.txt
    

Verify replication

On the Overview page of the db web console(http://host1:8080), note that the Replicas count is the same on all three nodes.

Image

This indicates:

  • There are this many “ranges” of data in the cluster. These are mostly internal “system” ranges since you haven’t added much table data.
  • Each range has been replicated 3 times (according to the CockroachDB default).
  • For each range, each replica is stored on different nodes.

Import or insert the TPC-C dataset

CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB’s version of the TPC-C workload.

  1. SSH to the VM where you want to run TPC-C. This can be a VM outside the CockroachDB cluster.

  2. Download the CockroachDB archive for Linux, extract the binary, and copy it into the PATH

    1
    
    $ curl https://binaries.cockroachdb.com/cockroach-v22.1.6.linux-amd64.tgz | tar -xz && sudo cp -i cockroach-v22.1.6.linux-amd64/cockroach /usr/local/bin/
    
  3. Import the TPC-C dataset

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    $ cockroach workload fixtures import tpcc --warehouses=3000 'postgres://root@host1:26257?sslmode=disable'
    I220901 21:01:34.540034 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 9 tables
    I220901 21:01:37.397596 137 ccl/workloadccl/fixture.go:483  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 1.949575087s, 0.08 MiB/s)
    I220901 21:01:39.118771 143 ccl/workloadccl/fixture.go:483  [-] 3  imported 7.9 MiB in item table (100000 rows, 0 index entries, took 3.670505412s, 2.14 MiB/s)
    I220901 21:01:40.234214 138 ccl/workloadccl/fixture.go:483  [-] 4  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 4.786096758s, 0.62 MiB/s)
    I220901 21:02:35.358777 142 ccl/workloadccl/fixture.go:483  [-] 5  imported 409 MiB in new_order table (27000000 rows, 0 index entries, took 59.910576709s, 6.82 MiB/s)
    I220901 21:08:46.746826 140 ccl/workloadccl/fixture.go:483  [-] 6  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 7m11.298610431s, 15.33 MiB/s)
    I220901 21:10:43.728536 141 ccl/workloadccl/fixture.go:483  [-] 7  imported 4.9 GiB in order table (90000000 rows, 90000000 index entries, took 9m8.280115404s, 9.12 MiB/s)
    I220901 21:45:25.079620 139 ccl/workloadccl/fixture.go:483  [-] 8  imported 52 GiB in customer table (90000000 rows, 90000000 index entries, took 43m49.631441777s, 20.16 MiB/s)
    I220901 21:48:29.329893 145 ccl/workloadccl/fixture.go:483  [-] 9  imported 52 GiB in order_line table (900015304 rows, 0 index entries, took 46m53.881701103s, 18.75 MiB/s)
    I220901 21:54:17.506086 144 ccl/workloadccl/fixture.go:483  [-] 10  imported 90 GiB in stock table (300000000 rows, 0 index entries, took 52m42.057629692s, 29.28 MiB/s)
    I220901 21:54:17.605151 1 ccl/workloadccl/fixture.go:326  [-] 11  imported 206 GiB bytes in 9 tables (took 52m43.064718388s, 66.51 MiB/s)
    I220901 21:54:21.913391 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
    I220901 21:54:22.005861 1 workload/tpcc/tpcc.go:517  [-] 13  check 3.3.2.1 took 92.298339ms
    I220901 21:55:12.597596 1 workload/tpcc/tpcc.go:517  [-] 14  check 3.3.2.2 took 50.591536345s
    I220901 21:55:18.425980 1 workload/tpcc/tpcc.go:517  [-] 15  check 3.3.2.3 took 5.828294303s
    I220901 22:09:04.970092 1 workload/tpcc/tpcc.go:517  [-] 16  check 3.3.2.4 took 13m46.543997152s
    I220901 22:10:34.086630 1 workload/tpcc/tpcc.go:517  [-] 17  check 3.3.2.5 took 1m29.116317621s
    I220901 22:18:38.482761 1 workload/tpcc/tpcc.go:517  [-] 18  check 3.3.2.7 took 8m4.395861507s
    I220901 22:19:19.015035 1 workload/tpcc/tpcc.go:517  [-] 19  check 3.3.2.8 took 40.532033557s
    I220901 22:20:08.785644 1 workload/tpcc/tpcc.go:517  [-] 20  check 3.3.2.9 took 49.770392014s
    

    or

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    
    $ cockroach workload init tpcc postgresql://root@host1:26257?sslmode=disable --warehouses 5000 --drop
    I220908 16:23:17.929051 1 workload/workloadsql/dataload.go:146  [-] 1  imported warehouse (0s, 5000 rows)
    I220908 16:23:18.503344 1 workload/workloadsql/dataload.go:146  [-] 2  imported district (1s, 50000 rows)
    I220908 17:07:16.747561 1 workload/workloadsql/dataload.go:146  [-] 3  imported customer (43m58s, 150000000 rows)
    I220908 17:21:27.967502 1 workload/workloadsql/dataload.go:146  [-] 4  imported history (14m11s, 150000000 rows)
    I220908 17:43:33.968710 1 workload/workloadsql/dataload.go:146  [-] 5  imported order (22m6s, 150000000 rows)
    I220908 17:45:25.967955 1 workload/workloadsql/dataload.go:146  [-] 6  imported new_order (1m52s, 45000000 rows)
    I220908 17:45:26.539448 1 workload/workloadsql/dataload.go:146  [-] 7  imported item (1s, 100000 rows)
    I220908 19:19:30.994627 1 workload/workloadsql/dataload.go:146  [-] 8  imported stock (1h34m4s, 500000000 rows)
    I220908 21:29:41.901690 1 workload/workloadsql/dataload.go:146  [-] 9  imported order_line (2h10m11s, 1500013787 rows)
    
    1
    2
    3
    
    $ df -h | grep cockroachdb
    Filesystem                      Size  Used Avail Use% Mounted on
    /dev/testvg/testvol             1007G  107G  850G  12% /mnt/cockroachdb_mnt1
    

    Image

Memory cache reload

The memory cache will be reloaded if the database is restarted. In the following vmstat output, we see the cache increases from ~1GB to ~100GB when we restart the CockroachDB cluster which runs 5000 warehouses tpcc database on each node. This would impact tpcc workload performance if we don’t wait for the cache load complete. Refer to this post on how to shutdown CockroachDB cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@host1 ~]# vmstat 60 -t
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st                 UTC
 1  0  24576 1051374656   2980 1012080    0    0  9703    84 5385 5115  0  1 99  0  0
 <omitted..>
 1  0  24576 739877376 195168 106159856    0    0    26  3532    0    0 10  1 88  0  0 2022-09-08 23:13:48
 6  0  24576 737796288 195332 106865944    0    0 11191 20505 11320 11812  3  0 97  0  0 2022-09-08 23:14:48
 3  0  24576 734534464 195460 108085112    0    0 16941 20944 13977 17926  3  0 97  0  0 2022-09-08 23:15:48
 0  0  24576 731609664 195580 109177408    0    0 16731 16587 12681 16702  3  0 97  0  0 2022-09-08 23:16:48
 1  0  24576 730459136 195676 109583376    0    0  6273 18268 7238 12809  1  0 99  0  0 2022-09-08 23:17:48
 0  0  24576 729931712 195784 109629192    0    0  3039 20403 6733 11904  0  0 100  0  0 2022-09-08 23:18:48
 0  0  24576 729257536 195880 109842848    0    0  3086 23863 7175 12834  0  0 100  0  0 2022-09-08 23:19:48
^C

Determine the ramp time

It takes time to stabilize the workload when to run tpcc. The ramp time to be used in tpcc run should be based on experiments.

In the following tpcc run on the 5000 warehouses, it takes about 5 minutes to get sustained throughput for newOrder operation. So we set ramp time to 5 minutes in the tpcc run later.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
$ grep newOrder tpcc_run_5000_5m_30m_run.log
    1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    5.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    6.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    7.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    8.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    9.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   10.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   11.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   12.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   13.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   14.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   15.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   16.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   17.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   18.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   19.0s        0           67.9            3.6     29.4     33.6     35.7     60.8 newOrder
   20.0s        0           72.0            7.0     28.3     71.3     79.7     83.9 newOrder
   21.0s        0           70.0           10.0     28.3     35.7     54.5     65.0 newOrder
   22.0s        0           78.9           13.1     28.3     32.5     33.6     35.7 newOrder
   23.0s        0           95.1           16.7     27.3     60.8     67.1     67.1 newOrder
   24.0s        0           86.0           19.6     28.3     32.5     35.7     35.7 newOrder
   25.0s        0           97.0           22.7     28.3     62.9     71.3     75.5 newOrder
   26.0s        0           86.0           25.1     28.3     48.2     75.5     75.5 newOrder
   27.0s        0           99.0           27.9     29.4     62.9     71.3     71.3 newOrder
   28.0s        0           89.9           30.1     27.3     56.6     71.3     75.5 newOrder
   29.0s        0           99.1           32.4     28.3     35.7     65.0     71.3 newOrder
   30.0s        0          103.0           34.8     28.3     71.3     79.7    100.7 newOrder
   31.0s        0           94.0           36.7     27.3     48.2     56.6     56.6 newOrder
   32.0s        0           98.0           38.6     26.2     35.7     56.6     62.9 newOrder
   33.0s        0          110.0           40.8     28.3     54.5     71.3     79.7 newOrder
   34.0s        0          101.9           42.6     26.2     35.7     58.7     60.8 newOrder
   35.0s        0           97.9           44.2     27.3     56.6     96.5    104.9 newOrder
   36.0s        0           94.0           45.6     26.2     30.4     56.6     60.8 newOrder
   37.0s        0          116.1           47.5     26.2     65.0     75.5     96.5 newOrder
   38.0s        0          124.9           49.5     26.2     50.3     65.0     71.3 newOrder
   39.0s        0          112.0           51.1     28.3     37.7     62.9     71.3 newOrder
   40.0s        0          134.0           53.2     26.2     41.9     60.8     71.3 newOrder
   41.0s        0          132.0           55.1     26.2     32.5     56.6     58.7 newOrder
   42.0s        0          115.0           56.5     27.3     52.4     71.3     71.3 newOrder
   43.0s        0          148.0           58.7     27.3     52.4     65.0     75.5 newOrder
   44.0s        0          146.0           60.6     26.2     50.3     62.9     62.9 newOrder
   45.0s        0          151.0           62.6     27.3     60.8     67.1     71.3 newOrder
   46.0s        0          149.9           64.5     26.2     30.4     30.4     50.3 newOrder
   47.0s        0          167.0           66.7     26.2     58.7     71.3     92.3 newOrder
   48.0s        0          169.9           68.9     27.3     65.0     75.5     83.9 newOrder
   49.0s        0          153.1           70.6     26.2     33.6     60.8     71.3 newOrder
   50.0s        0          162.9           72.4     26.2     31.5     52.4     62.9 newOrder
   51.0s        0          179.1           74.5     27.3     67.1     88.1    104.9 newOrder
   52.0s        0          179.9           76.6     26.2     54.5     60.8     62.9 newOrder
   53.0s        0          173.0           78.4     25.2     58.7     75.5     92.3 newOrder
   54.0s        0          181.9           80.3     25.2     29.4     31.5     48.2 newOrder
   55.0s        0          181.0           82.1     25.2     50.3     60.8     67.1 newOrder
   56.0s        0          188.0           84.0     27.3     67.1     88.1     92.3 newOrder
   57.0s        0          183.9           85.8     25.2     46.1     54.5     58.7 newOrder
   58.0s        0          189.0           87.6     26.2     52.4     58.7     83.9 newOrder
   59.0s        0          197.9           89.4     26.2     37.7     50.3     60.8 newOrder
   60.0s        0          195.0           91.2     25.2     44.0     56.6     58.7 newOrder
   61.0s        0          189.1           92.8     26.2     46.1     54.5     58.7 newOrder
   62.0s        0          210.0           94.7     26.2     58.7     71.3     71.3 newOrder
   63.0s        0          210.0           96.5     26.2     65.0     79.7    100.7 newOrder
   64.0s        0          208.6           98.3     25.2     54.5     62.9     65.0 newOrder
   65.0s        0          198.3           99.8     25.2     52.4     79.7    100.7 newOrder
   66.0s        0          220.0          101.6     25.2     62.9     79.7     88.1 newOrder
   67.0s        0          221.0          103.4     25.2     50.3     60.8     65.0 newOrder
   68.0s        0          242.7          105.5     26.2     54.5     67.1     71.3 newOrder
   69.0s        0          243.0          107.4     25.2     56.6     71.3     71.3 newOrder
   70.0s        0          241.1          109.4     25.2     44.0     60.8     65.0 newOrder
   71.0s        0          213.9          110.8     25.2     54.5     62.9     65.0 newOrder
   72.0s        0          266.9          113.0     25.2     54.5     65.0     71.3 newOrder
   73.0s        0          256.3          115.0     26.2     54.5     65.0     71.3 newOrder
   74.0s        0          251.5          116.8     25.2     62.9     67.1     79.7 newOrder
   75.0s        0          265.1          118.8     26.2     48.2     62.9     71.3 newOrder
   76.0s        0          244.5          120.4     25.2     52.4     60.8     60.8 newOrder
   77.0s        0          264.6          122.3     26.2     62.9     71.3     75.5 newOrder
   78.0s        0          253.7          124.0     25.2     52.4     60.8     67.1 newOrder
   79.0s        0          275.4          125.9     25.2     50.3     60.8     62.9 newOrder
   80.0s        0          282.9          127.9     25.2     56.6     65.0     67.1 newOrder
   81.0s        0          269.9          129.6     26.2     71.3     79.7     88.1 newOrder
   82.0s        0          298.8          131.7     26.2     60.8     79.7    100.7 newOrder
   83.0s        0          253.2          133.2     25.2     52.4     60.8     65.0 newOrder
   84.0s        0          276.7          134.9     25.2     54.5     60.8     67.1 newOrder
   85.0s        0          269.2          136.4     26.2     54.5     60.8     62.9 newOrder
   86.0s        0          288.9          138.2     26.2     56.6     60.8     65.0 newOrder
   87.0s        0          310.8          140.2     25.2     52.4     62.9     65.0 newOrder
   88.0s        0          265.1          141.6     26.2     58.7     71.3     71.3 newOrder
   89.0s        0          306.6          143.5     25.2     48.2     60.8     71.3 newOrder
   90.0s        0          285.3          145.1     26.2     58.7     62.9     67.1 newOrder
   91.0s        0          308.7          146.9     25.2     56.6     67.1     71.3 newOrder
   92.0s        0          302.3          148.5     26.2     56.6     65.0     71.3 newOrder
   93.0s        0          292.9          150.1     26.2     54.5     62.9     67.1 newOrder
   94.0s        0          307.7          151.8     26.2     56.6     75.5     83.9 newOrder
   95.0s        0          328.1          153.6     25.2     52.4     60.8     62.9 newOrder
   96.0s        0          336.3          155.5     25.2     60.8     79.7     88.1 newOrder
   97.0s        0          330.6          157.3     25.2     48.2     58.7     60.8 newOrder
   98.0s        0          320.4          159.0     25.2     56.6     71.3     83.9 newOrder
   99.0s        0          339.4          160.8     25.2     50.3     62.9     71.3 newOrder
  100.0s        0          348.3          162.7     26.2     56.6     62.9     71.3 newOrder
  101.0s        0          408.9          165.1     25.2     58.7     75.5     79.7 newOrder
  102.0s        0          337.8          166.8     26.2     56.6     67.1     75.5 newOrder
  103.0s        0          313.5          168.3     25.2     60.8     83.9     92.3 newOrder
  104.0s        0          362.3          170.1     26.2     58.7     71.3     83.9 newOrder
  105.0s        0          366.0          172.0     25.2     62.9     71.3     79.7 newOrder
  106.0s        0          357.1          173.7     25.2     52.4     62.9     88.1 newOrder
  107.0s        0          349.6          175.4     25.2     54.5     62.9     67.1 newOrder
  108.0s        0          369.5          177.2     25.2     48.2     62.9     67.1 newOrder
  109.0s        0          388.4          179.1     25.2     62.9     75.5     92.3 newOrder
  110.0s        0          380.4          180.9     25.2     54.5     62.9     79.7 newOrder
  111.0s        0          400.2          182.9     26.2     71.3     79.7     92.3 newOrder
  112.0s        0          357.5          184.5     25.2     52.4     62.9     75.5 newOrder
  113.0s        0          393.3          186.3     25.2     88.1    104.9    121.6 newOrder
  114.0s        0          356.1          187.8     25.2     71.3     96.5    100.7 newOrder
  115.0s        0          408.4          189.7     25.2     71.3     83.9    104.9 newOrder
  116.0s        0          393.7          191.5     25.2     65.0     83.9     92.3 newOrder
  117.0s        0          421.2          193.5     25.2     54.5     62.9    151.0 newOrder
  118.0s        0          423.3          195.4     24.1     52.4     62.9     71.3 newOrder
  119.0s        0          402.3          197.2     25.2     54.5     62.9     67.1 newOrder
  120.0s        0          459.6          199.3     25.2     58.7     62.9     79.7 newOrder
  121.0s        0          426.3          201.2     25.2     54.5     67.1     75.5 newOrder
  122.0s        0          421.6          203.0     25.2     60.8     71.3     79.7 newOrder
  123.0s        0          415.7          204.7     25.2     65.0     79.7     88.1 newOrder
  124.0s        0          442.0          206.7     25.2     52.4     60.8     62.9 newOrder
  125.0s        0          440.6          208.5     26.2     62.9     75.5     88.1 newOrder
  126.0s        0          405.3          210.1     25.2     58.7     65.0     71.3 newOrder
  127.0s        0          485.9          212.3     26.2     58.7     65.0     71.3 newOrder
  128.0s        0          458.7          214.2     25.2     54.5     67.1     71.3 newOrder
  129.0s        0          403.5          215.7     26.2     60.8     67.1     71.3 newOrder
  130.0s        0          487.2          217.7     27.3     62.9     75.5     96.5 newOrder
  131.0s        0          444.8          219.5     25.2     56.6     65.0     67.1 newOrder
  132.0s        0          414.7          221.0     25.2     58.7     79.7     88.1 newOrder
  133.0s        0          467.0          222.8     25.2     65.0     75.5     83.9 newOrder
  134.0s        0          495.1          224.8     27.3     67.1     83.9    100.7 newOrder
  135.0s        0          509.8          227.0     26.2     65.0     79.7     92.3 newOrder
  136.0s        0          427.3          228.4     26.2     58.7     65.0     67.1 newOrder
  137.0s        0          481.9          230.3     27.3     75.5     88.1     96.5 newOrder
  138.0s        0          496.6          232.2     26.2     60.8     75.5     79.7 newOrder
  139.0s        0          479.8          234.0     26.2     52.4     60.8     71.3 newOrder
  140.0s        0          478.6          235.7     28.3     60.8     71.3     79.7 newOrder
  141.0s        0          519.9          237.8     25.2     60.8     71.3     79.7 newOrder
  142.0s        0          520.0          239.7     25.2     71.3     88.1     92.3 newOrder
  143.0s        0          484.3          241.5     26.2     60.8     75.5     79.7 newOrder
  144.0s        0          481.3          243.1     25.2     65.0     79.7     96.5 newOrder
  145.0s        0          508.5          245.0     25.2     62.9     75.5     79.7 newOrder
  146.0s        0          453.2          246.4     26.2     56.6     71.3     79.7 newOrder
  147.0s        0          505.5          248.1     26.2     62.9     79.7     88.1 newOrder
  148.0s        0          453.9          249.5     26.2     54.5     65.0     67.1 newOrder
  149.0s        0          521.8          251.4     27.3     62.9     83.9     96.5 newOrder
  150.0s        0          535.7          253.3     27.3     60.8     67.1     83.9 newOrder
  151.0s        0          561.1          255.3     27.3     58.7     71.3     79.7 newOrder
  152.0s        0          549.4          257.2     25.2     52.4     62.9     65.0 newOrder
  153.0s        0          518.7          258.9     25.2     60.8     71.3     75.5 newOrder
  154.0s        0          555.3          260.9     26.2     54.5     62.9     65.0 newOrder
  155.0s        0          498.2          262.4     25.2     65.0     79.7     92.3 newOrder
  156.0s        0          556.0          264.3     25.2     56.6     62.9     62.9 newOrder
  157.0s        0          545.7          266.1     25.2     71.3     92.3    100.7 newOrder
  158.0s        0          531.4          267.8     25.2     54.5     62.9     71.3 newOrder
  159.0s        0          579.1          269.7     27.3     62.9     67.1     75.5 newOrder
  160.0s        0          531.9          271.3     25.2     56.6     62.9     67.1 newOrder
  161.0s        0          573.1          273.2     26.2     60.8     67.1     71.3 newOrder
  162.0s        0          584.8          275.1     25.2     56.6     67.1     75.5 newOrder
  163.0s        0          526.6          276.7     26.2     56.6     65.0    104.9 newOrder
  164.0s        0          570.8          278.5     25.2     56.6     60.8     65.0 newOrder
  165.0s        0          569.1          280.3     26.2     56.6     65.0     75.5 newOrder
  166.0s        0          530.9          281.8     25.2     58.7     65.0     75.5 newOrder
  167.0s        0          559.8          283.4     26.2     58.7     67.1     79.7 newOrder
  168.0s        0          612.2          285.4     27.3     62.9     71.3     79.7 newOrder
  169.0s        0          553.0          287.0     26.2     60.8     71.3     88.1 newOrder
  170.0s        0          576.6          288.7     25.2     60.8     71.3     79.7 newOrder
  171.0s        0          619.4          290.6     28.3     60.8     67.1     71.3 newOrder
  172.0s        0          576.3          292.3     26.2     62.9     71.3     83.9 newOrder
  173.0s        0          589.6          294.0     26.2     62.9     71.3     83.9 newOrder
  174.0s        0          648.3          296.0     29.4     71.3     88.1    125.8 newOrder
  175.0s        0          630.7          297.9     25.2     65.0     79.7     88.1 newOrder
  176.0s        0          614.8          299.7     27.3     71.3     83.9    130.0 newOrder
  177.0s        0          622.0          301.6     25.2     65.0     79.7     92.3 newOrder
  178.0s        0          617.4          303.3     25.2     65.0     75.5     79.7 newOrder
  179.0s        0          616.4          305.1     25.2     75.5     96.5    113.2 newOrder
  180.0s        0          662.0          307.1     25.2     88.1    104.9    113.2 newOrder
  181.0s        0          597.6          308.7     24.1     79.7     88.1    104.9 newOrder
  182.0s        0          558.9          310.0     25.2     75.5     92.3    100.7 newOrder
  183.0s        0          699.0          312.2     26.2     88.1    113.2    125.8 newOrder
  184.0s        0          659.2          314.1     25.2     79.7     96.5    113.2 newOrder
  185.0s        0          652.1          315.9     28.3     83.9    100.7    109.1 newOrder
  186.0s        0          640.0          317.6     27.3     65.0     75.5     83.9 newOrder
  187.0s        0          696.0          319.7     28.3     67.1     83.9    104.9 newOrder
  188.0s        0          659.3          321.5     26.2     60.8     71.3     83.9 newOrder
  189.0s        0          642.4          323.2     27.3     71.3     83.9     92.3 newOrder
  190.0s        0          693.0          325.1     28.3     60.8     75.5     92.3 newOrder
  191.0s        0          615.7          326.6     27.3     54.5     58.7     65.0 newOrder
  192.0s        0          665.3          328.4     27.3     67.1     79.7     88.1 newOrder
  193.0s        0          678.5          330.2     28.3     62.9     71.3     88.1 newOrder
  194.0s        0          668.7          331.9     29.4     71.3     83.9    100.7 newOrder
  195.0s        0          655.4          333.6     26.2     60.8     67.1     71.3 newOrder
  196.0s        0          683.8          335.4     27.3     71.3     83.9    100.7 newOrder
  197.0s        0          632.7          336.9     25.2     79.7    113.2    130.0 newOrder
  198.0s        0          723.6          338.9     25.2     75.5     83.9    113.2 newOrder
  199.0s        0          723.9          340.8     25.2     88.1     96.5    109.1 newOrder
  200.0s        0          698.6          342.6     25.2     88.1    117.4    121.6 newOrder
  201.0s        0          742.9          344.6     28.3     83.9    100.7    121.6 newOrder
  202.0s        0          655.7          346.1     28.3     56.6     65.0     71.3 newOrder
  203.0s        0          741.3          348.1     26.2     56.6     65.0     67.1 newOrder
  204.0s        0          729.1          349.9     28.3     75.5     92.3    142.6 newOrder
  205.0s        0          714.6          351.7     27.3     71.3     92.3    104.9 newOrder
  206.0s        0          691.4          353.4     29.4     58.7     65.0     71.3 newOrder
  207.0s        0          749.2          355.3     28.3     58.7     67.1     75.5 newOrder
  208.0s        0          721.3          357.0     30.4     58.7     65.0     75.5 newOrder
  209.0s        0          780.9          359.1     35.7     67.1     88.1    104.9 newOrder
  210.0s        0          664.7          360.5     26.2     65.0     79.7    109.1 newOrder
  211.0s        0          758.6          362.4     28.3     58.7     67.1     88.1 newOrder
  212.0s        0          761.6          364.3     32.5     65.0     75.5     88.1 newOrder
  213.0s        0          704.3          365.9     27.3     67.1     75.5     96.5 newOrder
  214.0s        0          781.0          367.8     26.2     79.7     92.3    109.1 newOrder
  215.0s        0          735.5          369.5     27.3     75.5     88.1     96.5 newOrder
  216.0s        0          784.3          371.5     30.4     56.6     65.0     71.3 newOrder
  217.0s        0          726.2          373.1     31.5     60.8     67.1     75.5 newOrder
  218.0s        0          706.4          374.6     26.2     58.7     71.3     88.1 newOrder
  219.0s        0          772.4          376.4     31.5     71.3     92.3    117.4 newOrder
  220.0s        0          731.3          378.0     32.5     71.3     88.1    113.2 newOrder
  221.0s        0          783.8          379.9     26.2     62.9     75.5     92.3 newOrder
  222.0s        0          784.7          381.7     26.2     65.0     75.5     92.3 newOrder
  223.0s        0          812.6          383.7     37.7     71.3     88.1    100.7 newOrder
  224.0s        0          761.1          385.3     29.4     62.9     75.5     88.1 newOrder
  225.0s        0          842.6          387.4     32.5     60.8     71.3     79.7 newOrder
  226.0s        0          728.6          388.9     30.4     60.8     71.3     88.1 newOrder
  227.0s        0          820.6          390.8     30.4     54.5     62.9     67.1 newOrder
  228.0s        0          789.8          392.5     31.5     56.6     65.0     71.3 newOrder
  229.0s        0          821.8          394.4     26.2     65.0     75.5     83.9 newOrder
  230.0s        0          776.8          396.1     27.3     67.1     79.7     92.3 newOrder
  231.0s        0          815.5          397.9     31.5     83.9    100.7    109.1 newOrder
  232.0s        0          824.9          399.7     29.4     75.5     83.9    151.0 newOrder
  233.0s        0          872.9          401.8     32.5     88.1    109.1    121.6 newOrder
  234.0s        0          797.3          403.4     27.3     62.9     75.5     96.5 newOrder
  235.0s        0          802.7          405.1     31.5     75.5     92.3    125.8 newOrder
  236.0s        0          837.1          407.0     27.3     83.9     96.5    151.0 newOrder
  237.0s        0          821.4          408.7     28.3     75.5     92.3     96.5 newOrder
  238.0s        0          860.0          410.6     35.7     75.5     92.3     96.5 newOrder
  239.0s        0          820.1          412.3     31.5     60.8     79.7    113.2 newOrder
  240.0s        0          829.4          414.1     27.3     65.0     79.7     88.1 newOrder
  241.0s        0          810.3          415.7     26.2     71.3     83.9     96.5 newOrder
  242.0s        0          856.7          417.5     29.4     71.3     83.9    104.9 newOrder
  243.0s        0          838.5          419.3     27.3     71.3     83.9    100.7 newOrder
  244.0s        0          823.4          420.9     35.7     67.1     79.7     88.1 newOrder
  245.0s        0          787.6          422.4     28.3     58.7     67.1     79.7 newOrder
  246.0s        0          883.1          424.3     33.6     62.9     71.3     79.7 newOrder
  247.0s        0          902.3          426.2     35.7     62.9     71.3    109.1 newOrder
  248.0s        0          920.8          428.2     35.7     65.0     79.7     92.3 newOrder
  249.0s        0          896.4          430.1     37.7     65.0     83.9    104.9 newOrder
  250.0s        0          847.9          431.8     35.7     60.8     67.1     75.5 newOrder
  251.0s        0          933.5          433.8     32.5     58.7     67.1     75.5 newOrder
  252.0s        0          846.9          435.4     33.6     58.7     67.1     71.3 newOrder
  253.0s        0          846.3          437.0     30.4     71.3     83.9    104.9 newOrder
  254.0s        0          866.3          438.7     33.6     67.1     79.7    125.8 newOrder
  255.0s        0          885.6          440.5     31.5     58.7     71.3     88.1 newOrder
  256.0s        0          893.9          442.3     32.5     58.7     71.3     83.9 newOrder
  257.0s        0          870.9          444.3     35.7     75.5     92.3    142.6 newOrder
  258.0s        0          913.4          445.7     32.5     75.5     92.3    109.1 newOrder
  259.0s        0          910.0          447.5     33.6     60.8     71.3     92.3 newOrder
  260.0s        0          919.5          449.3     35.7     60.8     71.3     75.5 newOrder
  261.0s        0          920.1          451.1     35.7     79.7     96.5    125.8 newOrder
  262.0s        0          968.1          453.1     35.7     62.9     71.3     83.9 newOrder
  263.0s        0          905.9          454.9     37.7     67.1     75.5     83.9 newOrder
  264.0s        0          963.0          456.8     35.7     58.7     71.3    121.6 newOrder
  265.0s        0          932.6          458.6     33.6     65.0     75.5     88.1 newOrder
  266.0s        0          946.4          460.4     37.7     65.0     75.5     83.9 newOrder
  267.0s        0          968.4          462.3     37.7     71.3     88.1    109.1 newOrder
  268.0s        0          882.0          463.9     35.7     71.3     83.9     96.5 newOrder
  269.0s        0          983.6          465.8     37.7     58.7     67.1     83.9 newOrder
  270.0s        0          962.4          467.6     35.7     62.9     79.7     88.1 newOrder
  271.0s        0         1034.0          469.7     44.0     75.5    104.9    121.6 newOrder
  272.0s        0         1005.8          471.7     37.7     58.7     67.1     75.5 newOrder
  273.0s        0          915.5          473.3     35.7     60.8     71.3     96.5 newOrder
  274.0s        0          967.6          475.1     37.7     67.1     83.9    100.7 newOrder
  275.0s        0          972.3          476.9     37.7     67.1     79.7     92.3 newOrder
  276.0s        0          886.9          478.4     35.7     60.8     71.3     88.1 newOrder
  277.0s        0          972.6          480.2     37.7     65.0     71.3     83.9 newOrder
  278.0s        0          943.8          481.9     35.7     58.7     71.3     88.1 newOrder
  279.0s        0          903.0          483.4     37.7     67.1     75.5     88.1 newOrder
  280.0s        0          986.6          485.2     39.8     67.1     75.5    109.1 newOrder
  281.0s        0         1064.7          487.2     39.8     71.3     88.1     96.5 newOrder
  282.0s        0          921.1          488.8     35.7     75.5     96.5    159.4 newOrder
  283.0s        0          953.7          490.5     35.7     71.3     88.1    100.7 newOrder
  284.0s        0         1029.8          492.3     39.8     71.3     88.1    113.2 newOrder
  285.0s        0          982.6          494.0     39.8     83.9    109.1    142.6 newOrder
  286.0s        0          938.2          495.6     37.7     67.1     83.9     92.3 newOrder
  287.0s        0          980.6          497.3     37.7     65.0     79.7    100.7 newOrder
  288.0s        0          976.1          498.9     37.7     65.0     75.5    100.7 newOrder
  289.0s        0         1108.2          501.1     41.9     67.1     75.5     88.1 newOrder
  290.0s        0         1029.4          502.9     39.8     88.1    100.7    134.2 newOrder
  291.0s        0          953.7          504.4     39.8     83.9    104.9    125.8 newOrder
  292.0s        0         1037.1          506.3     41.9     75.5     92.3    104.9 newOrder
  293.0s        0         1006.2          508.0     39.8     65.0     75.5    201.3 newOrder
  294.0s        0          986.2          509.6     37.7     67.1     88.1    100.7 newOrder
  295.0s        0         1099.4          511.6     37.7     71.3     88.1    100.7 newOrder
  296.0s        0          992.9          513.2     37.7     75.5     92.3    104.9 newOrder
  297.0s        0          989.1          514.8     39.8     71.3     79.7     88.1 newOrder
  298.0s        0         1086.2          516.7     37.7     65.0     79.7     92.3 newOrder
  299.0s        0         1032.4          518.5     39.8     65.0     75.5     88.1 newOrder
  300.0s        0         1041.4          520.2     39.8     79.7     96.5    121.6 newOrder
    1.0s        0         1041.5         1045.8     39.8     71.3     79.7     92.3 newOrder
    2.0s        0         1039.1         1042.4     41.9     79.7    104.9    134.2 newOrder
    3.0s        0         1049.3         1045.3     39.8     67.1     79.7     92.3 newOrder
    4.0s        0         1017.8         1040.4     37.7     67.1     83.9     96.5 newOrder
    5.0s        0         1112.4         1053.6     41.9     67.1     79.7    104.9 newOrder
    6.0s        0         1046.4         1051.7     39.8     79.7    104.9    134.2 newOrder
    7.0s        0         1133.5         1064.1     39.8     71.3     88.1    117.4 newOrder
    8.0s        0         1028.5         1059.4     39.8     67.1     79.7     96.5 newOrder
    9.0s        0         1083.8         1062.9     41.9     71.3     83.9    209.7 newOrder
   10.0s        0         1056.1         1061.6     39.8     62.9     71.3     83.9 newOrder
   11.0s        0         1053.4         1060.8     39.8     65.0     71.3    151.0 newOrder
   12.0s        0         1153.1         1069.0     39.8     71.3     88.1    109.1 newOrder
   13.0s        0         1083.1         1069.7     41.9     75.5     96.5    134.2 newOrder
   14.0s        0         1092.7         1071.3     41.9     75.5     88.1    104.9 newOrder
   15.0s        0         1136.7         1075.7     41.9     75.5     92.3    130.0 newOrder
   
<omitted..>

Run the benchmark

  1. Still on the same VM, create an addrs file containing connection strings to the 3 CockroachDB nodes:

    1
    2
    
    $ cat addrs
    postgres://root@host1:26257?sslmode=disable postgres://root@host2:26257?sslmode=disable postgres://root@host3:26257?sslmode=disable
    
  2. Run TPC-C for 60 minutes:

    1
    
    $ cockroach workload run tpcc --warehouses=5000 --ramp=5m --duration=30m $(cat addrs)
    

Interpret the results

Once the workload has finished running, you will see a result similar to the following. The efficiency and latency can be combined to determine whether this was a passing run. You should expect to see an efficiency number above 95%, well above the required minimum of 85%, and p95 latencies well below the required maximum of 10 seconds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Initializing 10000 connections...
Initializing 0 idle connections...
Initializing 50000 workers and preparing statements...

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
 1800.0s        0         188441          104.7     74.6     75.5    117.4    142.6   1543.5  delivery

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
 1800.0s        0        1889816         1049.9     39.8     37.7     71.3     88.1   2013.3  newOrder

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
 1800.0s        0         188547          104.7      9.2      6.8     23.1     30.4    335.5  orderStatus

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
 1800.0s        0        1886242         1047.9     23.3     21.0     46.1     60.8    318.8  payment

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
 1800.0s        0         188588          104.8     19.0     15.2     37.7     48.2    117.4  stockLevel

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
 1800.0s        0        4341634         2412.0     31.9     28.3     71.3     96.5   2013.3
Audit check 9.2.1.7: PASS
Audit check 9.2.2.5.1: PASS
Audit check 9.2.2.5.2: PASS
Audit check 9.2.2.5.5: PASS
Audit check 9.2.2.5.6: PASS
Audit check 9.2.2.5.3: PASS
Audit check 9.2.2.5.4: PASS

_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms)
 1800.0s    62993.3  98.0%     39.8     37.7     62.9     71.3     88.1   2013.3

tpmC - A Measure of Business Throughput

The throughput of TPC-C is a direct result of the level of activity at the terminals. Each warehouse has ten terminals, and all five transactions are available at each terminal. The RTE maintains the required mix of transactions over the performance measurement interval. This mix represents the complete processing of an order as it is entered, paid for, checked, and delivered. More specifically, the required mix is defined to produce an equal number of New-Order and Payment transactions and to produce one Delivery transaction, one Order-Status transaction, and one Stock-Level transaction for every ten New-Order transactions.

The tpmC metric reported by TPC-C is the number of New-Order transactions executed per minute. Given the required mix and the wide range of complexity and types among the transactions, this metric more closely simulates a complete business activity, not just one or two transactions or computer operations. For this reason, the tpmC metric is considered to be a measure of business throughput.

The RTE is also used to measure the response time of each transaction and to simulate keying times and think times. The keying time represents the time spent entering data at the terminal and the think time represents the time spent, by the operator, to read the result of the transaction at the terminal before requesting another transaction. Each transaction has a minimum keying time and a minimum think time. In addition, the response time of each transaction must be below a required threshold. These thresholds have been defined to give predominance to New-Order as the performance limiting transaction.

Use the built-in SQL client

Now that your cluster is live, you can use any node as a SQL gateway. To test this out, let’s use CockroachDB’s built-in SQL client.

Run the cockroach sql command against node 1:

1
2
3
4
5
6
7
8
9
10
11
12
[root@host1 ~]# cockroach sql --insecure --host=localhost:26257
root@localhost:26257/defaultdb> show databases;
  database_name | owner | primary_region | regions | survival_goal
----------------+-------+----------------+---------+----------------
  defaultdb     | root  | NULL           | {}      | NULL
  postgres      | root  | NULL           | {}      | NULL
  system        | node  | NULL           | {}      | NULL
  tpcc          | root  | NULL           | {}      | NULL
(4 rows)


Time: 6ms total (execution 6ms / network 0ms)

3000 warehouses

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@host1 ~]# cockroach sql --insecure --database tpcc --host=localhost:26257
root@localhost:26257/tpcc> show tables;
  schema_name | table_name | type  | owner | estimated_row_count | locality
--------------+------------+-------+-------+---------------------+-----------
  public      | customer   | table | root  |            90000000 | NULL
  public      | district   | table | root  |               30000 | NULL
  public      | history    | table | root  |            90000000 | NULL
  public      | item       | table | root  |              100000 | NULL
  public      | new_order  | table | root  |            27000000 | NULL
  public      | order      | table | root  |            90000000 | NULL
  public      | order_line | table | root  |           900000000 | NULL
  public      | stock      | table | root  |           300000000 | NULL
  public      | warehouse  | table | root  |                3000 | NULL
(9 rows)

Time: 93ms total (execution 92ms / network 1ms)

Show ranges for a database:

1
2
3
4
5
6
7
8
9
10
root@localhost:26257/tpcc> WITH x as (SHOW RANGES FROM database tpcc) SELECT * FROM x WHERE "start_key" NOT LIKE '%Prefix%';
  table_name |   start_key         |  end_key     | range_id |     range_size_mb     | lease_holder | lease_holder_locality | replicas | replica_localities
-------------+-----------------------------------------------------------------------+-----------------------------------------------------------------------
  customer   | /0/1/1              | /5/8/840     |      108 | 100.33181300000000000 |            2 |                       | {1,2,3}  | {,,}
  customer   | /5/8/840            | /21/1/1527   |      120 | 268.43538000000000000 |            3 |                       | {1,2,3}  | {,,}
  customer   | /21/1/1527          | /36/5/892    |      168 | 269.45613900000000000 |            3 |                       | {1,2,3}  | {,,}

(552 rows)

Time: 834ms total (execution 813ms / network 21ms)

Show ranges for a table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
root@localhost:26257/tpcc> WITH x as (SHOW RANGES FROM TABLE tpcc.order) SELECT * FROM x WHERE "start_key" NOT LIKE '%Prefix%';
    start_key    |    end_key     | range_id |     range_size_mb     | lease_holder | lease_holder_locality | replicas | replica_localities
-----------------+----------------+----------+-----------------------+--------------+-----------------------+----------+---------------------
  /0/1/-3001     | /65/7/-2327    |      146 | 83.262868000000000000 |            3 |                       | {1,2,3}  | {,,}
  /65/7/-2327    | /463/6/-2845   |      159 | 521.67653900000000000 |            2 |                       | {1,2,3}  | {,,}
  /463/6/-2845   | /858/9/-1145   |      182 | 525.36147800000000000 |            2 |                       | {1,2,3}  | {,,}
  /858/9/-1145   | /1000/1/-3001  |      222 | 187.54911500000000000 |            2 |                       | {1,2,3}  | {,,}
  /1000/1/-3001  | /1062/10/-2463 |      141 | 83.606528000000000000 |            2 |                       | {1,2,3}  | {,,}
  /1062/10/-2463 | /1466/9/-1100  |      150 | 536.77502200000000000 |            3 |                       | {1,2,3}  | {,,}
  /1466/9/-1100  | /1867/3/-888   |      173 | 532.07307100000000000 |            3 |                       | {1,2,3}  | {,,}
  /1867/3/-888   | /2000/1/-3001  |      231 | 176.37537900000000000 |            3 |                       | {1,2,3}  | {,,}
  /2000/1/-3001  | /2062/10/-2449 |      144 | 83.607130000000000000 |            2 |                       | {1,2,3}  | {,,}
  /2062/10/-2449 | /2466/9/-971   |      151 | 536.78022000000000000 |            1 |                       | {1,2,3}  | {,,}
  /2466/9/-971   | /2867/3/-889   |      206 | 532.06722600000000000 |            1 |                       | {1,2,3}  | {,,}
  /2867/3/-889   | NULL           |      279 | 176.37542400000000000 |            1 |                       | {1,2,3}  | {,,}
(12 rows)

Time: 72ms total (execution 70ms / network 1ms)

5000 warehouses

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@localhost:26257/tpcc> show tables;
  schema_name | table_name | type  | owner | estimated_row_count | locality
--------------+------------+-------+-------+---------------------+-----------
  public      | customer   | table | root  |           150000000 | NULL
  public      | district   | table | root  |               50000 | NULL
  public      | history    | table | root  |           150000000 | NULL
  public      | item       | table | root  |              100000 | NULL
  public      | new_order  | table | root  |            45000000 | NULL
  public      | order      | table | root  |           150000000 | NULL
  public      | order_line | table | root  |          1500000000 | NULL
  public      | stock      | table | root  |           500000000 | NULL
  public      | warehouse  | table | root  |                5000 | NULL
(9 rows)

Time: 52ms total (execution 51ms / network 0ms)

10000 warehouses

1
2
3
4
5
6
7
8
9
10
11
12
13
root@localhost:26257/tpcc> show tables;
  schema_name | table_name | type  | owner | estimated_row_count | locality
--------------+------------+-------+-------+---------------------+-----------
  public      | customer   | table | root  |           300000000 | NULL
  public      | district   | table | root  |              100000 | NULL
  public      | history    | table | root  |           300000000 | NULL
  public      | item       | table | root  |              100000 | NULL
  public      | new_order  | table | root  |            90000000 | NULL
  public      | order      | table | root  |           300000000 | NULL
  public      | order_line | table | root  |          3000058576 | NULL
  public      | stock      | table | root  |          1000000000 | NULL
  public      | warehouse  | table | root  |               10000 | NULL
(9 rows)

30000 warehouses

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@host1 ~]# ls -la /mnt/cockroachdb_mnt1 | wc -l
110426

root@localhost:26257/tpcc> show tables;
  schema_name | table_name | type  | owner | estimated_row_count | locality
--------------+------------+-------+-------+---------------------+-----------
  public      | customer   | table | root  |           900000000 | NULL
  public      | district   | table | root  |              300000 | NULL
  public      | history    | table | root  |           900000000 | NULL
  public      | item       | table | root  |              100000 | NULL
  public      | new_order  | table | root  |           270000000 | NULL
  public      | order      | table | root  |           900000000 | NULL
  public      | order_line | table | root  |          9000000000 | NULL
  public      | stock      | table | root  |          3000000000 | NULL
  public      | warehouse  | table | root  |               30000 | NULL
(9 rows)

Time: 46ms total (execution 45ms / network 1ms)

Troubleshooting

Too many open files

CockroachDB can use a large number of open file descriptors, often more than is available by default. This message indicates that the machine on which a CockroachDB node is running is under CockroachDB’s recommended limits.

For more details on CockroachDB’s file descriptor limits and instructions on increasing the limit on various platforms, see File Descriptors Limit.

1
2
3
4
5
6
7
8
9
10
11
12
[root@host1 ~]# vim /mnt/cockroachdb_mnt1/logs/cockroach.log
<omitted..>
E220902 02:14:03.445845 3866 kv/kvserver/pkg/kv/kvserver/replica_application_result.go:307 ⋮ [n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 9465 +(1) open ‹/mnt/cockroachdb_mnt1/auxiliary/sideloading/r0XXXX/r4›
E220902 02:14:03.445845 3866 kv/kvserver/pkg/kv/kvserver/replica_application_result.go:307 ⋮ [n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 9465 +Wraps: (2) too many open files
E220902 02:14:03.445845 3866 kv/kvserver/pkg/kv/kvserver/replica_application_result.go:307 ⋮ [n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 9465 +Error types: (1) *fs.PathError (2) syscall.Errno
E220902 02:14:03.800155 3776 kv/kvserver/pkg/kv/kvserver/replica_application_result.go:307 ⋮ [n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 9466  while removing sideloaded files during log truncation: open ‹/mnt/cockroachdb_mnt1/auxiliary/sideloading/r0XXXX/r4›: too many open files
E220902 02:14:03.800155 3776 kv/kvserver/pkg/kv/kvserver/replica_application_result.go:307 ⋮ [n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 9466 +(1) open ‹/mnt/cockroachdb_mnt1/auxiliary/sideloading/r0XXXX/r4›
E220902 02:14:03.800155 3776 kv/kvserver/pkg/kv/kvserver/replica_application_result.go:307 ⋮ [n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 9466 +Wraps: (2) too many open files
E220902 02:14:03.800155 3776 kv/kvserver/pkg/kv/kvserver/replica_application_result.go:307 ⋮ [n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 9466 +Error types: (1) *fs.PathError (2) syscall.Errno
E220902 02:14:03.866404 400163 1@util/log/logcrash/crash_reporting.go:174 ⋮ [n1,s1,r647/1:‹/Table/108/1/19{17/4/…-47/7/…}›] 9467  a panic has occurred!
E220902 02:14:03.866404 400163 1@util/log/logcrash/crash_reporting.go:174 ⋮ [n1,s1,r647/1:‹/Table/108/1/19{17/4/…-47/7/…}›] 9467 +open ‹/mnt/cockroachdb_mnt1/013282.sst›: too many open files
<omitted..>

Reference

This post is licensed under CC BY 4.0 by the author.

CockroachDB performance benchmarking

Using netstat to check network package retransmission

Comments powered by Disqus.