Benchmarks
Performance metrics for HotShot consensus in Espresso's Doppio testnet release
As a part of launching the Doppio testnet, we are publishing benchmarks related to performance of this release of the Espresso Sequencer. The benchmarks show that a network of 1000 nodes attains very high throughput comparable to that of 10 nodes.
The benchmarks for the Doppio testnet are listed below.
Network Size | Committee Size | Average View Time (s) | Throughput (MB/s) |
---|---|---|---|
1000 | 10 | 1.10 | 29.41 |
500 | 10 | 1.18 | 28.74 |
100 | 10 | 1.10 | 28.52 |
10 | 10 | 0.97 | 25.25 |
Experimental Setup
Node Information
Nodes are run as ECS Fargate tasks in 3 availability zones in the AWS US East 2 (Ohio) region
AWS does not release network bandwidth statistics for Fargate. According to this blog post trying to detect the bandwidth for Fargate, the bandwidth can vary during short-term runs, but is stable over the long term (as expected for a cloud service provider).
All benchmarks are run with 2 vCPUs and 16 GB memory ECS tasks.
ECS tasks are connected to the CDN-like nodes using an AWS VPC (virtual private cloud). They are abstracted such that we see them as being in the same network; in reality they are hosted on AWS VMs that we have little insight into.
Each node runs the HotShot examples inside a Docker container
The example application pre-generates transactions before the run starts. The transactions are randomly generated, dummy transactions about 100 bytes in size. The examples pre-generate transactions so the RNG doesn’t affect performance. Each transaction is then padded with zeros so it is the desired size.
We preselect 10 dedicated nodes to submit transactions throughout the benchmark.
CDN Information
Two CDN-like servers were run to simulate optimistic conditions: one to handle consensus messages and the other to handle data availability messages (including raw transactions)
Each server was an m6a.xlarge EC2 instance with 4 vCPUS, 16 GB memory, and up to 12.5 Gigabit bandwidth
Each server ran 1 instance of the HotShot web server and 1 instance of Nginx
The consensus web server also ran the HotShot orchestrator, which orchestrates the start of a run with certain parameters
Nodes poll the CDNs and orchestrator at regular intervals using HTTP/1.1. This interval is configurable, but is always set to 100ms unless otherwise noted
Benchmark Run Info
Benchmarks were run for 102 views (in order to reach 100 decide events)
At the end of each run each node outputs its statistics: total run time; how many transactions committed; how many views happened
The reported benchmarks are averaged across 3 runs of the same benchmark
Data Calculation
Total run time = average total run time from all nodes
Average view time = number of views completed / total run time
Total transactions submitted = total transactions submitted to the CDN-like node
Total transactions committed = total transactions committed by HotShot during the run
Transactions / sec = total transactions committed / total run time
The transactions in our benchmarks are configured to be quite large (on the order of 1 MB). We configure transactions to be large to more closely simulate the PBS scenario and to avoid extra tuning of parameters that could affect performance (e.g. the batch size of transactions that the CDN node returns per request)
Throughput = size per transaction * total transactions committed / total run time
Throughput uses 1000 as the divisor for KB, MB, etc. instead of 1024
Last updated