checkly/headless-benchmarks

Headless benchmarks

Benchmarking scripts for Puppeteer, Playwright, WebDriverIO (running against standalone Selenium server and devtools) and Cypress.

Setup

Local

Install the needed packages using npm i For WebDriverIO Selenium runs you will also need to install and run the Selenium Standalone Server:

./node_modules/.bin/selenium-standalone install
./node_modules/.bin/selenium-standalone start

For scripts running against Danube

Danube is a fully static demo website, and needs no further setup to be run against. Scripts running against Danube are already set up to export credentials for you as follows:

export [email protected]
export USER_PASSWORD=supersecure1

When running Cypress, remember to export the above variables with the CYPRESS_ prefix added, e.g. CYPRESS_USER_EMAIL=....

For scripts running against Checkly

Before kicking off any benchmarks against Checkly, create a placeholder check, as well as a placeholder snippet (note the id of the snippet, which is displayed at the end of the URL on the snippet page, e.g. https://app.checklyhq.com/snippets/6395)

Scripts running against Checkly will require you to set the following:

export URL=https://app.checklyhq.com
export EMAIL=<your checkly email>
export PASSWORD=<your checkly password>
export SNIPPET_ID=<your placeholder snippet id>

When running Cypress, remember to export the above variables with the CYPRESS_ prefix added, e.g. CYPRESS_EMAIL=....

Note: like most production web applications under active development, Checkly changes very often. Therefore, you might need to adjust your scripts (especially selectors) to reflect the latest changes.

Running the benchmarks

Scripts are in place for each website+tool combination.

You only need to provide a number of iterations and a label for your run, e.g.:

./bench-danube-plwr.sh 1000 my-label

Timings and logs for each run are stored under results.

Automation scripts

The goal is to have stable scripts that are as similar as possible across tools, i.e. scripts that perform the same high-level operations in the browser, regardless of the small differences under the hood (which are expected due to differences between the automation tools and APIs).