In this article, we will discuss how to monitor your website's performance using Playwright. If you are new to playwriting and testing in general, please read this article first.
Why is this important?
If the performance of your website is important to you and you want to know if there are any issues with it, there are plenty of tools available on the market for collecting real user data (Real User Monitoring). However, if there is a performance issue, this data is often not enough to resolve it. Performance tests allow you to measure metrics in an isolated environment, ensuring reproducibility and enabling you to track how changes in the code affect test results.
When and where should such tests be conducted?
The idea is to run these tests every time we want to push changes to the main branch. This can serve as the last check to ensure that performance has not been compromised. These tests should be run on CI on every preview deployment. This article does not cover how to set up CI for Playwright, but you can find information about that in this article.
What will we measure and how?
We are going to measure Core Web Vitals metrics in a browser environment. This is not quite the same as what we get in Lighthouse; instead, it’s what we can see in the Performance tab in Chrome. Firstly, we believe this data is more reliable than Lighthouse data. Secondly, we can control test conditions such as screen size, data transmission speed, CPU speed, etc. Ultimately, we want to measure LCP, CLS, and FCP for a mobile device under various network conditions: fast 3G, slow 3G, and slow 4G. We can also slow down the CPU to 6X, for example. This is optional since the CI worker is already slow enough.
Note: This is all set up as a test, so we should skip it for devices that do not use Chrome, as this functionality is only available there.
Test example
import { expect, test } from '@playwright/test';
import type { CDPSession, Page, TestInfo } from '@playwright/test';
// Define types for various structures used in the test
type Test = { name: string; value: number };
type Devices = {
mobile: Test[];
desktop: Test[];
};
type Results = {
slow3g: { normal: Devices };
fast3g: { normal: Devices };
slow4g: { normal: Devices };
};
type Network = {
download: number;
upload: number;
latency: number;
noThrottle?: boolean;
name: keyof Results;
};
type CPU = {
rate?: number;
noThrottle?: boolean;
name: keyof Results['fast3g'];
};
const SLUG = '/';
// Function to collect performance vitals
async function collectVitals(
page: Page,
session: CDPSession,
testInfo: TestInfo,
network: Network,
cpu: CPU,
prefix: 'no cache' | 'cache',
) {
// Collect LCP (Largest Contentful Paint) metric
const LCP: string = await page.evaluate(
() =>
new Promise((resolve) => {
new PerformanceObserver((list) => {
const entries = list.getEntries();
const lcp = entries.at(-1);
return resolve(Number(lcp?.startTime).toFixed(2));
}).observe({
type: 'largest-contentful-paint',
buffered: true,
});
}),
);
// Collect TTFB (Time to First Byte) metric
const TTFB = await page.evaluate(
() =>
new Promise((resolve) => {
new PerformanceObserver((list) => {
const entries = list.getEntries();
if (entries.length > 0) {
const navigationEntry = entries[0];
const ttfb =
//@ts-ignore
navigationEntry.responseStart - navigationEntry.fetchStart;
resolve(ttfb.toFixed(2));
}
}).observe({
type: 'navigation',
buffered: true,
});
}),
);
// Collect CLS (Cumulative Layout Shift) metric
const CLS: number = await page.evaluate(
() =>
new Promise((resolve) => {
new PerformanceObserver((list) => {
let total = 0;
for (const entry of list.getEntries()) {
// @ts-ignore
total += entry.value;
}
return resolve(+total.toPrecision(2));
}).observe({ type: 'layout-shift', buffered: true });
}),
);
// Collect long tasks metric
const longTasks = await page.evaluate(
() =>
new Promise((resolve) => {
new PerformanceObserver((list) => {
let total = 0;
for (const entry of list.getEntries()) {
total += entry.duration;
}
return resolve(total);
}).observe({ type: 'longtask', buffered: true });
}),
);
// Collect navigation metric
const L = await page.evaluate(
() =>
new Promise((resolve) => {
new PerformanceObserver((list) => {
let max = 0;
list.getEntries().forEach((entry) => {
if (max < entry.duration) {
max = entry.duration;
}
});
return resolve(max.toFixed(2));
}).observe({
type: 'navigation',
buffered: true,
});
}),
);
// Collect paint timings
const paintTimingJson = await page.evaluate(() =>
JSON.stringify(window.performance.getEntriesByType('paint')),
);
const paintTiming = await JSON.parse(paintTimingJson);
const perfData = {
CLS,
longTasks,
LCP,
FCP: 0,
DCL: '',
L,
TTFB: 0,
INP: 0,
};
// Measure INP (Interaction to Next Paint) for buttons
for (const button of await page.getByRole('button').all()) {
const className = await button.getAttribute('class');
const classWithLangSelector = className
?.split(' ')
.filter((e) => e.includes('StyledOpenButton'));
if (classWithLangSelector?.length) {
const selector = '.' + classWithLangSelector[0];
perfData.INP = await page.evaluate(async (selector) => {
return new Promise(async (resolve) => {
requestAnimationFrame(async () => {
const startTime = performance.now();
// @ts-ignore
document.querySelector(selector)?.click();
requestAnimationFrame(async () => {
const endTime = performance.now();
resolve(+(endTime - startTime).toFixed(2));
});
});
});
}, selector);
}
}
// Extract FCP (First Contentful Paint) from paint timings
for (const metric of paintTiming) {
if (metric.name === 'first-contentful-paint') {
perfData.FCP = metric.startTime.toFixed(2);
}
}
// Get performance metrics from CDP session
const performanceMetrics = await session.send('Performance.getMetrics');
results[network.name][cpu.name][testInfo.project.name as keyof Devices] =
performanceMetrics.metrics;
// Add paint timings to results
paintTiming.forEach((element: any) => {
results[network.name][cpu.name][
testInfo.project.name as keyof Devices
].push({
name: element.name,
value: element.startTime,
});
});
// Extract DCL (DOMContentLoaded) from performance metrics
for (const metric of performanceMetrics.metrics) {
if (metric.name === 'DomContentLoaded') {
perfData.DCL = metric.value.toFixed(2);
}
}
// Assert performance metrics against thresholds
expect(+perfData.LCP).toBeLessThan(2500);
expect(+perfData.FCP).toBeLessThan(1800);
expect(+perfData.CLS).toBeLessThan(0.1);
expect(+perfData.TTFB).toBeLessThan(800);
expect(+perfData.INP).toBeLessThan(200);
}
// Basic test function to set up and run performance tests
async function basicTest(
page: Page,
network: Network,
cpu: CPU,
testInfo: TestInfo,
) {
// Create a new connection to an existing CDP session to enable performance metrics
const session = await page.context().newCDPSession(page);
// Enable performance metrics collection in CDP session
await session.send('Performance.enable');
// Apply CPU throttling if needed
if (!cpu.noThrottle && cpu.rate) {
await session.send('Emulation.setCPUThrottlingRate', { rate: cpu.rate });
}
// Apply network throttling if needed
if (!network.noThrottle) {
await session.send('Network.enable');
await session.send('Network.emulateNetworkConditions', {
offline: false,
downloadThroughput: network.download,
uploadThroughput: network.upload,
latency: network.latency,
});
}
// Navigate to the specified URL
await page.goto(SLUG);
// Collect performance vitals
await collectVitals(page, session, testInfo, network, cpu, 'no cache');
}
// Define network conditions for different test scenarios
const networkConditions = {
slow3g: {
download: ((500 * 1000) / 8) * 0.8,
upload: ((500 * 1000) / 8) * 0.8,
latency: 400 * 5,
name: 'slow3g' as keyof Results,
},
fast3g: {
download: ((1.6 * 1000 * 1000) / 8) * 0.9,
upload: ((750 * 1000) / 8) * 0.9,
latency: 150 * 3.75,
name: 'fast3g' as keyof Results,
},
slow4g: {
download: ((1.6 * 1000 * 1000) / 8) * 1.5,
upload: ((750 * 1000) / 8) * 1.5,
latency: 150,
name: 'slow4g' as keyof Results,
},
};
// Define CPU conditions for different test scenarios
const cpuConditions = {
normal: {
noThrottle: true,
name: 'normal' as keyof Results['fast3g'],
weight: 10,
},
};
// Initialize results object to store performance metrics
const results: Results = {
slow3g: {
normal: {
mobile: [],
desktop: [],
},
},
fast3g: {
normal: {
mobile: [],
desktop: [],
},
},
slow4g: {
normal: {
mobile: [],
desktop: [],
},
},
};
// Describe the performance test suite
test.describe('Performance test', () => {
// Skip tests for non-Chromium browsers and non-mobile devices
test.skip(
({ browserName, isMobile }) => !isMobile || browserName !== 'chromium',
'Chromium only!',
);
// Loop through network and CPU conditions to run performance tests
for (const networkDetails of Object.values(networkConditions)) {
for (const cpuDetails of Object.values(cpuConditions)) {
test(`Get performance metrics for network: ${
networkDetails.name
} and cpu: ${`slow ${cpuDetails.name}`}`, async ({
page,
}, workerInfo) => {
await basicTest(
page,
networkConditions[networkDetails.name],
cpuConditions[cpuDetails.name],
workerInfo,
);
});
}
}
});
Note: The results of this test only give you a vague idea of the user experience, and you should only compare it to previous results. It does not cover all cases: for example, networking will be faster and the processor will be slower for a hosted worker than for the average consumer device. But it should give you a baseline with other monitoring tools to understand where you stand.
Summary
Performance tests using Playwright are another tool in your performance monitoring toolkit. However, unlike many others, this tool has an advantage—it immediately reflects changes in the code. If you have previous results, you can determine whether the performance has improved or worsened.
The preferred environment to run this test is CI. Here's an article on testing with Playwright on CI.