Playwright on Kubernetes: Shard a Testkube Workflow and Merge Reports
Running Playwright on Kubernetes is not just a matter of raising a replica count. A shard is a slice of the test suite; a pod is the Kubernetes unit that runs one slice; a worker is a separate browser-launching process inside that slice. Each layer adds capacity and each layer adds overhead. This guide walks through a Testkube workflow that checks out the suite once, transfers it to parallel shard pods, collects blob reports, and merges them into one HTML report. It also shows why a 32-shard run can finish slower than a single shard on the same cluster.
What the workflow does
The main execution pod checks out the repository and installs dependencies once. Testkube then starts one pod per shard, subject to a concurrency cap, and copies the prepared repository into each one. Every shard runs Playwright against its assigned slice and writes a blob report. The main pod fetches those reports and merges them into a single HTML report. The parallel, transfer, and fetch features of Testkube make this possible.
A Kubernetes Indexed Job is a reasonable alternative if you do not need Testkube's workflow steps. Its completion index identifies each shard and its parallelism field caps concurrent pods. Testkube adds setup, file transfer, report collection, and execution tracking around the same idea.
Kubernetes is a good fit when you already run a cluster and want test workloads to use its spare capacity. A single larger CI runner is often simpler for a small suite. Either way, measure CPU, memory, and startup overhead before you scale out.
Measure CPU and memory before choosing pod concurrency
For a reproducible sizing run, record container working-set memory, the unit, the sampling interval, the workload, the worker count, and the observed peak. Summing resident set size across browser processes can double-count shared pages, so use container-level measurements and keep the collection method with the results.
Kubernetes uses resource requests for placement. A low memory request lets more workloads share a node; it does not set the container's memory ceiling. A memory limit is enforced reactively and can end in an out-of-memory kill. Node memory pressure can also cause eviction. These are separate failure modes.
CPU requests matter as well. The tested workflow requested 250m, a quarter of a CPU, per shard while allowing two busy browser workers. That is a weak scheduling signal for this workload. The example keeps it so the benchmark configuration is explicit, but it should be measured and reconsidered before production use.
Choose concurrency from the remaining CPU and memory on each eligible node, leaving room for the main execution pod and other workloads. Total free memory alone does not tell you how many shard pods will fit or perform well.
Prepare the environment
The recorded test environment is summarized in the configuration table below. Treat it as a starting point for your own measurements, not as a sizing recommendation.
TABLE_MODULE: table0001
You need kubectl, the Testkube CLI, and whatever installation tools your agent setup requires. Install Node.js and npm locally to run the Playwright commands and open downloaded reports. The companion repository holds the sample application, both generated suites, the Playwright configuration, and the workflow. Clone or download it and run the commands from its root directory so paths such as k8s/testworkflow.yaml resolve.
The sample application serves a local product catalog and renders 20 items with JavaScript after a 150 ms delay. Each generated spec checks the item count, prices, and title. Playwright starts the application inside each shard pod, so page requests do not depend on an external site. Repository checkout, package installation, and Testkube communication still need their own network connections.
The revised workflow pins a source commit and the Playwright image by digest. Archived execution logs identify the commits used for the original benchmark runs. The revised source differs from the large-suite commit only in .gitignore.
Step 1: Connect the Testkube agent
This workflow needs an agent connected to a Testkube control plane, either cloud-hosted or installed on premises. Standalone mode does not support the parallel operation used here. A standalone attempt failed with this message:
CODE_MODULE: code0002
For Testkube Cloud, create an environment and register an agent through its setup flow, then follow the generated installation instructions for that environment and agent version. For a self-hosted control plane, follow the matching setup instructions.
Keep the agent secret out of inline helm --set flags. A command line like that can linger in shell history and process listings, and characters such as $ or quotes may be parsed in ways you did not intend. Feed the secret through the mechanism your chart documents instead — a protected values file or a pre-created Kubernetes Secret where the chart accepts one. Store credential files outside Git, and remember that a values file does not by itself shield Helm release data.
Point the Testkube CLI at the same control-plane environment and check the connection:
CODE_MODULE: code0003
In the connected setup used here, register workflow definitions through the Testkube API with testkube create testworkflow. Applying the custom resource locally with kubectl apply alone does not register the definition in this control plane.
Step 2: Configure the suite for sharding
The relevant settings live in playwright.config.ts:
CODE_MODULE: code0004
WORKERS caps worker processes and TEST_DIR switches between tests and tests-large. Supply a positive integer for WORKERS. The example keeps the original configuration, including server reuse. In shared CI environments, consider disabling reuse so an unintended existing server cannot satisfy the readiness check, then validate that change separately.
The Chromium launch flag carries over the shared-memory workaround from the original setup. Container runtimes often mount a small /dev/shm, yet Kubernetes does not promise any particular size, so confirm the actual mount inside the test container:
CODE_MODULE: code0005
With --disable-dev-shm-usage, Chromium writes that shared-memory traffic to /tmp instead, which shifts storage load and can change performance. A Kubernetes alternative is a memory-backed emptyDir mounted at /dev/shm, but its pages count against the container's memory budget, so include them when sizing. Playwright's Docker guidance recommends --ipc=host; in Kubernetes that means sharing the host IPC namespace, which is an isolation trade-off you must decide deliberately.
Step 3: Define the test workflow
Store the definition as k8s/testworkflow.yaml. The suite Testkube runs comes from the Git uri and revision inside the manifest, not from files sitting in your local working directory. After editing tests or Playwright configuration, commit and push them, then point uri and revision at that repository state. The definition keeps the original resource settings and adds an explicit guard for missing blob reports.
CODE_MODULE: code0006
count sets how many shard instances the execution creates. Testkube numbers instances from zero, so index + 1 produces the one-based shard number Playwright expects. parallelism: 5 admits at most five shard instances at once, so eight or 32 shards do not all run together.
The main pod installs dependencies a single time with npm ci. After that, the transfer block hands each shard pod its own copy of the source plus the installed dependencies. Version the lockfile and the browser image together: the image supplies the browser binaries, while the Playwright package still comes from the project. For the revised runs, the manifest references the image digest that was checked on both nodes.
The memory request is 1536Mi, or 1.5 GiB, and the limit is 2Gi. These values reproduce the tested fixture; they are not a sizing recommendation and do not guarantee enough memory for a different suite. There is no CPU limit, so a shard can use more than its 250m request when capacity allows.
Step 4: Register and run the workflow
Register the definition and start the smaller suite:
CODE_MODULE: code0007
Select the larger suite through configuration:
CODE_MODULE: code0008
Record the execution ID alongside its logs and artifacts. When you investigate pod placement or failures, scope your queries to resources from that execution rather than every recent pod in the namespace. A shared namespace may hold unrelated tests, and the main execution pod must be counted too.
Step 5: Collect and inspect the merged report
Every shard emits blob ZIP files. Their filenames carry shard identity and sometimes a configuration hash, so do not hard-code a single filename pattern. Keep the generated names and gather reports into a fresh directory for each execution.
The workflow's fetch block retrieves those files, and merge-reports builds the HTML report. condition: always attempts the merge after a shard failure. The guard raises a clear error when no ZIP files exist. If only some shards produced reports, check the execution status and the expected test count: an HTML report can be incomplete.
Retrieve artifacts using the exact execution ID:
CODE_MODULE: code0009
Swap the placeholder execution ID for the real one shown in the list. The download command writes that execution's files into its own directory, and REPORT_DIR targets the HTML report inside it. Verify both the Testkube execution status and the number of tests in the report.
In a September 12 smoke test, the downloaded report showed all 144 tests passing. The 17.0 seconds displayed in the Playwright report is the report duration; Testkube recorded 40.287 seconds for the complete workflow.
Deliberate failures were also exercised against the revised workflow. Adding one failing assertion yielded a report of 144 passing tests plus one failure, and the workflow itself remained failed. A failure before the tests ran left no ZIP files and set off the guard message. Running four workers under a 768Mi memory limit caused both shard containers to be observed as OOMKilled. None of these validation cases count toward the benchmark timings.
Benchmark results from a two-node cluster
The follow-up benchmark ran on September 12 on the same two-node cluster and resource settings described above. Each suite ran in five blocks, with one, four, eight, and 32 shards randomized within each block. The small suite has 48 files and 144 tests; the larger suite has 480 files and 1,440 tests. Each file has three tests against the local fixture. The small suite finished before the large suite, and benchmark executions did not overlap.
The pinned browser-image layers were already present on both nodes, making this a warm-image comparison. Every new main execution pod still performed a repository checkout and an npm ci. Neither package caches nor network latency were controlled. End-to-end timing on fresh nodes was not collected, and the earlier image-pull observations are kept apart from these figures.
CLI wall time runs from the execution launch request through watcher completion. It includes CLI and control-plane latency, setup, scheduling, transfer, tests, and merging. Testkube's own workflow duration is a separate metric. The tables use the CLI metric consistently, with five observations per configuration and 40 complete CLI timings in total. Speedup is the one-shard median divided by the selected configuration's median.
These figures are a dated reference from one shared cluster, not a current independent verification. Check your own cluster conditions before relying on them.
144 tests: four shards and one shard had similar medians
TABLE_MODULE: table0002
Before rounding, the medians came out at 59.370 seconds for a single shard and 59.386 seconds for four. A gap that small gives no reason to pick four shards for speed. Four shards were quicker than one in just one of the five paired blocks. Eight shards landed slightly higher, and 32 shards needed roughly 2.47 times the single-shard time.
1,440 tests: eight shards had the lowest median
TABLE_MODULE: table0003
Eight shards produced the lowest median in the larger-suite sample, a 1.95x speedup over one shard. Four shards had a median 21% below baseline; 32 shards were about 5% below. The observed ranges show the variation within each configuration and should be read alongside the medians. The unusually fast 187.8-second four-shard run had the same sampled placement as the other four-shard runs: three shard pods on node-1 and one on node-2. The retained diagnostics do not include continuous background-load telemetry, so the difference cannot be attributed to placement or background load.
The "Total execution pods" column adds the main pod to one pod per shard across the whole run; it does not represent pods running at the same moment. Five shard pods at two workers each put the configured ceiling at ten Playwright workers running concurrently.
All 40 Testkube workflows passed. Their downloaded HTML reports contained the expected 144 or 1,440 tests, with no unexpected failures, flakes, or skips. All 40 CLI timings are included in the summaries. One post-watch diagnostic query was interrupted by a local DNS failure after its CLI timing interval had ended. Execution-scoped polling observed every expected main and shard pod, with no containers seen in an OOMKilled state and no observed FailedScheduling events. These are polling observations, not a complete historical record of all container states or events.
What changed from the original measurements
In the original two-run sample, the midpoints of the two large-suite observations were 261.5 seconds for one shard, 236.5 for four, 164 for eight, and 314 for 32. Four and eight both improved on that baseline. The midpoints of the two initial small-suite observations were 47, 49, and 116.5 seconds at one, eight, and 32 shards; four shards were not tested in that sample. The original timings are retained separately in the benchmark data and methodology.
The seventh large-suite run used 32 shards. Its watcher finished cleanly and logged a CLI duration of 350.180 seconds. A status query afterward failed because the benchmark client lost DNS access, but that query happened after the CLI timing window had closed. The workflow was subsequently confirmed as passed. The timing stands, with the related post-run diagnostics flagged as interrupted, and the run was not repeated. Across five observations the 32-shard median is 350.180 seconds, about a 1.05x speedup over the one-shard baseline. Eight shards remained the lowest. The other configurations kept their original randomized order.
The new results are not pooled with the earlier observations. Other cluster workloads stayed active during the follow-up, and all five four-shard small-suite runs placed three shard pods on one node and one on the other. Small-suite durations also drifted upward across blocks. Thermal throttling was not controlled, and the observations do not isolate background load, CPU contention, thermals, or placement as the cause.
The repeated series carries more weight than the original two-run sample, yet it is still an exploratory study on a shared cluster. Randomizing the order lowers the influence of a fixed configuration sequence; it does not cancel shifts in background load. If a decision hinges on a small timing gap, a firm recommendation would call for further controlled measurements. When you repeat the experiment, keep the unsuccessful runs and incomplete diagnostics instead of reporting only the favorable timings.
Why adding shards can make the workflow slower
The work in this manifest happens at different frequencies:
TABLE_MODULE: table0004
In the original environment, image pulls took 34 and 52 seconds on the two nodes. Those timings are separate observations, excluded from the warm-cache benchmark, and cannot explain its 32-shard slowdown.
If capacity were unlimited, handing the same work to more shards would cut the tests each one runs. But every shard also pays a fixed startup bill, so the gains taper off. That simple picture still cannot explain why a run gets longer as shards are added.
The concurrency cap reshapes the math. Thirty-two shard instances have to queue through five slots at a time. Each time a slot opens, another instance boots and repeats its own startup and transfer work. Total orchestration effort climbs while the useful test work per shard shrinks. Uneven shard durations and resource contention can push the final completion time out even further.
Where pods land matters. In one original four-shard run, three pods ended up on a single node and one on the other. The isolated pod wrapped up in 59 seconds, while the three sharing a node each took roughly 199 seconds. That fits a contention story, though it does not prove CPU was the only factor.
A 250m request barely reflects what two browser workers actually consume. Measure a request that better matches real demand and rerun the benchmark before you recommend it. On multi-node clusters, look at topology spread constraints or suitable affinity rules so test work is not piled onto a few nodes. Any placement rule must target labels that select the intended workload, and overly strict rules can leave pods stuck pending.
The 144-test suite gained nothing at eight or 32 shards. How long a suite takes is not enough on its own to pick a shard count: another workload, runner, or existing CI setup can reach its break-even point somewhere else entirely.
Troubleshoot pending pods and browser failures
Start with the exact affected pod and container, using names from the selected execution:
CODE_MODULE: code0010
If a container has restarted, add --previous to pull the logs from its earlier instance. Make sure your Kubernetes context points at the environment that actually ran the workflow.
Pending pods: look at scheduling events, the resources requested on each node, taints, and placement constraints. Testkube's concurrency cap only limits how many shard instances are admitted; it cannot make the Kubernetes scheduler find room for them.
OOMKilled containers: inspect termination reasons, memory usage, and the configured limit. In the original forced failure, four workers ran with a 768Mi memory limit and both shard containers were reported as OOMKilled with exit code 137. That shows a limit-related failure in this workload. Changing the request at the same time does not prove the request caused it. Exit code 137 alone is also insufficient to diagnose OOM.
Evicted pods: check the eviction reason and any node-pressure events. Eviction and a container's OOMKilled termination reason are not the same thing and should not be treated as interchangeable.
Chromium or page crashes: read the browser output and check /dev/shm before concluding the container as a whole went over its memory limit. Under controlled conditions, weigh the /tmp workaround against giving the pod a dedicated shared-memory volume.
Timeouts under load: inspect contention, application readiness, waits, and test-data isolation. Retries can help diagnosis but should not replace fixing a repeatable failure. If you select tracing on the first retry, configure a retry; this example uses zero retries.
When a managed scraping API helps
The setup here fits testing an application you control. It measures nothing about reaching third-party sites, proxy throughput, or how well extraction works.
Extra pods do not automatically mean extra public IP addresses. If a cluster relies on network address translation so outbound traffic leaves through a single public address, every request can fall under the same IP-based rate limit, and adding browser capacity will not change that. One guide covers speeding up Playwright scraping scripts from inside the scraper; another covers the access constraints that come with scraping without getting blocked.
For permitted workflows such as monitoring public product prices and listings, a managed API can provide JavaScript rendering, proxy options, geolocation, and scripted page interactions. Select the features appropriate to the target and verify the returned content.
TABLE_MODULE: table0005
Nothing in the timing experiment settles which option costs less. To compare fairly, price out one successful test run or one correctly extracted record, operational effort included.
For third-party scraping, check authorization, site terms, privacy obligations, and applicable law, and use an official API where appropriate. Tooling does not grant permission to collect data. The Playwright image used here is intended for testing; deployment against untrusted sites also requires a separate browser-isolation and security review.
Start with a baseline and measure each change
Build one complete execution first: checkout, tests, blob collection, and an inspectable report. Then vary the shard count while holding the suite, resource settings, and cache conditions constant.
The follow-up small-suite medians were effectively tied at one and four shards, while eight shards produced the lowest median for the larger suite. The results also changed from the original sample, which makes repeat measurement essential. Check startup and transfer overhead, concurrent worker demand, pod placement, and the slowest shard before choosing a configuration.
Match the companion workflow and benchmark instructions to the precise source and configuration you actually run. Speeding up the test step only counts if the whole execution ends earlier and still yields complete, reliable results.
| Component | Recorded configuration |
|---|---|
| Kubernetes | K3s v1.34.5 with Cilium |
| Nodes | Two amd64 nodes, each with 4 CPUs and 16 GB memory |
| Allocatable memory | Approximately 9.9 GiB and 12.9 GiB |
| Testkube | 2.13.1, using a runner connected to Testkube Cloud |
| Playwright | 1.63.0 |
| Browser image | mcr.microsoft.com/playwright:v1.63.0-noble |
| Per-shard settings | Two workers; 250m CPU request; 1536Mi memory request; 2Gi memory limit |
| Concurrent shard pods | At most five |
Recorded configuration from the original test cluster; not a sizing recommendation.
| Cost | When it occurs |
|---|---|
| Repository checkout and npm ci | Once in the main execution pod |
| Pod startup and repository transfer | For each shard instance |
| Playwright runner and worker/browser startup | Within each shard execution |
| Blob retrieval and report merging | Collection across shards, followed by one merge |
| Browser image download | When an eligible node lacks the required image layers, subject to pull policy |
Frequency map of the tested manifest.
| Consideration | Self-managed Playwright | Managed scraping API |
|---|---|---|
| Test execution | Control over the runner, fixtures, assertions, and browser lifecycle | Check whether the API's interaction model fits the data task |
| Browser and proxy operations | Your team maintains the infrastructure | Provider operates the supported browser and proxy service |
| Application responsibility | Tests, infrastructure, extraction, validation, and monitoring | API integration, extraction requirements, validation, and monitoring |
| Cost comparison | Include compute, idle capacity, storage, network, tools, and engineering time | Include request credits, selected features, and integration work |
Qualitative comparison; the timing experiment does not establish which option is cheaper.
Frequently asked questions
How many Playwright shards should I run on Kubernetes?
Try a range of counts that fit the CPU and memory the workflow can actually use. Distinguish three numbers: total shards, shard pods running at once, and workers inside each pod. In the follow-up large-suite sample, eight shards gave the lowest median, but read that as a lead to test on your own workload rather than a fixed rule.
Can Testkube standalone mode run this workflow?
No. The workflow depends on parallel steps, and those need an agent connected to a control plane. That control plane may be Testkube Cloud or a deployment you host on premises.
How do I merge Playwright reports from different pods?
Turn on the blob reporter, gather that execution's ZIP files keeping the names they were generated with, then point npx playwright merge-reports --reporter=html at the directory you collected. Confirm no shard is missing before you accept the report as complete.
Does a low memory request cause OOMKilled?
No. A memory request only tells the scheduler how much room to reserve when placing a pod; it does not cap how much memory the container may consume. To diagnose an OOM, look at the container's memory limit together with its termination reason and exit details. Treat node-pressure eviction as its own separate failure mode, since it is not the same as a container being killed for exceeding its limit. A Chromium crash caused by a small shared-memory mount can also happen while the container stays well within its limit, so rule that out before blaming memory.
Why did 32 shards run slower than one shard for the small suite?
The concurrency cap admitted at most five shard instances at a time, so 32 shards had to queue through those slots. Each new instance paid its own startup and repository-transfer cost while the useful test work per shard shrank. The 144-test suite was too small to amortize that overhead, and the measured median was 146.7 seconds against 59.4 seconds for one shard.
| Shards | Total execution pods | Median | Observed range | Speedup |
|---|---|---|---|---|
| 1 | 2 | 59.4 s | 53.7 to 64.7 s | 1.00x |
| 4 | 5 | 59.4 s | 50.8 to 70.0 s | 1.00x |
| 8 | 9 | 63.7 s | 58.0 to 74.8 s | 0.93x |
| 32 | 33 | 146.7 s | 140.1 to 169.6 s | 0.40x |
Dated reference: September 12, five CLI observations per configuration, warm browser-image cache on both nodes, shared cluster.
| Shards | Total execution pods | Median | Observed range | Speedup |
|---|---|---|---|---|
| 1 | 2 | 368.3 s | 342.3 to 385.7 s | 1.00x |
| 4 | 5 | 292.3 s | 187.8 to 305.1 s | 1.26x |
| 8 | 9 | 188.8 s | 169.4 to 203.4 s | 1.95x |
| 32 | 33 | 350.2 s | 326.9 to 375.5 s | 1.05x |
Dated reference: September 12, five CLI observations per configuration, warm browser-image cache on both nodes, shared cluster.