Key Takeaways
Taken together, [4] and [65] indicate that running jobs that test different things in parallel in the same stage reduces overall runtime but needs more runners running simultaneously [4][65].
Spread across 12 parallel slots in a 720-hour month the 2,580-hour break-even point works out to those slots averaging about 30% utilization, below which hosted per-minute pricing is competitive and self-hosting isn't worth the operational overhead [4][65].
- Caching dependencies and build outputs to improve workflow execution time [23]; the build cache stores build outputs and allows fetching them when inputs have not changed, avoiding regenerating them [20]; a key can use the hash of package-lock.json so the key changes and a new cache is created when dependencies change, with restore-keys as alternate keys searched on a miss [21]. Setting up proper dependency caching can reduce build times by 50–90% depending on the project [36].
- Use rules to skip tests that aren't needed, for example skipping backend tests when only frontend code is changed [4]; without dynamic configuration, each change causes all jobs and workflows to run even if unrelated to the change [39]. Sharding splits tests into shards where each shard can run as a separate job to speed up test runtime [52]; cutting the wall-clock time of a large test suite by splitting it across jobs into balanced shards based on historical timing data [55]. Each job runs its shard and uploads timing reports used to balance the next run [55].
- Taken together, [4] and [60] indicate parallelism trades speed for spend. Running jobs that test different things in parallel in the same stage reduces overall runtime, with the downside that more runners are needed simultaneously [4]. Sharding increases total computing time and, in cloud CI services, can mean higher usage of minutes or CPUs, i.e. potentially higher cost [60]. Spread across 12 parallel slots in a 720-hour month, break-even averages about 30% utilization, and below that GitHub's per-minute pricing is competitive [65]. A small team of developers with possibly no dedicated DevOps engineer should consider a GitHub-hosted solution, as it simplifies work [16].
- Actual gains vary with project structure, dependencies, network, and CI environment, so teams should run their own benchmarks before adopting optimizations [15]. Overly specific keys may result in missed cache hits [35], while large caches can take longer to restore, reducing the performance gain [35].
[!WARNING] Cache contents are not signed or verified, and extracted caches may modify files that are subsequently executed in a workflow run, leading to malicious code execution [21]. Anyone who can open a pull request, from a fork included, can read the base branch's caches, so keep secrets and credentials out of cached paths [33].
Abstract
Taken together, [23][39] and [55] indicate caching dependencies and build outputs to improve workflow execution time [23], dynamically deciding the work to do in a pipeline based on the location of changes is beneficial when code or microservices are stored in a monorepo [39], and splitting a large test suite across jobs into balanced shards based on historical timing data cuts wall-clock time [55].
Spread across 12 parallel slots in a 720-hour month, that 2,580-hour break-even point works out to those slots averaging about 30% utilization; below that, GitHub's per-minute pricing is genuinely competitive and self-hosting isn't worth the operational overhead [65].
An exact match to the provided key is considered a cache hit, while no exact match is considered a cache miss that automatically creates a new cache if the job completes successfully [21], and creating a key using the hash of an npm package-lock.json file means the key changes and a new cache is automatically created when the dependencies that make up that file change [21]. Caching node_modules directly is unreliable [25]. Partial restores can leave orphaned packages that affect runtime behaviour [25], whereas the ~/.npm download cache avoids these problems because npm resolves and installs the correct versions from the lock file while using cached downloads to skip the network fetch [25]; restoring a stale CocoaPods download cache is safe because pod install re-resolves and re-links content from Podfile.lock regardless of what is already cached [25].
Dynamically deciding the work to do in a pipeline based on the location of changes is beneficial when code or microservices are stored in a monorepo [39], with a service build running only if a change was made in that service directory as determined by setup-phase path filtering [39], yet a workflow skipped by path filtering, branch filtering, or a commit message leaves its associated checks in a "Pending" state and blocks merging [38], while Predictive Test Selection trades testing comprehensiveness for faster feedback and additional test executions that run remaining tests or do not enable Predictive Test Selection should occur later in the change lifecycle, for example as part of post-merge, nightly or ready-for-release builds [31], and when enabled only a subset of tests may be executed, resulting in lower code coverage than usual [31].
Overall test time can shrink to about 1/N of the original plus overhead, trading additional compute resources for faster results [60], while sharding reduces the time to get results but increases total computing time [60]; a shard with disproportionately slow tests becomes the bottleneck that determines when the suite finishes [55], each shard carries overhead such as job startup and dependency install so there is a point of diminishing returns [60], the relationship between worker count and execution speed is not linear and the added benefit of each extra worker decreases as workers increase [59], and overhead for preparing tests such as starting a Docker container or seeding a database means the best case parallelized suite takes as long as the overhead time and no faster [59]. Depot gives each shard a different, non-overlapping subset of candidates balanced by how long each candidate took in past runs [55], and each job runs its shard and uploads a JUnit XML report whose times are used to balance the next run [55]. GitHub will remove any cache entries that have not been accessed in over 7 days, and total size of all caches in a repository is by default limited to 10 GB per repository [21], while large caches can take longer to restore, reducing the performance gain [35], and type=gha stores layers in the same Actions cache as actions/cache with the same quota so a large image with mode=max can crowd out dependency caches [33]. Caching node_modules, Docker layers, and build artifacts can cut build times and therefore costs by 50-70% [29] and proper dependency caching can reduce build times by 50-90% depending on the project [36], but actual gains vary with project structure, dependencies, network, and CI environment and teams should run their own benchmarks before adopting optimizations [15].
Table of Contents
- Key Takeaways
- Abstract
- 1. Introduction
- 2. Background
- 3. Findings
- 3.1 Measuring Queue Time and Critical-Path Bottlenecks
- 3.2 Dependency, Build and Container-Layer Caching Trade-offs
- 3.3 Test Selection and Changed-Path Filtering
- 3.4 Parallel Jobs and Balanced Test Sharding Effects
- 3.5 Runner Capacity and Self-Hosted Runner Decisions
- 4. Discussion
- 5. Conclusion
- 6. Hypotheses and possible implications
- References
1. Introduction
This chapter sets scope. This report asks what practical approaches help a small software team shorten CI build times and what trade-offs accompany each choice. Pipeline and runner terminology is covered elsewhere in this report. Extended compile and test cycles disrupt flow, encourage batched work, and increase the adverse impact of each CI failure, according to one Datadog analysis [18]. Buildkite notes other platforms struggle with the increased build volume from growing codebases [19]. One guide notes a GitHub Actions setup can have multiple runners so that jobs can be executed in parallel [16].
Measure first.
-
GitLab docs list repository size, total stages and jobs, dependencies between jobs, and the critical path among influences on total pipeline duration [4]. For quick triage, the same docs point to runtimes of jobs, stages, and total pipeline runtime as the easiest indicators for inefficient pipelines [4]. They also recommend monitoring global pipeline health alongside job and pipeline duration, with analytics giving visual representation [4].
-
Queue time covers time in queue before execution, execution time covers time actively running jobs, and approval wait time covers time waiting for manual approval, in Datadog docs definitions [1].
-
Harness notes changes in job counts, such as increased queued jobs or decreased completed jobs, can provide insights into efficiency, resource utilization, or execution issues [5].
-
To locate slowdowns, Depot docs describe a Top 5 Slowest Jobs list across repositories and workflows, a duration trends graph to spot recently slowed jobs, and step-level timing breakdowns with resource graphs for drill-down [2].
-
CircleCI, in its product description, spots slow workflows, queue bottlenecks, and failed jobs in real time with an insights dashboard [14]. Datadog docs describe CI Visibility as providing a pipeline-first view into CI health by displaying important metrics and results from pipelines [1]. The same docs note the tool helps troubleshoot pipeline failures, address performance bottlenecks, and track CI performance and reliability over time [1].
Scope stays narrow. Small-team capacity shapes the options; self-hosted ownership considerations are covered in full elsewhere in this report. CircleCI docs warn a fixed fleet of self-hosted runners can incur unnecessary costs as workload fluctuates with the job queue rate [12]. This report keeps broader pricing models, security hardening, and team process change outside scope except where they directly alter build-time choices. It covers parallel execution, caching, selective execution, and observability for small-team pipelines.
Background defines timing vocabulary and measurement points for later comparison. Findings surveys practical approaches that teams operate inside everyday CI systems. Discussion weighs trade-offs. Conclusion distills guidance for action and names open gaps for further work.
2. Background
A GitHub Actions setup can have multiple runners so that jobs can be executed in parallel [16].
-
Datadog defines execution time as the time for which a pipeline has been actively running jobs [1].
-
Datadog defines queue time as the time for which a pipeline or job was in the queue before execution [1].
-
Datadog defines approval wait time as the time for which a pipeline or job has been waiting for a manual approval [1].
-
GitLab documentation names the runtimes of jobs, stages, and the total runtime of the pipeline itself as the easiest indicators to check for inefficient pipelines [4]. Total pipeline duration is heavily influenced by the size of the repository, total number of stages and jobs, dependencies between jobs, and the critical path [4]. The critical path represents the minimum and maximum pipeline duration [4].
-
Viewing the needs dependencies in the full pipeline graph can help analyze the critical path in the pipeline and understand possible blockers [4]. Datadog's Critical path feature identifies CI jobs that are on the critical path of the pipeline [1]. It also offers step-level spans for more granular visibility [1].
GitLab documentation lists availability of the runners and the resources they are provisioned with as additional points to pay attention to, adding that if you use GitLab-hosted runners, to choose a machine type sized for the job instead of over-provisioning or under-provisioning [4].
CircleCI documentation states that maintaining a fixed compute fleet of self-hosted runners can incur unnecessary costs as workload can fluctuate depending on the rate at which jobs are queued [12]. It states that to help reduce this cost, the compute fleet can be scaled according to the demand [12]. If the container runner on Kubernetes is used, it will automatically spin up more pods as more work enters the queue [12]. Those pods are ephemeral and will be torn down after the job is done executing [12].
The comparison includes the statement "Is limited to only 2 core CPUs" [16]. It includes the statement "Is limited to 7 GB of RAM on a RAM-hungry Windows and Linux server" [16]. It states "you have better visibility into and control over logs and debug tooling," for example logging in via SSH to the machine to troubleshoot, while "When using GitHub-hosted runners, you will have some visibility via logs in the Actions panel" [16]. If a code base takes a long time to build and/or test and involves processes that are very multithreaded, or if a specific Linux distribution or even an old Windows version is needed, the comparison states you would be better off with a self-hosted solution [16].
Linux 2-core (x64) is listed as $0.006 [62]. Linux 1-core (x64) is listed as $0.002 [62]. Windows 2-core (x64) is listed as $0.010 [62]. macOS 12-core is listed as $0.077 and macOS 5-core (M2 Pro) is listed as $0.102 [62]. Docker layer caching uses 200 credits per job run in a pipeline (equivalent to $0.12 per job run) [26].
-
The actions/cache action allows caching dependencies and build outputs to improve workflow execution time [23]. Subsequent jobs that use the same cache don't have to download the files again, so they execute more quickly [24].
-
The build cache works by storing (locally or remotely) build outputs and allowing builds to fetch these outputs from the cache when it is determined that inputs have not changed, avoiding the expensive work of regenerating them [20]. By default, the build cache is not enabled; Gradle will use the build cache for this build only when run with --build-cache on the command-line, or it can be enabled by putting org.gradle.caching=true in gradle.properties [20]. Task types need to opt-in to task output caching using the @CacheableTask annotation, and custom task types are not cacheable by default [20].
-
You cannot change the contents of an existing cache; instead, you can create a new cache with a new key [21]. When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified, and for each instruction checks whether it can reuse the instruction from the build cache [22].
-
Because GitHub's runners are ephemeral, the BuildKit layer cache that makes local rebuilds instant is thrown away the moment the job ends, and the next run starts from zero, reinstalling system packages, re-resolving dependencies and recompiling everything [28]. The fix is to persist that layer cache somewhere outside the runner and feed it back in on the next build [28].
-
GitLab documentation states that
parallelruns a job multiple times in parallel in a single pipeline [11]. Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently [11]. -
Gradle runs only one task at a time by default [41]. To execute tasks from different subprojects in parallel, the
--parallelflag is used [41]. Gradle can run multiple test classes or methods in parallel, with parallel execution enabled by setting themaxParallelForksproperty on Test tasks [41]. Parallel test execution assumes that tests are isolated, and tests that share state or resources such as file systems, databases, or external services may fail intermittently due to race conditions or resource conflicts [41]. -
Playwright documentation describes sharding as splitting tests into smaller parts called "shards", where each shard is like a separate job that can run independently to speed up test runtime [52]. In a CI pipeline, each shard can run as a separate job, making use of the hardware resources available in the pipeline, like CPU cores, to run tests faster [52]. One Playwright example states that if these shards run in parallel on different jobs, the test suite completes four times faster [52]. Without the
fullyParallelsetting, Playwright Test defaults to file-level granularity, meaning entire test files are assigned to shards; if test files are not evenly sized, certain shards may end up running significantly more tests, while others may run fewer or even none [52]. -
Any monorepo approach must start with detecting changes, usually at folder level [43].
-
GitLab documentation describes
rules:changesas adding a job to a pipeline only when specified files are checked and found changed [11]. That rule resolves to true if any listed file matches, as an OR operation [11]. -
In one monorepo example, changing only one service out of 50 generates only that service's build and test steps [43]. Changing a shared library still requires rebuilding and testing every dependent service, not only services with direct file changes [43].
-
Gradle incremental build skips tasks whose inputs and outputs have not changed, while custom tasks must declare inputs and outputs to qualify [41]. Develocity documentation describes Predictive Test Selection as selecting only those tests for execution predicted to provide useful feedback on the changes under test [31]. It trades testing comprehensiveness for faster feedback and pairs with later comprehensive runs without selection, such as post-merge, nightly or ready-for-release builds [31]. Tests that are recently new, recently changed, recently failed or recently flaky are always chosen for execution [31].
3. Findings
3.1 Measuring Queue Time and Critical-Path Bottlenecks
Pipelines automate and orchestrate delivery through stages typically organized as build, test and deploy, according to Harness [5], while GitHub Actions runs such pipelines to build, test, lint and deploy code [16].
Taken together, [1] indicates queue time and execution time measure different waits: queue time is the time for which a pipeline or job was in the queue before execution [1], while execution time is the time for which a pipeline has been actively running jobs [1].
| Metric | What it measures | | Queue time | Time pipeline or job spent in queue before execution [1] | | Approval wait time | Time pipeline or job waited for manual approval [1] | | Execution time | Time pipeline actively ran jobs [1] |
A Harness Job is defined as a single run or execution of the pipeline which includes one or more stages such as building artifacts, executing test cases, deploy [5].
Buildkite documents the expected chronological order for job lifecycle measurement as created_at → scheduled_at → runnable_at → started_at → finished_at [3].
Buildkite's REST API flattens finished into passed or failed based on the job's exit status and lists limiting and limited as scheduled for legacy compatibility, while the GraphQL API uses finished for all completed jobs [3].
Datadog's Critical path feature identifies CI jobs that are on the critical path of the pipeline [1]. Total pipeline duration is heavily influenced by the size of the repository, the total number of stages and jobs, dependencies between jobs, and the critical path [4]. That critical path represents the minimum and maximum pipeline duration [4]. Viewing needs dependencies in the full pipeline graph can help analyze the critical path in the pipeline and understand possible blockers [4].
-
GitLab describes the pipelines used to build and verify GitLab as having more than 90 jobs, where some are simple tasks that take a few seconds to finish while others are long-running processes that must be optimized carefully [6]. For example, in December 2020, successful merge request pipelines had a duration of 53.8 minutes [6].
-
At the time of writing, GitLab had more than 700 pipelines running, each representing changes from team members and contributors from the wider community [6]. Given that GitLab ran around 500 merge request pipelines per day, it posed the question of whether the process can be optimized [6].
-
Job, stage and total runtime are the easiest inefficiency indicators to check for inefficient pipelines, according to GitLab [4]. Its Pipeline success and duration charts give information about pipeline runtime and failed job counts [4]. Global pipeline health is a key indicator to monitor along with job and pipeline duration, with CI/CD analytics giving a visual representation of pipeline health [4].
Datadog CI Visibility provides a pipeline-first view into your CI health by displaying important metrics and results from your pipelines [1]. It helps you troubleshoot pipeline failures, address performance bottlenecks, and track CI performance and reliability over time [1]. Analysis can help identify possible blockers in the CI/CD infrastructure, including analyzing job workloads, bottlenecks in the execution times, and the overall pipeline architecture [4].
-
Harness job duration reports can help you optimize build times by analyzing the total time it takes for jobs to run [5].
-
The CI/CD Job Duration Single Stat displays the median or average job execution duration as a numeric value but does not support drill-down [5].
-
The CI/CD Job Duration Report displays information regarding the duration of job execution, including maximum, median, and minimum values, as bar charts and supports drill-down for detailed analysis [5].
-
Pipeline job reports are functionally identical to job count reports and job duration reports, but pipeline job reports present heatmap visualizations by default, whereas job count and duration reports present bar charts [5].
-
The Harness CI/CD Job Duration Trend Report provides daily, weekly and monthly trends in job execution duration including minimum, median and maximum values, represented as a line graph [5].
-
To monitor and improve build time for a collection of builds, analysis of the range of performance metrics in a time series is needed, which differs from visualizing builds individually and seeking outliers using the Develocity performance dashboard [7]. In the described Develocity example, the author set the time granularity to “Week” or “Month” to filter out the noise of daily ups and downs [7].
-
Depot provides a duration trends graph to spot jobs that have recently slowed down or to identify patterns over time [2]. Daily successful and failed minute breakdowns are labeled as estimates, calculated from each day’s total elapsed time and job counts [2].
-
When a CI/CD Job Count report is added to an Insight, the Job End Date filter is set to a relative time frame by default, and setting Job End Date to Insight time makes the widget report all jobs run in the user-selected Insight time [5]. The Develocity overview shows the volume of builds and aggregate build time, with an example of 1700 builds (and climbing) executed taking over 42 days of serial execution time [7].
Depot displays a Top 5 Slowest Jobs list of your most time-consuming jobs across all repositories and workflows [2]. Clicking a job in the Top 5 Slowest Jobs list views the 50 slowest runs for that job [2]. Cypress Cloud's Slowest Tests analytics report sorts tests by duration, with the longest tests described as your best candidates to simplify or restructure [8]. CircleCI describes automatically identifying your flakiest and slowest tests with timing analytics and failure patterns [14]. Buildkite Test Engine describes identifying and optimizing slow or poor-performing tests using timing data analysis from all test runs [17].
-
Depot's detailed job view provides step-level timing breakdowns and resource utilization graphs [2]. Datadog offers step-level spans for more granular visibility [1] and supports using
datadog-cito send command-level events to CI Visibility to be incorporated into pipeline waterfall visualization [1]. -
Step-level breakdowns help identify common bottlenecks like dependency installation, test execution, or cache misses [2]. Dagger Traces shows how long a pipeline took to run, including every step [9] and provides detailed step-by-step execution of a pipeline and the associated telemetry to optimize Dagger Functions and modules as you author them [9]. The Dagger GitHub app shows the exact functions called for each commit, with their status, in real time [9]. Every CI build generates a build scan with a
CItag [7]. -
Cypress advises running Cypress App with debug logs enabled to print CPU and memory consumption every 10 seconds, where CPU consistently above 100% means the machine is saturated [8]. On a local machine, setting an env variable, and in a CI environment, using integrations, means each time the engine calls a function, a new trace will automatically appear in your Dagger Cloud account [9]. That tracing is equally useful for debugging pipelines running on a developer's local environment as it is for pipelines in CI [9]. 'Slowest SQL queries', 'Slowest HTTP requests' and 'Span timeline' are listed [17].
-
Depot's GitHub Actions main page shows queued, running, and completed jobs across all repositories with live runner status [2]. The current metrics refresh every 15 seconds [10], and each data point represents a snapshot at the end of a minute [10]. Use queue metrics and job investigation tools to operate an agent fleet and investigate why jobs are or are not running [10]. Connected is the number of agents currently connected to the queue, excluding agents that are stopping [10]. These tools are separate from the Queue Insights dashboard, which provides historical metrics across queue and cluster scopes [10].
-
GitLab's
resource_groupcreates a resource group that ensures a job is mutually exclusive across different pipelines for the same project; if multiple jobs that belong to the same resource group are queued simultaneously, only one of the jobs starts [11]. -
CircleCI lists Unclaimed Tasks, Running Tasks and List Resource Classes among API endpoints available to help set up a solution to scale machine runners or a Kubernetes cluster for container runners [12]. Use the queue-level activity chart to identify patterns in how fleet capacity responds to job demand and evaluate the efficiency of scaling rules [10].
Changes in the job count, such as an increase in queued jobs or a decrease in completed jobs, can provide insights into the pipeline's efficiency, resource utilization, or potential issues affecting its execution [5]. Identifying any cyclical or seasonal patterns in CI/CD job counts enables anticipation of resource allocation and workload fluctuations [5].
Datadog Logs Analysis uses LLM models on relevant logs to analyze the root cause of failed CI jobs [1]. To highlight failed jobs in the widget, Harness documents adding a Job Status filter set to Failed [5]. A Dagger Trace's details page can be used to debug failures and identify bottlenecks [9].
-
Datadog correlates host-level information for the Datadog Agent, CI pipelines, or job runners to CI pipeline execution data [1]. You can export your search query to a CI Pipeline monitor on the Executions page or the Test Runs page by clicking the Export button [1]. When creating a dashboard or a notebook, you can use CI pipeline data in your search query, which updates the visualization widget options [1].
-
You can fetch specific pipeline health metrics from the API, and external monitoring tools can poll the API and verify pipeline health or collect metrics for long term SLA analytics [4]. For example, the GitLab CI Pipelines Exporter for Prometheus fetches metrics from the API and pipeline events, can check branches in projects automatically and get the pipeline status and duration, and in combination with a Grafana dashboard helps build an actionable view for your operations team [4].
-
Buildkite Pipelines provides a unified dashboard that shows build health, queue metrics, and agent status across the entire organization [13]. The insights dashboard spots slow workflows, queue bottlenecks, and failed jobs in real-time [14], while CircleCI provides org-level Insights and dashboards for build performance [13]. Test insights can be collected from any CI/CD tool including Buildkite Pipelines, GitHub Actions, Jenkins, CircleCI, and more [17], with performance insights and analytics to get more out of existing CI compute [17].
CircleCI's performance and throughput are constrained by plan-based concurrency, queued capacity, and shared-platform limits [13]. As organizations scale, these limits show up as longer queue times and slower developer feedback loops [13]. Extended compile and test cycles disrupt flow, encourage batched work, and increase the adverse impact of each CI failure [18]. Other platforms struggle with the increased build volume from growing codebases [19], described as slow builds that don't scale [19]. Actual gains vary with project structure, dependencies, network, and CI environment [15]. Run your own benchmarks before adopting any of these optimizations [15].
3.2 Dependency, Build and Container-Layer Caching Trade-offs
The build cache stores build outputs locally or remotely and allows builds to fetch these outputs from the cache when it is determined that inputs have not changed, avoiding the expensive work of regenerating them [20]. Subsequent jobs that use the same cache don’t have to download the files again, so they execute more quickly [24]. This action allows caching dependencies and build outputs to improve workflow execution time [23].
-
Caching
node_modules, Docker layers and build artifacts can cut build times and therefore costs by 50-70% [29]. -
Setting up proper dependency caching can reduce build times by 50–90% depending on the project [36].
-
Caching dependencies and containerising builds to cut build times from 45 minutes to under 15 minutes [37].
-
Restoring the local build cache reduced build time from 32.3s to 28.9s, saving 3.4s or about 11%, reusing cacheable task outputs instead of recomputing them [15]; these numbers are not intended as universal performance claims and show the relative impact for one small Spring Boot project, one Gradle version, one JDK image and one local Docker environment [15], with individual runs varying by a second or two so small differences should be read as approximate [15].
One small Spring Boot example went from 32.3s to 28.9s, about 11% [15].
Data and sources
| Baseline (no caching) | 32.3 seconds [15] |
|---|---|
| Local build cache | 28.9 seconds [15] |
-
Existing cache contents cannot be changed; a new cache with a new key is required [21]. An exact match to the provided key is considered a cache hit, while no exact match is considered a cache miss that automatically creates a new cache if the job completes successfully [21]. If the provided key matches an existing cache, no new cache is created [23].
-
Keys have a maximum length of 512 characters, and longer keys cause the action to fail [21]. Creating a key using an expression that calculates the hash of a
package-lock.jsonfile means the cache key changes and a new cache is automatically created when the dependencies that make up that file change [21]. -
On a miss,
restore-keysallows a list of alternate restore keys, the action searches for keys prefixed with the restore key, and returns the most recently created cache when there are multiple partial matches [21]. Thecache-hitoutput is set to true when the cache is restored using the primary key and false when the cache is restored usingrestore-keysor when no cache is restored [27]. -
The example
runner.os-npm-github.shakey never hits on the next commit and writes a fresh entry on every push, and those entries fill the quota and evict caches other jobs still need [33]. Using unique cache keys ensures a cache is correctly restored or created, while overly specific keys may result in missed cache hits [35]. Native modules such assharporbcryptare built for one OS and Node.js ABI, so both must be put in the key [33]. -
Pip's download cache at
~/.cache/pipstores downloaded wheels before installing them, and partial restoration is safe because pip resolves and installs correct versions from the dependency file regardless of what wheels are present in the cache [25]. -
The
~/.npmdownload cache avoids these problems because npm resolves and installs the correct versions from the lock file, using the cached downloads to skip the network fetch [25]. Cache the npm download cache at~/.npm, notnode_modulesdirectly [25]. Cachingnode_modulesdirectly is unreliable because it breaks when Node or npm versions change between the cache write and the restore, and partial restores can also leave orphaned packages that affect runtime behaviour [25]. -
Partial restoration of an installed Python environment is only safe with a lock file pinning every dependency to an exact version, because without pinning pip may leave stale versions in place rather than upgrading them [25]. Pipenv generates a
Pipfile.lockthat pins every dependency to an exact version, and partial restoration of the installed virtual environment is safe because the lock file ensures pip installs correct versions even on top of a stale partial restore [25]. Poetry generates apoetry.lockfile that pins all dependencies to specific versions, and as with Pipenv the lock file makes partial restoration of the installed virtual environments directory safe [25]. -
Partial restoration is safe for Yarn because Yarn has always used a lock file [25]. Gradle centralizes its repository cache and adds to it over time, and partial restores do not affect which libraries end up on classpaths because Gradle fetches libraries into the cache and resolves them from it [25]. Maven centralizes its repository cache and adds to it over time the same way Gradle does, and Leiningen uses Maven under the hood and behaves the same way [25].
-
Partial restores are safe for Bazel because it reuses only the entries whose hash matches the current build and rebuilds everything else [25]. For CocoaPods, cache
~/Library/Caches/CocoaPodsrather than the installedPods/directory, and restoring a stale download cache is safe becausepod installre-resolves and re-links content fromPodfile.lockregardless of what is already cached [25]. -
Task types must opt in with
@CacheableTask, and custom types are not cacheable by default, Gradle documentation states [20]. The cache is disabled by default and enabled with--build-cacheororg.gradle.caching=trueingradle.properties[20]. When both caches are enabled Gradle loads local first then remote, stores remote hits locally, with local push enabled and remote push disabled by default [20]. -
Gradle fingerprints declared inputs, outputs and implementation details, then reuses local or Build Cache outputs instead of executing [15]. The task path is not a key input, so tasks with different paths can reuse each other's outputs as long as Gradle determines that executing them yields the same result [20]. Task-output caching reuses outputs from any earlier build in any location on the local machine, and with a shared cache across developer machines and build agents [20].
-
Missing inputs can cause incorrect hits, missing outputs can cause failures, and volatile data or absolute paths can lead to misses [20]. Sharing cache between developers or machines using different paths requires setting path sensitivity because inputs are identified by absolute path by default [20]. Environments with different
$PATHvalues will not share hits because only--action_envallowlisted variables enter the action definition, Bazel documentation explains [34]. -
Any
globalDependencieschange misses all Turborepo tasks [30], whilepassThroughEnvvalues do not contribute to keys unless also inenvorglobalEnv[30]. Test tasks store results in the Build Cache only if build caching is enabled, all tests were selected and successful [31].
Docker steps through Dockerfile instructions and reuses each layer when possible [22]. A .dockerignore file excludes irrelevant files without restructuring the repository [22]. GitHub runners are ephemeral, so the BuildKit layer cache is discarded when the job ends and the next build restarts from zero [28]. The fix is to persist that layer cache outside the runner and feed it back next build [28]. The cleanest wiring exports to GitHub's Actions cache service with cache-from: type=gha and cache-to in docker/build-push-action [28]. The first run only populates with no speedup; the second shows CACHED with 0.0s timing [28].
Docker layer-cache export options compared.
| Export path | What is cached | Trade-off |
|---|---|---|
mode=min default |
Only final-image layers [28] | Earlier stages rebuild on a fresh runner even when unchanged [33] |
mode=max |
Every stage including intermediates [28] | Large images can crowd out dependency caches under the shared 10 GB quota [33] |
| Registry backend | Cache as separate tag with no GitHub size limit [28] | Push/pull latency and registry storage cost on every run [28] |
Taken together, [28] indicates that layer caching differs from cache-mount caching because the layer cache captures the result of each Dockerfile instruction while the cache mount is scratch space that exists only during the build and is deliberately excluded from the image [28]. Those mounts are not saved by the gha, registry, or local backends [28], and BuildKit does not preserve cache mounts in the GitHub Actions cache by default [32].
The workaround is the buildkit-cache-dance action, which extracts the cache mount contents into a directory, persists that with actions/cache, and injects it back before the build [28]; this action creates temporary containers to extract and inject the cache mount data [32]. For compiled languages the compiler cache is often the single biggest win, because the compiler cache survives dependency changes that would otherwise invalidate the whole build layer [28].
-
Cache contents are not signed or verified, and extracted caches may modify files that are subsequently executed in a workflow run, leading to malicious code execution [21]. Anyone who can open a pull request, from a fork included, can read the base branch's caches, and secrets and credentials should be kept out of cached paths [33]. Only
push,workflow_dispatch,repository_dispatch,delete,registry_package,page_buildandschedulecan create or overwrite caches in the default branch's scope [21]. You may want only your CI system to be able to write to the remote cache [34]. Enabling this option is a security risk, as it allows any cache server to impersonate the intended server, and it should only be used as a temporary measure or in very tightly controlled network environments [20]. Running the Gradle Build Cache Node over HTTP for quick experimentation is not recommended for production environments—use HTTPS for security [15]. -
GitHub will remove any cache entries that have not been accessed in over 7 days, with total size limited by default to 10 GB per repository [21], and cache entries can be created at up to 200 uploads per minute per repository and downloaded at 1500 downloads per minute per repository [21].
-
Docker layer caching uses 200 credits per job run in a pipeline (equivalent to $0.12 per job run) [26]. Large caches can take longer to restore, reducing the performance gain [35]. When hitting GitHub's 10 GB limit, or letting a local cache balloon, eviction never shows up in the build log; if cached builds slow down with no Dockerfile change, check cache size [28].
3.3 Test Selection and Changed-Path Filtering
Using static configuration, each time a change is made to any file, all jobs and workflows would run even if they are unrelated to the actual change, CircleCI's dynamic-configuration guide reports [39]. Dynamically deciding the work to do in a pipeline based on the location of changes is beneficial when code or microservices are stored in a monorepo, the same guide states [39]. Running a full test suite for many projects is a barrier when someone submitting a PR only needs to see the results of the one they've worked on, one jest --changedSince discussion reports [44]. Conditionally executing only changed-path jobs drastically reduces CI/CD time and cost, that discussion claims [45].
GitLab rules:changes adds a job to a pipeline only when specified files are checked and found changed, according to GitLab's YAML reference [11]. Any listed-file match resolves true as an OR operation [11]. GitLab's pipeline-efficiency guide advises skipping unneeded tests, for example skipping backend tests when only frontend code changed [4].
One discussion asks about checking build artifacts in a condition instead of the git commit that triggered the change [50], wanting a build step to execute only if the dist folder contains the portal it is meant to deploy [50].
-
For new branch pipelines or when there is no Git push event,
rules:changesalways evaluates to true and the job always runs [11]. -
A maximum of 50 patterns or file paths can be defined per
rules:changessection [11]. -
Tag pipelines, scheduled pipelines, and manual pipelines do not have a Git push event associated with them, and to cover these cases use
rules:changes:compare_toto specify the branch to compare against the pipeline ref [11]. -
For performance reasons, GitLab performs a maximum of 50,000 checks against changes patterns or file paths, and after the 50,000th check rules with patterned globs always match, so the changes rule always assumes a match when more than 50,000 files changed [11]. Jobs or pipelines can run unexpectedly when using
rules:changes[11]. -
A pull request that only changes a file in the repository root will not trigger this workflow, and if build is required, the pull request is blocked with "Waiting for status to be reported" [38].
-
This workflow requires a successful build job, but runs only when a pull request changes files in scripts [38].
-
A workflow skipped by path filtering, branch filtering, or a commit message leaves its associated checks in a "Pending" state and blocks merging [38]. The guidance states "Avoid requiring workflows that can be skipped" [38]. One example describes a
pull_requestworkflow withpaths: - '**/*.py'that will skip the workflow if no file with the specified language was changed [49]. -
The
filter-pathsjob runs onubuntu-latestand runsactions/checkout@v4beforedorny/paths-filter@v3[45]. Its example triggers onpushandpull_requesttargetingmain[45]. -
The action outputs boolean
trueorfalseper path-map key indicating changed paths [45]. Downstream jobs gate withneeds: filter-pathsplusif: needs.filter-paths.outputs.KEY == 'true', so they queue only on detected changes [45]. Underneath it leveragesgit diffto set outputs usable inifconditions [45]. -
One excerpt shows
python: needs: plan if: needs.plan.outputs.python == 'true'[49]. -
One poster asks for a way to skip (or directly pass) a step if the portal is not in the build artifact, wanting the build step to execute only if the
distfolder contains the portal it is meant to deploy [50], noting the poster could not find anything about checking build artifacts in a condition instead of the git commit that triggered the change [50]. -
One excerpt states the
dorny/paths-filteraction is the industry standard for this use case [45]. -
If a repository uses GitHub Actions to perform required checks on pull requests, workflows need to include the
merge_groupevent as an additional trigger, otherwise status checks will not be triggered when a pull request is added to a merge queue [38]. -
Pull requests that are up-to-date and pass required status checks can be merged locally and pushed to the protected branch without running status checks on the merge commit itself [38]. For checks created by workflow jobs to be evaluated for a pull request, the workflow run must be triggered by
push,pull_request,pull_request_review,pull_request_target,deploymentordeployment_status[38]. -
A job skipped by a conditional reports "Success" [38], while a workflow skipped by path filtering, branch filtering or a commit message leaves associated checks in "Pending" and blocks merging [38].
contains(needs.*.result, 'failure')means a skipped job will not trigger a failure, only an actual failure will [49].if: always()makes sure it runs even when everything upstream was skipped [49]. -
Natively path-filtered workflows cannot satisfy GitHub required checks because a skipped required check blocks PR merging instead of passing [49]. GitHub staff added that issue to papercuts without committing to a timeline [49].
-
One documented workaround using an extra workflow creates N+1 files for each required workflow and doubles the checks in the PR UI [49].
-
A different single-workflow pattern puts everything in one workflow without
paths:, decides inside what changed, and makes a single always-running gate job the only required check [49]. A workflow cannot reference more than 20 reusable workflows [49]. The gate pattern works with merge queue because it usesmerge_groupand nopaths:filters at all [49]. Adding a new conditional job requires only touchingneeds:on the gate job, never branch protection settings [49]. -
CircleCI's path-filtering orb maps file paths with pipeline parameter values, and if a file change matches a path regex, the corresponding parameter is set [39]. It uses them to conditionally run workflows using the
whenkey [39]. -
Only one workflow that includes a continuation is allowed per pipeline [39]. When pipeline parameter
run-build-service-1-jobORrun-build-service-2-jobis true, therun-integration-testsjob is triggered [39]. -
config-pathis ignored if a mapping is in place, but is used if no mapping matches the path to modified files [39]. This will only happen if a change was made in the service-1 directory in the commit, as determined based on the path filtering in the setup phase (config.yml) [39]. In the event of changes to any file in the repository (.*) theshared-config.ymlconfiguration is included in the continuation configuration [39]. The script outputs a new configuration YAML based on some type of work it performs, for example inspecting git history or pipeline values that get passed to it [39]. -
Buildkite's monorepo guide describes a static approach where a single diff pipeline triggers different static pipelines in the monorepo based on what parts of the monorepo were changed [43], contrasting with dynamic pipelines that inject specific steps into a single pipeline based on the changes in the monorepo, requiring a generator script to inject steps according to the changes [43]. Additional steps can also be uploaded while a build is in progress [3].
-
Any approach to running monorepos needs to start with detecting changes in the monorepo, usually at the folder level [43]. Adding an
if_changedattribute to command, group, or trigger steps, then running the pipeline upload command with the--apply-if-changedoption, lets the Buildkite agent evaluate each step'sif_changedexpression against the changed files in the build [43]. -
Upload steps in batches, or trigger child pipelines, so that no single build exceeds a few hundred steps [43]. In the example, if only one service out of 50 gets changed, only that service's build and test steps will be generated [43]. What to run can be decided based on changed files, dependency graphs, repository state, or external signals, generating only the steps needed instead of producing a monolithic config [13].
-
The triggered pipelines must only be triggered by the dedicated triggering pipeline and never directly through the Buildkite interface, API, or other means, because direct execution bypasses the change detection logic and can generate build artifacts with incorrect library versions [43]. Dynamic pipelines can implement selective rebuilds with the
if_changedattribute for declarative path filtering, while for teams that want a turnkey solution, the Monorepo Diff plugin watches for changes across directories and triggers the appropriate pipelines automatically [13].
Changing a shared library means you need to rebuild and test every service that depends on it, not just the ones with direct file changes [43]. For that, you will need a pipeline generator script that resolves transitive dependencies [43]. Discussion guidance states that because a change in a shared library should also rebuild everything that depends on it, the plan job can compute that from the manifests instead of hand-written filters [49].
-
Turborepo treats
package.json,turbo.jsonand package manager lockfiles as always considered inputs, even if explicitly ignored [30]. WithoutaffectedUsingTaskInputs,--affectedoperates at the package level where any file change in a package selects all tasks in that package [30]. -
With
filterUsingTasks,--filterresolves at the task level by matching git-range changed files against each task's inputs globs and traversing the Task Graph instead of the Package Graph [30]. WithwatchUsingTaskInputs,turbo watchre-executes only tasks whose declared inputs match the changed files, rather than re-running all tasks in changed packages [30]. -
Changes to root configuration files, the package manager lockfile, or
globalDependenciesalways cause all tasks to be selected, regardless of individual task inputs [30]. WithaffectedUsingTaskInputs,--affectedselects only tasks whose declared inputs match the changed files, rather than all tasks in changed packages [30]. -
In the reference example, if only
README.mdchanged in a package, build would not run because it excludesREADME.mdfrom its inputs [30]. Without the flag,--filtermarks entire packages as matched if any file in the package changed and follows package-level dependencies [30].
The production pattern in Nx is defined once and reused across build and test [48]. Running nx affected -t build after editing one package narrows execution to that package and the packages that depend on it [48]. A change to a spec file won't invalidate the build cache because production explicitly excludes test files [48]. NX uses its dependency graph to run only test, build and lint for changed modules or modules depending on a changed module [49].
Develocity trains the predictive model on code changes and test outcomes from Build Scan data, and updates it with each new Build Scan [31]. With Predictive Test Selection enabled, builds select only those tests predicted to provide useful feedback on the changes under test [31]. It is available for Gradle and Apache Maven builds and is complementary to Build Caching and Test Distribution [31]. The tests selected for execution can be accelerated by distribution, further increasing build speed [31].
Predictive Test Selection profiles compared.
| Profile | Selection behavior | Time-saving consequence |
|---|---|---|
| Standard (default) [31] | Aims to strike a good balance between saving time and selecting relevant tests [31] | Aims to strike a good balance between saving time and selecting relevant tests [31] |
| Fast [31] | Favors saving time by using a higher selection threshold, which often selects fewer tests [31] | Often saves more time than the Standard profile [31] |
| Conservative [31] | Favors increasing the likelihood of selecting all tests that provide useful feedback, which often requires selecting more tests [31] | Reduces time savings compared to the Standard profile [31] |
-
Tests will always be chosen if they’re recently new, recently changed, recently failed, or recently flaky [31], and recently flaky tests will always be selected because flaky tests are inherently unpredictable [31]. Predictive Test Selection always selects tests that were skipped for the same inputs [31] and tests matching configured class-name filter patterns or annotation-driven criteria [31].
-
If the Gradle invocation filters tests using
--tests, Predictive Test Selection will always select all matching tests to ensure a consistent CLI and IDE user experience [31]. -
If the server is unreachable Predictive Test Selection is deactivated and all tests are executed, if processing lags behind it is temporarily deactivated and all tests are executed, and if selection requests take too long test selection is disabled and all tests are executed [31]. For test tasks or goals for which fewer than 14 days of code and dependency changes are available, Predictive Test Selection won’t attempt to make predictions, and in this case all tests except those having previously passed for identical inputs are selected [31].
-
For optimal prediction accuracy it’s vital that any additional file inputs to the test task are declared, and tracking additional input files by absolute paths convolutes the change history and negatively impacts its predictions [31]. Enabling flaky test retry functionality is encouraged as it allows test selection predictions to better account for non-deterministic flaky test outcomes [31]. Tests which have already passed against the same sources and dependencies are less likely to be selected [31].
-
Subset execution may result in lower code coverage than usual, so Develocity recommends enabling coverage measurement only for builds with Predictive Test Selection disabled [31]. The Build Scan indicates which tests were selected and which were not along with the reason why, and how much time was saved by skipping the not selected tests [31].
-
Predictive Test Selection trades testing comprehensiveness for faster feedback, with additional executions that run remaining tests or do not enable selection to occur later in the change lifecycle, for example as part of post-merge, nightly or ready-for-release builds [31]. A typical usage pattern uses "relevant tests" selection mode for early and frequent QA stages and "remaining tests" selection mode for later stages to ensure that all tests are ultimately executed, which avoids executing any given test more than once for a given change [31].
-
The simulator compares actual test results from Build Scan data to what would have happened if the build used Predictive Test Selection, visualizing task/goal failures predicted, savings potential, avoidable tests and unavoidable tests [31]. With
grepFilterSpecs: true, Cypress reads the spec files first, identifies which ones contain matching tests, and only loads and compiles those, and Branch Review, which compares test results between a PR branch and the base branch, loses accuracy when the test sets are asymmetric, for example when a PR branch runs only@smokewhile main runs the full suite [8].
Tests that depend on another test can fail when Predictive Test Selection is enabled because the depended-on test may not be selected [31].
-
Jest examples define
testasjest --coverageandtest:changedsinceasjest --changedSince=master --coverage[44]. -
A push workflow installs, lints and runs
test:changedsincewithactions/checkout@v2andactions/setup-node@v1[44]. -
Safe use fetches
masterthen restores the PR HEAD before testing [44]. The excerpt reports the suite failed withfatal: bad revision '^master'becauseactions/checkoutonly checks out PR-relevant code andmasteras a branch did not exist [44]. -
Bazel teams can run only the tests for a given package by feeding kind-filtered query results into
bazel test[47]. -
Standard
bazel querycannot revealselect()-based configurable dependencies because it operates on the loading phase without build flags, so accurate config-aware selection requirescquery[47]. -
Teams can identify tests affected by a shared-library change with
kind('go_test', rdeps(//..., //lib/metrics:metrics))[47]. History-based analysis leveraging failure and test execution history to detect faults in submitted code is a widely adopted approach, and tools like Bazel have gained popularity for tracking history and performing comparisons [46]. -
Using query results as input to
bazelor other shell commands composes dynamic workflows in delivery pipelines [47]. Before deleting apparently unused code, ifbazel queryfor dependencies reports nothing, it is likely safe to remove [47].
Gradle skips a task when its inputs and outputs have not changed since the last execution, and a custom task must declare inputs and outputs to be compatible [41].
Safe Buck2 incrementality requires:
- comparing previous-run inputs with current-run inputs to detect what changed [42]
- keeping an incremental state with at minimum a list of every source used and hash digests for those sources [42]
- keeping the previous result accessible with
no_outputs_cleanupon, while the user script detects which parts to delete and performs manual cleanup [42] - deleting before amending the result, because failure while changing the result can leave state out of sync and cause subsequent incorrect builds reported as "successful" [42]
Incremental actions produce the current result from the previous run's result [42]. Incrementality could significantly improve performance for actions such as packaging Apple App Bundles or MSVC incremental linking [42].
The failure rate of pre-merge checks compared to post-merge is 5:3 [46]. Unless skipping techniques provide trustworthy and explainable justifications, they are used with caution [46]. Diff- and dependency-based selection tools primarily operate in the post-merge and pre-release phases [46]. The ideal or safest scenario is to run all jobs at the pre-merge phase to ensure software security and reliability, but this approach can lead to inefficiencies and increased costs, as developers must wait longer for the pipeline to complete [46]. The study distinguishes “good” failures, which occur early in the development process (e.g., pre-merge), from “bad” failures, which arise at later phases (e.g., post-merge), with the cost of addressing issues at these phases varying significantly [46]. Two of the participants highlighted flaky tests and intermittent failures as major contributors to CI/CD job failures in their companies [46]. While techniques such as build skipping and test case prioritization can enhance productivity, many companies hesitate to adopt them due to concerns about undetected failures, and instead favor strategies that provide clear explanations and traceability [46]. Resolving issues after the merge can be significantly more time-consuming and complex, potentially outweighing the benefits of deferred execution [46]. There is hesitation in adopting early-phase build optimization techniques, especially those involving skipped builds or test cases, with participants preferring exposing failures earlier rather than later, aiming to detect as many issues as possible in the early phases [46]. While some companies have observed that postponing rarely failing jobs to later phases can improve overall efficiency, they approach this with caution, because post-merge failures could result in pre-merge effect such as merge stop, which impacts the entire team [46].
A pipeline hierarchy can contain up to 1000 downstream pipelines by default [40], and parent and child pipelines have a maximum depth of two levels of child pipelines [40]. With strategy: mirror, the trigger job always has the same status as the downstream pipeline [40]. Child pipelines do not directly affect the overall status of the ref the pipeline runs against, and the status of child pipelines only affects the status of the ref if the child pipeline is triggered with trigger:strategy [40]. You can’t use if: $CI_PIPELINE_SOURCE == "merge_request_event" to configure child pipeline jobs to run for merge request pipelines; instead, use if: $CI_MERGE_REQUEST_ID [40]. Child pipelines configured with interruptible are automatically canceled when a new pipeline is created for the same ref [40]. Without these strategies, the parent pipeline completes before child pipelines finish, and their reports don’t appear in the merge request [40]. This technique can be very powerful for generating pipelines targeting content that changed or to build a matrix of targets and architectures [40]. You can use rules with include to conditionally include other configuration files [11].
Matrix authoring, test splitting and sharding mechanics, container and dependency caching internals, queue analytics and runner sizing belong to Parallel Jobs, Caching, measurement and Runner Capacity sections and are not developed here.
3.4 Parallel Jobs and Balanced Test Sharding Effects
Playwright documentation defines sharding as splitting tests into smaller parts called shards where each shard can run independently, with the purpose of speeding up test runtime [52], while Bazel's test encyclopedia states the runner is launched once per shard when sharding is enabled [51]. Bazel tests can be parallelized via test sharding enabled with shard_count and --test_sharding_strategy [51], with runners selecting work via TEST_TOTAL_SHARDS as the number of shards and TEST_SHARD_INDEX as the shard index beginning at 0, for example with a round-robin strategy [51]. Gradle documentation reports that, by default, Gradle runs only one task at a time [41].
Parallel-job mechanisms:
- Gradle:
--parallelfor tasks from different subprojects [41] andmaxParallelForksfor test classes or methods [41]. - Since Gradle 9.4.0,
org.gradle.tooling.parallelcontrols Tooling API parallelism independently of task execution parallelism [41]. - GitLab
parallelclones the same job so copies run simultaneously [6]. - Buildkite
parallelismsplits one step into many jobs differing only by index and count while sharing dependencies and tags [56]. - One CircleCI example spins up four identical environments with
parallelism: 4[57]. - One GitLab account treats a normally 20-minute
frontend-fixturesjob as parallelizable because each fixture could be generated independently [6].
Sharding trades additional compute resources for faster results [60]. Each shard can run as a separate job, making use of available hardware resources such as CPU cores to run tests faster [52], and Playwright by default runs test files in parallel for optimal CPU-core use with further scaling by running tests on multiple machines simultaneously [52]. Depot guides state splitting a large suite across jobs into balanced shards based on historical timing data cuts wall-clock time [55]. Playwright documentation states that running shards in parallel on different jobs completes the test suite four times faster [52]. One Jest guide estimates overall test time can shrink to about 1/N of the original plus some overhead [60]. In one parallel_tests report, the suite took 86 seconds with 1 CPU, 47 seconds with 2 CPUs, and 26 seconds with 4 CPUs [54]. Jest's own test suite improved from about 10 minutes down to about 3 minutes on Linux by using shards [60]. One report found running 8 shards in parallel reduced total test time from 19 minutes to under 3 minutes [60]. In the reported GitLab pipeline, splitting the longest-running fixtures-generator file into smaller batches and optimizing it so more tests run in parallel shaved off an additional ~3.5 minutes [6].
Running jobs that test different things in parallel, in the same stage, reduces overall runtime, with the downside that more runners running simultaneously are needed to support the parallel jobs [4]. Each parallel job consumes an agent, so parallelism: 50 requires 50 available agents [56]. Use of parallel to run a job multiple times in parallel in a single pipeline requires multiple runners to exist, or a single runner configured to run multiple jobs concurrently [11]. The maximum number of jobs that can run simultaneously when using a matrix job strategy is set with jobs.<job_id>.strategy.max-parallel [53]; one example workflow runs a maximum of two jobs at a time, even if there are runners available to run all six jobs at once [53]. While sharding reduces the overall time to get results, it increases the total computing time, which in cloud CI services can mean higher usage of minutes or CPUs, i.e. potentially higher cost [60]. Ideally, the number of shards matches the number of parallel runners available, for example with 4 concurrent jobs use 4 shards, because more shards than available executors won’t speed up the overall time and would just queue up [60]. Each shard carries some overhead (job startup, dependency install, etc.), so there is a point of diminishing returns [60]. Overhead costs for preparing tests, whether starting up a Docker container or seeding a database and tearing things down, must be paid, so in the best case the parallelized suite takes as long as the overhead time to run, and no faster [59]. The relationship between worker count and execution speed isn’t linear, and as the number of parallel workers increases, the added benefit of each extra worker decreases [59]. A calculator reference gives realised speedup as 1 / ((1-p) + p/raw), with p (CPU-parallel fraction) of 0.85 for compile-heavy down to 0.4 for IO-bound workloads [61].
The slowest shard determines when the suite finishes.
- Depot warns a disproportionately slow shard becomes the bottleneck [55].
parallel_testsnotes groups often run for different times, making the run as slow as the slowest group [54].- After splitting into two parallel jobs, the shards took three and 17 minutes respectively for an overall decrease of about three minutes [6].
- Cypress distributes whole spec files, so one long spec can idle other machines [8].
- Without
fullyParallel, Playwright defaults to file granularity; uneven files can leave shards with many, few or even no tests [52]. - Playwright can only shard parallelizable tests, by default test files [52].
- By default Jest divides evenly by file count, not duration [60].
- Shardy McShardFace shards as evenly as possible with uneven splits in tail shards [56].
Generated jobs taking 3 and 17 minutes [6].
Data and sources
| Faster shard | 3 minutes [6] |
|---|---|
| Slower shard | 17 minutes [6] |
Historical timing evens the load: Depot gives each shard a different, non-overlapping subset of the candidates, balanced by how long each candidate took in past runs [55].
| Shard balancing compared [60][55] | Evenness limit [60][59] |
|---|---|
| Division evenly by number of test files, not by test duration [60] | If one test file is extremely slow, it could make its shard slower than others [60] |
| Depot gives each shard a different, non-overlapping subset of the candidates, balanced by how long each candidate took in past runs [55] | Actual timing results will differ due to random differences in test run time and retrying any failed tests, so evenness depends on estimate accuracy controlled by timing statistic and sample count [59] |
Playwright Test runs individual tests in parallel across multiple shards with fullyParallel: true, ensuring each shard receives an even distribution of tests [52]. Especially in CI environments, Playwright documentation recommends fullyParallel: true when aiming for balanced distribution across shards, otherwise manually organizing test files to avoid imbalances [52]. parallel_tests splits tests into balanced groups by number of lines or runtime and runs each group in a process with its own database [54]. Depot guides state using historical timing data to distribute tests evenly across shards helps reduce the suite's total run time [55]. Timing-based splitting has no timing data to use the first time the tests are run, but on subsequent runs the test time will be optimized [57]. The Test Engine Client determines optimal worker duration as total estimated run time divided by parallelism and marks any individual files accounting for more than 70% of a single worker's time to be split [59]. One Playwright proposal describes greedy longest-processing-time-first sharding from prior-run timings [58]. Buildkite Test Engine uses historic test timing data as input to a bin packing algorithm that distributes the work, optimized to ensure that all parallel workers finish in the same amount of time [59].
Gradle documentation warns parallel test execution assumes isolated tests, and tests sharing file systems, databases or external services may fail intermittently due to race conditions or resource conflicts [41]. If such problems are observed after enabling Parallel Execution, Gradle documentation states to consider disabling it to preserve the reliability of the build results [41]. All shards must run with the same codebase and configuration, with each parallel job checking out the same commit and identical environment setup, because sharding simply partitions tests and provides no synchronization between them [60]. Sharding may reveal implicit dependencies between tests if, for example, one test file seeds some global state that another relies on, and if tests are found failing only when sharded, it might indicate an order dependency that needs fixing [60]. All shards must receive the same timing data as input, otherwise tests will be missed [58]. Not all test runners support sharding; if a runner supports sharding, it must create or update the last modified date of the file specified by TEST_SHARD_STATUS_FILE, otherwise Bazel will fail the test if it is sharded [51]. GitHub Actions supports sharding tests between multiple jobs using the jobs.<job_id>.strategy.matrix option, running Playwright tests with --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} [52]. The number of processes defaults to available CPUs and can be overridden with -n [54]. Two environment variables can be used to divide application tests between the different jobs [56]. Flaky tests can often negate the speed performance gained by parallelizing a test suite, as the build or the job will need to be re-run until the flaky test passes, with impact driven by flaky proportion and flaky timing such as a test that is flaky 50% of the time but takes 100 seconds to fail being significantly more impactful than the same test that takes 1 second to fail [59]. The client will collect any failures from the initial run and retry these tests individually in the same job, and retry failing tests up to the number of retries specified, and integrates to ignore any failing test results from tests that have been marked with mute [59]. On CI, adding the blob reporter to the config when running on CI is the starting point, because the blob report contains information about all the tests that were run and their results as well as all test attachments such as traces and screenshot diffs [52]. Merging blob reports with npx playwright merge-reports --reporter html ./all-blob-reports will produce a standard HTML report into the playwright-report directory [52], and to ensure the execution order, the merge-reports job is made to depend on the sharded playwright-tests job by adding needs: [playwright-tests] [52]. Recording runs to Cypress Cloud unlocks the analytics needed to systematically improve performance: test duration trends to see which tests are getting slower over time and machine utilization to see if parallel machines are balanced or if one is a bottleneck [8]. The guide advises monitoring shard execution times; if one consistently runs much longer, you might need to balance them, which could involve moving some tests into a different file or using a custom test sequencer that distributes tests based on historical timing [60].
3.5 Runner Capacity and Self-Hosted Runner Decisions
Arnica describes runners as the apps that execute an assigned pipeline job after an event such as a push to a branch [16]. A GitHub Actions setup can have multiple runners so jobs execute in parallel [16]. Capacity and timing detail belongs in measurement and parallel-job sections.
GitHub's pricing reference lists standard macOS 3-core or 4-core (M1 or Intel) at USD 0.062/minute [62], standard Linux 2-core (x64) at USD 0.006/minute [62], standard Linux 1-core (x64) at USD 0.002/minute [62], standard Linux 2-core (arm64) at USD 0.005/minute [62], and standard Windows 2-core (x64) at USD 0.010/minute [62]. GitHub's billing concepts page lists Linux 2-core (x64) at USD 0.006/minute, Windows 2-core (x64) at USD 0.010/minute and macOS 3-core or 4-core (M1 or Intel) at USD 0.062/minute [63]. OpenMetal reports GitHub cut hosted-runner rates by up to 39% to USD 0.006 per minute for Linux, USD 0.010 for Windows and USD 0.062 for macOS for 2-core runners [65]. Taken together, [62] and [63] indicate the listed Linux 1-core (x64) rate is the lowest among the rates listed in this paragraph.
Comparison of GitHub runner charges and public-repository treatment
| Option | Per-minute charge | Public-repository treatment |
| Standard Linux 2-core (actions_linux) | USD 0.006/minute (≈ GBP 0.00/minute) [62] | Free and does not reduce included compute [63] |
| Larger macOS 12-core (macos_l) | USD 0.077/minute (≈ GBP 0.06/minute) [62] | Billed even for public repositories [63] |
| Self-hosted runner | Free and does not reduce included compute [63] | Free of GitHub platform charge as of mid-2026 [65] |
GitHub lists larger macOS 12-core at USD 0.077/minute and macOS 5-core (M2 Pro) at USD 0.102/minute [62]. CICDCalculator reports a 16-core runner costs 8x as much per minute but only finishes 3.5x faster — so the bill goes up [61].
Access and included-use notes:
- Larger runners are restricted to organizations and enterprises on GitHub Team or Enterprise Cloud [62].
- Larger runners are not free for public repositories [62] and are billed for public repositories [63].
- Their cost can draw from included compute for Team and Enterprise Cloud billing accounts [63], while CICDCalculator states larger runners never use free minutes [61].
- Taken together, [63] and [61] indicate conflicting statements on included usage.
Idle and image billing:
- Idle larger runners cost nothing and bill only for workflow execution time [62].
- No additional cost for public static IP addresses on larger runners [62].
- Custom images only work with larger runners, bill at the same per-minute rates, with storage billed separately [62].
- Larger runners need a valid payment method; usage is blocked without one once included compute is consumed [63].
- Custom-image storage bills on retained image data over time, not per pull [63].
Who pays and what is free:
- After monthly included compute is consumed, extra usage is billed and the allowance resets each cycle [63].
- Standard hosted runners in public repositories are free and do not reduce included compute [63].
- Self-hosted runners are free and do not draw from included compute [63].
- Runner compute cost is charged to the repository owner, not the workflow triggerer [63].
Right-size with utilization and host signals:
- Depot size-down recommendations flag jobs using far fewer resources that could run smaller without performance impact to save billable minutes [2].
- Depot size-up recommendations flag jobs constrained by high CPU or memory where larger runners can cut build times [2].
- GitLab ties efficiency to runner availability and provisioned resources and warns against over- or under-provisioning machine type [4].
- GitLab advises monitoring disk and disk IO, CPU, memory and runner process resources [4].
- Prometheus Node Exporter can monitor Linux runners and
kube-state-metricsruns in Kubernetes [4].
Added parallel machines cut latency, treated fully in the parallel-jobs section. Cypress docs report adding a second parallel machine cut a 1:51 serial run to 59 seconds, a 53% reduction, with 4–8 machines typically bringing large suites under 10 minutes [8].
Break-even and fit:
- Free tier runs out fast and self-hosted runners save money [29].
- Spread across 12 parallel slots in a 720-hour month, a 2,580-hour break-even works out to about 30% average utilization; below that GitHub per-minute pricing is competitive and self-hosting is not worth the overhead [65].
- Self-hosting fits well if you want costs that do not move when GitHub revisits pricing [65].
- Self-hosting does not fit well for low-volume pipelines well under the free tier or inside a small bill [65].
CircleCI does not charge per build minute on self-hosted runners [26]. Self-hosted GitHub Actions runner usage remains free of any GitHub platform charge as of mid-2026, with no announced timeline for reintroducing a fee [65]. A GitHub Actions self-hosted runner fee was announced then postponed [61].
CircleCI self-hosted limits and egress [26]:
- Self-hosted concurrency: 5 concurrent tasks on Free, 20 on Performance, Unlimited on Scale [26].
- Restoring caches and workspaces to self-hosted runners can incur egress at 420 credits per GB if usage exceeds the allowance [26].
- Even CircleCI self-hosted runners are limited by plan tier [13].
Fixed compute fleets can incur unnecessary costs when workload fluctuates [12]. CircleCI docs warn maintaining a fixed compute fleet of self-hosted runners can incur unnecessary costs as the workload can fluctuate depending on the rate at which jobs are queued [12]. The same docs state the compute fleet can be scaled according to demand to help reduce this cost [12]. OpenMetal reports CI load spikes around business hours and release windows and drops off overnight and on weekends, and self-hosted runner fleets that do not scale down during quiet periods end up paying for idle capacity [65]. Taken together, [12] and [65] indicate scaling down during quiet periods helps reduce idle-capacity cost.
Recycling by runner type [12]:
- Container runner on Kubernetes automatically spins up more pods as work enters the queue [12].
- Those pods are ephemeral and torn down after the job [12].
- Continuous machine runner polls for new tasks, requiring scaling to monitor workload and shut down unused runners [12].
- Setting a reasonable idle timeout enables automatic recycling during lower demand [12].
- For machine runners, add more runners to the resource class with more pending work [12].
Nomad workers scale up when jobs are queued and scale down when idle, controlled by the Nomad Autoscaler [64]. Cost factors include instance type configured in the Terraform module, number of concurrent jobs, job duration, and autoscaler configuration (min, max, cooldown) [64]. If the Nomad Autoscaler fails to scale down, for example due to nodes stuck in draining state, compute costs can increase, with monitoring of autoscaler logs and ASG/MIG instance counts stated as the response [64].
KEDA's github-runner scaler scales GitHub runners based on the number of queued jobs in GitHub Actions [66]. Its runnerScope can be org, ent or repo [66]. By default it scales on default labels (self-hosted, linux, x64) in addition to the ones defined in the labels parameter, and scaling on default labels can be disabled by setting noDefaultLabels to true [66]. OpenMetal lists GitHub's Actions Runner Controller for Kubernetes or simpler custom autoscaling against own infrastructure as handling this on the self-hosted side [65].
Self-hosting ownership burden [16][65]:
- Does not fit well without capacity to own fleet management, autoscaling and security isolation [65].
- Requires package, OS and runner updates plus hardware-failure handling [16].
- Requires provisioning servers, installing and configuring the runner, plus autoscaling, monitoring and possibly proxy setup [16].
- Can be more costly with engineer and IT/DevOps costs [16].
- Small teams with possibly no dedicated DevOps engineer should consider GitHub-hosted to simplify work [16].
Two options trim self-hosted cost [16]:
- Spot instances, because runners sit idle most of the time [16].
- Managed container services such as ECS/EKS, optionally with Fargate, to minimize maintenance and possibly reduce costs [16].
No savings amount is promised.
OpenMetal states that separating boot and data volumes, or running runners in isolated environments on the same host, keeps a bad build contained to itself, and that MicroVMs are a common way teams solve this for untrusted or semi-trusted code [65]. Arnica notes self-hosted runner servers could be only accessible via a VPN or even only local area for in-office usage as a way to implement the least privilege principle when it comes to accessing source code [16].
Visibility differs by hosting. Arnica states self-hosted runners allow SSH access for troubleshooting, versus more limited Actions-panel logs on hosted runners [16]. Debug access is better self-hosted.
Claimed hosted limits vs conflicting pricing entries:
- Arnica stated hosted runners run only on X86_64 with no ARM compatibility [16], are limited to 2-core CPUs [16], and are limited to 7 GB RAM on Windows and Linux [16].
- Arnica advised self-hosting for long multithreaded builds, specific Linux distros or old Windows [16] and legacy apps that cannot run on latest OS [16].
- The same source noted announced up-to-64-core and up-to-256 GB machines were still in beta and not generally available [16].
- GitHub pricing lists a Linux 2-core (arm64) runner [62] and larger macOS 12-core and 5-core (M2 Pro) runners [62].
- Taken together, [16] and [62] indicate those Arnica limits conflict with the GitHub entries for ARM Linux and larger macOS runners.
4. Discussion
Taken together, [23] and [55] indicate that workflow execution time improves by caching dependencies and build outputs [23], and by cutting the wall-clock time of a large test suite by splitting it across jobs into balanced shards based on historical timing data [55]. Sharding trades additional compute resources for faster results, with overall test time shrinking to about 1/N of the original plus overhead [60], while increasing total computing time which in cloud CI services can mean higher usage of minutes or CPUs, i.e. potentially higher cost [60]. Datadog defines queue time as time for which a pipeline or job was in the queue before execution, approval wait time as time for which a pipeline or job has been waiting for a manual approval, and execution time as time for which a pipeline has been actively running jobs [1], while step-level breakdowns help identify common bottlenecks like dependency installation, test execution, or cache misses [2] alongside a Top 5 Slowest Jobs list of the most time-consuming jobs across all repositories and workflows [2].
Taken together, [21] and [20] indicate reuse collides with correctness. Gradle fingerprints the task's declared inputs, outputs, and relevant implementation details and determines whether the task is already up to date locally, or whether matching outputs can be loaded from the Build Cache [15], yet missing task inputs can cause incorrect cache hits where different results are treated as identical because the same cache key is used by both executions, and wrongly declared task inputs can lead to cache misses especially when containing volatile data or absolute paths [20]. Caching node_modules directly is described as unreliable where partial restores can leave orphaned packages that affect runtime behaviour [25], while the ~/.npm download cache avoids these problems because npm resolves and installs the correct versions from your lock file while using the cached downloads to skip the network fetch [25]. Cache contents are not signed or verified and extracted caches may modify files that are subsequently executed in a workflow run, leading to malicious code execution [21]; only push, workflow_dispatch, repository_dispatch, delete, registry_package, page_build and schedule triggers can create or overwrite caches in the default branch's scope [21]; and GitHub will remove any cache entries that have not been accessed in over 7 days, with a default limit of 10 GB per repository [21]. Type=gha stores layers in the same Actions cache as actions/cache with the same branch scope rules and the same quota, so a large image with mode=max can crowd out your dependency caches [33]. Caching node_modules, Docker layers, and build artifacts can cut build times (and therefore costs) by 50-70% [29]. Large caches can take longer to restore, reducing the performance gain [35].
Skipping work risks undetected failures: many companies hesitate to adopt build skipping and test case prioritization due to concerns about undetected failures and favor strategies that provide clear explanations and traceability [46]. GitLab docs specify adding a job to a pipeline by checking for changes to specific files [11]. CircleCI dynamic configuration conditionally runs workflows using the when key [39], with the service-1 workflow running only if a change was made in the service-1 directory in the commit, as determined based on the path filtering in the setup phase [39], using a fallback configuration if no mapping matches the modified-file paths [39] and including shared configuration for changes to any file [39]. GitHub Docs states that a workflow skipped by path filtering, branch filtering, or a commit message leaves associated checks in a Pending state and blocks merging [38], and advises avoiding requiring workflows that can be skipped [38]. Develocity docs state that builds with Predictive Test Selection enabled select only those tests predicted to provide useful feedback on the changes under test [31] and that this trades testing comprehensiveness for faster feedback, with additional executions covering remaining tests or disabling selection that should occur later, for example as part of post-merge, nightly or ready-for-release builds [31]. Those docs also state that tests will always be chosen if they're recently new, recently changed, recently failed, or recently flaky [31], and that it's possible to configure the plugin to always select tests for execution based on custom criteria by specifying a class name filter pattern, via the includeClasses, or in an annotation-driven manner, via the includeAnnotationClasses [31]. A shared-library change requires rebuilding and testing every service that depends on it, not just services with direct file changes [43], which needs a pipeline generator script that resolves transitive dependencies [43]; similarly, a change in a shared library should also rebuild everything that depends on it, computed from manifests instead of hand-written filters [49], and dependency-graph selection runs only commands for changed modules or modules depending on a changed module [49]. The industrial study reports skipping techniques are used with caution unless they provide trustworthy and explainable justifications [46], hesitation about skipped builds or test cases in the early pre-merge phase with a preference for exposing failures earlier rather than later [46], and that resolving issues after the merge can be significantly more time-consuming and complex, potentially outweighing the benefits of deferred execution [46].
Parallelism involves cost and balance. Playwright sharding splits tests into smaller parts called "shards" where each shard runs independently to speed up total test runtime [52], a 1:51 serial run became a 59-second run by adding a second machine [8], and one report found that running 8 shards in parallel reduced total test time from 19 minutes down to under 3 minutes [60]. The general parallel-jobs resource trade-off is covered elsewhere in this section; ideally the number of shards matches the number of parallel runners because more shards than available executors won't speed up the overall time and would just queue up [60]. Test groups will often run for different times, making the full test run as slow as the slowest group [54], and a shard with disproportionately slow tests becomes the bottleneck that determines when the suite finishes [55]. By default, the division is evenly by number of test files, not based on test duration [60], a single extremely slow test file could make its shard slower than the others [60], and without the fullyParallel setting uneven test-file sizes may leave certain shards running significantly more tests while others run fewer or even none [52]. Overhead and diminishing returns are covered elsewhere in this section. Depot's history-based shard balancing is covered elsewhere in this section, while Buildkite Test Engine uses historic test timing data as input to a bin packing algorithm that distributes the work so all parallel workers finish in the same amount of time [59]. [62] lists "Linux 2-core (x64) | actions_linux | $0.006" [62] and "macOS 3-core or 4-core (M1 or Intel) | actions_macos | $0.062" [62], while one analysis notes a 16-core runner costs 8x as much per minute but only finishes 3.5x faster [61].
The steelman for early self-hosting on a fixed fleet runs: CircleCI does not charge per build minute on self-hosted runners [26], and self-hosted runners allow SSH login to the machine to troubleshoot, compared with more limited logs in the Actions panel for GitHub-hosted runners [16]. Maintaining a fixed compute fleet of self-hosted runners can incur unnecessary costs as the workload can fluctuate depending on the rate at which jobs are queued [12], while, if you use the container runner on Kubernetes, it will automatically spin up more pods as more work enters the queue [12] and those pods are ephemeral and will be torn down after the job is done executing [12]. Provisioning and maintenance burdens, and the small-team implication, are detailed elsewhere in this Discussion. Self-hosting doesn't fit well if the team doesn't have the capacity to own runner fleet management, autoscaling, and security isolation [65]. GitHub cut its own hosted-runner rates by up to 39%, with current 2-core rates, listed on GitHub's own Actions runner pricing reference, running $0.006 per minute for Linux, $0.010 for Windows, and $0.062 for macOS [65]. The utilization break-even analysis is detailed elsewhere in this Discussion. Self-hosting is described as better off for code bases that take a long time to build and involve very multithreaded processes, or that need a specific Linux distribution or an old Windows version [16], for legacy applications that cannot run on the latest OS [16], and for runner servers only accessible via a VPN or local area as a way to implement least privilege for accessing source code [16]. Self-hosting doesn't fit well for low-volume pipelines with CI usage well under the free tier or comfortably inside a small monthly bill [65].
Limits temper confidence. Proper dependency caching in your CI system (e.g., using restore_cache and save_cache in CircleCI, or GitHub Actions' cache action) can reduce build times by 50–90% depending on the project [36], while restoring the local build cache in one experiment reduced build time from 32.3s to 28.9s, saving 3.4s or about 11% by reusing cacheable task outputs instead of recomputing them [15]; actual gains vary with project structure, dependencies, network, and CI environment, and teams are advised to run their own benchmarks before adopting optimizations [15]. Larger runners, where available, when the billing account uses GitHub Team or GitHub Enterprise Cloud, their cost can draw from the included compute usage for billing accounts on GitHub Team and GitHub Enterprise Cloud [63], larger runners are billed for public repositories [63], and larger runners require a valid payment method, with usage blocked once the included compute usage is consumed if no valid method is on file [63]. One source notes "Is limited to only 2 core CPUs" [16] and "Can only run on X86_64 CPU (no ARM compatibility)" [16] and states that Github announced larger action runner machines, up to 64-cores for Windows and Linux and up to 256 GB of RAM, while this was still in beta and not yet available to the general public [16]. Predictive Test Selection won't attempt to make predictions for test tasks or goals for which fewer than 14 days of code and dependency changes are available, in which case all tests except those having previously passed for identical inputs are selected [31]; tests will always be chosen if they're recently new, recently changed, recently failed, or recently flaky [31]. Enabling flaky test retry functionality is encouraged as it allows test selection predictions to better account for non-deterministic flaky test outcomes [31], and the client will collect any failures from the initial run and retry these tests individually in the same job, up to the number of retries specified [59]. Tracking additional input files by absolute paths convolutes the change history and negatively impacts its predictions [31].
5. Conclusion
Caching dependencies and build outputs improves workflow execution time [23], where an exact match to the provided key is a cache hit and a miss automatically creates a new cache if the job completes successfully [21], and a key using the hash of package-lock.json changes the key and automatically creates a new cache when those dependencies change [21]. The path-filtering orb maps file paths with pipeline parameter values and sets the parameter when a file change matches a path regex [39], while rules:changes specifies when to add a job to a pipeline by checking for changes to specific files [11]. Sharding splits tests into smaller parts called shards where each shard can run independently to speed up test runtime [52], with each shard given a different, non-overlapping subset balanced by how long each candidate took in past runs [55], where overall test time can shrink to about 1/N of the original plus overhead by trading additional compute for faster results [60] while increasing total computing time and potentially higher minutes or CPU use [60]. Taken together, [65] and [16] indicate staying on a hosted solution for a small team of developers with possibly no dedicated DevOps engineer where slots average below about 30% utilization across 12 parallel slots in a 720-hour month, since hosted per-minute pricing is competitive below that level and self-hosted runners require more management and maintenance with engineer and IT and DevOps specialist cost [65][16].
The trade-offs sort by workload.
| Reader scenario | Recommended choice | Deciding factor |
|---|---|---|
| Repeat installs of dependencies [21][25] | Taken together, [21] and [25] indicate caching download caches keyed by lockfile hash with restore-key fallback | An exact match to the provided key is a cache hit and on a cache miss a new cache is automatically created if the job completes successfully [21], a key using hash of package-lock.json changes and a new cache is automatically created when dependencies that make up package-lock.json change [21], restore-keys allows a list of alternate restore keys to use when there is a cache miss on key [21], and partial restoration of the pip download cache is safe because pip resolves and installs correct versions from dependency file regardless of wheels present [25] and npm resolves and installs correct versions from lock file using cached downloads to skip the network fetch [25] |
| Monorepo with microservices stored in a monorepo [39] | Taken together, [39] and [11] indicate generating or including only changed-path work | The path-filtering orb maps file paths with pipeline parameter values and if a file change matches a path regex the corresponding parameter is set [39], dynamically deciding work based on location of changes is beneficial when code or microservices are stored in a monorepo [39], rules:changes specifies when to add a job to a pipeline by checking for changes to specific files [11], rules can be used with include to conditionally include other configuration files [11], and rules can skip unneeded tests such as skipping backend tests when only frontend code is changed [4] |
| Large test suite split across jobs [52][55] | Taken together, [52] and [55] indicate sharding across separate parallel jobs with history-based balancing | Sharding splits tests into smaller parts called shards where each shard is like a separate job that can run independently to divide tests to speed up test runtime [52], in a CI pipeline each shard can run as a separate job [52], Depot gives each shard a different, non-overlapping subset and uses the time each candidate took in past runs to balance how long shards take to run [55], and overall test time can shrink to about 1/N of the original plus some overhead trading additional compute resources for faster results [60] |
| Small team of developers with possibly no dedicated DevOps engineer and low-volume pipeline [16][65] | Taken together, [16] and [65] indicate staying on hosted runners while utilization remains below break-even and ownership capacity is lacking | If working in a small team of developers with possibly no dedicated DevOps engineer, a GitHub-hosted solution should be considered as it simplifies work [16], and spread across 12 parallel slots in a 720-hour month that break-even works out to those slots averaging about 30% utilization [65], where below that GitHub's per-minute pricing is genuinely competitive and self-hosting isn't worth the operational overhead [65] |
GitHub actions/cache allows caching dependencies and build outputs to improve workflow execution time [23], jobs using the same GitLab cache avoid re-downloading files and execute more quickly [24], and the Gradle build cache stores build outputs locally or remotely and fetches them when inputs have not changed, avoiding regenerating them [20]. One guide reports proper dependency caching can reduce build times by 50–90% depending on the project [36], another reports caching node_modules, Docker layers and build artifacts can cut build times by 50-70% [29], and a Gradle experiment reduced build time from 32.3s to 28.9s [15]. Actual gains vary with project structure, dependencies, network and CI environment, and teams are advised to run their own benchmarks before adopting optimizations [15]. Overly specific keys may result in missed cache hits [35]. Cache contents are not signed or verified and extracted caches may modify files subsequently executed in a workflow run, leading to malicious code execution [21], while anyone who can open a pull request, including from a fork, can read the base branch's caches [33].
Selection and sharding also carry conditions. GitLab Docs state rules:changes always evaluates to true and the job always runs for new branch pipelines or when there is no Git push event [11], and GitHub Docs state a workflow skipped by path filtering, branch filtering, or a commit message leaves associated checks in Pending and blocks merging [38]. Playwright can only shard tests that can be run in parallel [52], sharded jobs must check out the same commit with identical environment setup because sharding simply partitions tests without synchronization between shards [60], and tests failing only when sharded might indicate an order dependency that needs fixing [60]. Gradle assumes isolated tests and warns that shared file systems, databases or external services can cause intermittent failures from race conditions or resource conflicts [41], and Gradle advises considering disabling Parallel Execution if such problems are observed to preserve reliability of build results [41]. A shard with disproportionately slow tests becomes the bottleneck that determines when the suite finishes [55], each shard carries job-startup and dependency-install overhead with a point of diminishing returns [60], and Buildkite finds the relationship between worker count and execution speed is not linear with decreasing added benefit per extra worker [59]. Recording runs to Cypress Cloud provides test duration trends and machine utilization data showing whether parallel machines are balanced or one is a bottleneck [8].
On self-hosting, CircleCI does not charge per build minute on self-hosted runners [26], self-hosted runner usage remains free independently and does not reduce the included compute usage [63], operators can log in via SSH to the machine and troubleshoot, compared with some visibility via logs in the Actions panel for GitHub-hosted runners [16], self-hosted runner servers could be only accessible via a VPN or even only local area for in-office usage as a way to implement the least privilege principle when it comes to accessing source code [16], and self-hosting fits well if you want CI infrastructure costs that don't move when GitHub revisits its pricing model, whatever direction that takes next [65]. The trade-off is ongoing ownership: teams will have to maintain and manage self-hosted runners, including third-party package updates, OS updates, runner application updates and handling of hardware failures [16], including spending time provisioning and setting up servers and installing and configuring the runner application including desired behavior for autoscaling, monitoring and possibly establishing a proxy server [16], since self-hosted runners require more management and maintenance and the total cost can be higher since you need to pay engineers and IT and DevOps specialists [16]; self-hosting doesn't fit well if a team doesn't have the capacity to own runner fleet management, autoscaling, and security isolation, since those become its responsibility [65], self-hosted runner fleets that don't scale down during quiet periods end up paying for idle capacity because CI load spikes around business hours and release windows and drops off overnight and on weekends [65], and separating boot and data volumes, or running runners in isolated environments on the same host, keeps a bad build contained to itself, with MicroVMs a common way teams solve this for untrusted or semi-trusted code [65]. Taken together, [16] and [65] indicate a small team of developers with possibly no dedicated DevOps engineer should consider a GitHub-hosted solution as it simplifies a lot of the work [16], where spread across 12 parallel slots in a 720-hour month that 2,580-hour break-even point works out to those slots averaging about 30% utilization and below that GitHub's per-minute pricing is genuinely competitive and self-hosting isn't worth the operational overhead [65].
Develocity frames the central tension: predictive selection trades comprehensiveness for faster feedback and pairs it with later full runs post-merge, nightly or before release [31].
Currency equivalents are approximate, using reference rates dated 4 November 2026. They do not adjust for local prices, taxes, or purchasing power.
6. Hypotheses and possible implications
Unverified assumptions. These hypotheses combine the cited evidence; they are not established findings. Citations support the premises only.
No defensible new hypotheses were identified.
References
[1] CI Pipeline Visibility in Datadog — https://docs.datadoghq.com/continuous_integration/pipelines/ · general
[2] GitHub Actions metrics and analytics in Depot — https://depot.dev/docs/github-actions/observability/github-actions-metrics · general
[3] Defining your pipeline steps — https://buildkite.com/docs/pipelines/configure/defining-steps · general
[4] Pipeline efficiency — https://docs.gitlab.com/ci/pipelines/pipeline_efficiency/ · general
[5] CI/CD insights | Software Engineering Insights | Harness — https://developer.harness.io/software-engineering-insights/use-software-engineering-insights/analytics-and-reporting/efficiency/cicd-insights · general
[6] How we used parallel CI/CD jobs to increase our productivity — https://about.gitlab.com/blog/using-run-parallel-jobs/ · general
[7] Monitoring build performance trends using Develocity — https://develocity.ai/blog/build-performance-trends/ · general
[8] Optimizing test performance | Cypress Documentation — https://docs.cypress.io/app/guides/test-performance · general
[9] Introducing Dagger Traces: Debug and Monitor your Dagger Functions | Dagger Blog — https://dagger.io/blog/introducing-dagger-traces/ · general
[10] Queue metrics in clusters — https://buildkite.com/docs/pipelines/insights/queue-metrics · general
[11] CI/CD YAML syntax reference — https://docs.gitlab.com/ci/yaml/ · general
[12] Scaling self-hosted runners — https://circleci.com/docs/runner/runner-scaling/ · general
[13] Advantages of migrating from CircleCI — https://buildkite.com/docs/pipelines/advantages/buildkite-vs-circleci · general
[14] Drive CI performance with actionable insights - CircleCI — https://circleci.com/product/analytics-insights/ · general
[15] Essential Tips for Gradle on Ephemeral CI Environments - Part 3 — https://blog.gradle.org/gradle-on-ephemeral-ci-3 · general
[16] Choosing GitHub Runners vs Self-Hosted Runners — https://www.arnica.io/blog/github-hosted-or-self-hosted-runners · general
[17] Buildkite Test Engine | Remove flaky tests, optimize performance — https://buildkite.com/platform/test-engine/ · general
[18] How to measure developer experience (DevEx) in the AI era — https://www.datadoghq.com/blog/how-to-measure-developer-experience-in-the-ai-era/ · general
[19] Migration services — https://buildkite.com/resources/migrations/ · general
[20] Build Cache — https://docs.gradle.org/current/userguide/build_cache.html · general
[21] Dependency caching reference - GitHub Docs — https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching · general
[22] Building best practices — https://docs.docker.com/build/building/best-practices/ · general
[23] GitHub - actions/cache: Cache dependencies and build outputs in GitHub Actions — https://github.com/actions/cache · general
[24] Caching in GitLab CI/CD — https://docs.gitlab.com/ci/caching/ · general
[25] Caching dependencies — https://circleci.com/docs/guides/optimize/caching/ · general
[26] Pricing and Plan Information - CircleCI — https://circleci.com/pricing/ · general
[27] Cache - GitHub Marketplace — https://github.com/marketplace/actions/cache · general
[28] How to Cache Docker Images in GitHub Actions · Dash0 — https://www.dash0.com/faq/cache-docker-images-github-actions · general
[29] Best CI/CD Pipeline for Small Teams: A Practical 2026 Guide — https://dev.to/_d7eb1c1703182e3ce1782/best-cicd-pipeline-for-small-teams-a-practical-2026-guide-2ad0 · general
[30] Configuring turbo.json — https://turborepo.dev/docs/reference/configuration · general
[31] Develocity Predictive Test Selection Guide | Develocity Documentation 2026.3 — https://docs.develocity.ai/2026.3/guides/predictive-test-selection/ · general
[32] Cache management with GitHub Actions — https://docs.docker.com/build/ci/github-actions/cache/ · general
[33] GitHub Actions caching strategy: dependencies, builds and Docker — https://tenki.cloud/blog/github-actions-cache-strategy · general
[34] Remote Caching - Bazel Documentation — https://bazel.build/remote/caching · general
[35] Using GitHub Actions Cache with popular languages — https://www.warpbuild.com/blog/github-actions-cache · general
[36] How to reduce CI/CD costs: Complete optimization checklist — https://depot.dev/blog/how-to-reduce-cicd-costs-complete-optimization-guide · general
[37] What’s Slowing Down Your CI/CD Pipeline? 5 Fixes from Real DevOps Teams — https://deployflow.co/blog/whats-slowing-down-your-ci-cd-pipeline-5-fixes/ · general
[38] Troubleshooting required status checks - GitHub Docs — https://docs.github.com/en/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks · general
[39] Using dynamic configuration — https://circleci.com/docs/guides/orchestrate/using-dynamic-configuration/ · general
[40] Downstream pipelines — https://docs.gitlab.com/ci/pipelines/downstream_pipelines/ · general
[41] Improve the Performance of Gradle Builds — https://docs.gradle.org/current/userguide/performance.html · general
[42] Incremental Actions | Buck2 — https://buck2.build/docs/rule_authors/incremental_actions/ · general
[43] Working with monorepos — https://buildkite.com/docs/pipelines/best-practices/working-with-monorepos · general
[44] Jest and the --changedSince flag in GitHub Actions CI — https://dev.to/bnb/jest-and-the-changedsince-flag-in-github-actions-ci-468i · general
[45] How to efficiently skip unnecessary jobs in a monorepo GitHub Actions workflow? · community · Discussion #177835 — https://github.com/orgs/community/discussions/177835 · general
[46] ”Good” and ”Bad” Failures in Industrial CI/CD – Balancing Cost and Quality Assurance — https://arxiv.org/html/2504.11839 · academic
[47] A guide to Bazel query — https://buildkite.com/resources/blog/a-guide-to-bazel-query/ · general
[48] Nx vs Turborepo — https://nx.dev/docs/kb/nx-vs-turborepo · general
[49] Allow required checks to pass/skip, not fail, when using path filtering · community · Discussion #44490 — https://github.com/orgs/community/discussions/44490 · general
[50] Bitbucket pipelines - Condition: Run or skip step based on build output - Atlassian — https://community.atlassian.com/forums/discussion/1941662/bitbucket-pipelines-condition-run-or-skip-step-based-on-build-output · general
[51] Test encyclopedia - Bazel Documentation — https://bazel.build/reference/test-encyclopedia · general
[52] Sharding | Playwright — https://playwright.dev/docs/test-sharding · general
[53] Running variations of jobs in a workflow - GitHub Docs — https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations · general
[54] GitHub - grosser/parallel_tests: Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber — https://github.com/grosser/parallel_tests · general
[55] docs/content/ci/how-to-guides/split-tests.mdx at main · depot/docs — https://github.com/depot/docs/blob/main/content/ci/how-to-guides/split-tests.mdx · general
[56] Parallel builds — https://buildkite.com/docs/pipelines/best-practices/parallel-builds · general
[57] Test splitting and parallelism — https://circleci.com/docs/guides/optimize/parallelism-faster-jobs/ · general
[58] [Feature] Split shards via test timing data — https://github.com/microsoft/playwright/issues/17969 · general
[59] Beyond basic test splitting: Buildkite’s approach to test suite parallelization — https://buildkite.com/resources/blog/beyond-basic-test-splitting-buildkite-s-approach-to-test-suite-parallelization/ · general
[60] Speed Up Jest Test Suites with Sharding — https://learn.michaelguay.dev/blog/speed-up-jest-test-suites-with-sharding · general
[61] CI/CD Calculator 2026: Buildkite, GitHub Actions, GitLab, CircleCI — https://cicdcalculator.com/ · general
[62] Actions runner pricing - GitHub Docs — https://docs.github.com/en/billing/reference/actions-runner-pricing · general
[63] GitHub Actions billing - GitHub Docs — https://docs.github.com/en/billing/concepts/product-billing/github-actions · general
[64] Resource consumption and cost management — https://circleci.com/docs/server-admin/latest/operator/resource-consumption-cost-management/ · general
[65] The Real Cost Math Behind Self-Hosted GitHub Actions Runners — https://openmetal.io/resources/blog/the-real-cost-math-behind-self-hosted-github-actions-runners/ · general
[66] KEDA | Github Runner Scaler — https://keda.sh/docs/2.21/scalers/github-runner/ · general
Source quality: 1 academic, 65 general.