Key Takeaways
An exact match to the provided key is considered a cache hit [1], and Test Impact Analysis selects and runs only the relevant tests required to validate a commit [5].
- GitHub Docs state that an exact match to the provided key is considered a cache hit [1]; an example key includes the runner's operating system and a SHA-256 hash of the package-lock.json file, with a new cache when packages or OS change [1]; GitHub will remove any cache entries that have not been accessed in over 7 days [1]; CircleCI uses timing data from the previous test run to split a suite as evenly as possible, with no timing data on the first run and optimized time on subsequent runs [7]; Azure Pipelines includes existing impacted tests, previously failing tests, and newly added tests [5], and for commits and scenarios that TIA can't understand, it falls back to running all tests [5]; a workflow with the path filter will only run on push events that include at least one file outside the docs directory at the root of the repository [8].
- When you use a GitHub-hosted runner, machine maintenance and upgrades are taken care of for you [9]; current 2-core rates, listed on GitHub's own Actions runner pricing reference, run USD 0.006 per minute for Linux (GBP equivalent unavailable), all cheaper than they were in 2025 [6], against OpenMetal's current Medium v4 tier with 24 physical cores priced at USD 928.80 per month for the Ashburn configuration (GBP equivalent unavailable) [6], which works out to roughly 154,800 minutes, or about 2,580 hours [6], hosted as about 12 concurrent 2-core-equivalent job slots [6] averaging about 30% utilization across the month in a 720-hour month [6].
- In Playwright, greater parallelisation is achieved by running tests on multiple machines simultaneously, called sharding [10], with each shard able to run as a separate job to run tests faster [10], yet parallelism speeds up CI but introduces new flakiness vectors [11]; some teams report 20-30% of CI compute spent on retries alone [11].
- Limits narrow the promise: in order to use the test splitting feature, CircleCI requires test results to be uploaded as JUnit XML reports [7]; TIA is currently scoped to only managed code, and single machine topology [5]; cost math uses OpenMetal's current Medium v4 tier priced at USD 928.80 per month for the Ashburn configuration (GBP equivalent unavailable) and the current 2-core Linux rate of USD 0.006 per minute (GBP equivalent unavailable), cheaper than in 2025 [6].
[!WARNING]
GitHub Docs recommend that you don't store any sensitive information, such as access tokens or login credentials, in files in the cache path, noting that anyone with read access can create a pull request on a repository and access the contents of a cache [1]. Cache contents are not signed or verified, and any workflow run that can read a cache may extract its contents; extracted caches may modify files that are subsequently executed in a workflow run, leading to malicious code execution [1].
Abstract
Dependencies can be cached with a key that includes the runner's operating system and a SHA-256 hash of the package-lock.json file, creating a new cache when the packages in the package-lock.json file change, or when the runner's operating system changes [1], timing data from the previous test run is used to split a test suite as evenly as possible over a specified number of test environments [7], TIA selects and runs only the relevant tests required to validate a commit entering the CI/CD pipeline [5] and push and pull_request workflows can be configured to run based on what file paths are changed [8], where a workflow with the path filter will only run on push events that include at least one file outside the docs directory at the root of the repository [8], while the 2,580-hour break-even point works out to those slots averaging about 30% utilization spread across 12 parallel slots in a 720-hour month [6]. Below that, GitHub's per-minute pricing is genuinely competitive [6]. An exact key match is considered a cache hit and on a cache miss the action automatically creates a new cache if the job completes successfully [1]; use of the test splitting feature requires test results to be uploaded as JUnit XML reports [7] and the first time the tests are run there will be no timing data for the command to use [7]; TIA includes existing impacted tests, previously failing tests, and newly added tests [5] yet if the code commit contains changes to HTML or CSS files, it can't reason about them and falls back to running all tests [5]. With the exception of single-CPU runners, each GitHub-hosted runner is a new virtual machine hosted by GitHub [9] and when you use a GitHub-hosted runner, machine maintenance and upgrades are taken care of for you [9], while single-CPU runners are hosted in a container on a shared VM [9], while a workflow skipped due to path filtering leaves checks in a "Pending" state and a pull request that requires those checks to be successful will be blocked from merging [8]. Dividing the $928.80 per month for the Ashburn configuration for a dedicated server with 24 physical cores by the $0.006 per-minute hosted rate works out to roughly 154,800 minutes, or about 2,580 hours [6]. TIA is currently scoped to only managed code, and single machine topology [5].
Table of Contents
- Key Takeaways
- Abstract
- 1. Introduction
- 2. Background
- 3. Findings
- 3.1 Caching Dependencies and Build Artifacts
- 3.2 Parallelism and Sharding Test Suites
- 3.3 Slimming Test Suites and Handling Flakiness
- 3.4 Incremental Builds and Build Caching
- 3.5 Runner Hardware: Hosted versus Self-Hosted Trade-offs
- 3.6 Pipeline Structure, Fail-Fast and Merge Queues
- 3.7 Measuring and Profiling CI Time
- 4. Discussion
- 5. Conclusion
- 6. Hypotheses and possible implications
- References
1. Introduction
Continuous integration relies on frequent integrations verified with an automated build that runs regression tests to detect integration errors as soon as possible, according to Azure Pipelines documentation [5]. That documentation also reports full regression may require hours as code bases grow and mature [5]. Deferring longer-running tests to a separate stage to finish CI quickly defeats continuous integration, the same documentation cautions [5]. An OpenMetal cost analysis describes bills as thousands of small per-minute charges across every pull request [6]. Failed builds block deployment queues while other commits pile up, one analysis of flaky-test practice reports [11]. Taken together, [5][6] and [11] indicate delays compound. This report asks what practical approaches a small software team can use to reduce CI build times, and what the trade-offs involve.
Gradle documentation describes sharing and reusing unchanged build and test outputs to avoid rebuilding binaries unaffected by new code changes [4]. CircleCI documentation reports test splitting shortens testing time in pipelines [7]. Playwright documentation describes sharding as splitting tests into independently runnable parts to speed runtime [10]. Docker building guidance calls understanding build cache operation and invalidation critical for ensuring faster builds [2]. Taken together, [1] and [11] indicate speed carries costs. GitHub Docs warns nothing signs or verifies cache contents and extracted caches may enable malicious code execution in later runs [1]. One analysis reports parallelism speeds CI but introduces new flakiness vectors [11].
Scope stays narrow. The report covers dependency and build caching, Docker layer discipline, selective test execution and parallel execution, and runner economics for a team running hosted CI at modest volume. It excludes application rewrites, CI platform migrations, and large-scale self-hosted fleet engineering. Findings assess options only on build time, direct cost, and maintenance burden for that operating point.
Background defines the pipeline and pinpoints where small teams lose time. Findings detail each approach, its mechanics, and its limits. Discussion weighs combinations, failure modes, and mitigations. Conclusion distills operating guidance for immediate use. Read sequentially.
2. Background
See the CI pressures discussed elsewhere in this Background section [5][11]. Start with that pressure.
One report describes the build cache as storing previous build results and restoring them when needed. [4] The same source suggests sharing unchanged build and test outputs avoids rebuilding binaries unaffected by code changes in local and CI builds. [4] It also describes incremental builds as avoiding rebuild of other project files when a developer repeatedly changes a single file. [4] One Docker guide stresses understanding how the build cache works and how invalidation occurs as critical for faster builds. [2] Hold that idea.
The caching and Docker image-build details introduced here are covered in full elsewhere in this report, so this background keeps only the link to that coverage.
One GitHub account notes teams can use GitHub-provided runners or host their own. [9] The same source reports Linux and Windows runners run on Azure virtual machines, while macOS runners run in Azure data centers. [9] One workflow reference states jobs run in parallel by default, use needs to run sequentially, and GitHub maximizes parallel jobs depending on runner availability. [8] Keep parallelism in view.
One CircleCI account describes splitting a suite into buckets for parallel runs on configured environments. [7] One Playwright account defines sharding as splitting tests into independently runnable shards to speed runtime, with each shard running as a separate job using available hardware such as CPU cores. [10] One Azure Pipelines account reports Test Impact Analysis selects only relevant tests for a commit and falls back to all tests for commits it cannot understand. [5] One analysis reports parallelism speeds up CI but introduces new flakiness vectors. [11] That sets the baseline.
3. Findings
3.1 Caching Dependencies and Build Artifacts
Exact-key hits decide whether dependencies rebuild at all. An exact match to key restores files as a cache hit, while a miss automatically creates a new cache if the job completes successfully, according to GitHub's dependency-caching documentation [1]. Existing caches stay immutable. Contents cannot be changed and a new cache needs a new key [1]. Exactness is observable. The action outputs cache-hit as true on exact match, which can gate later steps [1].
Keys and paths misconfigured fail fast. key is required, can combine variables, context values, static strings and functions, and fails the action when longer than 512 characters, according to GitHub's dependency-caching documentation [1]. path is required too. It accepts a single path or multiple paths on separate lines, single files or directories, glob patterns, and absolute or workspace-relative paths [1].
Fallback lookup rescues near-misses in fixed order [1]. Optional restore-keys supply alternate keys one per line, searched sequentially when key misses, according to GitHub's dependency-caching documentation [1]. If there are no exact matches, the action searches for partial matches of the restore keys and restores the most recently created matching cache to the path directory [1]. The authoritative order is exact key, then partial matches of key, then sequential partial matches of restore-keys [1].
A cache key that includes the runner's operating system and a SHA-256 hash of the package-lock.json file creates a new cache when the packages in the package-lock.json file change, or when the runner's operating system changes, according to GitHub's documentation [1].
Low-configuration caching covers listed package managers. Using their respective setup-* actions requires minimal configuration and will create and restore dependency caches, according to GitHub's documentation [1].
Caches expire quickly. GitHub removes entries not accessed in over 7 days, and once the repository limit is reached it evicts by last-access date from oldest to most recent, according to GitHub's documentation [1]. Eviction is age-ordered [1].
Caches are a security boundary, not a vault. Sensitive information such as access tokens or login credentials should not be stored in cached path files because anyone with read access can create a pull request and access cache contents, according to GitHub's documentation [1]. Execution risk follows. Cache contents are neither signed nor verified, and extracted caches may modify subsequently executed files to cause malicious code execution [1].
Understanding how the build cache works, and how cache invalidation occurs, is critical for ensuring faster builds, according to Docker's best-practices documentation [2]. --no-cache disables the build cache, forcing Docker to rebuild all layers from scratch to get the latest available versions of dependencies from package managers like apt-get or npm [2]. It does not pull a fresh base image — for that, the documentation points to --pull [2]. That flag forces Docker to check for and download a newer version of the base image, even if you have a version cached locally [2].
Lean images build faster and fail less. Avoiding extra or unnecessary packages reduces complexity, dependencies, file sizes and build times, according to Docker's best-practices documentation [2]. RUN apt-get update must share a RUN statement with apt-get install; apt-get update alone causes caching issues and subsequent install failure [2]. Version pinning forces retrieval of a particular version regardless of cache and can reduce failures from unanticipated package changes [2]. Cleaning helps too. Removing /var/lib/apt/lists reduces image size, with Docker noting the apt cache is not stored in a layer [2].
Single-concern containers reuse better. Each container should have only one concern, and decoupling applications into multiple containers eases horizontal scaling and reuse, according to Docker's best-practices documentation [2]. Bind mounts beat copying. Bind mounts are more efficient than COPY for including build-context files, applying only temporarily for a single RUN instruction without persisting in the final image [2]. Avoid USER flip-flopping. Frequent back-and-forth USER switching should be avoided to reduce layers and complexity [2].
Reusing prior results avoids redundant work [4]. The build cache stores previous build results and restores them when needed, according to Gradle's optimization documentation [4]. Sharing and reusing unchanged build and test outputs across teams speeds local and CI builds by avoiding rebuilds of binaries unaffected by code changes [4]. It prevents redundant work and cost of time-consuming expensive processes [4].
Tasks are marked as FROM-CACHE when the build cache has been used to repopulate the local directory; once the local directory has been repopulated, the next execution marks tasks as UP-TO-DATE and not FROM-CACHE [4].
The GitHub Actions cache backend works only inside workflows. It requires $ACTIONS_RESULTS_URL and $ACTIONS_RUNTIME_TOKEN, which are populated only in a workflow context, according to Docker's GitHub Actions cache documentation [3]. The backend uses the GitHub Cache service API to fetch and upload blobs and should only be used in a GitHub Actions workflow [3].
Exporter choice controls cache coverage and placement. Max cache mode pushes the image and the cache separately using the registry cache exporter with the cache-to option, according to Docker's GitHub Actions cache documentation [3]. Inline export fits most cases but supports only min cache mode [3].
Comparison of exporter paths for Docker build caches:
| Exporter path | Behavior and limit |
|---|---|
| Max mode via registry exporter | Push image and cache separately with cache-to option [3] |
| Inline exporter | Preferred in most cases, but supports only min cache mode [3] |
| Registry exporter with manifest | Import and export from cache manifest or special image configuration on registry [3] |
Local exporter plus actions/cache |
Leverage GitHub cache by combining actions/cache with local exporter [3] |
Cache mounts need extra handling. BuildKit does not preserve cache mounts in the GitHub Actions cache by default, requiring the buildkit-cache-dance workaround that creates temporary containers to extract and inject cache-mount data between builds, according to Docker's cache documentation [3]. Local directories also grow. Exporting to a directory already holding a cache leaves prior blobs behind so the directory grows every run unless reset=true deletes them [3].
Write access determines export success. Read-only workflows should keep cache-from: type=gha, omit cache-to, and populate the cache from a push workflow on the default branch, according to Docker's cache documentation [3]. Export needs rights. Export requires write access, while issue_comment and pull_request_target runs in default-branch context get read-only access by default, can restore but may fail export with failed to reserve cache [3]. Setting ignore-error=true makes export failures non-fatal but suppresses all export errors without granting write access or saving cache when denied [3]. Overriding defaults with workflow- or job-level cache-mode to grant write access to untrusted workflows increases cache-poisoning risk [3].
The report mentions teams keeping build outputs, container layers, or test reports around longer than GitHub's default artifact retention [6].
You can extend the scope of TIA by explicitly providing the dependencies map as an XML file, then setting the build variable TIA.UserMapFile to point to it [5].
3.2 Parallelism and Sharding Test Suites
Parallel buckets beat serial runs on elapsed time [7]. According to CircleCI documentation, the mechanism is splitting the suite into buckets for parallel execution on configured execution environments [7]. According to Playwright documentation, greater parallelisation means running tests on multiple machines simultaneously, called sharding [10], where shards are smaller independently runnable parts whose division speeds runtime [10]. Suites grow. Microsoft Learn reports that as a codebase grows and matures, full regression might require hours [5], a selection problem owned by Slimming Test Suites.
According to CircleCI documentation, specify a number of identical execution environments for a job [7]. The same documentation introduces parallelism to spin up a number of identical test environments (4 in this example) [7].
GitHub jobs default to parallel and can add order. According to GitHub documentation, a workflow run is made up of one or more jobs, which run in parallel by default, and to run jobs sequentially, dependencies on other jobs can be defined using the jobs.<job_id>.needs keyword [8]. GitHub documentation states that, by default, GitHub will maximize the number of jobs run in parallel depending on runner availability [8].
A matrix has a ceiling. According to GitHub documentation, a matrix will generate a maximum of 256 jobs per workflow run [8]. Playwright documentation describes GitHub Actions sharding tests between multiple jobs using the jobs.<job_id>.strategy.matrix option, with Playwright tests run with --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} [10].
According to CircleCI documentation, timing-based splitting uses timing data from the previous test run to split a test suite as evenly as possible over a specified number of test environments [7]. The first time the tests are run there will be no timing data for the command to use, but on subsequent runs the test time will be optimized [7]. CircleCI documentation states the most recent timings data is always used to define where splits are made and timing splits are guaranteed to optimize with each test suite run [7]. The same documentation states this delivers the lowest possible test time for the compute power in use [7], and gives the most accurate split [7].
Caption: CircleCI split methods compared.
| Method | Split rule |
|---|---|
| Alphabetical | Alphabetically, by name, which is the default [7] |
| Size | By size [7] |
| Timing-based | Timing data from the previous test run is used to split a test suite as evenly as possible over a specified number of test environments [7] |
Platform and inputs gate CircleCI splitting [7]. According to CircleCI documentation, circleci tests run is available on CircleCI Cloud only and is not supported on CircleCI Server, where teams must use circleci tests split instead [7]. CircleCI requires test results to be uploaded as JUnit XML reports to use test splitting [7].
According to Playwright documentation, Playwright runs test files in parallel by default and strives for optimal utilization of CPU cores on your machine [10]. Playwright documentation notes it can only shard tests that can be run in parallel, which by default means it shards test files [10]. To shard the test suite, pass --shard=x/y to the command line, for example npx playwright test --shard=1/4 to split the suite into four shards, each running one fourth of the tests [10]. Playwright documentation states that if you run these shards in parallel on different jobs, your test suite completes four times faster [10].
With fullyParallel: true enabled, Playwright Test runs individual tests in parallel across multiple shards, ensuring each shard receives an even distribution of tests [10]. Without the fullyParallel setting, Playwright Test defaults to file-level granularity, meaning entire test files are assigned to shards, so certain shards may end up running significantly more tests, while others may run fewer or even none [10]. Playwright documentation states that without fullyParallel, tests are split at the file level, so to balance the shards, it is important to keep test files small and evenly sized [10]. Playwright documentation states that to run the same tests in multiple environments, as opposed to sharding tests onto multiple machines, environments need to be differentiated by specifying the testConfig.tag property to tag all tests with the environment name [10].
Distributed shards reunite in one report. According to Playwright documentation, merging means placing blob files in one directory and running npx playwright merge-reports --reporter html ./all-blob-reports to produce HTML output [10]. Names do not collide. Playwright documentation states blob report names contain the shard number so they will not clash [10]. For CI sharding, Playwright documentation instructs adding the blob reporter on CI, as in the conditional example below [10].
reporter: process.env.CI ? 'blob' : 'html',
blob-report holds the evidence. According to Playwright documentation, blob reports contain all run tests, results and attachments such as traces and screenshot diffs, and generate by default into blob-report [10]. Ordering is explicit. Playwright documentation describes a separate merge-reports job that depends on the sharded playwright-tests job via needs: [playwright-tests] [10].
According to CircleCI documentation, the circleci tests run command is used to run tests, split tests across parallel executors and take advantage of the Rerun Failed Tests options [7]. One report states that parallelism speeds up CI but introduces new flakiness vectors [11]. The same report lists as fixes unique test data with proper test isolation, dynamic port allocation with the OS assigning an available port, and worker-specific isolation giving each worker its own sandbox [11].
Adjacent mechanics belong elsewhere. Nested virtualization limits [9], runner-image log lookup [9], weekly image updates [9] and tool-version stability via actions [9] belong to Runner Hardware. Cache-restore branch scope [1] belongs to Caching Dependencies and Build Artifacts. Managed-code and single-machine scope [5], unsupported scenarios [5], parallel and coverage limits [5] and quarantined-test handling [11] belong to Slimming Test Suites. Deferring long tests [5] and blocked-queue effects [11] belong to Pipeline Structure.
3.3 Slimming Test Suites and Handling Flakiness
Main point: flakiness wastes substantial time even at modest rates [11].
- Flaky tests caused 21% of master branch build failures and wasted an estimated 150,000 developer hours annually [11].
- Sixteen percent of tests exhibit flaky behavior [11].
- 84% of pass-to-fail transitions come from flakiness rather than real bugs [11].
- A 30-minute suite with 10% flaky failure rate means roughly 3 minutes wasted per build, multiplied across dozens of daily builds [11].
CircleCI documentation describes splitting tests by using historic timing data [7], as covered in the parallelism discussion above. Playwright documentation adds that statically skipped tests with test.skip() or test.fixme() are not counted when balancing shards because they do not run [10].
Main point: catch flakiness before merge [11].
- Run new or modified tests multiple times: if any run fails, investigate before it enters main [11].
- When a test fails then passes on retry, flag it; this catches tests that became flaky over time [11].
- Track pass/fail rates across commits; the pattern reveals drift into flakiness [11].
pie.inc describes retrying twice before declaring failure as a practical balance, warning that more than three retries hides real problems [11]. The same guidance advises applying retries based on test characteristics, not blanket policies, with Checkout flow using retries: 0 where critical tests fail immediately and Tooltip animations using retries: 2 [11].
Details on same-job versus fresh-environment retry placement are compared below [11].
Caption: Retry placement — same-job immediacy versus fresh-job isolation [11]
| Approach | Reported benefit | Reported cost or limit |
|---|---|---|
| Re-run failed tests within the same CI job [11] | Fast execution, no extra job overhead [11] | Same environment conditions persist; if flakiness is environmental, retry won’t help [11] |
| Fresh environment [11] | Fresh environment isolates environmental flakiness; more reliable signal [11] | Longer feedback loop; more compute cost [11] |
pie.inc advises keeping quarantine under 5% of the total test suite, describes higher than 10% as signaling a systemic problem, and describes 30 days as a reasonable maximum after which tests need a fix-or-delete decision [11]. The same guidance states stable tests gate deployments while flaky tests run in parallel for monitoring, which preserves signal quality without losing coverage intent [11].
A smaller reliable suite beats a large flaky one, according to pie.inc [11]. That guidance says to fix tests validating critical functionality and delete edge-case tests that rarely matter in production [11].
Microsoft's Azure DevOps documentation describes enabling Test Impact Analysis (TIA) in the Visual Studio Test task to perform incremental validation by automatic test selection [5]. For a given code commit entering the CI/CD pipeline, TIA selects and runs only the relevant tests required to validate that commit [5]. The same documentation says selection includes existing impacted tests, previously failing tests, and newly added tests [5].
See the TIA selection discussion above for the fallback to running all tests [5]. Changes to HTML or CSS files provide one concrete case it cannot reason about, triggering fallback to all tests [5].
Microsoft documentation states setting DisableTestImpactAnalysis to true for a specific build disables TIA and forces TIA to run all tests for that build [5]. TIA can be configured to run all tests at a configured periodicity [5]. Scope tuning uses TIA_IncludePathFilters to include only specific paths, using the minimatch pattern when setting variables and separating multiple items with a semicolon [5].
TIA is integrated into existing test reporting at both the summary and details levels, including notification emails [5]. Selection can be checked by manually validating the selection or by running TIA-selected tests and then all tests in sequence, using two test tasks — one that runs only impacted tests (T1) and one that runs all tests (T2) [5].
3.4 Incremental Builds and Build Caching
Gradle's optimization guide states that if a developer continuously changes a single file, there is likely no need to rebuild all the other files in the project [4]. The same guide states that switching to a new branch created last week causes files to be rebuilt, even though the developer is building something that has been built before [4].
Docker's best-practices guide recommends using a .dockerignore file to exclude files not relevant to the build, without restructuring the source repository [2]. Broader cache-reuse logic is covered in Caching Dependencies and Build Artifacts. According to Docker's best-practices guide, the ADD instruction is best when downloading a remote artifact as part of the build because it ensures a more precise build cache than manually adding files using wget and tar [2].
Docker's best-practices guide directs splitting Dockerfile instructions into distinct stages to make sure that the resulting output only contains the files that are needed to run the application [2]. The same guide also recommends considering two types of base image: one for building and unit testing, and another typically slimmer image for production [2]. According to the same guide, using multiple stages can also let you build more efficiently by executing build steps in parallel [2]. Parallel runners are detailed in Parallelism and Sharding Test Suites.
Docker's best-practices guide recommends that when checking in a change to source control or creating a pull request, use GitHub Actions or another CI/CD pipeline to automatically build and tag a Docker image and test it [2]. Pipeline organization is covered in Pipeline Structure. Microsoft's Azure DevOps documentation states that integrations are frequent, and verified with an automated build that runs regression tests to detect integration errors as soon as possible [5], so frequent integration checks for errors early [5]. Test selection is covered in Slimming Test Suites and Handling Flakiness. Taken together, [2] and [5] indicate that these cited practices describe automatically building and testing on check-in or pull request [2] and frequent verification to detect integration errors as soon as possible [5], without describing rebuilding only changed work in those excerpts, so their relevance to build-system choice here is limited.
Docker's best-practices guide recommends the Alpine image as it is tightly controlled and small in size (under 6 MB), while still being a full Linux distribution [2]. The same guide describes Docker Official Images as a trusted starting point for many applications and Verified Publisher images as high-quality images published and maintained by the organizations partnering with Docker [2].
3.5 Runner Hardware: Hosted versus Self-Hosted Trade-offs
GitHub provides runners that you can use to run your jobs, or you can host your own runners [9].
GitHub hosts Linux and Windows runners on virtual machines in Microsoft Azure with the GitHub Actions runner application installed [9]. GitHub hosts macOS runners in Azure data centers [9]. GitHub-hosted runners are available with Ubuntu Linux, Windows, or macOS operating systems [9].
With the exception of single-CPU runners, each GitHub-hosted runner is a new virtual machine hosted by GitHub, according to GitHub documentation [9]. Single-CPU runners are hosted in a container on a shared VM, according to GitHub documentation [9].
Machine maintenance and upgrades are handled by GitHub when using GitHub-hosted runners, according to GitHub documentation [9]. Using GitHub-hosted runners requires network access with at least 70 kilobits per second upload and download speeds, according to GitHub documentation [9].
If on the GitHub Team or GitHub Enterprise Cloud plan, teams can provision a runner with more cores, or a runner powered by a GPU processor, according to GitHub documentation [9]. Custom images can only be used with larger runners, according to GitHub documentation [9]. Those larger runners also support custom images, which let you create and manage your own preconfigured VM images [9]. Taken together, the excerpts cited in this paragraph indicate the two custom-image statements describe the same larger-runner restriction from opposite sides.
Current 2-core rates, listed on GitHub's own Actions runner pricing reference, run USD 0.006 per minute for Linux, USD 0.010 per minute for Windows, and USD 0.062 per minute for macOS, all cheaper than they were in 2025 [6]. What did actually happen on January 1, 2026 is that GitHub cut its own hosted-runner rates by up to 39%, folding a smaller platform charge into the hosted price rather than the self-hosted one [6]. In December 2025, GitHub announced a new USD 0.002-per-minute platform charge that would apply to self-hosted runner usage starting March 2026; the backlash was immediate, and GitHub postponed the change within 48 hours "to re-evaluate our approach" [6]. As of mid-2026, that fee has never taken effect, self-hosted runner usage remains free of any GitHub platform charge, and there is no announced timeline for reintroducing it [6].
Capacity comparison for hosted versus self-hosted runners
| Attribute | GitHub-hosted | Self-hosted |
|---|---|---|
| Maintenance | Machine maintenance and upgrades are taken care of for you [9] | Doesn't fit well if the team doesn't have the capacity to own runner fleet management, autoscaling, and security isolation [6] |
| Isolation | With the exception of single-CPU runners, each runner is a new virtual machine hosted by GitHub; single-CPU runners are hosted in a container on a shared VM [9] | Separating boot and data volumes, or running runners in isolated environments on the same host, keeps a bad build contained to itself; MicroVMs are a common way teams solve this for untrusted or semi-trusted code [6] |
| Scaling control | If on the GitHub Team or GitHub Enterprise Cloud plan, you can provision a runner with more cores, or a runner that's powered by a GPU processor [9] | A dedicated server with 24 physical cores can host about 12 concurrent 2-core-equivalent job slots if that server runs a mix of parallel CI jobs sized to match GitHub's 2-core runner definition [6] |
| Image control | Custom images can only be used with larger runners; larger runners support custom images, which let you create and manage your own preconfigured VM images [9] | NVMe storage with high sequential read throughput keeps repository clones and artifact caching from becoming the bottleneck that CPU count alone won't fix [6] |
A dedicated server with 24 physical cores can host about 12 concurrent 2-core-equivalent job slots when jobs are sized to match GitHub's 2-core runner definition [6]. A team that outgrows 12 parallel job slots can move to Large v5 (32 cores/64 threads, currently USD 2,008.80 per month) or XL v5 (64 cores/128 threads, currently USD 3,859.20 per month) for more concurrent capacity, and the same break-even logic scales with it [6]. That 2,580-hour break-even point works out to those 12 parallel slots averaging about 30% utilization across a 720-hour month; below that GitHub's per-minute pricing is genuinely competitive and self-hosting isn't worth the operational overhead, while above it every additional minute of CI usage is functionally free, according to one OpenMetal calculation [6]. These are dedicated physical cores running full time, not shared vCPUs, so a given job may finish faster on this hardware than on an equivalent-sized cloud runner, which would push the real crossover point even lower [6].
Windows minutes push that crossover even lower, since the hosted rate is nearly double the Linux rate for the same workload [6]. Apple's licensing terms require actual Mac hardware to run macOS CI jobs, which OpenMetal doesn't sell [6]. Self-hosting doesn't fit well if the pipeline is low-volume, with CI usage well under the free tier or comfortably inside a small monthly bill, if a meaningful share of jobs specifically require macOS runners, or if the team doesn't have the capacity to own runner fleet management, autoscaling, and security isolation [6].
Separating boot and data volumes, or running runners in isolated environments on the same host, keeps a bad build contained to itself [6]. MicroVMs are a common way teams solve this for untrusted or semi-trusted code generally [6]. Self-hosted runner fleets that don't scale down during quiet periods end up paying for idle capacity, which undercuts the fixed-cost advantage that made self-hosting attractive in the first place [6]. The analysis mentions tools like GitHub's own Actions Runner Controller for Kubernetes [6]. For large repositories and dependency-heavy build steps that spend real time on disk I/O, not just CPU, NVMe storage with high sequential read throughput keeps repository clones and artifact caching from becoming the bottleneck that CPU count alone won't fix [6]. Included egress bandwidth is also worth factoring in at real CI volume, since pulling base images and pushing built container images and artifacts both consume bandwidth that's metered separately from compute on most hosted platforms [6].
Sharding and parallelism trade-offs are covered in 3.2, 3.3.
3.6 Pipeline Structure, Fail-Fast and Merge Queues
Jobs identified in jobs.<job_id>.needs must complete successfully before the dependent job will run [8]. If a job fails or is skipped, all jobs that need it are skipped unless the jobs use a conditional expression that causes the job to continue [8]. In the documented example, job3 uses the always() conditional expression so that it always runs after job1 and job2 have completed, regardless of whether they were successful [8].
Matrix builds cancel aggressively unless told otherwise. GitHub's workflow-syntax documentation states that when strategy.fail-fast is set to true or evaluates to true, GitHub cancels all in-progress and queued jobs in the matrix if any job in the matrix fails, and this property defaults to true [8]. GitHub's workflow-syntax documentation separately states that if jobs.<job_id>.continue-on-error is true, other jobs in the matrix continue running even if that job fails [8].
Failure-handling controls compared
| Control | Behaviour on failure |
|---|---|
strategy.fail-fast, default true |
Cancels all in-progress and queued matrix jobs if any matrix job fails [8] |
jobs.<job_id>.continue-on-error: true |
Other matrix jobs keep running even if that job fails [8] |
jobs.<job_id>.needs without continuation |
Dependent jobs are skipped if a needed job fails or is skipped [8] |
Documented example: job3 using always() |
In the example, job3 always runs after job1 and job2 have completed, regardless of whether they were successful [8] |
Concurrency groups can stop the active run. GitHub's workflow-syntax documentation states that specifying cancel-in-progress: true also cancels any currently running job or workflow in the same concurrency group [8].
Path filters keep docs-only changes out of the pipeline [8]. When using the push and pull_request events, you can configure a workflow to run based on what file paths are changed [8]. A workflow with the documented path filter will only run on push events that include at least one file outside the docs directory at the root of the repository [8].
Skipped workflows still block merges when checks are required. A workflow skipped by path filtering leaves its checks in Pending, and a pull request requiring those checks to be successful is blocked from merging, according to GitHub's workflow-syntax documentation [8]. Branch filtering carries the same penalty: a workflow skipped by branch filtering leaves its checks in Pending and blocks a pull request that requires successful checks [8].
When using the merge_group event, you can configure a workflow to run only for merge groups that target specific branches [8]. GitHub's workflow-syntax documentation includes an on: configuration with merge_group, types: [checks_requested] and branches for main and 'releases/**' [8]:
on:
merge_group:
types: [checks_requested]
branches:
- main
- 'releases/**'
Cache, post-merge and lifetime details belong elsewhere. Trigger-dependent cache permissions, with only listed triggers able to create or overwrite caches in the default-branch scope while other default-branch runs get read-only access, are developed in Caching Dependencies and Build Artifacts [1]. Docker's GitHub Actions cache documentation recommends push on the target branch instead of pull_request with types: [closed] after a merge because the closed-PR run uses the target-branch ref instead of the pull request merge ref [3]. Queue lifetime is a Runner Hardware concern: GitHub's hosted-runners documentation discards a run not queued within 30 minutes of triggering, plus a queued run not processed by a GitHub-hosted runner within 45 minutes [9]. Sorting multi-line arguments alphanumerically to ease maintenance and avoid duplication is a Dockerfile maintenance practice outside pipeline structure [2].
3.7 Measuring and Profiling CI Time
OpenMetal describes GitHub Actions bills as thousands of small charges, a few minutes here and a few minutes there, across every pull request a team opened [6]. CI load spikes around business hours and release windows and drops off overnight and on weekends [6]. OpenMetal describes a dedicated server with 24 physical cores, its Medium v4 tier, priced at USD 928.80 per month for the Ashburn configuration [6]. Current 2-core rates run USD 0.006 per minute for Linux [6]. See Section 4 for the break-even calculation.
Some teams report 20-30% of CI compute spent on retries alone [11]. Each false positive adds 20-60 minutes of investigation time [11]. For a given code commit entering the CI/CD pipeline, TIA selects and runs only the relevant tests required to validate that commit [5]. See Section 2 for test-suite treatment.
CircleCI documentation states allowed values for weighting the split are name, filesize, and timings (default name) [7]. CircleCI documentation states CircleCI attempts to detect the granularity of the test split (for example, to split by file name, or down to class name) based on the input to the split command [7]. See Section 1 for sharding detail.
GitHub documentation states you can create cache entries at a rate of up to 200 uploads per minute per repository, and download them at a rate of 1500 downloads per minute per repository [1]. It states that if you exceed this rate, subsequent cache upload or download attempts will fail until the relevant rate limit resets, and that the time until the rate limit resets is returned in the Retry-After header [1]. See Section 0 for caching design. Direct profiling instrumentation is not described here.
4. Discussion
Caching behavior [1], TIA test selection and fallback [5], and path-based workflow filtering [8] are covered in full elsewhere in this section and are not repeated here.
Taken together, [7] and [10] indicate parallel execution involves identical environments, compute-optimal timing and parallel-only sharding (3.2, 3.3). CircleCI specifies a number of identical execution environments for a job [7], with timing-based splitting as detailed in 3.2, which delivers the lowest possible test time for the compute power in use [7], while Playwright can only shard tests that can be run in parallel, which by default means Playwright will shard test files [10], with sharding granularity as detailed in 3.2, and with parallelism-flakiness and retry trade-offs as detailed in 3.3 [11].
Taken together, [6] and [9] indicate rented capacity wins until utilization justifies ownership (3.5, 3.1). Runner virtualization models are detailed elsewhere in this section [9], and when you use a GitHub-hosted runner, machine maintenance and upgrades are taken care of for you [9]. Teams on GitHub Team or Enterprise Cloud plans can provision a runner with more cores or a GPU processor [9], and custom images can only be used with larger runners [9]. The break-even utilization calculation is detailed elsewhere in this section [6]. Cache contents are not signed or verified, and extracted caches may modify files subsequently executed in a workflow run, leading to malicious code execution [1], and granting write access to untrusted workflows increases the risk of cache poisoning [3]. Some teams report 20-30% of CI compute spent on retries alone [11].
5. Conclusion
Taken together, [1][5] and [6] indicate redundant work on hosted runners can be cut with dependency caching, where an exact key match is considered a cache hit and a miss automatically creates a new cache if the job completes successfully [1], and with TIA selection that selects and runs only the relevant tests required to validate a given code commit entering the CI/CD pipeline [5], falling back to running all tests for commits and scenarios that TIA can't understand [5], adding self-run capacity only after sustained use passes about 30% average utilization across 12 parallel slots in a 720-hour month at the 2,580-hour break-even point, below which per-minute pricing is genuinely competitive and self-hosting isn't worth the operational overhead [6].
The table maps situations to choices.
| Reader scenario | Recommended choice | Deciding factor |
|---|---|---|
| Slow restores | Taken together, [1] indicates caching with a key that includes the runner's operating system and a SHA-256 hash of the package-lock.json file [1] | If there is an exact match to the provided key, this is considered a cache hit. If no cache exactly matches the provided key, this is considered a cache miss. On a cache miss, the action automatically creates a new cache if the job completes successfully [1] |
| Long suite, parallel budget | Taken together, [7] indicates timing-based splitting across identical execution environments [7] | This delivers the lowest possible test time for the compute power in use [7]; the first time the tests are run there will be no timing data for the command to use, but on subsequent runs the test time will be optimized [7] |
| Small changes, docs edits | Taken together, [5] and [8] indicate running only impacted tests with a docs path filter [5][8] | For a given code commit entering the CI/CD pipeline, TIA selects and runs only the relevant tests required to validate that commit [5] and for commits and scenarios that TIA can't understand, it falls back to running all tests [5]; a workflow with the following path filter will only run on push events that include at least one file outside the docs directory at the root of the repository [8] |
| Spiky low load vs steady high load | Taken together, [6] and [9] indicate stay hosted below about 30% use, self-host above it [6][9] | With the exception of single-CPU runners, each GitHub-hosted runner is a new virtual machine (VM) hosted by GitHub [9] and when you use a GitHub-hosted runner, machine maintenance and upgrades are taken care of for you [9]; spread across 12 parallel slots in a 720-hour month, that 2,580-hour break-even point works out to those slots averaging about 30% utilization across the month [6] |
GitHub Docs describes an exact key match as a cache hit, and a miss as automatically creating a new cache if the job completes successfully [1]. Docker describes avoiding extra or unnecessary packages as reducing complexity, dependencies, file sizes and build times [2]. CircleCI describes timings-based test splitting as giving the most accurate split and optimizing with each suite run, with the most recent timings data used to define splits [7]. Taken together, [5] and [8] indicate reduced work by running only relevant tests for a commit and by skipping workflows when only docs change, since for a given code commit entering the CI/CD pipeline TIA selects and runs only the relevant tests required to validate that commit [5] and a path filter will only run on push events that include at least one file outside the docs directory at the root of the repository [8]. The crossover rests on an Ashburn benchmark: a dedicated server with 24 physical cores, OpenMetal's current Medium v4 tier, priced at $928.80 per month for the Ashburn configuration, compared with a hosted rate of $0.006 per minute for Linux [6]; GitHub cut hosted-runner rates by up to 39% on January 1, 2026 [6] and lists current 2-core rates at $0.006 per minute for Linux, $0.010 per minute for Windows and $0.062 per minute for macOS [6]. Spread across 12 parallel slots in a 720-hour month, that 2,580-hour break-even point works out to those slots averaging about 30% utilization across the month; below that, GitHub's per-minute pricing is genuinely competitive and self-hosting isn't worth the operational overhead, while above it, every additional minute of CI usage is functionally free [6]. These are dedicated physical cores running full time, not shared vCPUs, so a given job may finish faster on this hardware than on an equivalent-sized cloud runner, which would push the real crossover point even lower [6]. Windows minutes push that crossover even lower, since the hosted rate is nearly double the Linux rate for the same workload, while Apple's licensing terms require actual Mac hardware to run macOS CI jobs, which OpenMetal doesn't sell [6]. TIA is currently scoped to only managed code, and single machine topology [5].
Parallelism speeds up CI but introduces new flakiness vectors [11].
Currency equivalents are approximate, using reference rates dated 16 October 2026. They do not adjust for local prices, taxes, or purchasing power.
6. Hypotheses and possible implications
Unverified assumptions. These hypotheses combine the cited evidence; they are not established findings. Citations support the premises only.
H2. Unverified hypothesis: Docker cache speed and package and base image staleness as unverified trade off
If faster builds are pursued through cache understanding while package and base image freshness are also sought, then combined update install practice with no cache and pull flags could interact in tension, but this is unverified
Evidence-backed premises
- Understanding how the build cache works and how cache invalidation occurs is critical for ensuring faster builds [2]
- Always combining RUN apt-get update with apt-get install in the same RUN statement is described as avoiding caching issues where subsequent install instructions fail [2]
- The no-cache flag disables the build cache forcing rebuild of all layers from scratch to get the latest available versions of dependencies without pulling a fresh base image [2]
- The pull flag forces checking for and downloading a newer version of the base image even if a version is cached locally [2]
Unproven assumptions
- If a single Docker build sought both faster builds and avoidance of both dependency staleness and base image staleness, then the need to consider both flags together might arise
Possible connection
Because faster builds are tied to cache understanding while cached layers can leave packages outdated and the two flags are described as addressing different staleness vectors, a build trying to keep speed and freshness might face tension between using cache and bypassing it, but whether both flags are needed together is not established
Alternative explanations
- Understanding cache invalidation with combined update and install in the same statement alone might avoid staleness without needing either flag
How to test or disconfirm
Examine Docker builds that follow combined update install practice and cache understanding to see whether outdated packages or base images are avoided without using no-cache and pull flags, which would disconfirm the need for both
References
[1] Dependency caching reference - GitHub Docs — https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching · general [2] Building best practices — https://docs.docker.com/build/building/best-practices/ · general [3] Cache management with GitHub Actions — https://docs.docker.com/build/ci/github-actions/cache/ · general [4] Incremental Builds and Build Caching Basic — https://docs.gradle.org/current/userguide/gradle_optimizations.html · general [5] Use Test Impact Analysis - Azure Pipelines — https://learn.microsoft.com/en-us/azure/devops/pipelines/test/test-impact-analysis?view=azure-devops · general [6] The Real Cost Math Behind Self-Hosted GitHub Actions Runners — https://openmetal.io/resources/blog/the-real-cost-math-behind-self-hosted-github-actions-runners/ · general [7] Test splitting and parallelism — https://circleci.com/docs/guides/optimize/parallelism-faster-jobs/ · general [8] Workflow syntax for GitHub Actions - GitHub Docs — https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax · general [9] GitHub-hosted runners - GitHub Docs — https://docs.github.com/en/actions/concepts/runners/github-hosted-runners · general [10] Sharding | Playwright — https://playwright.dev/docs/test-sharding · general [11] How to Fix Flaky Tests in CI/CD: Detection, Quarantine, and Pipeline Hardening — https://pie.inc/blog/flaky-tests-cicd/ · general
Source quality: 11 general.