Software Supply Chain Security: Building Verifiable Builds from Commit to Container

Modern applications are not written from scratch. They are assembled from open-source packages, base container images, CI/CD runners, build plugins, cloud services, and deployment pipelines. Every link in that chain is an opportunity for an attacker. A single compromised dependency, stolen build token, or tampered artifact can propagate through thousands of organizations before anyone notices. Software supply chain security is the discipline of making that chain visible, hardened, and verifiable from source code to production.

Why Software Supply Chain Security Matters

High-profile incidents have shown that attackers increasingly target the build and distribution layer rather than individual endpoints. SolarWinds demonstrated how a compromised build pipeline can distribute signed malware to customers. Log4Shell showed how a single transitive dependency can create global blast radius. Codecov and event-stream showed that CI scripts and maintainer accounts are valuable targets. The xz Utils backdoor showed how a patient attacker can social-engineer their way into a critical open-source project. These are not isolated cases; they are patterns.

The core problem is trust. We trust package registries, maintainers, build systems, and deployment tools. Supply chain security replaces blind trust with evidence: inventory, provenance, signatures, policy, and continuous verification.

Mapping the Software Supply Chain

A useful model breaks the supply chain into stages. Each stage has its own threats and controls.

• Source: Git repositories, branches, pull requests, commit signing, code review, and branch protection.
• Dependencies: Direct and transitive packages from npm, PyPI, Maven, NuGet, Go modules, Cargo, and container base images.
• Build: CI/CD runners, build scripts, compilers, packaging tools, caches, and secrets.
• Artifacts: Container images, binaries, JARs, wheels, tarballs, SBOMs, and attestations.
• Distribution: Registries, artifact repositories, CDNs, and update channels.
• Deployment: Kubernetes admissions, serverless deploys, VM provisioning, and configuration management.
• Runtime: Workload identity, network policy, monitoring, and incident response.

Security teams often focus on runtime. Supply chain security pushes controls left into source, dependencies, and build, then verifies them right at deployment.

Threat Model: How Supply Chains Break

Before choosing tools, define the threats you are defending against. Common supply chain attacks include:

• Typosquatting: Publishing a package with a name similar to a popular one, hoping a developer mistypes it.
• Dependency confusion: Publishing a public package with the same name as an internal private package and a higher version, causing build tools to fetch the public malicious package.
• Compromised maintainer: Stealing credentials or bribing a maintainer to publish a malicious update.
• Malicious package updates: Adding harmful code to a legitimate package through a compromised account or abandoned project takeover.
• Build system compromise: Injecting code into the build process so the artifact differs from the source.
• Credential theft: Stealing CI tokens, cloud keys, or registry credentials to publish artifacts or access production.
• Artifact tampering: Modifying an artifact after build, either in the registry or in transit.
• Registry poisoning: Uploading malicious images or packages to a registry that developers trust.
• Insecure CI/CD: Using unpinned actions, overly broad tokens, or long-lived secrets that allow lateral movement.

Your threat model should be specific. A startup may prioritize dependency and CI hygiene. A regulated enterprise may need provenance, signing, and admission control. A platform team may need to provide secure defaults for hundreds of services.

Core Principles

Supply chain security is not one product. It is a set of principles applied across the lifecycle.

• Least privilege: Build jobs, deploy jobs, and humans should have only the permissions they need, for as short a time as possible.
• Ephemeral builds: Builds should run in clean, isolated environments that are destroyed after use.
• Immutability: Artifacts should be immutable and identified by content digest, not mutable tags like latest.
• Verifiability: You should be able to verify who built an artifact, from what source, and with what dependencies.
• Defense in depth: Combine dependency scanning, SBOMs, signing, provenance, policy, and runtime controls.
• Transparency: Use transparency logs and attestations so evidence can be independently audited.
• Automation: Manual security gates do not scale. Policy must be encoded and enforced automatically.

SBOM: The Ingredient List for Software

A Software Bill of Materials is a machine-readable inventory of components in a software artifact. It lists direct and transitive dependencies, versions, licenses, and often package identifiers. An SBOM answers the question: what is inside this artifact?

The two dominant formats are SPDX and CycloneDX. Both support packages, relationships, licenses, and hashes. The NTIA minimum elements for an SBOM include supplier name, component name, version, unique identifiers, dependency relationships, author of the SBOM, and timestamp. Without these, an SBOM is difficult to use during an incident.

Generate SBOMs in the build pipeline, not after the fact. Tools such as Syft, Trivy, cdxgen, cyclonedx-gradle-plugin, cyclonedx-npm, and pip-cyclonedx can produce SBOMs from source, containers, or filesystems. Store the SBOM as an artifact next to the image or binary. Sign it. Attach it as an attestation when possible.

SBOMs are useful for:

• Vulnerability response: When a new CVE drops, query SBOMs to find affected services instead of scanning everything manually.
• License compliance: Identify licenses and obligations across dependencies.
• Incident investigation: Understand blast radius when a package or maintainer is compromised.
• Policy enforcement: Block artifacts that contain denied components or licenses.
• VEX: Pair SBOMs with Vulnerability Exploitability eXchange documents to communicate whether a vulnerability actually affects a product.

SBOMs have limitations. They can be incomplete for dynamically loaded code, vendored dependencies, or statically linked binaries. Package naming can be inconsistent. Transitive dependency resolution can be complex. An SBOM is evidence, not proof. It must be generated accurately and kept current.

Dependency Governance and SCA

Software composition analysis scans dependencies for known vulnerabilities and license issues. But SCA is only one control. Governance is broader.

• Lockfiles: Commit lockfiles for npm, Yarn, pnpm, Cargo, Go modules, and others. Lockfiles pin exact versions and hashes.
• Hash verification: Enable integrity checks so package managers verify hashes from lockfiles.
• Private registries: Proxy public registries through a controlled repository that can scan, cache, and apply policy.
• Scoped packages and namespaces: Use scoped names for internal packages and configure resolution order to prefer internal registries.
• Allowlists and denylists: Define approved registries, package names, and versions. Block unknown sources.
• Automated updates: Use Dependabot, Renovate, or similar tools with policy. Do not auto-merge blindly; require CI and review.
• Vulnerability triage: Prioritize based on reachability, exploitability, and exposure, not just CVSS score.
• Dependency review: Add a human or automated review step for new direct dependencies.

Dependency confusion is best mitigated by namespace control and registry configuration. If your build tool checks a public registry before an internal one, an attacker can win. If your internal packages use a scope that cannot be published publicly, the attack surface shrinks. Typosquatting is mitigated by automated package name checks, allowlists, and developer education.

Securing the Build

The build system is a high-value target because it has access to source code, secrets, and the ability to publish trusted artifacts. A secure build should be isolated, reproducible, and auditable.

• Ephemeral runners: Use fresh VMs or containers for each build. Do not reuse runners with persistent credentials or caches that can be poisoned.
• Isolated networks: Restrict egress from build jobs. Allow only required package registries, artifact stores, and APIs.
• OIDC federation: Replace long-lived cloud keys with short-lived tokens issued to the build job through OpenID Connect.
• Least privilege tokens: In GitHub Actions, set minimal GITHUB_TOKEN permissions. In Jenkins, use fine-grained credentials and roles.
• Pinned actions and plugins: Pin CI actions and build plugins to full commit SHAs, not mutable tags. Review updates.
• Build as code: Keep pipeline definitions in version control with review and branch protection.
• Hermetic builds: Where possible, build without network access after dependencies are fetched. This reduces supply chain injection.
• Reproducible builds: Aim for builds that produce the same artifact from the same source and inputs. This makes tampering easier to detect.
• Cache integrity: Treat build caches as untrusted. Sign and verify cache entries or avoid sharing caches across trust boundaries.
• Secret hygiene: Never echo secrets. Use secret scanning. Rotate credentials. Store secrets in a vault or cloud secret manager.

Provenance and SLSA

Provenance is evidence about how an artifact was produced. It typically includes the source repository, commit digest, build system, builder identity, build parameters, and dependencies. SLSA, the Supply-chain Levels for Software Artifacts, provides a framework for increasing confidence in provenance.

SLSA levels build on each other:

• Build L1: Provenance exists and is distributed. It may be self-declared.
• Build L2: Provenance is generated by a hosted build service and signed. This prevents tampering by individual developers.
• Build L3: Build runs in an isolated, ephemeral environment with strong guarantees against tampering during the build.
• Source L2-L4: Source provenance protections, such as verified history, branch protection, and two-party review.

In-toto provides a framework for attestations across the supply chain. An attestation is a signed statement about an artifact or process. SLSA provenance is a specific type of in-toto attestation. Tools such as cosign can generate and verify in-toto attestations. GitHub Actions can generate artifact attestations natively for builds.

Provenance enables policy decisions. For example, you can require that production images are built by a specific GitHub Actions workflow from the main branch of an approved repository, with a signed SLSA provenance attestation. If the provenance does not match, the image is rejected at admission.

Artifact Signing and Verification

Signing artifacts provides integrity and authenticity. Verification ensures the artifact came from a trusted builder and has not been tampered with. Signing without verification is security theater.

Common approaches include:

• GPG signatures: Traditional key-based signing for packages, commits, and releases. Key management is a burden.
• Sigstore and cosign: Keyless signing using OIDC identities, short-lived certificates, and the Rekor transparency log. Also supports key-based signing.
• Notary and TUF: The Update Framework protects software update systems against key compromise and rollback attacks.
• Registry-native signing: Some registries support image signing and verification as part of their platform.

With Sigstore, a developer or CI job can authenticate with an OIDC identity, obtain a short-lived signing certificate from Fulcio, sign the artifact, and record the signature in Rekor. Verification checks the signature, certificate identity, and transparency log entry. This avoids long-lived private keys for many use cases.

Sign more than container images. Sign SBOMs, provenance attestations, binaries, and Helm charts. Attach signatures and attestations to the image in the registry using cosign. Then verify them in CI, at admission, and before deployment.

Example cosign commands:

cosign sign --yes ghcr.io/example/app@sha256:abc123 ; cosign attest --yes --type slsaprovenance --predicate provenance.json ghcr.io/example/app@sha256:abc123 ; cosign verify --certificate-identity-regexp https://github.com/example/org/.github/workflows/build.yml@refs/heads/main --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/example/app@sha256:abc123

In Kubernetes, use admission controllers such as Kyverno, OPA Gatekeeper, or Sigstore Policy Controller to verify signatures and attestations before a workload is admitted. This ensures that only trusted artifacts run in the cluster.

Policy as Code: Enforce at Admission

Policy as code turns supply chain requirements into automated decisions. Policies can be evaluated in CI, at admission, or continuously at runtime.

Typical policies include:

• Images must be signed by an approved builder identity.
• Images must have a valid SLSA provenance attestation at Build L2 or higher.
• Images must include an SBOM attestation.
• No critical vulnerabilities with a fix available and reachable code path.
• Only approved base images and registries are allowed.
• No latest tags; use immutable digests.
• Containers must run as non-root and with read-only root filesystem where possible.
• Licenses must be from an approved list.
• Deployments must come from approved Git branches and workflows.

Start in audit mode. Observe what would be blocked. Fix legitimate issues. Then move to enforce mode for high-risk namespaces or production. Avoid breaking all deployments on day one. Provide clear error messages and remediation guidance.

CI/CD Hardening Checklist

Use this checklist as a baseline for pipelines.

• Pin third-party actions and plugins to full commit SHAs.
• Set minimal token permissions for each job.
• Use OIDC for cloud and registry access instead of long-lived secrets.
• Separate build and deploy credentials. Deploy jobs should not build untrusted code.
• Run builds on ephemeral, isolated runners.
• Restrict network egress from build jobs.
• Generate SBOMs, provenance, and signatures as part of the build.
• Store artifacts in a registry or repository that supports immutability and digest references.
• Scan for secrets in code, history, and pull requests.
• Protect branches and require reviews for pipeline changes.
• Monitor audit logs for unusual publishing or token use.
• Rotate credentials and revoke access automatically when employees leave or tokens leak.
• Test your incident response for a compromised dependency or build token.

Container and Kubernetes Considerations

Containers add layers and registries to the supply chain. Secure them end to end.

• Minimal base images: Use distroless, Alpine, or scratch images where practical. Fewer packages mean fewer vulnerabilities.
• Non-root users: Run containers as non-root and drop unnecessary capabilities.
• Read-only filesystems: Mount writable volumes only where needed.
• Image scanning: Scan images in CI and in the registry. Re-scan continuously because new CVEs are published daily.
• Digest references: Deploy by digest, not by mutable tag. Tags can be overwritten.
• Admission control: Verify signatures, attestations, and policies before workloads run.
• Runtime security: Detect unexpected process execution, network connections, and file changes.
• Registry security: Enable immutability, audit logging, and least-privilege access.

Incident Response with SBOMs and VEX

When a supply chain incident occurs, speed matters. SBOMs let you answer: which services use the affected component, which versions, and where are they deployed? Without SBOMs, you are guessing.

Build a workflow:

• Ingest SBOMs into a queryable store or dependency graph.
• Correlate vulnerabilities from CVE feeds, vendor advisories, and scanner results.
• Use VEX to record whether a vulnerability is exploitable in your context. VEX can reduce noise and focus remediation.
• Identify blast radius across environments, clusters, and customer deployments.
• Revoke compromised signing keys or identities. Rotate tokens. Publish fixed artifacts.
• Monitor transparency logs for unexpected signatures or attestations.
• Post-incident, add controls to prevent recurrence.

Implementation Roadmap

A practical supply chain security program can be built in phases.

1. Visibility: Inventory repositories, registries, build systems, and deployment targets. Identify owners.
2. Dependency hygiene: Enforce lockfiles, pin dependencies, use private registries, and enable SCA.
3. SBOM generation: Generate SBOMs for all build artifacts. Store and sign them.
4. CI/CD hardening: Ephemeral runners, OIDC, least privilege, pinned actions, secret scanning.
5. Artifact signing: Sign images, binaries, SBOMs, and attestations. Start with keyless where possible.
6. Provenance: Adopt SLSA Build L2, then L3 for critical services. Generate signed provenance.
7. Admission policy: Audit, then enforce signature and provenance verification in Kubernetes and other deploy targets.
8. Continuous assurance: Monitor, re-scan, rotate credentials, test incident response, and expand coverage.

Common Pitfalls

• Treating SBOM as a compliance checkbox: An SBOM that is never used during incidents provides little value.
• Signing without verifying: Signatures only help if admission or deployment checks them.
• Ignoring transitive dependencies: Most risk comes from indirect dependencies.
• Overblocking: Policies that block everything create workarounds and shadow deployments.
• No revocation plan: You need a way to revoke keys, identities, and artifacts when compromise happens.
• Securing build last: The build system is a primary target. Harden it early.
• No ownership: Every artifact and policy needs an owner.
• Assuming open source is insecure or proprietary is safe: Both can be vulnerable. Evidence matters more than origin.

Conclusion

Software supply chain security is a continuous practice, not a one-time project. Start with visibility and dependency hygiene. Add SBOMs, signing, provenance, and policy enforcement. Harden the build as a high-value asset. Verify artifacts before they run. Prepare for incidents before they happen. The goal is not to eliminate all risk, but to make trust explicit, evidence-based, and automatable from commit to container.