Long-term Reproducibility for Automotive Software

Executive Summary

Automotive OEMs must be able to build, integrate, and validate ECU software releases for many years after End of Production (EOP). This ensures vehicles remain safe, compliant, and functional throughout their lifecycle. These retention period for software covers from 10 to 20 years, depending on the legal regulation for different markets, automotive standards, customer expectations and other factors.

During this time, OEMs must be able to deliver software updates addressing:

• Security vulnerabilities
• Regulatory compliance changes
• Malfunction corrections and bug fixes
• Other operational or safety requirements

The vehicle’s software ecosystem must remain serviceable, auditable, and reproducible. A central challenge is ensuring that a software build generated today can be bit-perfectly reproduced a decade from now, despite the inevitable obsolescence of the underlying operating systems, build tools, and cloud infrastructure. Failure to do so risks non-compliance with homologation standards and an inability to patch critical safety or security vulnerabilities in aging fleets.

This document outlines the architectural concepts for establishing and maintaining long-term reproducible build environments for automotive software. This capability is a fundamental technical necessity driven by critical regulatory requirements, specifically the need to prove software integrity and functional safety (ISO 26262) and cybersecurity compliance over the vehicle’s entire service life, typically 10 to 20 years.

The proposed solution fundamentally decouples the volatile infrastructure from the static build environment through five key principles: Environment Decoupling, Network Isolation, Containerization, Configuration as Code, and Immutable Storage.

Background and Motivation

Legal and Standardization Obligations for Long-Term Retention

In the automotive sector, reproducibility is not merely a best practice; it is a regulatory mandate.

• ISO 26262 (Functional Safety): Requires evidence that the software development process is controlled and that tools used can be trusted, implying long-term stability of the toolchain.
• UN R156 (Software Update Management System – SUMS): Mandates that OEMs must be able to securely update vehicles throughout their lifecycle. To update a legacy ECU, one must be able to rebuild its software.
• ASPICE (Automotive SPICE): Requires strict configuration management and traceability, ensuring that every artifact can be traced back to its source and build environment.

Objectives and Challenges

The primary objective is bit-identical reproducibility. If a developer checks out a 10-year-old commit and runs the build pipeline, the resulting binary must be identical to the original release. This ensures that no unintended side effects are introduced when applying a “hotfix” to a legacy system.

Building the historical software introduces several challenges:

• Infrastructure Drift: Cloud providers deprecate APIs, open-source repositories reach end-of-life, and operating systems fall out of support. This breaks legacy build/test pipelines and exposes critical security vulnerabilities, making compliance with modern standards difficult.
• Tool Rot: Proprietary compilers and toolchains often depend on outdated operating systems (e.g., Windows 7) or legacy license servers that may no longer exist. These environments are unsupported, lack security updates, and introduce vulnerabilities into the build process.
• Dependency Hell: Third-party libraries, especially in infotainment and Linux ecosystems, disappear from public registries. Updating one vulnerable library can trigger a cascade of dependent updates, destabilizing the entire software stack.

Build & Test Environments for Automotive Software

Modern automotive software is built in environments that balance scalability, security, and rigorous testing. Development often begins in the cloud, where hyperscalers like AWS, Azure, and Google Cloud provide elastic compute and CI/CD pipelines. At the same time, many OEMs rely on private clouds to safeguard sensitive data, creating hybrid setups that combine flexibility with control.

CI/CD platforms such as Jenkins, GitHub Actions, and Tekton orchestrate automated builds, testing, and deployments, ensuring rapid iteration without compromising quality.

Each automotive domain uses its own toolchains. Powertrain engineers often lean on MATLAB/Simulink and dSPACE TargetLink for model-based design, while ADAS teams employ ROS, TensorFlow, and simulation tools. Infotainment developers work with Android Automotive, QNX, or Linux. Body and comfort systems depend on AUTOSAR toolchains and Vector tools like CANoe and CANalyzer to ensure reliable communication. Finally, integration testing validates the software before it reaches the road. Software-in-the-Loop (SiL) runs code in virtual ECUs using platforms like dSPACE VEOS, enabling fast iteration. Hardware-in-the-Loop (HiL) connects real ECUs to simulated environments with systems such as ETAS LABCAR or Vector CANoe.HIL, ensuring safety-critical behavior under realistic conditions.

Together, cloud infrastructure, CI/CD tools, specialized toolchains, and HiL/SiL testing form the backbone of modern automotive software development.

Core Strategy for Ensuring Long‑Term Reproducible Software Builds

Automotive compliance demands not only secure and auditable processes but also the ability to reproduce software builds reliably over the entire vehicle lifecycle. The core strategy is therefore to decouple volatile infrastructure from the static build environment, ensuring that builds remain reproducible even as tools, platforms, and cloud services evolve. This approach rests on five guiding principles:

 Environment Decoupling establishes a stable boundary between the static build environment and the ever‑changing CI/CD infrastructure. By isolating the build environment, teams can reproduce builds years later, independent of shifts in cloud APIs or orchestration platforms.

Network Isolation provides an additional safeguard by shielding IT infrastructure from vulnerabilities inherent in legacy operating systems and historical software. Isolating these environments prevents outdated components from being exposed to modern attack surfaces, allowing reproducible builds to run safely without introducing systemic risks.

Containerization reinforces reproducibility by encapsulating toolchains within containers such as Docker or Podman. Each container locks the environment to a specific version, providing an immutable runtime state that guarantees identical builds across time and infrastructure.

Configuration as Code ensures transparency and auditability. All CI/CD definitions and descriptors are stored in version control, making every pipeline reproducible, traceable, and verifiable. This practice supports compliance audits and provides non‑repudiation for long‑term build records.

Immutable Storage safeguards artifacts over extended retention periods. Archiving outputs in object storage systems with immutable storage policies prevents tampering and ensures that build artifacts remain intact and reproducible throughout the vehicle’s operational life.

Together, these principles establish a resilient foundation for long‑term reproducible software builds. They enable manufacturers to meet regulatory obligations, support forensic traceability, and guarantee that every build can be faithfully recreated years after its original release.

Applying the Core Principles to Common Platforms

While the strategy for creating reproducible software is platform-agnostic, its successful implementation relies on applying the core principles to specific cloud providers and CI/CD systems. In this chapter we are using AWS EKS / ECS, ECR and GitHub Actions, as an example infrastructure platform. Based on the actual environment of the Automotive OEM or supplier, the mentioned tools can be replaced by equivalent counterparts from other cloud providers like Azure, Google, or OpenShift / Kubernetes platforms etc.

Environment Decoupling is achieved by separating the orchestration logic (GitHub Actions workflows) from the environment that executes the build and test workloads. GitHub Actions defines the pipeline logic declaratively in YAML files stored in Git, ensuring that the orchestration is fully version-controlled and reproducible. The workflows trigger jobs that run inside isolated Agents provisioned on AWS infrastructure (such as EKS Pods or AWS CodeBuild containers). This separation guarantees reproducible and scalable builds, independent of changes in GitHub’s SaaS platform or AWS APIs.

The Agent executes the build inside isolated, ephemeral services (e.g., Kubernetes Pods in EKS or AWS CodeBuild containers). Each Agent pulls a static, version-controlled build environment (defined in a Docker image stored in Amazon ECR) and runs the pipeline steps. This ensures that the execution environment remains predictable and decoupled from changes in GitHub Actions or AWS orchestration services.

 Containerization reinforces reproducibility by encapsulating all necessary toolchains within Docker or Podman containers. Each container locks the environment to a specific version, providing an immutable runtime state that guarantees identical builds across time and infrastructure. These containers are stored in Amazon ECR, tagged with cryptographic hashes, ensuring that historical build images are always retrievable and verifiable.

 Network Isolation safeguards reproducibility by shielding legacy toolchains and operating systems from modern attack surfaces. Build environments often require outdated compilers or proprietary tools that cannot be continuously patched. Running these environments inside private AWS VPC subnets or enforcing Kubernetes Network Policies ensures they remain isolated, reducing systemic risk while allowing reproducible builds to run safely.

 Configuration as Code (CaC) ensures transparency and auditability. GitHub Actions workflows, infrastructure descriptors (Terraform, CloudFormation), and environment configurations are stored in Git, making every pipeline reproducible, traceable, and verifiable. This extends to “Everything as Code,” where OS base images, CI/CD pipeline logic, containerized toolchains, and even GitHub Actions workflow definitions are versioned. This guarantees that the precise orchestration logic used for historical builds can be faithfully re-created.

Immutable Storage safeguards artifacts over extended retention periods. Reproducibility demands that every component used in a build remains protected. AWS provides multiple layers of immutability:

• Base Images: Stored in Amazon ECR, locked by SHA-256 digests.
• Build Tools: Proprietary compilers and toolchains containerized and archived in ECR.
• Intermediate Libraries: Archived in artifact repositories like JFrog Artifactory or Sonatype Nexus, configured to prevent overwrites.
• Final Artifacts: Firmware binaries and deployable packages stored in Amazon S3 with Object Lock (WORM), ensuring tamper-proof archival for the vehicle’s operational life.

Orchestration and Tooling Flow

Within this highly controlled and secured architecture, the CI/CD system orchestrates the tooling workflow. Source code managed in Git triggers pipelines defined by orchestrators like Jenkins or Github Actions. These pipelines schedule jobs into the containerized Agents where systems like CMake/Make, Bazel, or the Yocto Project execute the necessary compilation and image generation. The segregation provided by the Controller-Agent Principle allows this entire workflow to be frozen in time, ensuring that the same sequence of validated, immutable components yields an identical result every time a historical build is invoked.

Non-Linux containers

In modern automotive software pipelines, Linux containers have become the popular. They are lightweight, reproducible, and perfectly suited for headless compilers and automated test frameworks. Not every tool fits neatly into this model. Some of the most critical build tools in the automotive world, older compilers, calibration environments, or proprietary IDEs, were designed for Windows, often for versions that are now considered legacy. These tools sometimes demand a full graphical interface, making them difficult to run in the stripped‑down, command‑line environments that Linux containers excel at. Supporting these tools requires a different approach. Instead of abandoning containerization altogether, the infrastructure can be extended to include Windows containers or virtualized Windows environments. A build pipeline might spin up a Windows container that hosts an older Tasking Compiler or a calibration suite, allowing engineers to interact with the tool through a remote desktop session. In this way, the container provides both isolation and reproducibility, while still exposing the UI that the tool requires. Integration into CI/CD pipelines follows the same principles as with Linux workloads. A commit in Git can trigger Zuul or Jenkins, which then schedules a Windows container alongside Linux builds. Inside that container, the legacy tool runs exactly as it did years ago, but now it is wrapped in modern orchestration.

License Management

Re-creating historical build environments with proprietary tools requires license management. The greatest non-technical risk to long-term reproducibility lies in managing proprietary software licenses for tools like compilers. Historic builds rely on injecting the original license key or file into an ephemeral Agent (the build container) just-in-time and then destroying it upon completion. This strategy is excellent for technical security and auditability, treating the license as a secured, versioned artifact. However, this technical solution does not guarantee legal compliance. The End-User License Agreements (EULA) of all commercial tools needs to be considered. Following key legal constraints need to be investigated:

• Node-Locked vs. Floating: Many historic licenses are node-locked, meaning they are cryptographically bound to the MAC address or CPU ID of a single, specific machine. Running this on dozens of different, temporary Agents (the cloud containers) could constitute a violation of the “single machine” clause.
• Activation/Transfer Limits: Some agreements limit the number of times a license can be activated or transferred. Ephemeral CI/CD environments could easily exceed these limits over time, causing the license to fail and the build to halt.
• Support for Historic Versions: The vendor might have ceased support, activation, or validation services for the specific older version of the tool required for the historic build, regardless of whether your key is “perpetual.”

To ensure long-term reproducibility is both technically sound and legally viable, the license conditions and EULAs must be carefully investigated and in some cases negotiated. The preferred technical solution is the use of a dedicated, isolated License Server that manages a pool of floating licenses, which the Agents can check out and return over a controlled network, satisfying the need for ephemeral use without violating node-binding restrictions.

Governance and Audits

Maintaining long-term reproducible builds is impossible without robust governance and an unyielding focus on auditability. The entire process must be managed to ensure that the integrity and functional safety required by standards like ISO 26262 are preserved over the vehicle’s 10-to-20-year service life^.

The Environment Registry is a core element of governance and is understood as a logical concept, not a single physical component, that must be implemented to maintain long-term reproducibility. It functions as the single source of truth for all dependencies required to recreate a specific historical software build.

Technically, the Registry is implemented as an immutable, version-controlled repository of metadata. This repository tracks and stores the exact state of the entire build ecosystem at the moment of a software release. Key data points recorded include the unique SHA-256 hash or immutable tag for every build container image, which are stored in dedicated registries like Amazon ECR or Azure Container Registry (ACR). It also captures the pinned versions of all proprietary toolchains, which is crucial for overcoming Tool Rot.

The Registry is intrinsically linked to the principle of Configuration as Code (CaC). The infrastructure definitions and the CI/CD pipeline code, which are stored in Git, represent the codified input that defines how the software is built. The Environment Registry captures the final, successful, auditable state of that CaC definition and the resulting artifacts. By providing this comprehensive, auditable snapshot, the Registry ensures that the build environment can be faithfully reconstructed years later, directly addressing challenges like Infrastructure Drift

Restoration Validation is the periodic, proactive check that confirms the system’s ability to reproduce historical software. This process is the ultimate proof of concept for the long-term reproducibility strategy: a scheduled task is run to retrieve an archived build’s metadata from the Environment Registry and attempt to recreate its final binary. If the resulting binary is not bit-identical to the original release, the validation fails, triggering an immediate forensic investigation to correct the broken pipeline or update the registry’s records. This mechanism ensures ongoing compliance with homologation standards and prevents the inability to patch critical vulnerabilities in aging fleets.

Change Control

All modifications to the registered build environments and CI/CD pipelines must be strictly managed through formal Change Control. Any update, whether a minor security patch to a base operating system or a toolchain version upgrade, must be fully traceable, documented, and stored in version control (e.g., Git). This practice, aligned with the Automotive SPICE requirement for strict configuration management, guarantees that every resulting software artifact can be traced back to its specific source code, build environment, and toolchain versions. This high level of traceability prevents unintended side effects when applying a hotfix to a legacy system.

Audit Reports

The culmination of this governance is the Audit Report. These reports synthesize the data from the Environment Registry, Restoration Validation, and the Change Control logs into a cohesive, auditable package. The reports provide verifiable evidence to regulatory bodies that the OEM can prove software integrity and cybersecurity compliance over the vehicle’s entire service life. By demonstrating the use of pinned, immutable container images, network-isolated builds, and WORM-enforced immutable storage for artifacts, the audit reports ensure nonrepudiation for long-term build records, which is crucial for meeting regulatory mandates like UN R156 (SUMS).

Workflow for Reproducing a Legacy Build

Reproducing a software build from years ago is a critical governance requirement, often called Restoration Validation, which proves that the long-term reproducibility strategy is effective. This is the pre-requisite to applying changes to old software. The following conceptual workflow details the process of creating a hotfix for a historical software using a C++ application, CMake build system, CUnit tests. As an example, an AWS infrastructure stack (Amazon EKS, S3, ECR) orchestrated by GitHub Actions is used. The model combines GitHub for developer usability and version control with S3 Object Lock for immutable archival compliance.

1. Entrypoint: Identifying the Artifact
   The reproduction process begins with the Artifact Identifier (e.g., ECU-A-v1.3.1-2018Q2).

• The manifest is retrieved from GitHub, where it is version-controlled and easy to execute in GitHub Actions workflows.
• At the same time, the manifest exists in Amazon S3 with Object Lock as an immutable reference.
• GitHub provides the working copy for developers, while S3 ensures tamper-proof archival.
• When a developer checks out the manifest from GitHub, the GitHub Actions runner verifies its integrity against the S3 copy, ensuring both usability and compliance.

Example Manifest

2. Build Execution
   A temporary, network-isolated workspace is provisioned as an Amazon EKS Pod.

• The GitHub Actions runner checks out the source code and pipeline definition from GitHub according to the manifest.
• The toolchain container is pulled from Amazon ECR using the immutable SHA-256 hash recorded in the manifest.
• The original CI/CD workflow is executed, running legacy CMake to configure the build, invoking the historical C++ compiler, and running unit tests with CUnit.

3. Validation
   Once the build is executed, Restoration Validation confirms reproducibility:

• A new hash (e.g., SHA512) is calculated for the recreated binary.
• This hash is compared against the original checksum in the manifest.
• A match proves bit-identical reproducibility, confirming that the environment and toolchain have been faithfully restored.
• If validation fails, a forensic investigation is triggered to correct the pipeline or update registry records.

4. Hotfix Application
   After successful validation, the hotfix can be applied:

• Developers introduce intentional changes (e.g., security patch, regulatory compliance update).
• The hotfix build runs inside the same validated environment to ensure consistency.
• The resulting binary will differ from the original, but the process guarantees controlled and reproducible changes.

5. HOTFIX Manifest Creation
   Once the hotfix build is complete, a HOTFIX manifest is generated to capture the updated state of the build environment and artifacts.

• Explicitly labeled as a HOTFIX to distinguish it from original release manifests. It includes
  • Unique artifact ID and release date for the hotfix version.
  • Git commit SHA of the hotfix source code.
  • Immutable SHA-256 digest of the toolchain container image used.
  • Updated build tool versions (if applicable).
  • New binary checksum (e.g., SHA512).
  • A description of the hotfix reason.
• Stored in GitHub for developer usability and archived in Amazon S3 with Object Lock for compliance.

6. Audit Reporting
   The audit report synthesizes the entire workflow:

• Entrypoint manifest retrieval and verification.
• Successful bit-identity validation of the unmodified build.
• Controlled application of the hotfix.
• Creation and archival of the HOTFIX manifest.

This provides regulators with verifiable evidence that every change is intentional, reproducible, and clearly marked as a hotfix, ensuring compliance with standards like ISO 26262, UN R156, and ASPICE.

Conclusions

Automotive Original Equipment Manufacturers (OEMs) face a unique and demanding challenge: they must be able to build, integrate, and validate Electronic Control Unit (ECU) software releases for 10 to 20 years after a vehicle’s End of Production (EOP). This long-term retention is not merely a best practice but a regulatory mandate essential to ensure vehicles remain safe, compliant, and functional throughout their service life. The industry must be capable of delivering updates that address critical issues like security vulnerabilities, regulatory compliance changes, and malfunction corrections for aging fleets.

The core objective is achieving bit-identical reproducibility. If a developer attempts to rebuild a 10-year-old commit, the resulting binary must be the same as the original release, ensuring no unintended side effects are introduced when applying a “hotfix” to a legacy system. Failure to achieve this risks non-compliance with homologation standards and an inability to patch critical safety or security flaws.

This mandate is driven by key standards: ISO 26262 (Functional Safety) requires evidence of a controlled development process and trusted toolchains; UN R156 (SUMS) mandates the ability to securely update vehicles; and ASPICE demands strict configuration management and traceability.

 The greatest technical challenge lies in overcoming Infrastructure Drift, Tool Rot, and Dependency Hell, where libraries are deprecated and operating systems fall out of support.

The proposed solution fundamentally addresses this by decoupling the volatile infrastructure from the static build environment. This core strategy is based on five guiding principles:

1. Environment Decoupling establishes a stable boundary between the CI/CD orchestration and the build environment itself.
2. Containerization encapsulates proprietary toolchains and operating systems in immutable containers (like Docker or Podman). This locks the build environment to a specific, versioned state, guaranteeing identical builds across time.
3. Network Isolation runs the build process in private subnets with restricted access (e.g., using Amazon VPC or Azure VNet). This shields the IT infrastructure from the security vulnerabilities inherent in legacy operating systems and tools.
4. Configuration as Code (CaC) ensures all CI/CD pipelines and infrastructure definitions are version-controlled, providing transparency, traceability, and auditability for compliance.
5. Immutable Storage archives final build artifacts, binaries, and logs in WORM (Write Once, Read Many) object storage (like Amazon S3 Object Lock or Azure Blob Storage Immutability Policies). This prevents tampering and guarantees the artifacts remain intact for the vehicle’s entire operational life.

In essence, these principles enable the entire automotive toolchain to run consistently and reproducibly for decades, even accommodating legacy Windows-based tools within modern containerized or virtualized environments. This capability is the fundamental technical foundation for meeting critical regulatory obligations and supporting traceability across the long automotive lifecycle.

Share