[Youngwoo Kim (김영우), executive director at MathWorks Korea] As agentic AI moves beyond simple support to take on execution tasks inside workflows, trust is no longer a challenge that can be avoided. Agentic systems reason across multiple domains, call engineering tools and perform repetitive work across domains. At the same time, they must operate in environments that require strict safety, performance and verification standards. When AI leads planning, execution and iteration, outputs that appear correct on their own can trigger unintended behaviour at the overall system level because interactions among components create additional errors.

In software-defined products, this risk does not stem from AI acting independently. The fundamental problem lies in engineering workflows that cannot express system-level behaviour in an executable form.

More than 1 out of 5 vehicle recalls are linked to software-related fixes, and similar integration failures are appearing in other software-defined systems. For example, even a timing change that passes unit tests can cause problems when integrated into a larger control loop that depends on the original behaviour. This is because system-level behaviour is not evaluated consistently as the system evolves.

Engineering teams can build trust in agentic AI by adopting system-first engineering. System-first engineering is a mindset that prioritises system-level behaviour from the start of development, and it is implemented through model-based design.

By using a shared executable system model instead of assumption-based handoffs, teams create system-level criteria that both engineers and agentic AI can reference. These models serve as common behavioural criteria across mechanical, electrical and software domains. They anchor AI to verified engineering workflows rather than abstract requirements. This allows behaviour to be interpreted consistently even as development scales across teams and functions.

System-first engineering keeps agentic AI operating in a structured environment while reducing unintended risk. Leading teams virtualise systems before hardware exists and continuously evaluate system models through automated workflows. This is mainly implemented through CI/CD pipelines and closed-loop simulations.

Agentic AI accelerates development workflows by coordinating and directly executing modelling, testing and verification activities. It does so by invoking automatable parts of model-based design and following predefined process models that define how each step is executed. In this environment, agent-led changes are evaluated early against system behavioural criteria before they lead to integration failures later. Engineers monitor agentic processes and results and are responsible for approving final verification and validation (V&V). This ensures AI-driven execution is operated responsibly based on system-level evidence rather than individual outputs.

Deterministic verification builds trust

Trust in agentic AI comes from repeatability. Repeatability depends on deterministic verification, which provides consistent, auditable evidence for traceability and safety reviews. Teams applying system-first engineering use executable specifications that run from system architecture and design through code generation and testing, rather than documents that can produce differences in interpretation. This allows every change made by an agent to be assessed against the same system behavioural criteria.

Software can be developed iteratively every day, but hardware takes weeks to months. In that situation, relying on hardware for verification is not realistic. Teams can reduce dependence on expensive hardware tests by continuously running simulations on established system models. All changes in software, architecture and control logic are deterministically verified based on a model that represents the whole system. System-level errors such as missed timing targets or excessive memory use can be caught before reaching the hardware stage. This repeatable verification evidence allows teams to review and certify agentic AI changes even as task complexity increases.

Agentic AI operating within a verified system Once trust is established through deterministic verification, the next question is how agentic AI operates within that environment. Agentic AI runs inside engineering workflows and handles outputs such as system models, component models, test cases and scenario variations, many of which can be produced by generative AI. Agentic AI can also contribute to creating these outputs, but its core role is to handle them within workflows where changes are verified. For example, if an agent changes a timing parameter, the change may pass tests, but a lower-level controller tuned to receive signals at the original rate may misinterpret signals.

To prevent such failures, agentic AI must operate in workflows that separate generation from execution. Agentic AI further raises the level of automation in existing model-based design workflows, including simulation, code generation, analysis and testing. These workflows can be implemented through tools such as the Simulink Agentic Toolkit. Defining what is executable and verifiable is the role of system-first engineering, and deciding when and how to apply automated steps is the role of agentic AI.

Applying system-first engineering and introducing agentic AI into engineering workflows happens gradually. Engineers typically start with a single function and a single team, verify the system model with automated simulation and test workflows, and then expand to adjacent teams. Agentic AI generates and evaluates changes, and the verification stage checks that changes were tested before moving to the next stage. Many teams set up a common starting environment that provides the same CI pipeline and verification guardrails to all teams to prevent fragmented adoption in which only a specific group runs models and there is no CI or cross-domain use.

This common environment ensures agentic AI is not introduced in isolation from the outset and operates within a consistent system-level context. As a result, it can scale reliably without fragmenting workflows and can establish the foundation needed to build trust in AI-driven execution.

Scaling software-defined product development fails when teams treat only a single domain as the system rather than engineering as a whole. Trust in agentic AI also collapses under the same condition. Building robust products has always required uncompromising principles, and those principles now apply equally to agentic AI.

If it is not an executable model, it is only an opinion. Code can define the behaviour of individual components, but an executable system model provides a common baseline that shows how components interact at the system level. If a change fails verification in the overall system model, that change is not applied. Even a change that passes local tests can cause regression in other parts of the system.

If behaviour has not been verified through simulations that reflect real operating conditions, it is only a guess. As a result, potential failure modes are discovered late, only at the hardware test or field operation stage, and rework at that point incurs enormous costs.

If requirements are not linked to models, tests and data, alignment with system implementation breaks down. It becomes difficult to trace which models or tests verify each requirement, and implemented behaviour can drift from the original intent. If an agent-led change cannot be verified through an executable model, deterministic analysis and human review, it should not be accepted. Without this verification, the review and approval process cannot explain what changed, why it changed and whether it meets system-level requirements.

Engineers retain control over agent-led changes, review results based on system-level evidence and give final approval. If problems occur, they are traced, diagnosed and fixed through models and tests, as with any other engineering issue, before moving to the next step. Teams that engineer systems with the right tools, processes and practices can introduce agentic AI while delivering trustworthy software-defined products without losing system-level confidence. Teams that do not may face the limits of agentic behaviour at the integration, certification or recall stage.