HPC AI Workloads Need Runtime Security. The Architecture Already Exists.

The US Federal Government is committing $600 million to build one of the world’s most advanced AI infrastructure systems. Executive Order 14363, the Genesis Mission, connects national laboratory supercomputers across nuclear simulation, biodefense, energy grid modeling, and every major scientific domain. Fifty-one organizations signed on, including NVIDIA, OpenAI, IBM, Microsoft, AWS, Google, and Oracle.

The security framework governing these workloads was not written for this scale of use.

NIST SP 800-234, the High-Performance Computing Security Overlay, is well-constructed, tailoring 60 controls across four security zones, building on the SP 800-53B moderate baseline. It was designed for deterministic HPC workloads such as climate simulations, finite element analysis, and computational fluid dynamics. These workloads share a common attribute: code that runs the same way, every time, and behaves predictably under well-understood inputs. The security controls governing those workloads assume you can scan at the perimeter, clear memory between jobs, and attest to integrity at load time.

AI workloads break every one of those assumptions.

SentinelOne has submitted a formal proposal to the NIST HPC Security Working Group regarding this gap, and NIST has acknowledged it. We have a post on LinkedIn to share our proposal, and welcome commentary from across the industry.

The supply chain problem just got a lot more dangerous

This spring, in just three weeks, three AI-driven supply chain attacks targeted widely deployed software: LiteLLM, the most-used AI infrastructure package in Python development environments, Axios, the most-downloaded HTTP client in the JavaScript ecosystem, and CPU-Z, a trusted system diagnostic tool with a legitimate signed binary from the official vendor domain.

SentinelOne stopped all three on the same day each attack launched, with no prior knowledge of any payload.

The most important aspect of this outcome is how these attacks were stopped, and why signature-based detection couldn’t work. Each attack arrived through a trusted delivery channel. LiteLLM was compromised after credentials were stolen via Trivy, a security scanner. The attacker published two malicious versions to the PyPI repository. In at least one confirmed case, an AI coding agent with unrestricted permissions auto-updated to the infected version, meaning there was no human review or approval step before the payload ran. The Axios attacker exploited a legacy access token that the project maintainers had forgotten to revoke, bypassing every npm security control. CPU-Z attackers targeted the vendor’s distribution infrastructure directly; anyone who downloaded from the official website received a properly signed binary containing a payload. In all three cases, while the authorization chain was legitimate, the intent was not.

This is the defining characteristic of modern supply chain attacks: the workflow is verified, but the intent has been subverted. Every perimeter control, signature library, and reputation lookup checks authorization and passes. These attacks were designed to exploit that gap, and they ran at machine speed through automated pipelines with no human checkpoint.

To put this into the context of HPC and AI workloads running at scale, a compromised Python package in a developer’s environment is a serious incident; a poisoned training pipeline on classified biodefense data on a national laboratory supercomputer is on a different order of magnitude. The model it produces may be correct 99.9 percent of the time and adversarially wrong under precisely targeted conditions. No perimeter scan, signature check, or load-time integrity verification will catch it after training completes.

Where the current framework falls short

Of the 60 controls SP 800-234 tailors, three bear directly on AI workload protection, and each carries a documented gap. In a fourth area, supply chain, the overlay does not tailor at all.

• SI-3 (Malware scanning): The control acknowledges that real-time scanning is most effective but explicitly permits tailoring for performance on HPC systems, deferring to perimeter scanning before data reaches the compute zone. For traditional HPC workloads, that tradeoff may be defensible, but for AI workloads, it leaves behavioral analysis of the execution process completely unaddressed. A poisoned training run that executes within the expected statistical range of a training job looks like legitimate compute to a perimeter scanner.
• SI-4 (System monitoring): The control notes that high-speed data flows in HPC environments can overwhelm standard monitoring tools, and lacks AI-specific monitoring requirements or telemetry collection requirements from execution pipelines. The practical interpretation of this is: monitor what you can, accept the gap for what you can’t. On infrastructure running AI at scale, that gap creates a primary attack surface.
• SC-4 (Information in shared resources): Requires GPU memory clearing between user reassignments. It addresses data residency at the transition but does not address runtime behavioral monitoring of workloads during execution, side-channel attack detection, or anomalous compute-pattern identification while training is active.
• SR family (Supply chain risk management): The overlay carries all 12 moderate-baseline SR controls forward from SP 800-53B, with no HPC or AI-specific guidance, and supply chain is not among the 14 categories it tailors to. The SR controls still address only the conventional software and hardware supply chain; they say nothing about training-data provenance, model-weight integrity, or pre-trained-model validation, and the framework defines no AI equivalent of a software bill of materials. LiteLLM, Axios, and CPU-Z all arrived through legitimate software supply chain channels. AI workloads carry that same exposure one layer deeper, in the data and model artifacts that software trains on, which is exactly where the overlay is silent

AI Runtime Threats

The attacks against AI workloads on HPC are not theoretical, and they are not detectable at the perimeter.

• Training data poisoning scales at rates most security teams are not equipped to respond to. Research¹ across 41 studies documents attack success rates exceeding 60 percent from manipulation of 100 to 500 training samples, a fraction of a percent of a typical dataset. Poisoning as little as 3 percent² of training data achieved 41 percent attack success rates in code-generating models. OWASP’s LLM Top 10³ documents the consequence. Backdoors leave model behavior intact until a specific trigger activates adversarial outputs. The model ships, it gets deployed, and operates correctly, until it doesn’t. No post-training audit reliably catches a well-designed poisoning attack.
• GPU side-channel attacks are executed remotely by a co-tenant workload on shared GPU infrastructure; no physical access is required. The NVBleed research demonstrated covert channel attacks on NVIDIA NVLink, achieving over 91 percent accuracy in recovering data-dependent information from co-tenant GPU workloads on a shared fabric. The BarraCUDA research demonstrated the extraction of neural network weights via electromagnetic side channels from NVIDIA hardware. Both attack classes execute during active training, not at job transition. If your HPC environment runs multiple projects or security classifications on shared accelerators, the co-tenancy model is an active attack surface today.
• Inference pipeline compromise survives load-time integrity checks. A model with clean weights at deployment faces attacks through three vectors: hot-swap modification of serving configurations while inference runs; preprocessing and postprocessing layer injection that alters inputs before they reach the model or modifies outputs before delivery; and adversarial input manipulation that triggers targeted misbehavior in a model that appears fully operational. For AI serving safety-critical inference, each is a security risk, not just a research concern.

The characteristic that makes AI workloads uniquely difficult is persistence. A compromised simulation may produce visibly wrong results, but a compromised model can produce correct results the overwhelming majority of the time and adversarially wrong results under precisely

Key Takeaways

• HPC AI workloads require runtime security for protection.
• Existing architecture can support these security needs.
• US government invests $600 million in AI infrastructure.