Running AI models on your own infrastructure gives you more control over costs, data privacy and performance. Tools such as Ollama and Ray make it possible to serve models from your own servers instead of relying entirely on external AI providers. But taking control of the infrastructure also means taking responsibility for securing it.
Self-hosted AI is already closely tied to mainstream cloud infrastructure. According to the Cloud Native Computing Foundation’s latest Cloud Native Survey, 66% of organizations hosting generative AI models use Kubernetes to manage some or all of their inference workloads.
That means AI teams are managing production servers, GPU nodes and clusters that can contain valuable model files, sensitive data and significant computing resources.
Protecting a self-hosted AI model therefore means protecting the server behind it while that server is live. This requires runtime security, which monitors activity inside workloads and detects or stops attacks as they happen. Some runtime tools can now recognize model-serving workloads specifically. Wiz Sensor, for instance, identifies servers running tools such as Ollama and Ray and detects attacks against them, and can stop them.
What Changes When You Host Your Own Models
Using a hosted AI service generally leaves the underlying model-serving infrastructure to the provider. When you deploy a model yourself, that infrastructure becomes your responsibility.
A typical deployment packages the model in a container or virtual machine, often with Kubernetes, the open-source system that schedules containers across a cluster of servers. Servers with GPUs do the heavy computation, storage holds the model weights (the trained model itself) and its data, and an inference endpoint, the address applications send requests to, returns the model’s answers.
The GPU nodes represent valuable computing capacity that attackers could misuse. Model weights and fine-tuning datasets may contain proprietary intellectual property. And the credentials the model server uses can often reach other systems in the same cloud account.
Inference endpoints deserve particular attention. Teams frequently create them during development to test models, connect applications or evaluate performance. An endpoint configured for easy access during experimentation can become a liability if it remains publicly reachable.
Even organizations using a unified multi-LLM gateway to route requests between hosted and self-hosted models still need to secure the infrastructure behind their locally deployed models. A gateway may centralize model access, authentication and traffic management, but it does not monitor the operating system beneath the inference service.
Ultimately, a self-hosted model is another production workload, with many of the same risks as a web application or database server. Its computing resources and stored information can make a compromise particularly costly.
How Attackers Go After AI Model Servers
Attackers targeting AI infrastructure can take advantage of familiar server weaknesses, including exposed services, vulnerable software and compromised credentials.
Often, the trouble starts with an inference endpoint nobody meant to leave open to the internet. A developer might temporarily expose a model server to simplify testing or connect a remote application. If the endpoint lacks appropriate authentication and network restrictions, unauthorized users will be able to interact directly with the service.
Vulnerabilities in model-serving software create another path into the environment.
In June 2024, Wiz Research disclosed a remote code execution vulnerability in Ollama, tracked as CVE-2024-37032 and nicknamed “Probllama.” The flaw could allow an attacker to execute code on a vulnerable server, particularly in Docker deployments, where Ollama’s API server ran as root and listened on all network interfaces by default. Ollama subsequently addressed the issue, with users advised to upgrade to version 0.1.34 or later.
Once attackers gain execution access, they may attempt to install cryptomining software to exploit expensive computing resources. They might establish a reverse shell, allowing them to issue commands remotely, or attempt to escape a compromised container and reach the host operating system.
Other objectives include stealing model weights, extracting sensitive data or using credentials available to the compromised workload to access additional cloud resources.
Why Model-Level Protections Leave the Server Uncovered
Many AI security controls concentrate on a model’s inputs and outputs. They inspect prompts, filter responses and apply policies to how an AI application handles requests. These protections are useful for managing model behavior and application-level risks. However, they provide limited visibility into the infrastructure executing the model.
A prompt-filtering system cannot generally tell whether an unexpected process has started inside a container, whether a server is connecting to an unfamiliar external address or whether executable files have been modified.
An attacker who exploits a vulnerability in the inference service may bypass the model’s normal input-processing path entirely. Someone using stolen infrastructure credentials could potentially access the host without interacting with the model at all.
The distinction matters because a model can continue responding normally while malicious activity takes place on its server.
What Runtime Protection Looks Like on an AI Server
Runtime security tools monitor workloads while they are executing, providing visibility into activity that configuration checks and periodic vulnerability scans may miss.
A lightweight sensor installed on a server can observe process execution, network connections and file activity. Security policies use this information to identify suspicious behavior and, where blocking is enabled, prevent certain malicious actions from continuing.
For AI infrastructure, relevant detections include unexpected processes running inside model-serving containers, malware, remote code execution, reverse shells and attempts to escape container boundaries.
Runtime visibility also helps teams prioritize vulnerabilities. A software library identified in a container image might never actually be used by the running application. Identifying vulnerable libraries loaded into memory provides additional evidence about which weaknesses deserve attention.
Wiz Sensor uses this approach. The agent-based sensor monitors runtime activity on workloads, including servers running AI-serving tools such as Ollama and Ray. On supported Linux workloads, it uses eBPF to observe activity at the operating-system level.
When response policies are set to block, it can terminate malicious processes such as cryptominers and reverse shells, containing an attack that began through vulnerable serving software.
The broader cloud context is also important. Wiz combines runtime observations with information about the surrounding infrastructure, including exposed endpoints, workload identities and accessible resources.
For example, a suspicious process on a model server becomes more significant if that server is publicly accessible and its cloud identity has permissions to reach sensitive storage. Connecting those details helps security teams understand the potential impact of an incident.
Separately, Wiz’s cloud detection capabilities analyze cloud audit events for suspicious activity, including unauthorized downloads from cloud storage. These cloud-level detections complement the process and network activity observed by the runtime sensor.
This combination illustrates what runtime-capable cloud-native application protection platforms, or CNAPPs, can provide. A CNAPP brings together cloud security capabilities, and those equipped with workload sensors can extend protection to attacks occurring on running AI infrastructure.
Where to Start
AI teams do not need an entirely separate infrastructure security strategy for every model they deploy. Most of the foundational practices already apply to other production services.
A practical starting point includes:
- Inventory model servers and endpoints. Include development environments, temporary test deployments and GPU nodes that may fall outside existing asset inventories.
- Restrict public access. Keep inference endpoints private wherever possible and enforce authentication and network controls when external access is necessary.
- Patch serving software. Monitor security updates for tools such as Ollama and Ray, and prioritize known vulnerabilities affecting deployed versions.
- Monitor running workloads. Deploy runtime protection across model servers and GPU nodes, including the containers hosting inference services.
- Validate response policies. Test automated blocking against legitimate AI workloads before enabling enforcement in production.
Conclusion
Running your own AI models offers meaningful control over how data is processed, where workloads operate and how computing resources are allocated. That flexibility also introduces infrastructure security responsibilities that can be easy to overlook when the team’s attention is focused on model performance.
Self-hosted AI infrastructure needs runtime security just like any other production workload. Some CNAPPs, including Wiz through Wiz Sensor, provide protection for the servers running AI models.
As self-hosted deployments expand, runtime monitoring belongs in the standard model deployment process.
Frequently Asked Questions
Do self-hosted AI models need runtime security?
Yes. A self-hosted model runs on production infrastructure that can be targeted through vulnerable software, exposed services or stolen credentials. Runtime security monitors activity on that infrastructure while the workload is executing. It can identify suspicious processes, network connections and other behaviors associated with attacks, including cryptomining and remote code execution. Depending on the tool and configured policies, it can also stop malicious processes before they cause further damage.
What are the security risks of running Ollama or Ray on your own servers?
Key risks include publicly exposed inference endpoints, vulnerabilities in model-serving software, compromised credentials and unauthorized access to model files or associated data. The 2024 Probllama vulnerability in Ollama demonstrated how flaws in serving software could enable remote code execution. Attackers who compromise these environments may also attempt to misuse GPU resources for cryptomining, establish persistent remote access or move from a container into other parts of the infrastructure.
Does a CNAPP cover runtime threats on AI infrastructure?
Some CNAPPs provide runtime threat detection and response through sensors deployed on workloads. The agent-based Wiz Sensor monitors running workloads and detects or stops attacks, while Wiz’s agentless cloud scanning maps the exposure, identities and relationships around them. Both feed the same Wiz Security Graph, so runtime detections arrive with that cloud context. CNAPP capabilities vary, so teams should verify that a platform supports runtime monitoring and response on their specific AI infrastructure.
How is securing an AI model different from securing the server it runs on?
Model security focuses on how an AI system processes inputs, generates responses and behaves when interacting with users or applications. Server security focuses on the infrastructure executing that model, including operating-system processes, network connections, file activity and access permissions. Model-level controls can help manage unsafe or unauthorized interactions, while runtime protection detects attacks affecting the underlying workload. Organizations operating self-hosted models need both approaches to protect the complete deployment.

Sandeep Kumar is the Founder & CEO of Aitude, a leading AI tools, research, and tutorial platform dedicated to empowering learners, researchers, and innovators. Under his leadership, Aitude has become a go-to resource for those seeking the latest in artificial intelligence, machine learning, computer vision, and development strategies.





