How to secure and apply rules to all your locally running LLMs

Running a language model on your own computer or server is one of the best things you can do for privacy. Prompts, documents and customer data stay with you. That is why we build local AI for clients, and why our own tools like LOKIDocs work offline.

But "local" does not mean "offline". The moment you give a local model tools, such as web search, a browser, a terminal or an MCP server, it starts talking to the internet. From then on one question matters more than which model you run: which addresses may your AI reach, and which not?

This guide shows how anyone can answer that with their own rules: from a 15-minute setup on one computer to a gateway for a whole team.

Who actually goes online?

The model itself does not. Ollama, LM Studio or llama.cpp run the model and stay quiet, apart from downloading a model when you ask for one. The traffic comes from everything around the model:

• the agent that gives the model its tools (Claude Code, Open WebUI, your own Python or Node agent)
• web fetch and search tools
• MCP servers
• package managers like pip and npm, which an agent with a terminal will happily run
• a plain curl in a shell

That is the traffic to control. And it is a network question, not a model question: no prompt and no system message reliably stops an agent from calling an address. A firewall does.

The one limit to know: HTTPS hides the URL

Almost all traffic is encrypted. On the network you see the domain (github.com), but not the full address (github.com/myorg/project), the method or the content.

That gives two levels of rules:

• Domain rules ("only pypi.org, github.com and our own server"): simple, no decryption, enough for most people.
• URL and content rules ("only our own GitHub organisation", "never send a private key"): the gateway has to decrypt the traffic. For that you create your own small certificate authority (CA), and the AI tools trust it.

If you go for the second level, two things to know up front:

• Node, Python and Go often ignore the system's certificate store. Point them to your CA with NODE_EXTRA_CA_CERTS, SSL_CERT_FILE and REQUESTS_CA_BUNDLE.
• Apps that pin their certificates refuse decrypted connections. Keep a short list of them that passes through on domain level only.

Four ways to catch the traffic

Approach	How strong	Effort
HTTP_PROXY environment variables	Weak: any tool can ignore them	Minutes
VPN for the whole device	Catches everything, your own browsing too	Low
Filter per app (e.g. a macOS Network Extension)	Strong, only the AI processes	High: needs Apple signing and entitlements
AI in a container, VM or separate user whose only way out is the proxy or tunnel	The strongest, and the easiest to reason about	Medium

We recommend the last row. The idea: don't ask the AI to use the proxy. Give it a place to run where the proxy is the only exit.

Level 1: a filter in 15 minutes with Docker

For one computer and domain rules. All you need is Docker.

Two networks: an internal one with no route to the internet, where the agent lives, and a normal one. Only the proxy sits in both.

# docker-compose.yml
services:
  proxy:
    image: ubuntu/squid
    volumes:
      - ./squid.conf:/etc/squid/squid.conf:ro
    networks: [ai-internal, internet]

  agent:
    image: python:3.12-slim      # or the image of your agent
    command: sleep infinity
    environment:
      HTTP_PROXY: http://proxy:3128
      HTTPS_PROXY: http://proxy:3128
      NO_PROXY: localhost,ollama
    networks: [ai-internal]

networks:
  ai-internal:
    internal: true               # no route to the internet
  internet: {}

And the allowlist:

# squid.conf
http_port 3128
acl allowed dstdomain .pypi.org .files.pythonhosted.org .github.com
http_access allow allowed
http_access deny all

Tools that respect the proxy now reach only the listed domains. Tools that ignore it reach nothing at all, because the internal network has no way out. Squid also logs every request, so you can see what your agent tried.

If your model runs in Ollama, put an Ollama container into the internal network too. Download the models before you lock it, or add registry.ollama.ai to the list.

Level 2: a VM, a tunnel and a kill switch

When the agent should run as a normal program, with a terminal and its own files, give it a virtual machine (UTM, Parallels, VirtualBox) or a small Linux box. The VM sends everything through a WireGuard tunnel to your gateway:

# /etc/wireguard/wg0.conf in the VM
[Interface]
PrivateKey = <private key of the VM>
Address = 10.66.0.2/32
DNS = 10.66.0.1                  # DNS only through the gateway

[Peer]
PublicKey = <public key of the gateway>
Endpoint = gateway.example.com:51820
AllowedIPs = 0.0.0.0/0, ::/0     # everything, IPv6 included

The important part is the kill switch. An agent with terminal access could simply switch the tunnel off. So the firewall drops everything from the agent's user that does not go through the tunnel:

# Kill switch for the user "aiagent": loopback and the tunnel only
iptables  -A OUTPUT -o lo  -m owner --uid-owner aiagent -j ACCEPT
iptables  -A OUTPUT -o wg0 -m owner --uid-owner aiagent -j ACCEPT
iptables  -A OUTPUT        -m owner --uid-owner aiagent -j REJECT
ip6tables -A OUTPUT -o lo  -m owner --uid-owner aiagent -j ACCEPT
ip6tables -A OUTPUT -o wg0 -m owner --uid-owner aiagent -j ACCEPT
ip6tables -A OUTPUT        -m owner --uid-owner aiagent -j REJECT

Two rules make this hold:

• The agent runs as aiagent without admin rights. Anyone with root can also delete firewall rules. If the agent needs root, put the kill switch on the host instead of in the VM.
• DNS and IPv6 go through the tunnel too (DNS = and ::/0 above). Otherwise they become a side door.

On a Mac without a VM, the same works with a separate macOS user and the built-in pf firewall, which can filter by user as well.

Level 3: your own gateway with URL rules, logs and key injection

Now the tunnel ends at a gateway you control: a small VPS, a server in your office or a home lab. It decrypts, checks every request against your rules and logs the result.

[AI sandbox] ──WireGuard──▶ [Gateway server]
 kill switch                      │
 trusts your CA                   ▼
                       Proxy (decrypts TLS)
                                  │
                       Policy (YAML / Python)
                                  │
                 allowed → internet │ denied → 403 + log
                                  │
                         Audit log, dashboard

Why a separate server is worth it:

• Tamper-proof. Rules and logs live where the agent cannot change them, even if it gets root on its own machine.
• Credential injection. The gateway adds the real API keys on the way out. The agent holds only dummies, so there is nothing to steal. This is probably the single biggest security gain.
• One policy for several machines, one fixed exit IP and one audit trail.

The quickest start is mitmproxy, which has a WireGuard mode built in: start it, scan or copy the WireGuard configuration it shows, connect the VM, done. Install its CA (~/.mitmproxy/mitmproxy-ca-cert.pem) in the VM and set the three variables from above. Your rules are a short Python file:

# policy.py, start with:  mitmdump --mode wireguard -s policy.py
import os, re
from mitmproxy import http

ALLOW = {                                  # host -> allowed path prefixes
    "pypi.org": ["/"],
    "files.pythonhosted.org": ["/"],
    "docs.python.org": ["/"],
    "api.github.com": ["/repos/myorg/"],   # only your own organisation
}
READ_ONLY = {"docs.python.org"}
SECRETS = re.compile(rb"-----BEGIN [A-Z ]*PRIVATE KEY-----|AKIA[0-9A-Z]{16}")

def request(flow: http.HTTPFlow) -> None:
    req = flow.request
    paths = ALLOW.get(req.pretty_host)
    if paths is None or not any(req.path.startswith(p) for p in paths):
        return block(flow, "host or path not allowed")
    if req.pretty_host in READ_ONLY and req.method != "GET":
        return block(flow, "read-only site")
    if req.content and SECRETS.search(req.content):
        return block(flow, "secret in request body")
    if req.pretty_host == "api.github.com":
        # the agent only has a dummy token; the real one is added here
        req.headers["Authorization"] = f"Bearer {os.environ['GITHUB_TOKEN']}"

def block(flow: http.HTTPFlow, reason: str) -> None:
    flow.response = http.Response.make(
        403, f"Blocked by AI policy: {reason}".encode(), {"Content-Type": "text/plain"})

Later you can replace mitmproxy with a small Go proxy as one binary, and the Python rules with a YAML file or OPA if your rules grow.

Rules beyond allow and deny

Once the gateway sees whole requests, much more becomes possible:

• Method and path: only GET on documentation sites, no POST to unknown hosts.
• Data-leak checks: block request bodies with secrets, .env contents or private keys.
• LLM API control: which models are allowed, token or cost budgets, a log of prompts and answers.
• Rate limits and an approval mode: an unknown domain waits until you approve it.
• Response checks: flag fetched pages that contain prompt-injection patterns.

Close the side doors

A filter is only as good as its weakest route. Check these:

• DNS: DNS over HTTPS and direct connections to IP addresses. Force all DNS through your gateway.
• QUIC / HTTP/3 on UDP port 443: block it, clients then fall back to TCP.
• IPv6: filter it like IPv4, or switch it off.
• WebSockets and other long-lived connections.
• Leaks through allowed domains: pastebins, gists, cloud storage. An allowlist alone does not stop that, only content checks do.

Which level is right for you?

Your situation	Start with
You are trying an agent on your laptop	Level 1
The agent has a terminal or works with client data	Level 2
Several machines or a team, or you need an audit trail	Level 3

Start small. Level 1 takes an afternoon and already shows you what your AI wants to reach. That list is the best starting point for any stricter rule.

Need a hand?

We set up local AI for small and medium businesses: models running on your own hardware, assistants that work with your data, and rules for where that AI may go. If you want local AI with clear limits, tell us what you'd like to run, or read more about our AI integration.