GDPR-Compliant AI Agents with Local LLMs: llama.cpp, Ollama & LM Studio

Why your AI agents don’t have to go to the cloud

Autonomous AI agents search documents, answer customer inquiries, generate code and coordinate workflows across several steps. Along the way they almost always process personal data. For companies in Germany, Austria and Switzerland, one question comes before every other: how do we deploy agents without violating the GDPR (DSGVO in German)?

Cloud APIs like those from OpenAI, Anthropic or Google send the data to US servers. That means data processing agreements, third-country transfer assessments and, in the worst case, the risk of fines. Data protection authorities in Austria, France and Italy, among others, have already ruled Google Analytics non-compliant, and AI APIs may be next.

The alternative is a local setup: models run on your hardware, data never leaves your network, and you need no processing agreement. Our stack uses llama.cpp as the engine, Qwen 3.6 or 3.8 as the model, Unsloth for fine-tuning and the web interface that llama-server ships with. Further down we place Ollama, LM Studio and Unsloth Studio, with a focus on privacy and security.

For many agent workloads, local models cost less and carry less legal risk than cloud APIs.

What the GDPR requires of AI agents

The legal obligations come first. For full advice, talk to your data protection officer.

Art. 6 GDPR: legal basis

Every processing of personal data needs a lawful ground. When your agent reads email addresses, names or contract data, you must document why it may do so. Depending on the case, that is performance of a contract, legitimate interest or consent.

Art. 5(1)(c): data minimization

Only what the specific request needs belongs in the context window. An agent that loads all customer data to answer a single question violates this principle. The fix is to filter by context before anything goes to the model.

Art. 35: data protection impact assessment

New technologies with a high risk for data subjects trigger a DPIA. Agents that build profiles, decide automatically or use new technology almost always fall under it. A DPIA documents the risks and how you manage them. It does not block the project.

Art. 17: deletion in the agent’s memory

The right to erasure applies not only to databases but also to vector stores, agent logs and conversation histories. If a customer requests deletion, it must reach all agent components.

Art. 22: automated decisions

If an agent makes decisions with legal effect without human intervention, stricter requirements apply: exceptions, safeguards and a path to human review.

The paperwork stays manageable when you plan for it in the architecture from the start, and local models cover part of it because data sovereignty is built into the setup.

If you want to tick off a few basics before launching a website, our GDPR checklist before launch helps. It lists a handful of items, each with a check step.

llama.cpp: the engine for local models

is an inference engine in C and C++ that runs open-source models in GGUF format on CPU and GPU. Ollama and LM Studio build on llama.cpp or on its library ggml.

For the GDPR, the main point is that the model runs on your hardware, so no external API call happens at all. The code is also open source and therefore auditable, and deployment stays flexible, whether Docker, Kubernetes or bare metal.

The quick start in three steps

1. Build llama.cpp

One clone and two cmake commands are enough.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

2. Load a model and test it

llama-cli starts a model directly in the console. Enter the repository you want to load a quantized GGUF model from, for example a Qwen 3.x published by Unsloth.

./build/bin/llama-cli -hf <your-gguf-repo> -ngl 99 -p "What is the most important GDPR principle?"

3. Start the server

llama-server makes the model available to agents and to the web interface. We bind it to 127.0.0.1 and protect it with an API key. If you want to open the server to other machines, put a reverse proxy with TLS and authentication in front of it instead of exposing it directly on 0.0.0.0.

./build/bin/llama-server -hf <your-gguf-repo> --port 8080 --host 127.0.0.1 --api-key <long-random-value>

The web interface of llama-server

Then you open http://localhost:8080 in your browser and land in the built-in interface of llama-server. You can chat right away without installing a single tool. The interface lets you switch models and load new GGUF models from the Hugging Face Hub or upload your own. Besides the chat, there are views for testing chat and completion requests as well as the tokenizer.

The same server also exposes the OpenAI-compatible endpoints. You use the browser interface for quick tests, and your agents call http://localhost:8080/v1/chat/completions on the same address, so one setup covers both jobs.

llama-server ships with its web interface, and for day-to-day work with a single agent that is often enough.

Qwen 3.6/3.8 with Unsloth: our recommendation

Not every model handles German. For agents that process German text, we use Qwen 3.6 or 3.8 combined with Unsloth. Unsloth is an open-source library that speeds up fine-tuning and quantization. In a reasonable amount of time, a base model turns into a fine-tuned, quantized GGUF for the llama server. With Unsloth Studio there is now also a web interface for this, which we place in the comparison below.

A customized model fits on hardware that sits next to a desk, because Unsloth also lowers memory requirements. Fine-tuning on your own data usually beats even a long prompt chain, and Qwen 3.6 and 3.8 are a strong base for German text. After fine-tuning, Unsloth exports the model to GGUF, the format llama.cpp reads, which keeps the path from customization to production short.

Our default for German agents is Qwen 3.6/3.8 with Unsloth. If your use case stays small, a more compact German model is enough.

What is Ollama? Local AI with one command

Ollama is an open-source tool under the MIT license that downloads language models and runs them on your own machine, on macOS, Windows and Linux. One command is enough to load a model and chat in the console:

ollama run <model>

It adds model management, a background service and a local REST API that other programs use to talk to the model. Running Ollama locally gets you a working local AI in a few minutes, which is why many people meet the topic through Ollama.

Technically, Ollama builds on ggml, the library behind llama.cpp, and reads models in GGUF format. You can therefore use the same quantized models as with llama.cpp, including your own GGUF files, for example from an Unsloth fine-tune. The difference is in operation. Ollama handles the build, model search and server start for you, while llama.cpp gives you more settings and a built-in web interface.

The project publishes release news and new models on the Ollama blog. This article covers what matters for operation under the GDPR.

Ollama privacy: local and cloud

Whether Ollama complies with privacy law depends on the mode. Ollama offers local models and hosted cloud models in one product, and their risk profiles differ.

Ollama local: no content reaches the vendor

According to the privacy policy of Ollama Inc. (as of March 2026), the vendor does not collect or store prompts, responses or other content you process when you run locally. Limited device and usage data such as the app version and the number of requests may be collected, but no content. For the GDPR that means: with local operation, personal content does not flow to a service provider, and a processing agreement for the inference does not apply.

Your obligations stay the same: legal basis, data minimization and deletion in logs and conversation histories apply to local agents too.

Ollama Cloud: privacy with a US connection

With Ollama’s cloud models, processing runs on third-party infrastructure. By its own account, Ollama does not log prompts and responses there and does not use them for training. According to Ollama, its hosting partners are NVIDIA Cloud Providers, and they must guarantee no logging, no training and zero data retention. Hosting is mostly in the US and, when capacity requires it, also in Europe and Singapore.

From a GDPR perspective, this is still commissioned processing by a US provider. You need a data processing agreement under Art. 28 and an assessment of the third-country transfer, and Ollama may route requests between regions. A promise on a web page does not replace a contract, so for agents with personal data we stay with local models.

Checklist: running Ollama in line with the GDPR

• Use only local models as long as personal data is involved, and check before you start that the chosen model runs locally.
• Keep Ollama bound to 127.0.0.1 or run it behind a reverse proxy with authentication and TLS.
• Include conversation histories and agent logs in your deletion concept (Art. 17).
• Keep Ollama up to date so that known security flaws are closed.
• Document the use in the record of processing activities and, if needed, in the DPIA.

Is Ollama safe?

Run locally, Ollama is a usable tool, and the risk comes from configuration. By default Ollama binds to 127.0.0.1, but the REST API has no built-in authentication. Many guides for Docker and home labs say OLLAMA_HOST=0.0.0.0, and with that the API is open to anyone who can reach the machine.

In January 2026, security researchers counted around 175,000 publicly reachable Ollama instances in 130 countries. In May 2026, the flaw CVE-2026-7482 (“Bleeding Llama”, CVSS 9.1), reported by Cyera Research, made headlines. Through the GGUF loader, an attacker without a login could read the memory of a reachable server, including prompts, conversations, API keys and environment variables. Cyera estimates the affected servers at around 300,000. The fix is in Ollama 0.17.1 from February 25, 2026, but the release notes did not label it as a security update. Many operators therefore never realized they should update.

With personal data in the mix, this goes beyond IT: a leak of conversations is a data breach that you generally must report to the authority within 72 hours under Art. 33, and Art. 32 requires appropriate technical safeguards. Researchers also reported two more flaws in Ollama’s Windows updater in May 2026 (CVE-2026-42248 and CVE-2026-42249), for which no fix was available at that time. Check the current status in the release notes.

What helps:

• Update Ollama to the current version and check how automatic updates behave.
• Do not put the API on the internet or in the guest Wi-Fi.
• Put a reverse proxy with authentication and TLS in front and restrict the port to known clients with a firewall.
• Log access without storing prompt contents in the logs.

The same holds for every local inference server, llama-server included, where --api-key protects access, as in the quick start above.

llama.cpp, Ollama, LM Studio and Unsloth Studio compared

All four tools run models locally and send no prompts to a vendor as long as you use local models. They differ in license, operation, server use and whether you can train models too.

llama.cpp is the base the others build on. It gives you the most settings and a small attack surface, at the cost of more manual work during setup.

Ollama is the fastest way to a running model. The trade-offs are an API without authentication and cloud models sitting next to the local ones, which you should avoid for personal data.

LM Studio has the most convenient interface and has been free for business use since July 2025. Its source code is closed, which can rule it out for companies that must review source code. For servers without a screen, llmster is a separate daemon.

Unsloth Studio is the only one of the four that also covers training, and it exports to GGUF. The interface is under AGPL-3.0. For internal use that is usually fine, but if you modify it and offer it to third parties as a service, check the license (this is not legal advice).

Our recommendation: train with Unsloth, run with llama.cpp

We combine the tools according to their strengths. Fine-tuning of Qwen 3.6/3.8 runs with Unsloth, the export lands as GGUF in llama-server, and the agents talk to its OpenAI-compatible API. We use Ollama for quick tests on the developer laptop, and we recommend LM Studio to teams that want to start without a terminal. The models stay interchangeable between all four tools, because they all read GGUF.

The agent frameworks at a glance: Pi, Hermes, OpenClaw, Atomic Agents

An agent needs a framework that ties the model to tools such as the file system, API calls and database access. Four are worth a look.

Pi (TypeScript)

A lean agent for the command line. Pi gives agents tools to read, write, edit and execute bash, with a vendor-neutral LLM layer. For developer workflows it is the easiest entry point. You connect Pi to a local model through the llama server, with no cloud involved.

Hermes Agent (Python/TypeScript)

A cloud-oriented autonomous agent with persistent, multi-layer memory and continuous self-learning. It integrates Slack and Teams. For local operation you need more configuration, and you should keep an eye on the memory component for deletion requests, so that a deletion also reaches the agent’s memory.

OpenClaw Foundation (TypeScript)

By far the largest community project, with a complete sandbox for autonomous agents. After the security crisis in early 2026, the code is better audited. Use versions from 2026.3 on and disable telemetry. The Docker sandbox lowers the risk, and the rest is standard hygiene.

Atomic Agents (Python)

Each agent has one responsibility, with typed inputs and outputs. The framework is type-safe through Pydantic, which also makes it easy to test and reuse. You build the agents yourself and keep control over the data flows. It takes more effort but gives you clear structure, which matters with sensitive data.

Three examples from practice

Customer service without giving up data

An agent answers customer inquiries by email without personal data ever seeing a cloud API.

[Customer email] -> [PII filter] -> [local LLM (Qwen 3.6/3.8)]
   -> [Draft reply] -> [Human review] -> [Send]

This includes PII detection before the model input, logs that are deleted after 30 days, and a deletion path that also covers the agent logs.

Searching contracts instead of skimming them

An agent reads contract PDFs, finds termination clauses and summarizes them. OCR runs locally, and so does the vector database.

[Contract PDF] -> [Local OCR] -> [Local RAG vector database]
   -> [Local LLM] -> [Summary]

Deletion requests also delete the vectors here, and the procedure is documented in the DPIA.

Code reviews that stay in your own network

An agent checks pull requests for security flaws and coding standards without any code leaving the network.

[Git repo] -> [Pi with local LLM] -> [Code analysis]
   -> [Review comments] -> [Pull request]

The agent gets access only to the repo, not to customer data. Logs are kept for 14 days.

Since August 2026: AI content must be labeled

Since August 2, 2026, the transparency obligations of the EU AI Act (Regulation 2024/1689, Article 50) apply. Synthetically generated audio, image, video or text content must be marked as AI-generated in a machine-readable way when it is put into circulation. Chatbots must disclose that they are AI, and deepfakes must be recognizable as such. Violations can carry fines of up to 15 million euros or 3 percent of worldwide annual turnover. For systems already in use before August 2, a transition period until December 2, 2026 applies to the marking.

Who is responsible: provider or deployer

The AI Act distinguishes two roles: the provider, who places an AI system on the market (develops it or makes it available under their own name), and the deployer, who uses it in their own operation. The provider is responsible for the machine-readable marking of AI-generated content (Art. 50(2)). The deployer is responsible for deepfake disclosure and chatbot notices (Art. 50(1),(4)). A startup that uses an AI chatbot on its own website is a deployer and must tell users that they are talking to an AI. A startup that develops its own AI system and offers it to others for use is a provider and carries additional obligations.

A local model makes two things easier:

• You control the labeling. If the model runs on your hardware, the entire output pipeline belongs to you. You build the machine-readable marking in where the content is created, and you document it. With a cloud API, you depend on the provider’s implementation.
• No dependence on GPAI documentation obligations. Large foundation models have been subject to their own requirements since August 2025, including technical documentation and a training data summary. With a locally operated open-source model you do not buy in this chain, and models that are only used internally do not put you in the provider role.

The labeling obligation applies to everyone who publishes AI content, wherever the model runs. Anyone who operates a chatbot must disclose that it is an AI, whether inference happens locally or in the cloud. Local models give you control over the whole process, and the obligation stays.

Mini PC or cloud server: both work

The hardware question is rarely the bottleneck today. For models in the 8B class, a mini PC that fits next to the monitor is enough. If you run several agents in parallel or need larger models, use a cloud server.

For our customers we mix both options and pick the hardware that fits the use. The same agent runs on a local mini PC for one and in the cloud for another, and a later distribution across more devices is possible without a rebuild. We decide the right mix together with you.

Our article on Rybbit and Matomo covers cookie-free web analytics, in case you need that too. The first step is the same in both cases: a system you control yourself.

Do you have questions about your local agent setup or want to talk through your specific situation? Contact our team.