Picture this: A patient in critical condition arrives at a hospital. In the trauma room, a dozen doctors and medical personnel strive to stabilize the patient’s condition, discussing pressing issues and trying to keep each other informed. Keeping track of everything is difficult at the best of times, let alone at the end of a 12-hour shift.
As development partner of T-Systems we designed and implemented an agentic system to provide support in exactly this situation: TraumaAgent. It keeps track of the entire treatment session and provides a live structured overview of the patient’s condition and administered treatments to to help doctors maintain an overview in highly stressful situations and to support documentation post-treatment.
For production use, it is imperative that such a system conforms to data protection regulations – after all, the data being processed is highly personal. However, this should not come at the cost of the system’s accuracy and capability. To ensure maximum privacy, we deploy it completely on-premises. Doing so introduces several challenges, which we describe in this blog post. We hope the lessons we learned may provide useful pointers for anyone considering local deployment and what to take into account for their particular use case.
Note: There are two major backend components enabling the TraumaAgent: an automated speech recognition (ASR) model and a large language model (LLM). Both rely on the transformer architecture, and some of the challenges laid out in this post apply to both components in equal measure. We will therefore focus on the large language model, which drives all the text-based agentic workflows.
LLM Inference Primer
Before we begin, we will present a short primer on LLM inference (for a more detailed overview consult e.g. this Huggingface article). Conceptually, an LLM generates one token (think of it as part of a word; on average, one word consists of around 1.3 tokens at a time. In a chatbot, the LLM takes the user’s prompt and generates tokens successively to answer it.
There are therefore two inference phases: processing of the prompt and answer generation. The first stage is simple to perform – the entire sequence of prompt tokens is processed in parallel. GPUs, the hardware processors used to deploy LLMs, are highly efficient at such parallel computations.
Answer generation, however, cannot be parallelized in the same way, as each generated token depends on the previous tokens. In order not to have to re-process all previous tokens for each token to be generated, intermediate computation results are stored in the GPU memory – this is called KV caching. This dramatically speeds up token generation, but it necessitates a high memory bandwidth on the GPU, as the KV cache needs to be read and written to for every token.
Moving from Cloud-based LLMs to a Local Deployment
On a large commercial GPU server, both stages can be performed quickly, as the hardware is fast in prompt processing and, due to high memory bandwidth, token generation as well.
Thanks to access to the Open Telekom Cloud (OTC), we could take advantage of this speed and develop our highly performant agent system and trial it under ideal conditions. But of course, the challenge of data protection remains.
Through T-Systems, we gained access to Nvidia DGX Spark devices. Advertised as a desktop supercomputer, the DGX Spark is based on a cutting-edge GPU chip with 128GB of memory at a small form factor. This enables it to run sizeable language models at high speeds, while not requiring an entire server setup of its own – in theory, perfect for our use case.
However, we identified two main challenges when using the device:
- An incomplete software ecosystem
- Limited memory bandwidth.
The Software Ecosystem
The specific LLM we selected is OpenAI’s open-weight GPT-OSS-120b. With 120 billion parameters, it is a mid-size model suitable for edge deployment while retaining strong reasoning and agentic capabilities. We found it to work very well for our agentic system when deployed in the OTC, in part due to its configurable reasoning effort.
This allows us to select how much the model “thinks” before returning an answer or performing an action. Another particularly useful feature of the model is the native weight quantization to MXFP4. Compared to a regular language model, it requires only a quarter the amount of GPU memory and enables faster computation on recent GPU architectures.
A regular 120B model would require at least 240GB of memory, whereas this one can work with 60GB. The GB10 GPU in the DGX Spark is thus, in principle, an architecture capable of leveraging both advantages.
But this is where we run into the first issue: NVIDIA supplies versions of the popular inference frameworks (we use vLLM) compatible with the DGX Spark. While these work, they are not fully optimized to exploit this particular quantization method. As a result: both the prompt processing and token generation phases are slower than they could be – a major drawback in our setting, where low latency is crucial for the system to be useful.
There is a way around this problem, though: a community of DGX Spark put significant effort into developing more optimized versions of inference frameworks. A good example of this is this one on GitHub, which we used to boost token generation speed by 50%.
The Memory Bandwidth
While the GB10 chip is fast in the prompt processing stage, the memory it uses has a significantly lower bandwidth than server-grade GPUs. As explained above, this affects the token generation speed in theory, and we observed this in practice: our measured token generation throughput is only around a quarter of what is achievable on a single server-grade GPU.
The practical effect was immediately noticeable when we ran our application in the simulation trauma room of the clinic in Köln-Merheim – information extracted from the doctors’ conversation took too long to appear on the dashboard.
To reduce this latency, we lowered the model’s reasoning effort, meaning that it generated fewer tokens dedicated to its thinking process. While doing so reduced the latency to acceptable levels again, it came at the cost of reduced precision in the information presented on the dashboard. The reason is that t speech in a trauma room often contains relevant information that is not explicitly stated, but rather obscured by the heavy use of jargon or simply left implicit.
Another common practice for dealing with memory bandwidth limits is to quantize the KV cache. In other words, the stored intermediate computations are kept in a lower precision datatype, which means reading and writing are sped up for each sequence being processed. However, that can, in theory ,come at the cost of quality, as this quantization is lossy. In our case, we chose FP8 as the data type, which results in a negligible quality drop.
Using Multiple Sparks
In addition to the above optimizations and workarounds, there is the option to connect two DGX Spark devices, effectively sharing the GPU memory and using the two GPUs for computation in parallel.
The aforementioned limitations still apply, and unfortunately, twice the number of devices does not result in half the latency in our case of single, non-concurrent requests – in fact, generation throughput increases by only ~30% (For detailed performance benchmarks, see https://www.storagereview.com/review/nvidia-dgx-spark-cluster-review-distributed-inference-on-dell-gigabyte-and-hp). While not groundbreaking, it provided a useful boost.
Moving to General Use Cases
There are more advantages to such a cluster setup and to illustrate those it is useful to consider more general use cases, in which. To illustrate them, it is useful to consider more general use cases in which latency may not be as critical, but the capability of the model to solve very complex tasks is relatively more important.
Examples include a coding assistant for confidential codebases or an involved agentic setup to analyze contracts, again subject to high data protection requirements. In addition, such applications may be used by an entire team rather than a single user. This places different demands on the computing hardware.
Compared to the real-time single-user use case of TraumaAgent, a cluster of DGX Spark devices offers advantages that are particularly relevant to these requirements.. First, twice the usable memory means models with more parameters and thus more capabilities can be served, for example the popular DeepSeek-V4-Flash at 284B parameters. Secondly, more memory enables more KV cache to be stored. Since every conversation thread with the model results in KV cache usage proportional to the number of tokens in the conversation, this means that more users can be served concurrently at an acceptable speed.
More generally, there are many ways to achieve, there are many ways to achieve completely local LLM deployments.
These include a proper server-grade setup on premises, which comes at significantly higher cost and requires space, cooling solutions, maintenance and more. Alternatively, alternative desktop-class AI devices can be used – Apple Mac Minis or AMD Strix Halo devices are popular alternatives to the DGX Spark.
Each of these options has pros and cons. We hope that this post sheds light on some of the limitations and challenges one may encounter, so that the reader may make an informed decision on how to proceed.
Conclusion
Compared with one or two years ago, it is now feasible to operate complex LLM-powered applications locally without a huge upfront investment in AI hardware. As open-weight LLMs become even more competitive and low-footprint AI hardware becomes more capable, the barrier to entry is likely to drop further. Currently, however, moving from cloud to edge is not entirely straightforward and limitations apply. Our specific, though highly relevant, use case shows that these limitations have practical consequences – but also that they can be mitigated.
