Local AI & Self-Hosting

Introduction

Local deployment of artificial intelligence models has seen considerable growth since the advent of open source LLMs. Where organizations previously relied exclusively on proprietary cloud APIs — with recurring costs, latency constraints, and privacy concerns — it is now possible to host, run, and optimize performant models on their own infrastructure. This shift, made possible by advances in quantization, distillation, and optimized inference engines, is fundamentally redefining the relationship between businesses and artificial intelligence.

Sovereignty and Costs

Data sovereignty is the primary driver of self-hosting adoption. When an organization uses a cloud API to process its documents, emails, voice recordings, or knowledge bases, it entrusts its most sensitive data to a third party. For regulated sectors — healthcare, finance, defense, legal — this transfer is simply impossible: GDPR, sector-specific regulations, and internal security policies require data to remain on the organization's infrastructure. Local hosting removes this constraint: no data leaves the controlled perimeter, models run on internal servers, and complete processing traceability is ensured. This total control over the data lifecycle is, for many organizations, a non-negotiable prerequisite.

Cost reduction is the second decisive argument. Cloud APIs charge per usage: each generated token, each submitted query, each processed document is billed. For modest volume, the cost remains acceptable. But as usage intensifies — document assistance for a hundred employees, automated processing of thousands of documents per day, continuous voice analysis — the monthly bill can reach several thousand or even tens of thousands of euros. Local hosting replaces this recurring cost with an initial hardware investment and controlled maintenance. A server equipped with two professional GPUs, amortized over three to five years, costs a fraction of the equivalent cloud API volume. The break-even point depends on volume and model choice, but our benchmarks show that beyond a few hundred thousand requests per month, local hosting systematically becomes more economical.

Minimal latency is the third competitive advantage of self-hosting. Cloud APIs introduce incompressible network latency: the round-trip time between the organization's server and the cloud provider, added to the model's inference time. This latency, typically between 200 and 800 milliseconds, is acceptable for asynchronous use but problematic for real-time applications — live voice transcription, conversational assistant, video stream analysis. Locally, network latency is nearly zero, and inference time becomes the only determining factor. For a quantized 7 to 14 billion parameter model on a modern GPU, response times drop below 50 milliseconds, paving the way for high-quality interactive applications.

Optimization

Quantization techniques are at the heart of model optimization for local deployment. A language model is traditionally stored in 16-bit (FP16) or 32-bit (FP32) precision, meaning each parameter occupies 2 or 4 bytes in memory. A 70-billion parameter model in FP16 thus requires 140 GB of GPU memory — the equivalent of multiple high-end GPUs. Quantization reduces this precision: an 8-bit (INT8) quantized model halves memory, 4-bit (INT4) quarters it. The GGUF format, popularized by llama.cpp and Ollama, is the most widespread in the open source ecosystem: it supports asymmetric quantization, storage on CPU and GPU, and a wide range of quantization levels (Q2_K to Q8_0). Each level offers a different trade-off between memory size and generation quality. A Llama 3 70B model quantized in Q4_K_M occupies about 40 GB, the memory of a single professional GPU like the RTX 6000 Ada.

AWQ (Activation-Aware Weight Quantization) is a more recent format that adapts quantization based on each weight's importance for inference quality. Unlike GGUF which applies uniform quantization, AWQ analyzes weight activation during calibration passes and preserves higher precision for the most influential weights. The result is better generation quality at equivalent model size. GPTQ (GPT Post-Training Quantization) is another optimized format, particularly suited to Llama and Mistral architectures. It uses batch quantization and error compensation to minimize quality loss. In practice, a model quantized in 4-bit AWQ or GPTQ offers quality close to the original FP16 model, with 75% reduced memory consumption. The choice between GGUF, AWQ, and GPTQ depends on the inference engine used: Ollama favors GGUF, vLLM supports AWQ and GPTQ, while frameworks like text-generation-webui offer multiple compatibility.

Distillation is another fundamental optimization technique, distinct from but complementary to quantization. While quantization reduces the numerical precision of weights, distillation involves training a smaller model (the student) to reproduce the behavior of a larger model (the teacher). The student learns not only to produce correct answers but also to imitate the teacher's probability distribution, capturing the nuances of its reasoning. Distilled models, such as the Llama 3.2 series (1B, 3B), Mistral Small (7B), or Qwen 2.5 (1.5B, 7B), achieve performance comparable to models two to three times larger, with a fraction of the memory and inference time required. The combination of distillation and quantization is particularly powerful: a distilled 7-billion parameter model quantized in 4 bits can run on a consumer GPU with surprisingly high response quality.

Inference engines constitute the software layer that orchestrates model execution on hardware. Ollama is the most accessible solution for getting started with self-hosting: it encapsulates model management complexity in a simple command-line tool, with automatic download, version management, OpenAI-compatible REST API, and native GGUF support. Its modular architecture allows changing models with a single command and configuring inference parameters — temperature, top-p, context, batch size. Ollama is ideal for single-model deployments, rapid testing, and development environments. Its main limitation is single-GPU management: for models requiring multiple GPUs, other solutions are more suitable.

vLLM is the high-performance inference engine designed for production deployments. Its core innovation is the PagedAttention algorithm, which manages KV attention cache memory like operating system pages: pages are dynamically allocated and freed, eliminating the memory waste of traditional approaches. This optimization increases the number of simultaneous requests handled per GPU by 2 to 5 times compared to conventional engines, without loss of quality or latency. vLLM supports GPU parallelism (tensor parallelism, pipeline parallelism), dynamic batching, token streaming, and AWQ and GPTQ quantization formats. It exposes an OpenAI-compatible API, allowing a cloud API call to be replaced by a local call without modifying application code. For high-volume deployments — document assistance, batch document analysis, production chat — vLLM is the reference engine.

Deployment with Docker and Kubernetes constitutes the standard infrastructure for production environments. Each inference engine is packaged in a Docker image with its GPU dependencies (CUDA, cuDNN, TensorRT), ensuring reproducibility across development, test, and production environments. Docker Compose orchestration enables linking multiple services: a vLLM container for inference, a Redis container for response caching, an Nginx container for load balancing, and a Prometheus container for metric collection. For larger-scale deployments, Kubernetes orchestrates clusters of GPU nodes with auto-scaling based on load, model distribution across nodes, zero-downtime updates, and fault tolerance. Helm Charts simplify deployment of complex inference stacks with declarative, versioned configuration.

CUDA optimization is a key performance factor. Modern inference engines leverage CUDA to compile custom kernels tailored to each model and GPU. TensorRT-LLM, NVIDIA's optimization library, fuses computation graph operations, applies on-the-fly quantization, and generates specialized CUDA kernels that fully exploit the target GPU architecture. Gains over naive execution are considerable: up to 4 times more tokens per second on an A100 GPU, and significant memory consumption reduction. Advanced optimizations include FlashAttention for speeding up attention computation on long sequences, vLLM's PagedAttention for memory management, and JIT (Just-In-Time) compilation that adapts kernels to the exact dimensions of the model and batch. GPU choice is decisive: professional GPUs (A100, H100, RTX 6000 Ada) have more memory and bandwidth, but consumer GPUs (RTX 4090, RTX 5090) offer the best price-to-performance ratio for moderately sized models.

Cloud vs local benchmarking is a central focus of this project. We have developed a systematic protocol to compare costs and performance of both approaches on twenty representative models, three cloud providers (OpenAI, Anthropic, Mistral API), and four local configurations (RTX 4090, RTX 5090, 2× RTX 4090, 2× RTX 6000 Ada). Collected metrics cover six dimensions: cost per million generated tokens, throughput in tokens per second, first response latency, total completion latency, quality measured by automatic and human metrics, and reliability measured by error rate and response variance. Results confirm that the local break-even point lies between 500,000 and 2 million tokens per day depending on the model, and that latency is systematically 60 to 80% lower compared to cloud APIs.

The quality difference between quantized open source models and proprietary models deserves nuanced analysis. On common tasks — classification, extraction, short summarization, simple Q&A — quantized 7 to 14 billion parameter models achieve scores comparable to GPT-4 or Claude 3.5, with a difference of less than 2-3% on standardized metrics. On complex tasks — multi-step reasoning, legal analysis, advanced coding — the gap widens in favor of proprietary models, but distillation and 8-bit quantization of large models (70B+) significantly reduce this gap. The choice between cloud and local is therefore not binary: a hybrid architecture, where simple queries are handled locally and complex queries delegated to the cloud, offers the best quality-cost trade-off.

Monitoring with Prometheus and Grafana is essential for managing a local inference infrastructure. Prometheus continuously collects metrics exposed by inference engines: GPU consumption (memory, utilization, temperature, power), inference throughput (tokens per second, requests per minute), latency (average, P50, P95, P99), error rate, queues, and saturation. Each Docker container exposes a /metrics endpoint in Prometheus format, and specialized exporters (nvidia-exporter for GPU metrics, node-exporter for system metrics) complete the collection. Grafana aggregates this data into dynamic dashboards: cluster overview, per-model analysis, cost tracking, threshold breach alerts. Alerts are configured to notify teams of performance degradation — saturated GPU memory, excessive latency, failing model — enabling intervention before users are impacted.

Use Cases

Concrete use cases cover a broad spectrum of professional applications. Internal document assistance is the most common case: a locally deployed LLM with vLLM and a Qdrant vector database powers a RAG chatbot that answers questions about the company's knowledge base. Sensitive data — contracts, customer data, financial information — never leaves the infrastructure. Batch document processing is a second use case: analysis of hundreds of reports, structured data extraction from invoices, automatic classification of incoming mail. Batch inference with vLLM enables processing large volumes with optimal throughput. Writing assistance and content generation — emails, reports, meeting minutes, sales proposals — benefit from local's minimal latency for a smooth interactive experience.

Real-time voice transcription combines a local Whisper model (via Ollama or the WhisCPP server) with an LLM for understanding and summarization. The complete pipeline runs locally without any cloud dependency, ensuring conversation confidentiality. Code analysis and development assistance are also relevant: a model like Code Llama or DeepSeek Coder deployed locally with Ollama offers programming assistance without sending source code to external servers, a decisive argument for software companies concerned about protecting their intellectual property. Content moderation and sentiment analysis on internal text streams complete the application landscape.

Technical Challenges

Technical challenges of self-hosting are real and must be anticipated. GPU resource management is the first challenge: multiple models or users can contend for the same GPU, causing performance degradation. vLLM handles this contention through dynamic batching, but capacity planning is necessary. Model maintenance is the second challenge: models evolve rapidly, new versions bring quality and security improvements, and quantization formats need updating. An automated update pipeline, with regression testing before deployment, is essential. Scaling is the third challenge: local infrastructure must be dimensioned for peak activity, which can lead to over-provisioning during off-peak periods. Kubernetes with horizontal auto-scaling mitigates this by dynamically allocating and releasing resources.

Local infrastructure security is an often underestimated concern. Models themselves can contain vulnerabilities — recent research has shown it is possible to inject malicious behaviors into a quantized model. Verification of downloaded model integrity via checksums and signatures is a minimum practice. Docker container isolation prevents a compromised model from accessing the rest of the system. Limiting internal API token permissions reduces the attack surface. Logging model access and submitted queries ensures necessary traceability in case of incident. Finally, securing API endpoints exposed by inference engines — authentication, rate limiting, TLS encryption — is essential as soon as the system is accessible from the internal network.

Perspectives

Self-hosting evolution prospects are particularly promising. The arrival of GPUs with ever more memory (48 GB, 80 GB, soon 144 GB) pushes back the limit of models deployable on a single card. Quantization formats are advancing: 3-bit and 2-bit formats are emerging, offering additional memory reductions with increasingly well-controlled quality loss. CPU inference, with models optimized for execution without a GPU, is improving thanks to AVX-512 and AMX instructions in modern processors, making self-hosting accessible even without a dedicated graphics card for small models. Standardized protocols like MCP facilitate integration of local models into existing information systems, unifying access to AI capabilities regardless of provider or deployment mode.

Multi-model orchestration is a natural evolution of self-hosting. A single server can host multiple specialized models: an LLM for text generation, an embedding model for vectorization, a vision model for image analysis, an STT model for voice transcription. The orchestrator routes each request to the competent model, pools GPU resources across models, and manages priorities. This approach transforms the local server into a complete AI platform, comparable to cloud offerings but with total data control. Initial feedback from organizations that have adopted self-hosting confirms the trend: tool maturity, open source model quality, and the relative decline in hardware costs make local deployment accessible not only to large enterprises but also to SMEs and organizations with modest IT budgets.

Conclusion

This research project on local AI and self-hosting establishes a complete framework for evaluating, deploying, and optimizing artificial intelligence models on one's own infrastructure. GGUF, AWQ, and GPTQ quantization techniques, distillation, Ollama and vLLM inference engines, Docker and Kubernetes orchestration, CUDA optimization, systematic benchmarking, and Prometheus-Grafana monitoring form a coherent technology stack that makes self-hosting viable and performant. Data sovereignty, cost reduction, and latency control are no longer trade-offs: they are the concrete benefits of an approach where the organization regains control of its AI infrastructure.