LLM Benchmark

Introduction

The landscape of large language models (LLMs) is experiencing exponential growth. Each month brings its share of new models — fine-tuned versions, distilled architectures, quantized variants, promises of breakthroughs. Platforms like Hugging Face now host over 500,000 models, and announcements of new LLMs — whether open source or proprietary — number in the dozens each week. This proliferation, while reflecting the dynamism of the field, makes choosing a model suited to a specific business need increasingly complex. Technical teams face a daunting question: which model to choose for which use, with what confidence, and at what cost?

Objectives

The vision of this project is to create a systematic evaluation framework that puts business needs at the center of measurement. The goal is not to produce yet another generalist ranking, but to answer a practical question: for a given task, in a given context, with given constraints, which model offers the best quality-cost-reliability ratio? The protocol is designed to be reproducible: each test is documented, each dataset is versioned, each metric is unambiguously defined. Results are traceable and verifiable. Another lab, another team, can reproduce the same tests and obtain comparable results, ensuring the objectivity and transparency of the process.

Project Overview

Traditional academic benchmarks — MMLU, HellaSwag, GSM8K, HumanEval, BIG-Bench — have played a foundational role in providing standardized references for comparing models. They measure general capabilities such as logical reasoning, language understanding, basic mathematics, or code generation. However, these indicators have significant structural limitations. A high MMLU score does not guarantee that a model will correctly extract structured data from a legal document, nor that it will respect a specific output format required by a business information system. Academic benchmarks do not measure response reliability in a professional context, nor inference cost, nor real latency, nor regulatory compliance, nor the ability to follow complex constrained instructions. Worse still, training data contamination — entire benchmarks potentially having been ingested during training — distorts the real meaning of scores. A model can excel on MMLU while failing on simple tasks not present in its training corpus.

The divergence between academic performance and actual execution quality on business tasks is a recurring observation. A model ranked first on a generalist leaderboard can produce incoherent responses on a specialized corpus in law, finance, or technology. Domain nuances, specific vocabulary, format constraints, precision requirements — all aspects that no academic benchmark captures. This gap creates an urgent need: a rigorous, business-oriented, reproducible evaluation protocol that measures what truly matters for an organization deploying LLMs in its operational processes.

Architecture

The benchmark architecture is based on a modular Python framework designed to orchestrate the entire evaluation process automatically and reproducibly. The core of the system is an asynchronous execution engine that handles parallel model calls, optimizes API and GPU resource usage, and ensures complete traceability of each test. Each step — test submission, execution, result collection, metric calculation, report generation — is isolated and configurable. The framework uses LlamaIndex to orchestrate model calls, manage execution contexts, and standardize interactions with heterogeneous APIs. Everything is containerized with Docker, ensuring tests run in an identical environment for each campaign, regardless of the machine or infrastructure provider.

Asynchronous orchestration is a key element of the architecture. Benchmark campaigns often involve several dozen models, each tested on multiple scenarios with repetitions to ensure statistical significance. The framework distributes tasks across multiple workers, manages queues, timeouts, error retries, and result collection. An intelligent caching system avoids re-executing tests already performed with the same parameters. Evaluation pipelines can run from a few hours to several days, depending on the number of models and test complexity. The asynchronous architecture ensures optimal resource utilization without human intervention.

Each unit test is defined in versioned YAML files, forming a living and collaborative evaluation library. A test file describes the entire scenario: the prompt or input set, measurable success criteria, metrics to collect, execution parameters (temperature, top-p, max tokens), and comparison references. This approach offers several decisive advantages. Versioning in Git allows tracking test evolution over time. Tests are readable by non-developers: a business expert can understand and validate a scenario without programming skills. Reuse is immediate: a test created for one model can be run on any other model without modification. Transparency is total: evaluation criteria are explicit, documented, and discussable.

Functioning

Collected metrics cover four complementary dimensions. Quality measures the relevance, accuracy, and reliability of responses: exact match score for determinate-answer tasks, semantic similarity score for open-ended tasks, completion rate for multi-step instructions, hallucination detection to assess the propensity to invent information, and inter-response consistency to measure model stability across phrasing variations. Cost calculates cost per request, cost per token, cost per completed task, and projects monthly cost for different usage volumes — from a few hundred to several million requests per month. Latency measures time to first response (TTFR), total completion time, variation under load, and degradation with context length. Sovereignty evaluates data location, GDPR compliance, provider certification, on-premise deployment feasibility, and guaranteed confidentiality level.

Evaluation combines automatic scoring and human evaluation to cover all quality aspects. Automatic scoring is used for objectively measurable tasks: factual answer accuracy, output format compliance, instruction completion, execution time. For each test, a scoring function automatically determines the grade by comparing the model's response to a reference or a set of formal criteria. Human evaluation handles subjective tasks where perceived quality is difficult to automate: writing quality, explanation relevance, tone and style, creativity, dialogue fluidity. A panel of qualified evaluators scores responses according to a standardized grid, with explicit criteria and a calibration system to ensure inter-rater consistency. The combination of both approaches offers a complete evaluation: the machine ensures reproducibility and speed, the human brings nuanced and contextual judgment.

Technologies

Models included in the study scope cover a representative spectrum of the current ecosystem, constantly evolving. On the open source side, the Llama family (Meta) across its various sizes — 8B, 70B, 405B — and fine-tuned variants. Mistral, with its Small, Medium, and Large models as well as Mixtral variants, is evaluated for its performance on European languages and its ability to run on accessible hardware. The Qwen family (Alibaba) is analyzed for its multilingual capabilities and performance on code and reasoning tasks. Gemma (Google) completes the panel with its compact models suited to deployment constraints. On the proprietary side, OpenAI's GPT-4 and its variants (GPT-4 Turbo, GPT-4o, GPT-4o mini) constitute the market reference for general quality. Claude (Anthropic) is studied for its reasoning capabilities, long context handling, and compliance with complex instructions. Gemini (Google) is evaluated for its multimodal capabilities and performance on vision and document tasks.

Use Cases

Evaluation scenarios are designed from real business use cases, identified with the lab's project teams and clients. Code generation assesses models' ability to produce functional code from natural language descriptions: function creation, bug fixing, refactoring, unit test generation, code documentation, security analysis. Tests cover multiple languages (Python, TypeScript, Rust, Go, SQL) and complexity levels. Document analysis measures the ability to extract structured information from heterogeneous documents: named entity recognition, long document summarization, tabular data extraction, contract clause classification. Technical writing evaluates the quality of professional content generation: reports, emails, specifications, technical documentation, articles.

Contextual question-answering (Q&A) is tested on business corpora: a model receives a document context and must answer questions whose answers are explicitly or implicitly found in the context. Metrics include answer accuracy, ability to cite relevant sources, appropriate refusal when the answer is not in the context, and resistance to hallucinations. Content classification evaluates the ability to automatically categorize texts according to business taxonomies: support ticket classification, document categorization, sentiment analysis, spam detection, theme identification. Each scenario comes with a test set representative of real usage conditions, including edge cases, ambiguities, and counterexamples.

Storage and Visualization

The storage architecture uses a PostgreSQL database designed for longitudinal performance tracking. The database schema is optimized for cross-analysis queries: by model, by scenario, by metric, by campaign. Each result is timestamped and associated with the exact version of the test, model, and evaluation framework, enabling reconstruction of the complete performance history of a model over time. Pre-computed analysis views break down results by task category, metric type, and model family, offering a synthetic view without complex queries. The Grafana interface will expose this data in dynamic dashboards: score evolution charts across campaigns, comparative rankings by scenario, quality-cost-latency matrices, and alerts on significant regressions.

Technical Challenges

The design of a business-oriented benchmark protocol raises several technical challenges. Test reproducibility over time requires strict version control of models, datasets, and the execution environment. Models evolve, their APIs change, versions succeed one another — each campaign must be reproducible identically to ensure comparability of results. Statistical significance of measurements requires repeating each test multiple times and calculating confidence intervals, which multiplies execution time and campaign cost. Managing hallucinations and biases requires specific evaluation protocols going beyond simple precision metrics. Data sovereignty constitutes a determining selection criterion, particularly for organizations subject to strict regulatory constraints (GDPR, regulated sectors, health data, financial data). The benchmark aims to systematically evaluate deployment options: cloud API with contractual guarantees, sovereign European infrastructure, or full on-premise deployment, documenting for each model the available options, provider certifications, and geographic location of processing servers.

Perspectives

The benchmark's evolution prospects are ambitious. Integration of new models will continue, with particular attention to domain-specialized models: legal, medical, financial, technical. Extension of evaluation scenarios is an ongoing effort: business teams regularly identify new use cases requiring specific test creation. The project will also explore evaluation of multi-model architectures (agents, RAG, model chains) where several models collaborate to accomplish a complex task, an increasingly common configuration in professional applications.

The evaluation protocol itself will undergo continuous improvement. Metrics will be refined based on feedback: new indicators may be added — such as robustness to phrasing variations, ability to politely refuse out-of-scope requests, or fidelity to system customization — and performance thresholds adjusted. Collaboration with other labs and research teams is encouraged to enrich the protocol, share test sets, and cross-validate results. Ultimately, the benchmark aims to become an open reference for evaluating LLMs on professional tasks, contributing to better transparency and more informed adoption decisions across the entire ecosystem.

Conclusion

The LLM benchmark is not a one-time exercise but a continuous process. The model landscape evolves rapidly, business needs transform, and selection criteria become more complex with the emergence of new architecture types. This research project lays the foundations for a rigorous, business-oriented, and reproducible evaluation framework, enabling organizations to make informed choices in a constantly evolving ecosystem. Mastering LLM evaluation is a decisive competitive advantage for any organization integrating generative AI into its operational processes.