OmniScore
OmniScore resource page for deterministic multilingual generative text evaluation metrics, released data, models, Python package, and reproducible scoring.
Evaluation toolkit
OmniScore: Deterministic Metrics for Multilingual Generative Text Evaluation
OmniScore is a family of deterministic learned metrics for multilingual generated-text evaluation. It is designed as a practical alternative to costly and prompt-sensitive LLM-as-a-judge workflows, while keeping the speed and consistency of model-based scoring.
Quick install pip install omniscore
Overview
LLM-as-a-judge evaluation can be expensive, slow, and sensitive to prompt design, language, and aggregation choices. OmniScore addresses this with small deterministic learned metrics that approximate LLM-judge behavior while providing reproducible, low-latency scores for generated text.
The paper reports large-scale synthetic supervision across 107 languages and evaluates the approach on manually annotated examples. The released resources include a Python package, model checkpoints, and the OmniScore-Data dataset for training and evaluation workflows.
Resource at a glance
Hugging Face dataset downloads
Hugging Face dataset likes
rows in OmniScore-Data
language values in the data viewer
task values in OmniScore-Data
languages reported for training supervision
manual annotations reported in the paper
package license
What the resource includes
Python and CLI scoring
The PyPI package supports both Python APIs and a command line interface for scoring generated text with hosted OmniScore checkpoints.
Released data
OmniScore-Data contains train, validation, and test splits with input, output, task, language, and multi-dimensional label fields.
Model collection
The Hugging Face collection links the paper, released data, and checkpoints such as QCRI/OmniScore-mxbai and QCRI/OmniScore-deberta-v3.
Dataset and scoring dimensions
The Hugging Face data viewer lists 581k rows with train, validation, and test splits. It exposes six task values and 99 language values, with labels for scoring dimensions such as clarity, faithfulness, informativeness, and plausibility.
Quick usage
from omniscore import OmniScorer, get_example
example = get_example("QCRI/OmniScore-deberta-v3")
scorer = OmniScorer("QCRI/OmniScore-deberta-v3")
result = scorer.score(
predictions=example.prediction,
sources=example.source,
references=example.reference,
tasks=example.task,
)
print(result.to_list())
omniscore \
--model QCRI/OmniScore-deberta-v3 \
--prediction "Microsoft releases detailed model documentation." \
--source "Full article text goes here." \
--task headline_evaluation \
--pretty
Citation
@article{alam2026llmasajudgedeterministicmetricsmultilingual,
title={Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation},
author={Alam, Firoj and Bhatia, Gagan and Laskar, Sahinur Rahman and Chowdhury, Shammur Absar},
journal={arXiv preprint arXiv:2604.05083},
year={2026},
url={https://arxiv.org/abs/2604.05083}
}