Multimodal Reward-Guided Decoding (MRGD), explained

The problem it solves

Multimodal language models hallucinate: ask one to describe an image and it may name objects that are not there. The classic fixes sit at two extremes. Training-time alignment (RLHF, DPO) works but is frozen — every behavior change means retraining. Prompting is adjustable but coarse. MRGD occupies the useful middle: it steers the model while it generates, using reward models as judges, and exposes a single weight w that trades precision against recall per request.

How it works

y ← ""                                    # the response so far
while y has no end-of-sequence token:
    sample k candidate continuations      # each ≈ one sentence
    score each candidate:
        s = w · r_hal + (1 − w) · r_rec   # weighted reward mix
    y ← y + argmax(s)                     # keep the best candidate

• r_hal — a learned reward model that sees the image and penalizes text not grounded in it (in the paper: a PaliGemma-3B scorer trained on ~40k preference pairs in about nine minutes on 8×H100).
• r_rec — a programmatic recall scorer built from off-the-shelf parts (open-vocabulary object detection + embedding matching); no training at all.
• w ∈ [0, 1] — the runtime dial: 1.0 maximizes grounding (precision), 0.0 maximizes coverage (recall).

Scoring happens every T sentences (T=1 by default). Set T to the whole response and MRGD degenerates into plain best-of-k rejection sampling — which makes MRGD a strict generalization of best-of-n, sitting between token-level guided decoding (ARGS, controlled decoding) and whole-response reranking.

What the paper measured

From “Controlling Multimodal LLMs via Reward-guided Decoding”(Mañas, D’Oro, Sinha, Romero-Soriano, Drozdzal, Agrawal — Mila / Meta FAIR, ICCV 2025), evaluating LLaVA-1.5, Llama-3.2-Vision, and SmolVLM-2 on COCO and AMBER:

• ~70% fewer hallucinated objects than greedy decoding (CHAIRi 15.05% → 4.53% on COCO at w=1.0).
• Sentence-level guidance with k=5 candidates beat best-of-30 whole-response sampling — over 6× more sample-efficient than rejection sampling.
• Latency grows sublinearly in k thanks to batched candidate generation (k=30 ≈ 11× wall clock, not 30×).
• It stacks: applied on top of a DPO-aligned model, hallucination still dropped further.

MRGD in Maestro IDE

Maestro IDE implements MRGD as a per-node decoding mode in its agent orchestration engine, generalized to any number of weighted scorers: learned reward endpoints, programmatic scorers, LLM-as-judge rubrics, or your own scripts. You set k, T, and the weights in the inspector; presets cover the precision↔recall spectrum; a live candidate inspector shows every round, every score, every selection; and the ~k× cost is estimated before you run. The same idea extends to media as best-of-k selection over generated images and audio.

Related terms

Reward-guided text generation · inference-time alignment · decoding-time alignment · best-of-n / rejection sampling · segment-level guidance · process reward models · hallucination mitigation for vision-language models · CHAIR metric · guided decoding (ARGS, RAD, controlled decoding). MRGD is the segment-level member of this family, and the first built on multimodal reward models.