Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
Abstract
Masked Diffusion Language Models (MDLMs) exhibit locality bias and are negatively impacted by appended mask tokens, but a mask-agnostic loss function improves their context comprehension.
Masked Diffusion Language Models (MDLMs) have recently emerged as a promising alternative to Autoregressive Language Models (ARLMs), leveraging a denoising objective that, in principle, should enable more uniform context utilisation. In this work, we examine the context comprehension abilities of MDLMs and uncover two key limitations. First, despite their more global training objective and bidirectional attention mechanism, similarly to ARLMS, MDLMs exhibit a strong locality bias: performance is highly sensitive to the position of relevant information within the input, favouring local over distant context. Second, we show that appending a large number of mask tokens--required for generation--can significantly degrade context comprehension. Through systematic ablations, we find that these masks act as distractors, reducing the model's ability to process relevant information. To address this, we introduce a mask-agnostic loss function that encourages predictions to remain invariant to the number of appended masks. Fine-tuning with this objective substantially mitigates the distracting effect of masks, improving robustness of MDLMs. Overall, our findings reveal critical limitations of the current MDLM training paradigm and provide actionable insights for building diffusion-based language models with stronger context comprehension.
Community
If you find this paper and its results interesting or insightful, please consider upvoting for us. Thank you!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Attention Sinks in Diffusion Language Models (2025)
- Soft-Masked Diffusion Language Models (2025)
- From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model (2025)
- Closing the Data-Efficiency Gap Between Autoregressive and Masked Diffusion LLMs (2025)
- UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models (2025)
- No Compute Left Behind: Rethinking Reasoning and Sampling with Masked Diffusion Models (2025)
- CDLM: Consistency Diffusion Language Models For Faster Sampling (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper