Fundamental Limits of Prefill-Based Decoding Length Prediction

Analysis shows that predicting LLM decoding length using prefill-phase hidden states is unstable because current benchmarks conflate predictor quality with task hardness. The research identifies that the inherent stochasticity of autoregressive decoding causes reported accuracy to collapse when models or datasets change.