KnowRL：探索知识增强强化学习以提升事实准确性
原文标题：KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
作者：Baochang Ren, Shuofei Qiao, Wenhao Yu, Huajun Chen, Ningyu Zhang
原文摘要：Large Language Models (LLMs), particularly slow-thinking models, often exhibit severe hallucination, outputting incorrect content due to an inability to accurately recognize knowledge boundaries during reasoning. While Reinforcement Learning (RL) can enhance complex reasoning abilities, its outcome-oriented reward mechanism often lacks factual supervision over the thinking process, further exacerbating the hallucination problem. To address the high hallucination in slow-thinking models, we propose Knowledge-enhanced RL, KnowRL. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. This targeted factual input during RL training enables the model to learn and internalize fact-based reasoning strategies. By directly rewarding adherence to facts within the reasoning steps, KnowRL fosters a more reliable thinking process. Experimental results on three hallucination evaluation datasets and two reasoning evaluation datasets demonstrate that KnowRL effectively mitigates hallucinations in slow-thinking models while maintaining their original strong reasoning capabilities. Our code is available at https://github.com/zjunlp/KnowRL.
中文翻译：大型语言模型（LLMs），特别是慢思考模型，经常表现出严重的幻觉现象，由于在推理过程中无法准确识别知识边界而输出错误内容。虽然强化学习（RL）可以增强复杂推理能力，但其面向结果的奖励机制往往缺乏对思考过程的事实监督，进一步加剧了幻觉问题。为了解决慢思考模型中的高幻觉问题，我们提出了知识增强强化学习（KnowRL）。KnowRL通过将基于知识验证的事实性奖励整合到RL训练过程中，引导模型进行基于事实的慢思考，帮助它们识别知识边界。KnowRL通过将基于知识验证的事实性奖励整合到RL训练过程中，引导模型进行基于事实的慢思考，帮助它们识别知识边界。这种在RL训练期间的有针对性的事实输入使模型能够学习并内化基于事实的推理策略。通过直接奖励在推理步骤中对事实的遵守，KnowRL培养了更可靠的思考过程。在三个幻觉评估数据集和两个推理评估数据集上的实验结果表明，KnowRL有效缓解了慢思考模型中的幻觉现象，同时保持了它们原有的强大推理能力。我们的代码可在https://github.com/zjunlp/KnowRL获取。
智能问答：
Q: 这篇论文试图解决什么问题？
A: 这篇论文试图解决大型语言模型（特别是慢思考模型）中的严重幻觉问题。具体来说：1）知识边界识别问题：慢思考模型在推理过程中无法准确识别自己的知识边界，导致输出错误或不准确的内容；2）强化学习的局限性：虽然传统的强化学习可以增强复杂推理能力，但其面向结果的奖励机制缺乏对思考过程的事实监督，这进一步加剧了幻觉问题；3）事实性验证缺失：现有的训练方法缺乏有效的机制来确保模型在推理过程中遵循事实，导致模型可能产生看似合理但实际错误的输出；4）推理质量与事实准确性的平衡：如何在保持模型强大推理能力的同时，显著降低幻觉现象，确保输出内容的事实准确性。论文提出的KnowRL方法通过集成基于知识验证的事实性奖励到强化学习训练过程中，帮助模型识别知识边界并进行基于事实的推理。
阅读全文 →
