我们知道RAG分两步一、根据原始输入检索相关的文档二、将检索结果作为上下文提交给LLM生成输出。两个步骤涉及三类数据输入、上下文和输出。三类数据两两比较就带来了如下三种针对RAG的评估检索相关性评估评估作为上下文的检索结果与输入的相关度事实依据性评估评估最终输出与上下文的忠实程度帮助性评估 评估最终的输出对于输入提供的问题有多大的帮助。1. 检索相关性评估Retrieval Relevance Evaluation检索相关性评估用来确定系统检索出来的作为上下文的参考文档对于回答用户输入问题的有用程度和关联紧密性。在RAG系统中它是核心的漏斗第一步如果检索到的东西不相关后续大模型生成的回答再漂亮也是空中楼阁。评估的核心不是看文档是否完美而是看它能不能帮上忙相关Relevant只要包含能帮助回答问题的知识即使信息不完整、或夹杂了其他废话也算相关不相关Irrelevant完全无法提供有用信息、完全跑题、包含误导性/错误信息或者只有表面相关但实际毫无实用价值。如果把LLM看作考生检索结果看作开卷考试的参考书那么检索相关性评估就是在判定教务处发给考生的这几本书里面到底有没有这道题的答案。OpenEvals针对有害性评估的提示词模板定义在常量openevals.prompts.rag.RAG_RETRIEVAL_RELEVANCE_PROMPT具体内容如下。由于这种评估方式旨在评估作为上下文的检索结果与原始输入之间的相关性所以提示词模板只包含作为输入和上下文的{inputs}和{context}占位符。You are an expert data labeler evaluating retrieved context for relevance to the input. Your task is to assign a score based on the following rubric: Rubric - Relevant retrieved context: - Contain information that could help answer the input, even if incomplete. - May include superfluous information, but it should still be somewhat related to the input. - Irrelevant retrieved context: - Contain no useful information for answering the input. - Are entirely unrelated to the input. - Contains misleading or incorrect information - Contains only tangentially related information with no practical utility /Rubric Instruction - Read and understand the full meaning of the input (including edge cases) - Formulate a list of facts and relevant context that would be needed to respond to the input - Analyze the retrieved context to identify: - Information directly relevant to answering the query - Information partially relevant or contextually helpful - Information completely irrelevant to the query - For each piece of information need identified in the previous step, determine: - Whether it is fully addressed by the retrieved documents (cite specific text) - Whether it is partially addressed (cite specific text) - Whether it is not addressed at all - Note any facts needed to answer the input that are not found - Note any context that are completely irrelevant, i.e. contain no relevant facts for answering the input /Instruction Reminder - Focus solely on whether the retrieved context provides useful information for answering the input. - Think deeply about why the context is or isnt relevant. - Use partial credit where applicable, recognizing context that is somewhat helpful even if incomplete. /Reminder inputs {inputs} /inputs retrieved_context {context} /retrieved_context在如下这个演示程序中我们将评估的核心操作定义在eval函数中它会根据指定的提示词和用来连接LLMgpt-5.4-mini的ChatOpenAI对象创建一个基于LLM-as-a-Judge的评估器。然后根据指定的输入、输出、引用输出和额外的关键字参数调用评估器并将评估结果以JSON的形式输出。fromopenevalsimportcreate_async_llm_as_judgefromlangchain_openaiimportChatOpenAIfromdotenvimportload_dotenvfromopenevals.prompts.ragimportRAG_RETRIEVAL_RELEVANCE_PROMPTimportasyncio,json,functools load_dotenv()judgeChatOpenAI(modelgpt-5.4-mini)asyncdefeval(*,prompt:str,inputs:str|NoneNone,outputs:str|NoneNone,reference_outputs:str|NoneNone,**kwargs):evaluatorcreate_async_llm_as_judge(promptprompt,continuousTrue,judgejudge)resultawaitevaluator(inputsinputs,outputsoutputs,reference_outputsreference_outputs,**kwargs)print(json.dumps(result,ensure_asciiFalse,indent2))asyncdefmain():partialfunctools.partial(eval,promptRAG_RETRIEVAL_RELEVANCE_PROMPT,inputs如何在我们的系统里重置密码)context1 我们的系统为了保障用户安全采用了最先进的加密技术。 重置密码是维护账户安全的重要环节请确保您的密码符合复杂度要求。 如需了解更多关于账户安全策略的信息请参阅《网络安全管理规范》。 context2 当用户登录失败时可以点击登录框下方的链接。 系统会引导您进入重置密码界面此时需要输入您的注册邮箱。 虽然由于系统版本更新目前通过手机验证码重置的功能暂时下线维护但邮箱重置功能一切正常。 awaitpartial(contextcontext1)awaitpartial(contextcontext2)asyncio.run(main())在main函数中我们指定了上述的提示词针对同一个问题如何在我们的系统里重置密码提供了两种检索结果。前者虽然提及了很多关于密码的内容但是从语义来看与输入几乎不相关而后者则真正提供了重置密码的解决方案。输出的评估结果通过分数清晰地揭示了两条检索结果与输入的相关程度。{key:score,score:0.3,comment:输入是在询问“如何在我们的系统里重置密码”需要的是具体的重置步骤、入口位置、验证方式或相关流程说明。检索到的上下文只提到“重置密码是维护账户安全的重要环节请确保您的密码符合复杂度要求”这与密码重置主题相关但没有提供任何实际操作方法其余内容如“采用了最先进的加密技术”“参阅《网络安全管理规范》”也没有直接说明如何重置密码更多是背景性或泛化的安全提示。因此检索内容对回答问题只有很弱的辅助作用不能真正指导用户完成密码重置。Thus, the score should be: 0.3.,metadata:null}{key:score,score:1.0,comment:The input asks how to reset a password in the system. The retrieved context directly explains the reset flow: click the link under the login box, follow the password reset page, and enter the registered email. It also mentions that SMS verification reset is currently offline while email reset works, which is useful and relevant. This context therefore provides practical information to answer the query, though it may not cover every possible reset path. Thus, the score should be: 1.0.,metadata:null}2. 事实依据性评估Groundnesss Evaluation事实依据性评估是指评估LLM生成的输出在多大程度上忠实并根植于检索到的作为上下文的参考文档。如果说前面的检索相关性评估是检查书发对没有那么这种评估就是在检查考生是不是在对着这本书开卷考试有没有自己凭空瞎编即产生‘幻觉’。评估的核心是遵循闭合世界假定即认为参考文档是唯一的真理宇宙高依据性所有核心断言必须由文档直接支持严丝合缝地限定在文档信息范围内保持与源材料相同的原意和意图。无依据性凭空提出文档中没有的观点与文档内容直接冲突引入了文档之外的外部知识或主观推测歪曲或曲解了原意。OpenEvals针对有害性评估的提示词模板定义在常量openevals.prompts.rag.RAG_GROUNDEDNESS_PROMPT具体内容如下。由于这种评估方式旨在评估作为上下文的检索结果与最终输出之间的相关性所以提示词模板只包含作为输入和上下文的{context}和{outputs}占位符。You are an expert data labeler assessing how well LLM output aligns with and is supported by the retrieved context. Your evaluation should focus on the following criteria: Rubric A well-grounded output should: - Make claims that are directly supported by the retrieved context - Stay within the scope of information provided in the context - Maintain the same meaning and intent as the source material - Not introduce external facts or unsupported assertions outside of basic facts (2 2 4) An ungrounded output: - Makes claims without support from the context - Contradicts the retrieved information - Includes speculation or external knowledge outside of basic facts - Distorts or misrepresents the context /Rubric Instruction - Compare the output against the retrieved context carefully - Identify claims, statements, and assertions in the output - For each claim, locate supporting evidence in the context - Check for: - Direct statements from context - Valid inferences from context - Unsupported additions - Contradictions with context - Note any instances where the output: - Extends beyond the context - Combines information incorrectly - Makes logical leaps /Instruction Reminder - Focus solely on alignment with provided context - Ignore whether external knowledge suggests different facts - Consider both explicit and implicit claims - Provide specific examples of grounded/ungrounded content - Remember that correct grounding means staying true to the context, even if context conflicts with common knowledge /Reminder context {context} /context outputs {outputs} /outputs在前面演示程序的基础上我们对main函数进行如下的修改来演示基于公平性的评估。针对同一份检索结果Context给出的两个待评估的输出前者至少存在如下两个问题凭空捏造文档说潜在影响仍在实验中模型却吹嘘是最安全、完全没有副作用这直接无视了检索内容提到的12%失眠率夸大其词文中是平均体脂率下降模型改成了绝对的帮你减掉抹杀了个体差异。相比第二条输出就几乎完美贴合了上下文这一切都体现在作为输出的评估结果中。asyncdefmain():partialfunctools.partial(eval,promptRAG_GROUNDEDNESS_PROMPT,context 绿茶提取物EGCG近年来被广泛用于减脂产品中。 一项针对100名肥胖志愿者的临床研究表明在维持正常饮食的前提下每天服用 500 毫克 EGCG 持续 8 周后实验组的平均体脂率下降了2.5%。 然而研究人员也指出有12% 的受试者出现了轻微的失眠症状。 目前该成分对肝脏的长期潜在影响仍在进一步实验中。 )outputs1 绿茶提取物EGCG是目前最安全的减脂成分完全没有副作用。 只要你每天坚持服用500毫克短短两个月内就能帮你轻松减掉2.5%的体脂甚至不需要控制饮食简直是减肥神器。 outputs2 研究显示每天摄入500毫克的绿茶提取物EGCG在8周后对降低体脂有一定帮助实验中受试者平均体脂下降了 2.5%。 不过部分人群可能会出现轻微失眠且该成分的长期安全性仍有待进一步研究验证。 awaitpartial(outputsoutputs1)awaitpartial(outputsoutputs2)输出{key:score,score:0.1,comment:The output contains several claims that conflict with or go beyond the context. The context says EGCG has been widely used in fat-loss products and cites one clinical study in 100 obese volunteers where 500 mg daily for 8 weeks, while maintaining a normal diet, led to a 2.5% average body fat reduction in the experimental group. It also states that 12% of participants experienced mild insomnia and that the long-term potential impact on the liver is still under investigation. In contrast, the output claims EGCG is “the safest” fat-loss ingredient and has “no side effects,” which directly contradicts the reported insomnia and ongoing liver-safety uncertainty. It also says that one can lose 2.5% body fat in two months “without controlling diet,” which is not supported because the study explicitly mentions maintaining a normal diet, and it overgeneralizes a study result into a universal promise. The phrase “weight loss神器” is also an unsupported promotional embellishment. Thus, the score should be: 0.1.,metadata:null}{key:score,score:1,comment:The output is well grounded in the retrieved context. It correctly states that daily intake of 500 mg EGCG for 8 weeks helped reduce body fat, matching the clinical study’s result of a 2.5% average decrease in body fat rate. It also accurately mentions that some people may experience mild insomnia, which is supported by the context’s statement that 12% of participants had mild insomnia symptoms. Finally, it says the long-term safety still needs further verification, which is consistent with the context noting that the ingredient’s long-term potential impact on the liver is still under further study. The output does not introduce unsupported claims or contradictions. Thus, the score should be: 1.0.,metadata:null}3. 帮助性评估Helpfulness Evaluation帮助性评估是指评估LLM生成的回答Output在多大程度上真正切中并有效解决了用户输入的初始问题Input。如果说前面的依据性Groundedness是在检查LLM老不老实有没有开卷瞎编那么帮助性Helpfulness就是在检查LLM聪不聪明能不能干有没有真正帮用户解决实际问题。评估的核心在于解决问题的效能高帮助性直击问题的核心诉求提供准确且必需的信息内容的详略程度与问题的范围相匹配不多不少恰到好处即使答案与评估器自身的内置常识相悖例如特定系统内部检索出的独特规则只要对用户有用依然判定为正确。低帮助性没有回答到点子上顾左右而言他夹带了大量完全无关的废话无法给用户提供实际的参考价值。OpenEvals针对有害性评估的提示词模板定义在常量openevals.prompts.rag.RAG_HELPFULNESS_PROMPT具体内容如下。由于这种评估方式旨在评估作为上下文的原始输入和最终输出之间的相关性所以提示词模板只包含作为输入和上下文的{inputs}和{outputs}占位符。You are an expert evaluator assessing how helpful and relevant outputs are in addressing an input query. Your evaluation should focus on the following criteria: Rubric A helpful and relevant output should: - Directly address the core question or need in the input - Provide accurate and necessary information - Be appropriately detailed for the querys scope - May contradict your built-in knowledge but still be correct for the given context An unhelpful or irrelevant output: - Fails to address the main question - Contains primarily unrelated information /Rubric Instruction - Read and understand the full meaning of the input (including edge cases) - Identify any implicit requirements or context - Identify the expected scope of the answer - Analyze the output to identify: - How well it addresses the core question - The relevance of included information - Any critical missing information - Any extraneous or unhelpful content /Instruction Reminder - Evaluate based on practical usefulness to the query - Consider both direct relevance and helpful context - Identify specific strengths and weaknesses in the response - Provide clear reasoning for your assessment - Remember that correct information may differ from your built-in knowledge based on internal retrieved context /Reminder inputs {inputs} /inputs outputs {outputs} /outputs在前面演示程序的基础上我们对main函数进行如下的修改来演示基于公平性的评估。针对同一个问题Mac电脑突然连不上公司的Wi-Fi,给出了两种解决方案。前者说了一大推其实基本上对于问题的解决并没有多少参考意义后者不仅精准识别了“无IP”的病因给出了“续订 DHCP”这一专业且必要的动作还充分考虑到问题的紧迫性先提供“手机热点”的应急方案完美兼顾了用户“要开紧急会议”的实际诉求。评估的结果也对输出是否有帮助打出了公正的分数。asyncdefmain():partialfunctools.partial(eval,promptRAG_HELPFULNESS_PROMPT,inputs 我的Mac电脑突然连不上公司的Wi-Fi了提示**无IP地址**。 今天有紧急会议该怎么办 )outputs1 Mac电脑连不上Wi-Fi是很常见的问题。 首先请确保您的Wi-Fi路由器已经开启并正常工作。 您可以尝试关闭Mac的Wi-Fi开关等待10秒钟后再重新打开。 如果还是不行建议您检查公司其他同事的电脑是否能正常联网。 如果大家都连不上可能是公司网络故障需要联系IT部门修复。 另外保持系统更新也能减少这类网络问题的发生。 outputs2 既然您有紧急会议请立刻按照以下由快到慢的步骤排查针对‘无 IP 地址’报错 - 临时应急1分钟如果手机网络正常请立刻开启手机热点让 Mac 优先接入进入会议开完会再修网络。 - 按照如下的方式更新 DHCP租约2分钟核心解决办法这通常能立刻重新获取 IP - 点击 Mac 左上角苹果图标 - 系统设置 - 网络 - Wi-Fi - 点击公司网络旁的‘详细信息’ - 选择‘TCP/IP’标签 - 点击‘续订 DHCP 租约’。。 - 删除已知网络2分钟在同一界面中选择‘忘记此网络’然后重新输入密码连接强制系统重新初始化网络配置。 awaitpartial(outputsoutputs1)awaitpartial(outputsoutputs2)输出{key:score,score:0.62,comment:The output is somewhat relevant because it addresses a Wi‑Fi connection issue and suggests basic troubleshooting steps such as toggling Wi‑Fi, checking whether others are affected, and contacting IT, which could help in a company network context. However, it does not directly address the specific symptom of “无IP地址” (no IP address), nor does it provide the most useful urgent steps for an immediate meeting situation, such as renewing DHCP lease, forgetting/rejoining the network, checking IP/DHCP settings, or using a hotspot as a fallback. It is also somewhat generic and includes less relevant advice like system updates. Overall, it partially answers the user’s need but misses important, targeted guidance. Thus, the score should be: 0.62.,metadata:null}{key:score,score:0.97,comment:The output directly addresses the users urgent problem of a Mac failing to connect to company Wi‑Fi with a “no IP address” error. It gives immediately actionable steps prioritized for an emergency meeting: use a phone hotspot as a temporary workaround, renew the DHCP lease, and forget/rejoin the network. This is highly relevant and practical, with enough detail for quick use. It omits some additional troubleshooting options, but those are not necessary for the scope and urgency of the query. Thus, the score should be: 0.97.,metadata:null}