从0到1上手NVIDIA NemotronLabs VoiceChat-11B离线推理与流式部署完整指南【免费下载链接】NVIDIA-NemotronLabs-VoiceChat-11B项目地址: https://ai.gitcode.com/hf_mirrors/nvidia/NVIDIA-NemotronLabs-VoiceChat-11BNVIDIA NemotronLabs VoiceChat-11B是一款革命性的11B参数端到端实时语音全双工模型专为对话式AI设计能够同时执行流式语音理解和语音生成任务。与传统的级联模型ASR→LLM→TTS不同该模型采用统一架构实现全双工、实时、无缝的语音交互无需多模型或API切换显著降低了端到端延迟。 模型核心亮点与优势突破性技术特性全双工实时交互支持自然的对话轮替约450ms响应延迟和用户打断功能实现类人化交流体验一体化架构将语音编码、语言理解和语音合成为单一模型避免传统级联系统的延迟累积工具调用能力首个支持工具调用的开源全双工模型在工具执行期间保持自然对话流程高效性能在VoiceBench基准测试中位列开源全双工模型第二名平衡智能与延迟的最佳选择关键技术参数参数数值模型参数11B轮替延迟~450 ms输入类型文本提示、音频用户语音输入格式String、WAV/WebAudio输入采样率16 kHz输出类型文本代理、音频代理语音、文本用户转录输出格式String、WAV/WebAudio输出采样率22.05 kHz 快速开始环境准备与安装硬件与软件要求推荐GPUNVIDIA A100/H100/H200/B100/B200/RTX-6000操作系统Linux运行时引擎vLLM依赖工具conda、Git、Python 3.12一键安装步骤克隆仓库并切换分支git clone https://gitcode.com/hf_mirrors/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B cd NVIDIA-NemotronLabs-VoiceChat-11B git clone https://github.com/NVIDIA-NeMo/Speech.git cd Speech git switch nemotron-labs-voicechat export NEMO_DIR$(pwd)创建并激活conda环境conda create -y -n voicechat python3.12 conda activate voicechat pip install torch2.10.0 torchvision0.25.0 torchaudio2.10.0 pip install -e .[all] pip uninstall -y nvidia-resiliency-ext pip install transformers4.56.0 tokenizers0.22.0 lhotse1.32.2 \ huggingface-hub0.34.4 hf-xet1.1.9 torchcodec0.10.0 \ torch_audiomentations jinja2 pip install ninja packaging wheel einops pip install --no-build-isolation --no-deps causal-conv1d1.6.2.post1 mamba-ssm2.3.2.post1下载模型 checkpointhf download nvidia/NVIDIA-NemotronLabs-VoiceChat-11B \ --local-dir /path/to/checkpoint 离线推理快速测试与验证离线推理适用于非交互式批处理语音转语音测试只需简单几步即可体验模型能力。基本对话推理conda activate voicechat export NEMO_DIR/path/to/Speech python $NEMO_DIR/examples/speechlm2/offline_voicechat_infer.py \ --checkpoint /path/to/checkpoint \ --wav $NEMO_DIR/examples/speechlm2/sample_audio/sample_general.wav \ --output-dir /path/to/output⚠️ 注意使用自定义音频文件时需包含足够的尾随静音以便模型有时间生成响应。工具调用推理python $NEMO_DIR/examples/speechlm2/offline_voicechat_fc_infer.py \ --checkpoint /path/to/checkpoint \ --wav $NEMO_DIR/examples/speechlm2/sample_audio/sample_fc.wav \ --api-response-json $NEMO_DIR/examples/speechlm2/function_calling/random_number_response.json \ --output-dir /path/to/output运行后在输出目录中查看JSON文件可看到模型预测的工具调用格式TOOLCALL[{name: generate_random_number, arguments: {min: 1, max: 50}}]/TOOLCALL 交互式流式部署实时语音对话体验对于低延迟的交互式语音对话推荐使用优化的NVIDIA推理容器它封装了模型与NVIDIA推理栈CUDA、Triton、vLLM并提供双向WebSocket接口支持工具调用。部署主要步骤查看系统要求确认硬件、软件和驱动满足Prerequisites启动容器并运行对话按照Deploy and Run指南操作构建模型仓库如需要从本地NeMo checkpoint构建Triton模型仓库详见Generate Model RepositoryAPI参考WebSocket和HTTP接口详情请参见API Reference️ 工具调用系统提示示例默认Jinja模板会将可用工具和工具调用协议附加到提供的系统消息中。系统提示和API/工具响应必须是纯ASCII格式避免Unicode标点和符号。以下是渲染后的提示示例You are an AI voice assistant developed by NVIDIA. Your name is NVIDIA Voice Chat. Your job is to be helpful and harmless and have engaging conversations in English. Maintain a warm and friendly tone. Keep the dialogue open and ongoing. When you receive a request, follow this decision process: 1. Does the request match one of your available tools below? If yes, you MUST call that tool - never answer it directly from your own knowledge. 2. Is it a general knowledge question? If yes, answer directly from your own knowledge - do not call any tool. 3. Does it require an external action none of your tools cover? If yes, politely say you dont have that capability. You can use the following tools: AVAILABLE_TOOLS[{name: get_weather, description: Get the current weather for a city, parameters: {type: object, properties: {city: {type: string}}, required: [city]}}, {name: get_stock_price, description: Get the current stock price for a given ticker symbol, parameters: {type: object, properties: {symbol: {type: string}}, required: [symbol]}}]/AVAILABLE_TOOLS If you decide to call any tool(s), use the format: TOOLCALL[{name: tool_name1, arguments: tool_args1}]/TOOLCALL完整的默认函数调用系统提示和提示构建逻辑请参见offline_voicechat_fc_infer.py模板文件为template.jinja。 性能基准测试结果VoiceBench 性能NVIDIA NemotronLabs VoiceChat在VoiceBench基准测试中表现优异该基准专注于真实世界语音交互评估包含开放式问题、多项选择QA、指令遵循和对抗性案例等任务。Full-Duplex-Bench 1.0 结果指标数值合成语音暂停处理(TOR↓)0.153真实语音暂停处理(TOR↓)0.255平滑轮替(TOR↑)0.82平滑轮替延迟↓448 ms用户打断(TOR↑)1用户打断延迟↓480 ms用户打断主观评分(GPT-4o↑)4.33工具调用性能 (AU Harness BFCL-v3)工具调用类型准确率简单调用58.5%多工具调用62.5%并行调用42.5%并行多工具调用27.5%无关请求识别89.6%平均准确率56.1%⚠️ 已知限制与注意事项在使用过程中请注意以下限制完整列表参见Known Limitations目前仅支持英语语音交互嘈杂环境下识别准确率可能下降长对话可能出现上下文理解偏差工具调用参数复杂时可能需要多次确认 相关文档与资源许可证信息OpenMDW License Agreement, version 1.1训练数据集包含550k小时音频数据混合了真实语音和合成语音数据集技术论文SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language ModelOpen Full-duplex Voice Agent with Speech-to-Speech Language ModelAudio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models通过本指南您已掌握NVIDIA NemotronLabs VoiceChat-11B的基本安装、配置和使用方法。无论是离线批处理推理还是实时交互式部署这款模型都能为您提供高性能的语音对话体验。开始探索语音AI的无限可能吧 ️【免费下载链接】NVIDIA-NemotronLabs-VoiceChat-11B项目地址: https://ai.gitcode.com/hf_mirrors/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考