ArxivWatcher

📡 eess.AS / cs.SD
Audio and Speech Processing, Sound
2026年10月08日
LLM: deepseek-v4-flash
58
论文总数
39
跨领域
10
成功解读
48
待处理
#1
cs.RO
Stanford University (QS Top 100)NVIDIA (World Famous IT Company)Microsoft (World Famous IT Company)

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang, Zijian Du 等 (9 人)
cs.RO
查看摘要
Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.

📖 深度解读

RoboRender 论文解读

1. 一句话总结

这篇论文提出了 RoboRender,用视频生成模型把仿真器渲染的"深度视频+机器人掩码+语言指令"转换成照片级真实的 RGB 视频,让在仿真里训练的策略能零样本直接迁移到真实机器人上,平均成功率从 10% 提升到 71%。

2. 研究背景与动机

核心问题:机器人策略在仿真中训练成本低、可大规模并行,但仿真画面和真实世界画面差异巨大(即"视觉 sim-to-real gap"),导致仿真训练的策略一到真实机器人上就失效。

为什么重要:如果这个 gap 能被弥合,就意味着可以用几乎零成本的仿真数据训练出能直接部署的真实机器人策略,极大降低数据采集成本。

现有方法的不足:
- Real2Sim2Real(数字孪生):需要逐场景采集和标定,难以规模化,交互几何也常不准确。
- 几何/物体中心方法(用深度、点云、物体位姿替代 RGB):丢失了丰富的语义外观信息,且部署时需要额外的感知模块。
- 视觉域随机化(Domain Randomization):随机纹理和光照,但调参困难,生成的画面仍与真实观测差距很大,在需要精细视觉的任务(如识别抽屉把手)上完全失效。

论文的洞察是:视频生成模型已经编码了大量关于真实世界外观、语义和时序动态的先验,可以拿来"渲染"仿真轨迹。

3. 核心方法

整体框架:RoboRender 以 Wan2.1-T2V-1.3B 为基础,改造成一个"面向机器人的视频生成模型"。输入是仿真器提供的深度视频、语言指令、机器人 RGB 掩码视频,输出是照片级真实的 RGB 视频。生成的视频与仿真器提供的状态和动作配对,用于训练策略。

关键创新点:

  1. 多条件可控生成:深度视频保留场景几何和物体运动(支持多视角一致性);语言指令控制纹理、材质、光照和背景语义;机器人 RGB 掩码保留机器人运动并支持未见过的机器人形态(embodiment)。三者结合,既保留了仿真的物理基础,又注入了真实世界的视觉先验。

  2. 多视角一致性生成:将多个视角的 latent 在通道维度拼接后堆叠成统一的多视角 latent,由 diffusion transformer 联合去噪。消融实验证明,仅靠推理时的 latent 拼接不够,必须专门做多视角训练才能保证跨视角一致。

  3. 干扰物插入(Distractor Insertion):训练时随机遮挡深度图的某些区域,让模型学会在无几何约束处合成合理外观。推理时,在目标区域丢弃深度并提示新物体,就能插入跨视角一致的干扰物,作为数据增强提升策略对杂乱场景的鲁棒性。

  4. 自回归长视频生成:通过前一帧条件支持变长、长时程的机器人演示生成,推理时逐块自回归延续。

直觉解释:可以把 RoboRender 理解为一个"翻译器"——仿真器提供的是"骨架"(几何、运动、动作),视频生成模型负责给骨架"长出血肉"(真实纹理、光照、背景)。骨架不变,所以动作标签和物理正确性得以保留;血肉由生成模型填充,所以看起来像真实世界。

4. 实验与结果

视频生成质量实验:
- 数据集:500 条来自 DROID 和 AgiBot 的留出片段。
- 基线:Wan-Fun-Control(深度条件)、Cosmos-Transfer2.5(深度控制)、RoboTransfer(深度+法线+外观参考)。
- 结果(Table I,first-chunk):RoboRender 在所有指标上领先——FID 25.7(vs 基线 42.8–62.5),FVD 93.9(vs 164.5–275.4),PSNR 22.28,SSIM 0.784,LPIPS 0.142。值得注意的是 Cosmos 用了更高的 480p 输入仍不如 RoboRender。
- 人类 2AFC 研究(50 名评分者):RoboRender 在生成质量上以 68.9%–94.6% 的偏好率胜过所有基线,多视角一致性上以 74.9%–85.7% 胜出。
- 15 秒自回归 rollout:RoboRender 在每个时间步的 PSNR 和 LPIPS 都领先。

真实机器人实验:
- 任务:5 个 pick-and-place、2 个关节物体开启(冰箱、抽屉)、1 个移动操作(R1Pro 捡收音机)、1 个 YAM 机械臂任务。
- 策略:π0.5,每个任务 10 次 rollout。
- 结果(Table III):RoboRender 平均成功率 71%,Sim 仅 10%(7.1×),Sim+DR 为 20%(3.6×)。开启任务上 RoboRender 是唯一非零的方法(冰箱 70%,抽屉 90%)。
- 干扰物鲁棒性:无干扰物时 Sim/Sim+DR/RoboRender 分别为 48%/70%/80%;加干扰物后基线大幅下降,RoboRender 仅小幅下降。
- 未见机器人形态:R1Pro 和 YAM 在视频模型训练中从未见过,RoboRender 仍显著优于基线。
- 对比深度策略(Table IV):RoboRender 70%/68%/70% vs 深度策略 30%/25%/10%,说明 RGB 语义线索对识别正确物体很重要。
- 对比其他视频模型生成的数据(Table V):RoboRender 90%/70% vs Cosmos 30%/0% vs Wan 0%/0%,说明生成质量越高策略越好。

消融实验揭示:
- 每个仿真轨迹生成多个视频(用不同语言提示)能显著提升策略:开启任务从 1× 的 0% 提升到 2× 的 65%(+65 个百分点),PnP 从 48% 提升到 58%。且这是在只用一半仿真轨迹的情况下实现的,说明视觉多样性可以部分替代物理轨迹多样性。
- 3-View Stacked 消融:给基线同样的多视角 latent 拼接,仍无法达到 RoboRender 的一致性,证明优势来自专门的多视角训练而非推理技巧。

5. 优势与局限

优势:
1. 物理基础与视觉真实的结合:保留仿真器的几何、动力学和动作标签,同时用生成模型注入真实外观,避免了纯生成方法(如 DreamGen)动作需要反推的问题。
2. 多功能性:支持多视角一致生成、干扰物插入、未见机器人形态泛化,这些能力对机器人学习特别实用。
3. 数据效率:同一条仿真轨迹可以生成多条视觉多样的视频,无需成比例增加仿真数据就能提升策略性能。

局限:
1. 受限于仿真器动力学保真度:只解决视觉 gap,动力学和接触建模的误差仍然存在,可变形物体任务尤其困难。
2. 泛化验证范围有限:虽然展示了 R1Pro 的泛化,但更广泛的机器人形态、相机配置和真实环境验证仍不足。
3. 长时程复杂任务尚未验证:论文提到未来希望扩展到更复杂的长时程任务。

6. 关键结论与启发

最重要的 takeaway:视频生成模型可以作为"视觉渲染器",把仿真轨迹翻译成照片级真实视频,从而有效弥合视觉 sim-to-real gap,实现零样本策略迁移。生成质量越高,策略性能越好;从同一轨迹生成多条视觉多样的视频,可以显著提升策略学习效率。

对后续研究的启发:
- 视觉多样性作为数据增强的新维度:传统上提升策略靠增加轨迹数量,本文表明增加"同一轨迹的视觉变体"同样有效,甚至更高效,这为数据生成提供了新思路。
- 多视角一致性是机器人视频生成的关键要求:仅靠推理时的 latent 拼接不够,需要专门的多视角训练,这对未来机器人视频生成模型的设计有指导意义。
- 干扰物插入作为可控数据增强:通过深度遮挡实现可控的干扰物合成,是一种优雅的鲁棒性增强手段。
- 可能的延伸方向:扩展到可变形物体、更复杂的接触丰富任务、更多机器人形态,以及将视觉 gap 的解决与动力学 gap 的解决结合。


说明:论文全文提取完整,以上解读基于提供的完整论文内容。部分实验细节(如附录中的具体数值)在正文中有引用但未完全展开,已在相应位置注明。

🔥 推荐必读
🏷️ 领域多模态视听 > 音视频同步/对齐
💡 创新面向机器人视觉Sim-to-Real的视频生成——基于Wan2.1-T2V-1.3B改造,引入深度视频、语言指令与机器人掩码多条件控制及多视角一致性训练
⭐ 评分8.5
#2
eess.AScs.SD

Is Word Error Rate Enough? Rethinking Privacy Evaluation in Speech with Entity-Aware Metrics 跨领域

Anjana Rajasekhar, Jule Pohlhausen, Nayana Jacob Alappattu, Anna Leschanowsky
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Multimedia (cs.MM); Sound (cs.SD)
查看摘要
As the use of smart devices continues to increase, their potential to capture sensitive speech content raises growing privacy concerns. It is therefore critical to develop techniques that prevent information leakage while preserving the utility of the audio, and evaluation metrics that accurately quantify the level of privacy without overestimating it. In this work, we evaluate the effectiveness of two obfuscation techniques in protecting speech content, with particular emphasis on named entities, by adapting entity-aware privacy metrics from the Natural Language Processing field to the speech privacy domain. Further, we investigate several attack scenarios and show that fine-tuning on entity-rich data improves attack performance for some entity categories but not others. Finally, we provide guidance on metric selection based on whether the obfuscation method preserves temporal alignment.

📖 深度解读

论文解读:词错误率足够吗?用实体感知指标重新思考语音隐私评估

1. 一句话总结

这篇论文指出用词错误率(WER)来评估语音混淆技术的隐私保护效果会失真,并借鉴自然语言处理领域的实体感知指标,提出了两种更适合评估"命名实体是否泄露"的新指标,同时揭示了攻击者若用实体丰富的数据微调模型会显著提升对敏感实体的识别能力。

2. 研究背景与动机

核心问题:智能设备(智能家居、养老监护、智慧城市等)常处于"始终监听"状态,采集的语音中可能包含人名、地点、日期、机构等敏感命名实体。为保护隐私,研究者提出了多种轻量级混淆技术(如时间平滑、声音碎片化)。但问题是——我们如何准确衡量这些技术到底保护了多少隐私?

为什么重要:随着 ASR 系统性能提升,敏感信息被提取和滥用的风险增大,GDPR 等法规要求保护这类内容。如果评估指标本身失真,就可能高估或低估隐私保护水平,导致错误的技术选型或合规判断。

现有方法的不足:
- 以往研究几乎都用 WER 来评估隐私。但 WER 对所有词一视同仁——把功能词(如 "the"、"is")的错误和敏感实体(如人名、地址)的错误同等计分。而实际上只有一小部分词是敏感的。
- 这会导致两种失真:非敏感词的错误会虚高整体错误率,让人误以为隐私保护很好;而敏感实体可能仍然可识别,却被平均掉了。
- 虽然 NLP 领域已有实体感知指标(如槽位错误率 SER),但尚未被引入语音隐私评估。

3. 核心方法

论文的核心贡献是将 NLP 中的实体感知评估思想迁移到语音隐私领域,提出了一套新的评估指标,并系统分析了攻击者知识对隐私的影响。

关键创新点:

  1. 命名实体错误率(NEER)——对齐依赖型指标
    - 只统计参考文本中属于命名实体位置的替换和删除错误,忽略插入(因为插入无法归因到参考实体位置)。
    - 直觉理解:就像考试只批改"重点题",而不是所有题目平均扣分。

  2. 1 − 命名实体泄露率(1−NELR)——对齐无关型指标
    - 不依赖时间对齐,直接在 ASR 输出中搜索参考实体是否出现。每个唯一实体只计一次,只要出现在假设中就视为"泄露"。
    - 值为 0 表示全部泄露,为 1 表示完全保护。
    - 直觉理解:不管实体出现在句子的哪个位置,只要"冒出来"就算泄露——这更适合碎片化这类打乱时间顺序的方法。

  3. 模糊匹配版 1−NELR
    - 用 rapidfuzz 库做字符级相似度匹配,容忍拼写小错误。设定阈值 θ(10 到 100),相似度 ≥ θ 即视为泄露。

  4. 命名实体命中率(NEHR)——用于分析攻击模型强度
    - 全局统计测试集中有多少唯一实体至少被攻击模型识别过一次,用来衡量攻击者能力,而非评估混淆效果。

方法核心思路的直觉解释:论文认为,评估隐私保护不能"眉毛胡子一把抓"。应该像安检一样,只关注真正危险的物品(敏感实体),而不是统计所有行李的错误。同时,不同的混淆方式(是否打乱时间顺序)需要不同的"检查方式"——保持时间对齐的用 NEER,打乱对齐的用 1−NELR。

4. 实验与结果

数据集:SLUE-VoxPopuli 基准(基于 VoxPopuli 语料),专为命名实体识别设计。微调集 5000 条录音(14.5 小时,139 位说话人),测试集 1842 条(5 小时,275 位说话人)。

混淆技术:
- 声音碎片化(shredding):块长 0.5s、1s、2s、5s、10s
- 时间平滑+下采样(temporal smoothing):τ = 125/250/375/500 ms

攻击模型:
- 无知攻击者:Whisper Base、SpeechBrain-TransformerLM
- 知情攻击者:在混淆后的 LibriSpeech 上重训
- 完全知情攻击者:再在混淆后的 SLUE 微调集上微调

主要结果:

  1. WER 表现(表 2):碎片化块长越大 WER 越低(10s 时约 35%,0.5s 时超 100%);时间平滑 τ 越大 WER 越高(125ms 时约 99.9%)。

  2. 微调显著提升实体识别:在 τ=125ms 时,完全知情攻击模型比知情模型 WER 低约 20%。关键发现是——这种提升主要来自实体识别改善:非实体命中率从 59% 升到 68%,而实体命中率从 38% 猛增到 63%。

  3. 微调提升的是"领域特有实体":对两个训练集都出现过的实体,命中率几乎不变(60%→62%);但对仅出现在 SLUE 微调集的实体,命中率从 13% 飙升到 46%。

  4. 见过 vs 未见实体(图 1b):测试集中约 60% 实体在训练中见过。未见实体平均命中率比见过实体低 28%。但 PERSON 类别即使见过也只有约 5% 被识别,说明通用攻击模型对某些类别力不从心。

  5. 指标对比(图 2):
    - 高度混淆时,WER 虚高,可能高估隐私保护。
    - 轻度混淆时,实体指标可能高于 WER,说明 WER 可能低估了对敏感信息的保护不足。
    - 对时间平滑(保持对齐),NEER 和 1−NELR 结果相近。
    - 对碎片化(破坏对齐),1−NELR 显示泄露更多,说明 NEER 会漏掉部分泄露、高估隐私。

  6. 模糊匹配阈值(图 3):θ < 40 时几乎所有实体都被判为泄露(因短字符串偶然重叠);θ 在 60–80 之间是合理折中;θ ≥ 90 时趋近精确匹配。

5. 优势与局限

优势:
1. 填补空白:首次将实体感知评估指标系统引入语音隐私领域,指出 WER 的根本缺陷。
2. 实用性强:给出了明确的指标选择指南——保持对齐的混淆用 NEER 或 1−NELR 均可,破坏对齐的必须用 1−NELR。
3. 攻击分析深入:区分了"知情"与"完全知情"攻击者,揭示了实体丰富数据微调对隐私威胁的放大效应。

局限:
1. 方法覆盖有限:只评估了一种保持对齐(时间平滑)和一种破坏对齐(碎片化)的方法,结论未必能推广到其他变换策略。
2. 数据集单一且领域狭窄:SLUE 仅基于欧洲议会录音,缺乏大规模、多样化的实体标注语音数据集,影响泛化性。
3. 模糊匹配阈值缺乏定论:θ 的合理取值区间(60–80)是经验性的,是否会实际增加隐私风险仍需人类研究验证。
4. 碎片化未考虑知情攻击:作者承认因难以与微调 ASR 模型公平比较,碎片化只用了无知攻击模型,可能低估了实际威胁。

6. 关键结论与启发

最重要的 takeaway:

WER 不足以评估语音隐私保护。 它既可能高估(高度混淆时)也可能低估(轻度混淆时)隐私水平。评估敏感内容保护必须用实体感知指标,且指标选择要匹配混淆方法是否保持时间对齐。

对后续研究的启发:
1. 攻击者视角的重要性:用实体丰富数据微调能大幅提升攻击效果,尤其对领域特有实体。这提示隐私评估必须考虑"完全知情攻击者"这一最强威胁模型。
2. 类别差异化风险:不同实体类别(如 PERSON vs DATE)的可识别性差异巨大,未来可能需要类别特定的攻击模型和评估。
3. 数据集建设需求:亟需大规模、多领域、带实体标注的语音数据集,以提升评估的泛化性。
4. 模糊匹配的标准化:需要进一步研究确定合理的相似度阈值,甚至结合人类感知研究。
5. 更广泛的混淆方法验证:未来应在更多类型的混淆技术上验证这些指标的适用性。


说明:本解读基于论文提供的全文内容。论文结构完整,包含摘要、引言、实验设置、结果讨论和结论,未发现因提取不完整导致的关键信息缺失。

👀 推荐值得看
🏷️ 领域语音安全与隐私 > 语音隐私保护/匿名化
💡 创新实体感知隐私评估指标——基于 NLP 的槽位错误率思想迁移到语音隐私领域,提出 NEER 与 1−NELR 两种指标以替代失真的 WER
⭐ 评分7.5
#3
eess.AScs.SD

Intonation Perception in Real and Synthetic Speech across Varying Familiarity Levels: A Pilot Study of Equivalence Assessment 跨领域

Hanrui Zhou, Gaoyuan Zhang, Yixiang Chen, Yujie Xing, Feng Xu 等 (7 人)
Audio and Speech Processing (eess.AS); Human-Computer Interaction (cs.HC); Sound (cs.SD)
Comments: Accepted by Interspeech 2026
查看摘要
Language training relies on a corpus constructed by a large number linguistic materials. AI-powered voice clones provide a way to construct the corpus with relatively low cost. Singing voice conversion (SVC) model is used to generate synthetic voices. This study compares participants' performances on natural and synthetic speech in two experiments, similarity perception and intonation recognition. In the accuracy of similarity perception task, a significant interaction between speech type and intonation is found, suggesting that question may serve as a cue for speaker identification but may be influenced by synthetic features. In the accuracy of intonation recognition task, a significant interaction between speech type and familiarity is observed, indicating that speech type affects how much familiarity contributes to voice processing.

📖 深度解读

论文解读报告

1. 一句话总结

这篇论文通过两个行为实验(说话人相似度感知和语调识别),初步检验了AI语音克隆生成的合成语音能否在语料库建设中替代自然语音,发现合成语音在语调和说话人熟悉度相关的感知任务中与自然语音存在系统性差异。

2. 研究背景与动机

核心问题: 语言训练需要高质量语料库,但传统语料库建设成本高昂(需要专业配音演员、受控录音环境、大量数据)。AI语音克隆技术提供了一种低成本替代方案,但现有评估主要基于陈述句,尚未检验合成语音能否准确再现汉语中承载疑问功能的语调特征。

为什么重要: 在汉语普通话中,疑问句有时没有词汇标记,语调(尤其是句末上升或高平的F0曲线)是区分陈述与疑问的关键线索。如果合成语音无法准确再现这些韵律特征,就无法用于语言训练语料库的建设。

现有方法的不足:
- 已有研究主要评估合成语音的可懂度、自然度和说话人相似度,但多基于陈述句
- 语调是语音克隆系统持续面临的挑战,合成语音在韵律细节上可能存在压缩或失真
- 说话人熟悉度对语音处理的促进作用(熟悉声音加工更快、更高效)在合成语音条件下是否仍然成立,尚不清楚

3. 核心方法

方法框架: 采用2×2×2被试内设计,三个自变量为:语音类型(自然 vs. 合成)、熟悉度(熟悉 vs. 陌生)、语调(陈述 vs. 疑问)。因变量为反应时(RT)和准确率(ACC)。研究包含两个实验:说话人身份相似度感知和语调识别。

关键创新点:
1. 首次系统考察合成语音在汉语语调感知中的表现——不同于以往仅关注陈述句的研究,本文聚焦于疑问语调这一语音克隆的难点
2. 引入熟悉度变量——利用参与者对身边熟人的声音熟悉度,模拟语言训练中反复接触语音的自然过程
3. 双任务范式——同时考察说话人身份感知和语调识别,从两个维度评估合成语音的感知可行性

核心思路的直觉解释: 想象你要用AI生成的声音来制作语言学习材料。你需要知道:AI声音听起来像真人吗?(相似度感知)AI能正确表达"这是陈述还是疑问"吗?(语调识别)而且,当学习者对某个声音越来越熟悉时,AI声音的表现会和真人声音一样好吗?本研究就是通过让参与者听真人和AI的声音,完成上述两个任务来回答这些问题。

4. 实验与结果

数据集/刺激材料:
- 11个中性汉语陈述句(6词),1个用于练习,10个用于正式实验
- 2名陌生人 + 2名参与者熟悉的人录制
- 每句录制陈述和疑问两种语调
- 合成语音由F0条件化的歌声转换(SVC)模型生成
- 说话人相似度客观评分:陈述句平均0.55,疑问句平均0.51

参与者: 17人参与相似度感知实验,其中11人参与语调识别实验。年龄22-41岁,母语为汉语普通话。

主要实验结果:

实验一:说话人身份相似度感知
- 准确率: 语音类型主效应显著(χ²=11.244, p<0.001),纯自然语音对(91.7%)准确率高于自然+合成混合对(85.1%)。语音类型与语调交互显著(χ²=5.988, p=0.014):纯自然语音对中,疑问句准确率(94.6%)显著高于陈述句(87.5%);混合对中无显著差异。
- 反应时: 熟悉度主效应显著(F=6.713, p=0.010),陌生说话人反应时更长(0.848s vs. 0.748s)。
- 相似度评分: 熟悉度主效应显著(χ²=5.448, p=0.020),陌生说话人评分更高(3.93 vs. 3.56);语音类型主效应显著(χ²=3.906, p=0.048),纯自然语音评分更高(3.90 vs. 3.60)。

实验二:语调识别
- 准确率: 存在天花板效应(平均>95%)。熟悉度与语音类型交互显著(LR χ²=4.078, p=0.043):自然语音中陌生准确率略高(98.7% vs. 98.2%),合成语音中熟悉准确率略高(99.2% vs. 97.3%),但简单效应均不显著。
- 反应时: 语调边缘显著(F=3.844, p=0.050),疑问句反应时更长(0.270s vs. 0.249s)。

消融实验揭示: 本文未进行传统消融实验,但通过简单效应分析揭示了交互作用的来源。

5. 优势与局限

优势:
1. 生态效度高——使用参与者真实熟悉的人的声音,而非实验室训练形成的"熟悉感",更贴近语言训练的实际场景
2. 多维度评估——同时考察相似度感知和语调识别,从说话人身份和韵律两个层面评估合成语音
3. 统计方法严谨——对有序评分数据使用累积链接混合模型(CLMM),对高准确率数据使用Firth偏差校正逻辑回归,处理了天花板效应和分离问题

局限:
1. 样本量小——仅17名(相似度任务)和11名(语调任务)参与者,属于初步探索性研究,统计效力有限
2. 语调识别任务过于简单——准确率超过95%的天花板效应使得准确率指标区分度不足,主要依赖反应时
3. 合成语音质量有限——SVC模型生成的语音相似度评分仅0.51-0.55,合成语音本身的自然度可能影响结果解释
4. 仅使用女声——所有说话人均为女性,结果可能无法推广到男声

6. 关键结论与启发

最重要的takeaway: 当前AI语音克隆技术生成的合成语音在语调感知和说话人身份感知方面与自然语音仍存在可检测的差异。合成语音在疑问语调的说话人识别线索上表现不同,且熟悉度对合成语音加工的影响模式与自然语音相反。这表明,虽然合成语音在受控实验条件下具有一定的感知可行性,但用于构建高质量语言训练语料库仍需进一步改进。

对后续研究的启发:
1. 扩大样本量和多样性——需要更大规模、更多样化的参与者群体和语料库来验证初步发现
2. 改进合成语音质量——随着语音克隆技术迭代,需要持续评估新一代模型在韵律特征上的表现
3. 拓展到其他韵律特征——如情感韵律、焦点重音等,这些同样是语言训练语料库的重要组成部分
4. 探索训练范式的长期效果——熟悉度效应在语言训练中具有实际意义,需要纵向研究考察长期接触合成语音的效果
5. 关注个体差异——不同学习者对合成语音的感知可能存在差异,值得进一步探索

补充说明: 本文为初步探索性研究(pilot study),作者在结论中明确承认样本量有限、需要更大规模研究验证。论文全文提取完整,以上解读基于提供的完整文本。

👀 推荐值得看
🏷️ 领域评估与口语学习 > 韵律评测
💡 创新合成语音语调感知评估——基于F0条件化歌声转换(SVC)模型生成汉语陈述/疑问语调语音,结合说话人熟悉度变量,通过相似度感知与语调识别双任务范式检验合成语音能否替代自然语音用于语料库建设
⭐ 评分5.5
#4
eess.AScs.SD
University of Zurich (QS Top 100)

UZH-CL at ArA-DF 2026: Prompt-Tuned Foundation Models and Track-Adaptive Score Fusion for Arabic Speech Deepfake Detection 跨领域

Aref Farhadipour, Teodora Vukovic, Petr Motlicek
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: this is the submitted system of the UZH-CL team for ArA-DF 2026 challenge
查看摘要
Detecting synthetic and voice-converted speech remains difficult for low-resource languages with dialectal diversity, where systems must generalize across regional dialects and unseen acoustic channels. We present the UZH-CL submission to the ArA-DF 2026 Shared Task on Arabic speech deepfake detection, covering Track~1 (dialect generalization) and Track~2 (acoustic robustness). We freeze a W2V-BERT-2.0 backbone and adapt it with \emph{Wavelet Prompt Tuning}, updating under 1% of parameters, and aggregate multi-layer representations with cross-layer attention and a general attentive-statistics pooling head rather than a specialized graph backend. Complementary detectors are obtained by varying adaptation strategy, training data, augmentation, and encoder family. We find that the two shift types require different fusion regimes: a broad multi-window ensemble for dialect generalization, and a compact, channel-matched, center-crop ensemble for acoustic robustness. Official evaluation yields 1.96% EER on Track~1 (6th place) and 1.04% EER on Track~2 (3rd place), corresponding to 87% and 96% relative reductions over the XLS-R+AASIST baselines.

📖 深度解读

论文解读:UZH-CL 在 ArA-DF 2026 阿拉伯语语音深伪检测中的方案

1. 一句话总结

这篇论文提出了一套面向阿拉伯语语音深伪检测的系统:冻结大型语音模型 W2V-BERT-2.0,仅用"小波提示微调"更新不到 1% 的参数,再针对"方言泛化"和"声学鲁棒性"两类不同挑战采用不同的分数融合策略,最终在两个赛道上分别把错误率降到 1.96% 和 1.04%,相对官方基线降低 87% 和 96%。


2. 研究背景与动机

核心问题:如何检测阿拉伯语的合成语音(TTS 生成)和语音转换(VC)伪造,尤其是在方言多样、录音信道复杂(压缩、噪声、混响、传输滤波)的条件下。

为什么重要:阿拉伯语包含现代标准阿拉伯语和多种地区方言,在音系、形态、韵律上差异很大;同时真实录音常经过编解码压缩和信道干扰。这使得在英语场景下有效的检测方法难以直接迁移。这类技术可被用于媒体伪造、冒充和诈骗,因此检测具有防御价值。

现有方法的不足:
- 主流方案(如官方基线 XLS-R + AASIST)通常只取自监督模型的单一层特征,再用图神经网络后端分类,忽略了不同层携带的信息差异(低层偏声学伪影、高层偏语言内容)。
- 对大型自监督模型做全量微调计算昂贵,且在小规模目标数据上容易过拟合、覆盖预训练特征。
- 现有系统往往用一套固定策略应对所有类型的分布偏移,没有区分"方言偏移"和"信道偏移"这两种本质不同的挑战。


3. 核心方法

整体框架:两阶段流水线——先由多个"检测器"各自打分,再做"赛道自适应"的后期分数融合(见图 1)。

关键创新点:

  1. 小波提示微调(Wavelet Prompt Tuning, WPT):冻结 W2V-BERT-2.0 主干(24 层 Conformer),在每一层输入前拼接两类可训练提示向量——6 个标准提示 + 4 个小波提示。小波提示通过离散 Haar 小波变换把可训练参数分解成低频近似和高频细节子带,再投影回原维度。这样只更新不到 1% 的参数,既避免过拟合又保留预训练知识。

直觉理解:相当于给一个博学的"专家"(冻结的主干)配了一副可调节的"眼镜"(提示向量),眼镜同时捕捉粗粒度的整体特征(低频)和细粒度的突变特征(高频),而不去改动专家本身的知识。

  1. 跨层注意力 + 通用池化头:用 4 头跨层注意力对所有 24 层输出做内容相关的加权聚合,再用多头注意力统计池化(4 头、128 维瓶颈、256 维嵌入)产生分类 logits。作者强调:因为用上了所有层,所以可以用通用池化头替代 AASIST 这类专用图后端。

  2. 检测器多样化:沿四个维度构造互补检测器——适配策略(WPT vs. 部分微调 FTN)、编码器家族(W2V-BERT、XLS-R、Dasheng、Qwen3-Omni)、训练数据规模(官方 22.5k / 阿拉伯语约 100k / 多语言约 264k)、数据增强组合(DSU、RawBoost、MUSAN 噪声、RIR 混响、有损压缩、语速扰动)。

  3. 赛道自适应融合:这是论文的核心发现——两类偏移需要不同的融合方式:
    - 赛道 1(方言泛化):用多窗口重叠切分(4.04 秒窗、2.02 秒步长)取平均,融合更广、相关性更低的检测器集合,对外部数据模型(FTX、MEGA、FTXA)加权 1.5。
    - 赛道 2(声学鲁棒):用单个 4 秒中心裁剪,融合紧凑、信道匹配的核心集合(权重 1.0),辅助模型降权到 0.5。

融合前对每个检测器分数做 z-score 标准化(式 3),再加权平均。


4. 实验与结果

数据集/基准:ArA-DF 2026 共享任务,22.5k 训练、21k 开发语句,覆盖 MSA 和地区方言。评估分两赛道,各含开发测试集(赛道 1:16.0k;赛道 2:14.2k)和最终测试集(赛道 1:144.2k;赛道 2:127.7k)。指标为等错误率(EER)。

对比基线:官方基线 XLS-R 300M + AASIST(赛道 1 最终 EER 14.54%,赛道 2 为 27.11%)。

主要结果:
- 官方最终测试:赛道 1 为 1.96% EER(第 6 名),赛道 2 为 1.04% EER(第 3 名),相对基线分别降低 87% 和 96%。
- 融合始终优于任何单一检测器:赛道 1 从最佳单模型 3.01% 降到 1.96%;赛道 2 从 1.39% 降到 1.04%。1000 次自助重采样显示融合与最佳单模型的 95% 置信区间不重叠。
- W2V-BERT 系列稳定优于其他主干:XLS-R(赛道 1 为 5.84%)、Dasheng(12.10%)在方言偏移下表现差;Qwen3-Omni 在赛道 2 达到 3.03%,对融合有贡献。

消融/分析揭示的关键现象:
- 分布内开发 EER(0.04–1.37%)完全不能预测分布外鲁棒性,测试 EER 会上升到 2.76–17.32%(5–200 倍差距)。
- 相关性分析:W2V-BERT 模型之间高度相关(r = 0.85–0.97),而外部语料训练的模型相关性低(r = 0.46–0.62),正是这些"去相关"模型提升了赛道 1。
- 赛道 2 中把不匹配的检测器全权重加入反而使 EER 升高(1.10%),只用方言模型 DIA 也升高(1.09%),说明紧凑的信道匹配集合更优。


5. 优势与局限

主要优势:
1. 参数高效:只更新不到 1% 的参数即可适配大型语音模型,训练成本低、过拟合风险小。
2. 性能显著:相对官方基线实现 87%/96% 的相对错误率下降,且融合稳定优于所有单模型(统计显著)。
3. 洞察有价值:明确区分并验证了"方言偏移"与"信道偏移"需要不同融合策略,这一发现对后续系统设计有指导意义。

局限性(论文自述):
1. 融合是转导式的(transductive):z-norm 依赖整个评估集统计量,单个文件的分数取决于批次中其他文件,无法逐文件独立复现;论文未报告使用开发集统计量的归纳式版本。
2. 单次训练、无种子方差:每个检测器只跑一次,表 1 中接近的 EER(如 3.01–3.15)不能视为稳定结果。
3. 外部数据未去重:因盲评未公开说话人/录音/合成系统身份,外部阿拉伯语和多语言池未与 ArA-DF 评估材料去重,赛道 1 融合包含外部数据模型,需谨慎解读;此外未报告准确率和宏 F1 等次要指标。


6. 关键结论与启发

最重要的 takeaway:面对语音深伪检测中的分布偏移,"一刀切"的融合策略行不通——方言偏移偏好"广覆盖 + 多窗口 + 去相关模型",信道偏移偏好"紧凑 + 中心裁剪 + 信道匹配模型"。同时,冻结主干 + 轻量提示微调 + 全层聚合,是一种兼顾性能与效率的有效范式。

对后续研究的启发:
- 可探索归纳式(inductive)融合,用开发集统计量替代转导式 z-norm,使系统可实际部署。
- 需要多种子重复实验来确认相近 EER 差异的稳定性。
- 外部数据与评估集的去重问题值得在共享任务设计中解决,以更公平地评估泛化能力。
- "不同偏移需要不同融合"这一结论可推广到其他低资源、多方言语言的深伪检测任务。
- 不同编码器(波形前端 vs. 频谱前端)对偏移类型的敏感性差异(如 Dasheng/Qwen3-Omni 保留更多信道线索)值得进一步机理研究。


说明:本解读基于论文提供的全文(含摘要、正文、表格、参考文献),内容完整,未出现因提取不全导致的缺失。

👀 推荐值得看
🏷️ 领域语音安全与隐私 > AI 生成检测
💡 创新小波提示微调与赛道自适应分数融合——基于冻结的 W2V-BERT-2.0 主干,在每层注入可训练的小波提示向量并跨层聚合,再针对方言偏移与信道偏移分别设计不同的融合策略
⭐ 评分7.5
#5
eess.AScs.SD
University of Zurich (QS Top 100)

Unified Shared Encoder in Spoof-Aware Speaker Verification with Hybrid Wavelet Prompt Tuning 跨领域

Aref Farhadipour, Srikanth Madikeri, Teodora Vukovic, Volker Dellwo, Petr Motlicek
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: accepted at IEEE SLT 2026
查看摘要
Spoofing-aware speaker verification (SASV) must confirm who is speaking and that the speech is genuine, but current systems gain one at the expense of the other. Modular cascades are accurate yet run two large self-supervised encoders, whereas end-to-end models are efficient but lose accuracy when a single embedding has to serve two conflicting objectives. We show that this trade-off between efficiency and specialization is avoidable. A single frozen W2V-BERT~2.0 backbone is adapted per task by deep hybrid Wavelet Prompt Tuning (WPT), so each branch obtains its own view of the shared encoder through dedicated prompts and a task head, and the scores are combined only at inference. Training only about 7M parameters, a small fraction of those a strong two-encoder cascade requires, the system surpasses it on SpoofCeleb evaluation with 0.03% CM-EER and 0.038 min a-DCF against 0.16% and 0.047. Because the backbone is shared and frozen, new branches attach without retraining the heads and prompts already in place. On ASVspoof5, with adversarial attacks and codec distortions, it reaches 0.092 min a-DCF and 3.60% CM-EER using only the provided data, indicating that the design holds under realistic in-the-wild threats.

📖 深度解读

论文解读:统一共享编码器 + 混合小波提示调优的欺骗感知说话人验证

1. 一句话总结

这篇论文提出了一种"共享一个冻结大模型、用轻量提示分别适配说话人验证和伪造检测两个任务"的框架,只用约 700 万可训练参数就超过了需要两个大模型级联的强基线,同时保持了架构的可扩展性。


2. 研究背景与动机

核心问题:欺骗感知说话人验证(SASV)需要同时完成两件事——确认"说话人是谁"(ASV)和确认"这段语音是不是真的"(CM,反欺骗)。现有系统往往顾此失彼。

为什么重要:生成式 AI 让 TTS、语音转换、神经深度伪造越来越逼真,ASV 系统面临严重安全威胁。SASV 是把 ASV 和反欺骗统一起来的防御范式,直接关系到生物特征安全。

现有方法的不足:
- 模块化级联(两个独立子系统):精度高,但要跑两个大型自监督编码器(如 WavLM + XLSR),计算冗余严重,且两个子系统内部表示从不交互,容易误差传播;扩展性差,加一个任务就要加一个骨干网络。
- 端到端单嵌入模型:效率高,但要求同一个嵌入同时满足身份和真实性两个相互冲突的目标,精度往往不如级联方案。

论文的核心主张:效率与专业化之间的权衡是可以避免的。


3. 核心方法

框架名称:Unified Shared Encoder(USE),即统一共享编码器。

核心思路(直觉版):想象一个博学多才的专家(冻结的 W2V-BERT 2.0 大模型),他知识渊博但不针对具体任务。论文不去重新训练他,而是给每个任务配一套"专属提示卡片"——ASV 任务递上身份相关的提示,CM 任务递上真伪相关的提示。专家看着不同的提示,就会把注意力引向不同的特征。两个任务共用同一个专家(省内存),但各自看到不同的"视角",最后只在打分阶段把两个分数相乘合并。

关键创新点:

  1. 共享冻结骨干 + 任务专属提示:只训练提示(prompt)和任务头,骨干 6 亿参数完全冻结。这既避免了单嵌入冲突,又避免了双骨干开销。选择提示调优而非 LoRA/Adapter/全微调,正是因为提示不碰任何骨干权重,才能让多个任务分支独立挂载。

  2. 混合小波提示调优(Hybrid WPT):每层注入两类提示——常规提示(15 个 token,提供灵活的任务适配)和小波提示(8 个 token)。小波提示先经过二维 Haar 离散小波变换,分解出低频近似和三个高频细节子带再拼回。直觉是:合成语音的伪造痕迹常表现为局部高频扰动,小波基提供了多分辨率的先验,引导注意力关注这些细节。

  3. 深层提示注入:提示不是只加在输入层,而是在全部 24 层每一层都重新注入,且每层有自己独立的提示。只有音频帧作为 Query,提示作为 Key/Value 提供"虚拟上下文",处理完后提示被丢弃,只传音频帧到下一层。

  4. 两个任务头采用不同的层融合策略:
    - ASV 头用静态加权求和融合 24 层(因为说话人身份是全局稳定属性,固定权重更稳、方差更低)。
    - CM 头用跨层注意力自适应融合(因为伪造线索在不同攻击中出现在不同抽象层级,需要动态重标定)。

分数整合:ASV 余弦相似度经 sigmoid 校准到 [0,1],CM 直接输出真实语音后验概率,最终 SASV 分数 = 两者乘积(相当于一个 AND 门,两个任务都认可才高分)。这个规则固定、无参数、与试次无关。


4. 实验与结果

数据集/基准:
- SpoofCeleb(主基准):源自 VoxCeleb1,保留真实环境噪声、混响、信道失真,1251 个说话人,约 15 万真实语音 + 约 240 万合成语音(23 种合成系统)。
- ASVspoof5(Track 2):源自多语言 LibriSpeech,1922 个说话人,约 68 万条语音,评估集含 32 种攻击(含对抗样本)和 7 种编解码条件。

对比基线:
- E2E SASV(SKA-TDNN,WildSpoof 官方基线,31M 参数)
- 模块化级联(WavLM-ECAPA-TDNN + W2V-AASIST,320M 参数)

主要结果(SpoofCeleb 评估集):

系统 可训练参数 CM-EER ASV-EER min a-DCF
E2E 基线 31M 49.62% 11.64% 0.230
级联基线 320M 0.16% 3.86% 0.047
USE(本文) 7.0M 0.03% 3.65% 0.038

USE 仅训练约 700 万参数(不到级联基线的 1/45),CM-EER 从 0.16% 降到 0.03%,min a-DCF 从 0.047 降到 0.038,在官方榜单上排名第二。

ASVspoof5 评估集:min a-DCF 0.092,CM-EER 3.60%,仅用官方提供数据、单一 W2V-BERT 2.0 骨干,在开放条件下排名第二,优于所有闭集系统。

消融实验揭示:
- 提示贡献最大:去掉所有提示,eval min a-DCF 从 0.038 恶化到 0.086;仅常规提示恢复到 0.044,加上小波提示进一步到 0.038(CM-EER 从 0.15% 降到 0.03%)。
- 去掉 Refiner 模块:min a-DCF 退化到 0.055。
- 去掉多层融合:SASV-EER 降到 4.93%。
- 跨层注意力换成加权求和:参数从 7.0M 降到 3.2M,性能仅轻微下降(0.038→0.039),说明该组件性价比不高。

可扩展性验证:额外加一个在 VoxCeleb2 上训练的 ASV 头(仅增 1.5M 参数),两个 ASV 头分数平均后,eval SASV-EER 降到 2.73%,min a-DCF 到 0.035。作者明确说明这是模块化能力的演示,因辅助头用了域外数据,不能与基础配置直接对比。

ASVspoof5 细分:编解码维度误差在 2.88%–4.48% 之间(最压缩的 C11 最高);攻击维度差异大,最好的 A29 仅 0.48%,最难的 A28 达 7.74%,说明整体误差主要集中在少数困难攻击上。


5. 优势与局限

主要优势:
1. 极高的参数效率:仅调约 1% 参数即超越 320M 参数的级联系统,内存占用小,适合部署。
2. 模块化可扩展:骨干冻结,新任务头可独立挂载,无需重训已有模块——这是级联和端到端方案都难以做到的。
3. 任务适配设计有针对性:ASV 用静态层融合、CM 用动态跨层注意力,且小波提示为伪造检测提供多分辨率先验,符合两个任务的本质差异。

局限性:
1. ASV 性能提升有限:ASV-EER 3.65% 相比级联基线的 3.86% 提升不大,主要增益集中在 CM 侧。
2. ASVspoof5 上误差集中:部分生成式攻击(如 A28 达 7.74%)仍难处理,整体 CM-EER 远高于 SpoofCeleb,说明在困难攻击子集上鲁棒性不足。
3. 可扩展性结论需谨慎:扩展实验用了域外 VoxCeleb2 数据,作者自己也承认基础配置与扩展配置不可直接比较,核心架构本身的增益证据有限。
4. 推理需两次前向:虽然共享骨干内存,但 ASV 和 CM 各需一次提示条件化的前向传播。


6. 关键结论与启发

最重要的 takeaway:不必在"高效但精度受损的端到端"和"精确但昂贵的双骨干级联"之间二选一。共享一个冻结的通用编码器,用参数高效的提示为每个任务提供独立视角,只在分数层面合并,可以同时拿到效率和专业化——这是本文最核心的架构洞察。

对后续研究的启发:
1. 提示调优作为多任务共享骨干的通用范式:冻结骨干 + 任务专属提示的思路可推广到其他需要多任务共享大模型的安全场景。
2. 层融合策略应任务定制:论文发现 ASV 和 CM 依赖不同的层分布(ASV 偏深层语义,CM 兼顾浅层频谱和深层长程依赖),提示多任务系统不应统一融合策略。
3. 小波先验在伪造检测中的价值:将小波变换作用于提示 token(而非音频本身)是一个新颖的注入多分辨率先验的方式,值得在其他细粒度检测任务中探索。
4. 困难攻击子集应成为优化重点:论文指出不应只看榜单聚合分数,攻击维度的分解揭示了真正的短板所在。


说明:本解读基于提供的论文全文,内容完整,未出现因提取不完整导致的信息缺失。文中"论文声称"与"实际展示"已尽量区分——例如可扩展性增益作者本人标注为演示性质而非严格对比。

👀 推荐值得看
🏷️ 领域语音安全与隐私 > 反欺骗/伪造检测说话人技术 > 说话人识别/验证
💡 创新欺骗感知说话人验证——基于冻结的 W2V-BERT 2.0 共享编码器,引入混合小波提示调优(常规提示+Haar 小波提示)与任务定制层融合,实现 ASV 与 CM 双任务参数高效适配
⭐ 评分7.5
#6
eess.AS

Indic-CLAP: Text--Audio Understanding in Indic Languages via Cross-Lingual Distillation

Sanat Kumar Agrawal, Srikanth Raj Chetupalli
Audio and Speech Processing (eess.AS)
Comments: Submitted to ICASSP 2027
查看摘要
Contrastive Language--Audio Pretraining (CLAP) learns a joint text--audio representation space, which enables zero-shot audio understanding from natural-language descriptions. However, its English-only text encoder restricts CLAP to tasks and datasets expressed in English. We present Indic-CLAP, a multilingual text--audio model that extends CLAP to nine Indic languages spoken by over a billion people. Exploiting CLAP's dual-encoder structure, we propose to train only an Indic text encoder by distillation while keeping the audio encoder frozen, using machine-translated AudioCaps captions. We compare distillation from CLAP's text encoder, audio encoder, and a novel hybrid objective combining both. Our evaluation shows that the Indic-CLAP encoder inherits the cross-modal alignment from the CLAP teacher, as evidenced by the performance on cross-modal retrieval and zero-shot classification tasks. Further, we study the cross-modal alignment using modality gap analysis, which explains the performance trends.

📖 深度解读

Indic-CLAP 论文解读

1. 一句话总结

这篇论文提出了 Indic-CLAP,通过"跨语言蒸馏"的方式,把原本只会英语的 CLAP 文本-音频模型扩展到九种印度语言,让超过十亿使用者可以用母语进行零样本音频理解。

2. 研究背景与动机

核心问题:CLAP(对比语言-音频预训练)能学习一个文本和音频共享的表示空间,从而支持零样本音频分类、检索、标注等任务。但它的文本编码器基于 RoBERTa,只懂英语,训练数据也全是英文音频描述,因此无法处理非英语任务和数据集。

为什么重要:印度语言(印地语、孟加拉语、泰米尔语等九种)使用者超过十亿,但据作者所述,目前没有任何公开的印度语言音频描述数据集,也没有任何模型针对印度语言音频-文本任务进行训练或评估。这是一个明显的空白。

现有方法的不足:
- GLAP 虽然加入了多语言能力,但只覆盖四种非英语语言,且不包含任何印度语言;
- RA-CLAP 等其他工作同样没有多语言能力;
- 已有的"翻译描述"思路只在文本-音频检索和音频描述生成中被初步研究过,印度语言方向完全缺失。

3. 核心方法

方法框架:Indic-CLAP 利用 CLAP 的双编码器结构(音频编码器 A + 英文文本编码器 T_en),只训练一个新的印度语言文本编码器 T_in,让它的输出落在 CLAP 已有的共享嵌入空间中,从而"继承"与音频编码器的对齐关系。音频编码器和英文文本编码器全程冻结,训练时甚至不需要读取音频(嵌入已预计算缓存)。

关键创新点:

  1. 跨语言蒸馏 + 冻结双教师:用机器翻译的 AudioCaps 描述作为训练数据,让印度语言学生编码器模仿 CLAP 的教师编码器。这是 M-CLIP 思路在音频领域的延伸,也是首次用于印度语言。

  2. 四种蒸馏目标的系统比较:包括文本教师 MSE(TT-MSE)、音频教师 MSE(AT-MSE)、音频教师 InfoNCE(AT-NCE),以及一个混合目标——同时从文本编码器(用余弦相似度)和音频编码器(用 InfoNCE)蒸馏,通过超参数 λ 平衡两者。

  3. 用"模态鸿沟"(modality gap)做机理解释:作者不仅报告性能,还用模态质心余弦相似度和匹配对相似度来量化文本空间与音频空间之间的错位,解释为什么混合目标在检索任务上更好。

  4. 架构无关性验证:把同样的蒸馏策略应用到 GLAP 上,证明方法不依赖于特定架构。

直觉解释:CLAP 的文本和音频编码器虽然被训练到同一个空间,但两个模态实际上各自"抱团"(存在模态鸿沟)。如果只模仿文本教师,学生会继承文本侧的概念结构(分类强)但离音频侧远(检索弱);只模仿音频教师则相反。混合目标相当于让学生同时靠近两个"极点",缩小鸿沟,从而在检索上表现更好。

4. 实验与结果

数据集/基准:
- 训练:AudioCaps(52.25K 音频,英文描述,机器翻译成九种印度语言)
- 检索评估:AudioCaps 测试集、Clotho 测试集
- 零样本分类:ESC-50(50 类)、UrbanSound8K(10 类)
- 翻译系统对比:Helsinki-NMT vs GPT-4o-mini

对比基线:CLAP 教师模型、GLAP(零样本 + 微调)、以及四种蒸馏目标之间的对比。

主要结果:

  • 翻译质量至关重要:用 Helsinki-NMT 翻译的印地语模型 DF 仅 37.2、ZSC 57.7%;换成 GPT-4o-mini 后 DF 升到 63.6、ZSC 84.8%,检索也全面提升。原因是 NMT 常把拟声词(sizzling 等)译错或漏译,而 LLM 能保留声音事件结构。

  • 多语言联合训练优于单语言模型:平均 DF 从 57.4% 提升到 70.5%,A→T 检索从 30.6% 到 29.5%(略降),但整体更稳。有趣的是,Indic-CLAP 在 A→T 检索上甚至超过了 CLAP 教师(29.5% vs 24.5%),作者归因于数据不对称——多个描述对应同一音频,使 A→T 任务更容易。

  • 混合目标在检索上最佳:λ=0.10 时 A→T R@1 达 38.8%、T→A R@1 达 25.4%,明显优于纯 TT-MSE(29.5%/22.1%)和纯 AT-NCE(28.2%/23.4%),但 ZSC 略降(73.3% vs 75.3%)。

  • 与 GLAP 对比:GLAP 零样本在印度语言上检索很强(A→T 40.6%),但 ZSC 只有 38.5%;用本文目标微调后 ZSC 提升到 75.0%,检索也接近 GLAP 教师水平。

  • 跨数据集泛化:在 Clotho 上,混合目标仍最好,但与教师存在明显差距(作者归因于 CLAP 训练时见过 Clotho,而 Indic-CLAP 只见过 AudioCaps)。

消融实验揭示:
- 模态鸿沟分析(表 5)显示,CLAP 文本和音频质心位于相反半球(质心 CS = −0.34)。TT-MSE 学生 Gap = +1.02(完全偏向文本极),AT-MSE 学生 Gap = −0.07(偏向音频极),混合目标 Gap = +0.96 但匹配对相似度更好,说明它略微改善了跨模态对齐,这正是检索提升的机制来源。
- λ 扫描显示:λ=0 退化为纯文本对齐,λ=1 退化为 AT-NCE,中间值(尤其 0.10)在检索和分类间取得最佳平衡。

5. 优势与局限

优势:
1. 填补空白:首个针对九种印度语言的文本-音频模型,覆盖超十亿人口的语言需求。
2. 训练高效:只训练一个文本编码器,音频编码器冻结且嵌入预计算,训练时无需处理音频,计算成本低。
3. 方法可迁移:蒸馏策略在 GLAP 上同样有效,说明不绑定特定架构;混合目标配合模态鸿沟分析提供了可解释的机理。

局限:
1. 依赖机器翻译质量:性能对翻译系统高度敏感,且印度语言音频描述数据本身不存在,只能靠翻译英文数据,可能引入文化/语言偏差。
2. 零样本分类仍有明显差距:与英文 CLAP 教师相比 ZSC 差约 17 个百分点,DF 饱和在 75% 左右,学生未能完全复现教师的概念结构。
3. 跨域泛化有限:在 Clotho 等未见数据集上检索性能明显下降,说明模型对训练域(AudioCaps)依赖较强。

6. 关键结论与启发

最重要的 takeaway:通过跨语言蒸馏,可以让一个冻结的英文 CLAP 模型"教会"印度语言文本编码器,使其继承跨模态对齐能力;而同时从文本和音频两个教师蒸馏(混合目标)能缩小模态鸿沟,在检索任务上甚至超过教师本身。此外,LLM 翻译明显优于传统 NMT,是这类低资源语言音频任务的关键设计选择。

对后续研究的启发:
- 模态鸿沟可以作为设计蒸馏目标的显式优化信号,而不仅是事后分析工具;
- 该框架可推广到更多低资源语言,或结合语音而非纯文本的跨语言表示;
- 未来可探索如何突破 DF 饱和(75%)瓶颈,例如引入更多样化的翻译数据、多教师集成或对齐正则化;
- 印度语言音频描述数据集的构建本身就是一个值得投入的方向。


说明:论文全文提取完整,以上解读基于提供的完整文本。个别数值(如表 1 中多语言模型 A→T 平均 29.5% 略低于单语言 30.6%)在原文中即呈现这种非单调趋势,解读中已如实反映。

👀 推荐值得看
🏷️ 领域音频理解 > 音频-文本检索
💡 创新跨语言蒸馏——基于冻结的英文 CLAP 双编码器,训练印度语言文本编码器并系统比较四种蒸馏目标(含文本+音频混合目标)以缩小模态鸿沟
⭐ 评分6.5
#7
eess.AS

Timestamped Hindi speech transcription using Whisper

Sanat Kumar Agrawal, Srikanth Raj Chetupalli
Audio and Speech Processing (eess.AS)
Comments: Submitted to ICASSP 2027
查看摘要
Time accurate speech transcription is essential in critical appli- cations, such as medical conversation transcription, reading as- sessment, and atypical speech recognition. Encoder-decoder auto- matic speech recognition models, such as Whisper, do not produce word-level timestamps out-of-the-box for all languages. Existing approaches to word-level time stamp estimation either use an exter- nal CTC model capable of frame level phoneme/character prediction which is computationally expensive, or use the cross attention scores internal to the model which gives coarse time stamps. We consider timestamped speech transcription for Hindi in this paper. We pro- pose a CTC-based approach, in which we train a low-complexity character level CTC head directly on the Whisper encoder output, to address the complexity and granularity limitations of existing works. We evaluate the system using manually-annotated word-level times- tamps. Experiments show that the proposed internal-CTC based approach performs similar to using an external CTC model and better than the cross attention score-based approaches. Further, we study the usage of CTC-head output to detect and mitigate hallu- cinations in conversational speech. Our results show that a simple token length criterion derived from the CTC output can improve the transcription quality for short segments where hallucinations are more probable.

📖 深度解读

论文解读:基于 Whisper 的印地语时间戳语音转写

1. 一句话总结

这篇论文在 Whisper 编码器上"外挂"了一个极轻量的字符级 CTC 头,用极少的额外参数实现了印地语语音转写的词级时间戳估计,精度与需要额外大模型(wav2vec2.0)的 WhisperX 相当,同时还能顺便检测和缓解短语音片段中的"幻觉"问题。


2. 研究背景与动机

核心问题:Whisper 这类编码器-解码器 ASR 模型虽然转写能力强,但开箱即用时无法给出词级时间戳(尤其是印地语等低资源语言)。而时间戳在医疗对话转写、阅读流利度评估、字幕生成、关键词检索等场景中至关重要。

现有方法的不足:

  • 基于交叉注意力 + DTW 的方法(如 whisper-timestamped、CrisperWhisper):直接利用模型内部的注意力分数做对齐,计算简单、通用性强,但时间戳粒度粗、误差大。
  • 基于外部 CTC 强制对齐的方法(如 WhisperX):用另一个独立训练的 wav2vec2.0 模型做帧级对齐,精度高,但需要第二个模型,计算开销大、复杂度高,且第二个模型往往也是语言特定的(WhisperX 的 wav2vec2.0 就是英语/特定语言模型),两个模型的误差还会叠加。
  • 印地语/印度语系空白:CrisperWhisper 和 WhisperX 都是为英语设计的,印地语目前没有现成的时间戳转写方案。

动机:能否不引入第二个模型,直接在 Whisper 编码器输出上训练一个轻量 CTC 头,既省算力又能获得细粒度时间戳?


3. 核心方法

方法框架:在 Whisper 编码器输出(50Hz 帧率、1280 维表示)之上,接一个单层线性层作为 CTC 头,输出字符级后验概率。然后用强制对齐(forced alignment,基于 DTW)把 CTC 帧级后验与 Whisper 解码器输出的转写文本对齐,从而得到每个词的起止时间。

关键创新点:

  1. 极轻量的字符级 CTC 头:词表仅 73 个符号(71 个天城文字符 + 空格 + CTC blank),额外参数仅约 9.35 万。相比之前工作([9] 用 8k 词表、[10] 用约 50k 词表),参数量和计算量都大幅降低。
  2. 字符级而非 BPE 级对齐:作者指出 Whisper 用的天城文 BPE token 平均只覆盖 0.77 个字符,token 边界与字符边界不对齐,不适合做精细时间戳;字符级单位让对齐分辨率更高、边界更规整。
  3. 无需第二个模型:直接复用 Whisper 编码器,避免了 WhisperX 那种"双模型误差叠加"和语言特定对齐器的限制。
  4. CTC 头兼作幻觉检测器:利用 CTC 输出的字符数作为"预期转写长度",用于截断/纠正 Whisper 在短片段上的幻觉性输出。

直觉解释:可以把 Whisper 编码器想象成一个已经"听懂"了语音的模块,作者只是在它后面加了一个很小的"标尺",把这个标尺的刻度(字符后验)和 Whisper 写出的文字对齐,就能知道每个字、每个词大概在音频的什么位置。因为标尺很小,训练和推理都很便宜。


4. 实验与结果

数据集:
- IndicVoices-R(印地语子集):26.3K 训练语句,125 条测试语句(1.89K 参考词),并人工标注了词边界(100 条双人标注,25 条单人标注,共 3784 个评分边界)。
- MPS-Hindi:儿童朗读语料(域外)。
- GramVaani:方言印地语电话 spontane 语音。
- DISPLACE-M:农村医疗对话,含重叠语音、背景人声、方言(对话场景)。

基线方法:WhisperX(外部 wav2vec2.0 对齐)、CrisperWhisper(适配到印地语)、NeMo 强制对齐器(NFA),以及原始 Hindi-Whisper。

主要结果:

  • 时间戳精度(表 2,对人工标注):wav2vec2.0 原始误差 70.9ms,提出的 M1–M6 在 73–80ms 之间,差距 <10ms;经过固定偏移校准后,提出方法(约 41–43ms)与 wav2vec2.0(43.4ms)基本持平。NFA 反而更差(79.5ms 原始 / 52.1ms 校准)。
  • 联合 ASR + 时间戳(表 3):所有配置的中位边界偏差都小于一个 Whisper 帧移,超过 92% 的边界误差在 50ms 以内。CrisperWhisper 偏差极大(中位 60.1ms,p90 高达 236.9ms)。
  • 跨语料泛化(表 4):在 DISPLACE-M 对话数据上,加入 CTC 头(M3/M6)相比仅微调的 M4 显著降低 WER(如 M3 35.60% vs M4 39.10%),主要来自插入错误大幅减少(7.28% vs 11.73%),说明 CTC 头起到了抑制幻觉的正则化作用。
  • 错误纠正(表 5):在 DISPLACE-M 上,beam search 反而让 WER 变差(35.60%→41.15%);基于 CTC 字符数截断的 "cap" 策略把 WER 降到 32.88%(M3),接近 oracle 上限 31.25%,插入错误从 7.28% 降到 3.66%。

消融发现:
- 只训练 CTC 头(M1)不改变转写质量,但能提供时间戳。
- 只微调编码器(M2/M5)会同时恶化 WER 和边界误差。
- 微调解码器(M3/M4)对适配新数据、提升泛化很关键。
- 在对话语音上,CTC 头与 Whisper 联合训练(M6)比先微调再加头(M4)更鲁棒。


5. 优势与局限

优势:
1. 参数与计算高效:仅增加约 9.35 万参数,无需第二个对齐模型,训练和推理都轻量。
2. 精度可比 WhisperX:在印地语上时间戳精度与使用外部 wav2vec2.0 的 WhisperX 相当,且优于 CrisperWhisper 和 NFA。
3. 附带幻觉抑制能力:CTC 头输出可用于检测和纠正短片段幻觉,在对话语音上显著降低插入错误。

局限:
1. 存在系统性时间偏移:CTC 倾向于提前预测字符,导致起始/结束时间有固定偏差(onset +36~44ms,offset −100~−114ms),需要额外校准才能达到最佳精度。
2. CTC 头本身的转写质量差:作者明确说明 CTC 头输出的 WER 通常很高,因此不报告、也不直接用作转写,只能作为对齐和长度参考。
3. 语言/模型绑定:方法依赖已有的 Hindi-Whisper 预训练模型,迁移到其他语言需要相应的 Whisper 微调版本;测试集规模也较小(125 条)。


6. 关键结论与启发

最重要的 takeaway:不必引入第二个大型对齐模型,仅在 Whisper 编码器上加一个极小的字符级 CTC 头,就能以极低成本获得与 WhisperX 相当的时间戳精度,并且这个 CTC 头还能"顺手"用来抑制 Whisper 在短语音上的幻觉——一举两得。

对后续研究的启发:
- 可以探索联合 CTC/注意力解码的变体,让 CTC 头不仅用于对齐,还能直接改善转写质量。
- 字符级 CTC 头的思路可推广到其他印度语系及低资源语言。
- 用 CTC 输出长度作为"预期转写长度"来约束解码,是一个简单但有效的幻觉缓解范式,值得在更多对话/短片段场景中验证。
- 系统性时间偏移的自动校准(无需人工标注)也是一个值得研究的方向。


说明:本文全文提取完整,以上解读基于论文提供的摘要、方法、实验与结论部分。论文发表于 arXiv(编号 2610.08847v1),作者为 IIT Bombay 的 Sanat Kumar Agrawal 和 Srikanth Raj Chetupalli。

👀 推荐值得看
🏷️ 领域语音识别 (ASR) > 标准/端到端 ASR
💡 创新词级时间戳估计——基于 Whisper 编码器外挂极轻量字符级 CTC 头,无需第二个对齐模型即可实现与 WhisperX 相当的时间戳精度并抑制幻觉
⭐ 评分7.5
#8
eess.AS

Unheard but Recognizable: Ultrasonic Signatures for Musical Instrument Recognition

Izhak Kapash, Uri Rom, Ram Zamir
Audio and Speech Processing (eess.AS)
Comments: 5 pages, 3 figures, 3 tables. Submitted to IEEE ICASSP 2027
查看摘要
Musical-instrument recognition is a fundamental music information retrieval (MIR) task supporting transcription, source separation, and audio understanding. Despite significant algorithmic progress, there is still room for improvement in recognizing acoustically similar instruments, particularly in polyphonic mixtures. We examine the contribution of ultrasonic components, i.e., frequencies above the audible 0-20 kHz range. Although most of this content cannot be represented at conventional 44.1/48-kSPS sampling rates under the Nyquist criterion, it is available in high-quality 96-kSPS recordings. Using a 96-kSPS multitrack corpus covering 15 classes across performers, instruments, studios, and sessions, we compare full-band, low-pass, and ultrasonic-only inputs using classical machine-learning and deep-learning classifiers. Our results show that source-dependent ultrasonic extensions and broadband transients contain instrument-specific information that significantly improves isolated and polyphonic recognition and motivates wider-bandwidth studies of other MIR tasks.

📖 深度解读

论文解读:听不见但可识别——用于乐器识别的超声波特征

1. 一句话总结

这篇论文发现,乐器声音中那些"人耳听不见"的超声波成分(20 kHz 以上)其实携带着乐器身份信息,保留这些成分能显著提升乐器识别(包括复音混音场景)的准确率。

2. 研究背景与动机

核心问题:乐器识别是音乐信息检索(MIR)的基础任务,但现有系统在区分"音色相近"的乐器(尤其是多个乐器混在一起时)仍然吃力。论文追问:那些被常规录音采样率"丢掉"的超声波信息,能不能帮上忙?

为什么重要:绝大多数 MIR 数据集和系统都使用 44.1 或 48 kSPS 采样率,根据奈奎斯特准则,这意味着 22.05 kHz 以上的频率在进入模型之前就被彻底删除了。如果这部分信息有用,那么整个领域可能一直在"自断一臂"。

现有不足:
- 物理声学测量早就发现乐器有超声波成分(如 Boyk 1997),但从未在受控条件下验证它是否真的能提升"机器学习"的识别性能。
- 直接比较 44.1 kSPS 和 96 kSPS 是不公平的——采样率提高同时改变了采样点数量和保留带宽,无法判断性能提升到底来自"超声波内容"还是"更密的采样"。论文用匹配的低通/超声-only 对照解决了这个归因问题。

3. 核心方法

方法框架:作者录制了一个原生 96 kSPS(奈奎斯特上限 48 kHz)的多轨语料库,覆盖 15 类乐器/人声,然后用同一批录音派生出三种带宽条件进行对比:

  • 全带宽 96 kSPS(0–48 kHz)
  • 低通 96 kSPS(在 22.05 kHz 处滤波,模拟 44.1 kSPS 的带宽但保留采样密度)
  • 超声-only(只保留 20–48 kHz,把可听部分完全去掉)

关键创新点:

  1. 受控的带宽归因设计:通过"96 kSPS 低通"这个对照,把"采样密度"和"保留带宽"两个因素解耦。结果显示低通 96 kSPS 的性能几乎和 44.1 kSPS 一样,说明提升确实来自超声波内容而非采样率本身。

  2. 原生 96 kSPS 多轨语料库:6 首完整多轨作品(爵士、蓝调摇滚、安达卢西亚、小调抒情曲),超过 1 小时混音 + 近 15 小时对齐的独立音轨,跨越多个录音棚、演奏者、乐器实体和麦克风配置。

  3. 双范式验证:同时使用特征工程随机森林(RF)和 CNN6(改编自 PANNs),两种截然不同的学习机制都得到一致结论,降低了"效果依赖某种特定表示"的可能性。

  4. 超声-only 分类:把 20 kHz 以下内容全部删除后仍能达到 82% 准确率(12 类,随机基线 8.33%),直接证明超声波本身具有判别力。

直觉解释:可以把乐器声音想象成一座冰山——我们平时听到的 0–20 kHz 只是露出水面的部分。水下部分(超声波)来自三种机制:谐波序列的延伸、瞬态激励产生的宽带能量、以及金属打击乐器的类噪声辐射。不同乐器的"水下形状"不同:萨克斯能延伸到 30 kHz 以上,钢琴几乎没有;原声吉他保留超声,而磁拾音电吉他则被"低通"了。这些差异本身就是识别线索。

4. 实验与结果

数据集:自建 96 kSPS 多轨语料库,15 类(Bansuri、古典吉他、低音提琴、长笛、Kanun、次中音萨克斯、小号、小提琴、人声、原声吉他、单簧管、电吉他、口琴、钢琴、鼓)。采用"录音级"划分(整首 take 分配,训练和评估不共享 take),类别平衡窗口采样。

基线/对照:44.1 kSPS、96 kSPS 低通(22.05 kHz)、96 kSPS 全带宽、超声-only(20–48 kHz)。

主要结果:

任务 指标 44.1 kSPS 96 kSPS 低通 96 kSPS 全带宽
15 类孤立源(CNN6) 准确率 87% ~87% 91%
15 类孤立源(RF) macro-F1 81.11% 82.26% 88.38%
复音检测 整体准确率 — 与 44.1 几乎相同 +约 4 个百分点
超声-only(12 类) 准确率 — — 82%(随机 8.33%)

关键数字:
- 全带宽 vs 低通的 RF macro-F1 差距为 6.12 个百分点,这是归因于超声波的核心证据。
- 人声提升最大(+17.1 F1),小号和萨克斯各 +9.9,鼓 +8.7,Kanun +5.79。
- 复音场景中,古典吉他精确率从 68.02% 升到 92.74%,原声吉他召回率从 60.27% 升到 82.13%,萨克斯准确率 +8.70,人声 +4.60。
- 超声-only 中鼓、Kanun、萨克斯的类别 F1 超过 90%,长笛和单簧管约 70%。

消融/对照揭示:
- 低通 96 kSPS 与 44.1 kSPS 差异仅 1.15 个百分点(可能来自重采样和滤波),说明"更密的采样"本身贡献很小。
- 安静窗口对照实验:对未编辑原始录音中低于 -80 dBFS 的安静窗口做同样协议,性能回到随机水平,排除了"录音棚/设备链的固定签名"这一伪影解释。
- 电吉他和钢琴在全带宽下也有提升,但这不代表它们发射超声波——"没有超声波"本身也是判别特征,同时超声波活跃源的更好分离减少了误判。

5. 优势与局限

优势:
1. 归因严谨:匹配的低通对照设计干净地把"带宽"从"采样率"中分离出来,这是此前物理测量研究没有做到的。
2. 多范式 + 多任务验证:RF 和 CNN 两种范式、孤立源和复音两种任务、以及超声-only 极端条件,结论互相印证。
3. 实际可行性:96 kSPS 只需 44.1 kSPS 的 2.18 倍存储,现代硬件下已不构成瓶颈;且宽频母带可以向下兼容生成常规副本,反之则不可逆。

局限:
1. 语料规模小:仅 6 首作品,尽管有近 15 小时音轨,但曲目数量有限,作者自己承认"群体泛化性仍未解决"。
2. 统计效力有限:n=3 的种子重复只能说明"可重复性",不能说明"群体显著性";作者对此表述诚实。
3. 分离任务的证据薄弱:NMF 案例研究只有一个打击乐片段(SDR 从 -4.07 到 2.15 dB),作者明确说这"不能确立通用分离性能"。
4. 架构覆盖有限:只测了 RF 和 CNN6,循环网络和 Transformer 留作未来工作。

6. 关键结论与启发

最重要的 takeaway:乐器声音中 20 kHz 以上的超声波成分不是"无用的高频噪声",而是携带乐器身份信息的结构化特征。在受控条件下,保留这部分带宽能带来 4–6 个百分点的稳定提升,且这种提升在孤立源和复音混音中都成立。这意味着 MIR 领域长期使用的 44.1/48 kSPS 标准可能正在丢弃有用的信息。

对后续研究的启发:
1. 更宽带宽的 MIR 研究:作者呼吁对其他 MIR 任务(如音高估计、和弦识别、情感识别)也开展带宽受控研究。
2. 被动检测与定位:超声-only 识别成功暗示了"机器听不见但能识别"的可能性,可能用于被动声学监测。
3. 更大的语料库:需要更多曲目、更多演奏者和录音棚来验证群体泛化性。
4. 更广的架构测试:Transformer、CRNN 等架构在带宽受控条件下的表现值得探索。
5. 录音实践建议:作者建议未来 MIR、声学和归档分析采用原生 96 kSPS 母带 + 宽频前端,因为过滤掉超声波后无法恢复。

一句话评价:这是一篇"用严谨对照证明一个反直觉假设"的工作——它没有提出新架构,而是通过精心设计的带宽归因实验,让整个 MIR 社区重新审视一个被默认丢弃的频率区间。

#9
eess.AScs.SD

Phoneme-Guided Initialization for LLM-based Speech Recognition 跨领域

Ryo Magoshi, Shinsuke Sakai, Tatsuya Kawahara
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD)
Comments: Accepted at IEEE SLT 2026
查看摘要
Speech large language models (speech LLMs) perform well on automatic speech recognition (ASR) when sufficient paired speech-text data is available, but their performance degrades in low-resource settings. A cascaded pipeline that performs speech-to-phoneme (S2P) conversion followed by phoneme-to-grapheme (P2G) conversion has been shown to outperform end-to-end speech LLMs in this regime, suggesting that phoneme-mediated processing is beneficial when paired data is scarce. We propose \textit{phoneme-guided initialization}, a simple method that uses this insight within an end-to-end framework: we pre-train the audio encoder on S2P and the LLM on P2G tasks, then connect them and fine-tune the full model end-to-end on the target ASR task. Experiments on Japanese (CSJ), Chinese (AISHELL-1), and two low-resource languages from Common Voice 25.0 (Tatar and Urdu) show that our method matches or outperforms both the cascaded S2P-P2G baseline and the end-to-end model without P2G initialization.

📖 深度解读

论文解读:Phoneme-Guided Initialization for LLM-based Speech Recognition

1. 一句话总结

这篇论文提出了一种"音素引导初始化"方法:在训练语音大模型做语音识别之前,先用大量纯文本数据让 LLM 学会"音素→文字"的转换,同时让音频编码器学会"语音→音素"的转换,从而在低资源场景下显著提升识别效果。


2. 研究背景与动机

核心问题:基于 LLM 的语音识别系统(speech LLM)在配对语音-文本数据充足时表现很好,但在低资源场景下性能急剧下降。

为什么重要:配对语音-文本数据在很多语言和领域中非常稀缺,而纯文本数据则丰富得多、也更容易获取。如果能利用纯文本数据来改善语音识别,就能大幅降低对昂贵语音标注的依赖。

现有方法的不足:
- 端到端语音 LLM:连续表示方法存在"模态不匹配"问题——音频编码器的输出空间按声学相似度组织,而 LLM 的嵌入空间按语义相似度组织。解决这个问题通常需要大量配对数据,即使使用 LoRA 等参数高效技术也难以避免。
- 级联方案(S2P→P2G):虽然低资源下优于端到端,但 CTC 音素识别阶段的错误会传播到后续阶段(尤其在日语自发语音上表现差)。
- 音素优先预测(Phoneme-First):要求模型在训练时同时输出音素和文字序列,在数据稀缺时反而增加模态匹配难度;迭代方法还只适用于中文字符-音节一一对应的语言。
- 已有工作(Li et al.) 发现用 S2P 初始化音频编码器能缓解模态不匹配,但没有探索用 P2G 训练 LLM 的效果——这正是本文的切入点。


3. 核心方法

方法名称:Phoneme-Guided Initialization(音素引导初始化)

核心思路(直觉解释):与其让模型从零开始学习"声音→文字"的映射,不如先把两个组件各自"预训练"到音素这个中间层:
- 音频编码器先学会"听音辨音素"(这比直接辨文字简单,因为音素数量远少于字符);
- LLM 先学会"看音素写文字"(这只需要纯文本,用 G2P 工具把文本转成音素即可);
- 然后把两者用投影器连起来,在目标 ASR 任务上端到端微调。

打个比方:这就像教一个翻译,先分别训练"听力"和"拼写"两项基本功,再让它们配合工作,而不是一上来就练整段听写。

三个关键创新点:

  1. 双向音素对齐初始化:不仅初始化音频编码器(S2P),还初始化 LLM(P2G),让两个组件在端到端训练前都朝向音素级表示对齐。这是与 Li et al. 的关键区别。

  2. 纯文本驱动 LLM 初始化:P2G 训练完全不需要语音数据,用 G2P 工具从文本语料生成音素序列即可,极大扩展了可用数据来源。

  3. 只改初始化,不改推理格式:与音素优先预测等方法不同,本文方法不约束推理时的输出格式,模型可以只用文字输出,也可以音素+文字输出,灵活性更高。

三阶段流程:
- Stage 1:编码器用 CTC 损失做 S2P 训练;LLM 用纯文本做 P2G 微调。
- Stage 2:连接投影器,只更新编码器和投影器(warmup)。
- Stage 3:全部参数端到端联合微调。


4. 实验与结果

数据集/基准:
- 日语:CSJ(10小时子集 + 36小时 core 集),文本用 BCCWJ(自带音素标注)
- 中文:AISHELL-1(10h / 50h / 全集),文本用 THUCNews,G2P 用 g2pW
- 低资源语言:Common Voice 25.0 的鞑靼语(8.3h)和乌尔都语(8.4h)

模型配置:音频编码器 XLS-R 300M,LLM 用 Qwen3.5-2B,投影器为下采样+两层 MLP。

对比基线:XLS-R+CTC(非 LLM)、Cascade(级联)、Base(端到端无 P2G)、PF(音素优先)、PF+Joint(联合训练)。

主要结果:
- CSJ 10小时:Base+P2G 将平均 CER 从 20.8% 降到 11.7%,相对降低 44%;36小时 core 集相对降低 27%。
- AISHELL-1:10小时相对降低 72%(39.1%→11.1%),50小时降低 49%,全集降低 37%。数据越少,增益越大。
- 关键反转:无 P2G 初始化时,端到端 Base 甚至不如非 LLM 的 XLS-R+CTC;加入 P2G 后,Base+P2G 在所有 CSJ 评测集上反超 XLS-R+CTC,并在所有语言上超过 Cascade。
- 低资源语言:鞑靼语无提升(41.6%),乌尔都语仅小幅提升(29.3%→28.4%)。

消融/交互发现:
- 音素优先预测效果因语言而异:中文上有效(字符-音节一一对应),日语上反而变差(汉字读音变长且依赖上下文)。加入 P2G 初始化后,音素优先预测一致变差,说明 P2G 已提供足够音素知识,显式输出音素变得冗余。
- 错误分析:P2G 初始化显著提高了"同音替换"占比(即预测了正确读音但选错字),说明错误从"语义漂移"转向"声音忠实但字形混淆",残余错误主要是正字法消歧问题而非声学问题。这一效应在中文最强,日语次之,乌尔都语/鞑靼语几乎无变化。


5. 优势与局限

主要优势:
1. 数据效率高:仅用纯文本即可初始化 LLM,在配对数据越少时增益越大,非常适合低资源场景。
2. 通用性强:不约束推理输出格式,可兼容文字输出和音素优先输出两种模式。
3. 可解释的增益来源:错误分析定量证明 P2G 初始化让错误从"语义漂移"转向"声音忠实混淆",与增加配对数据的效果方向一致,说明它在一定程度上"替代"了配对数据。

局限性:
1. 对低资源语言效果有限:鞑靼语和乌尔都语几乎无提升,作者归因于 G2P 工具质量较低,以及这些语言的正字法本身更接近音素(音素-字形差距小,P2G 提供的信息增量少)。
2. 未完全隔离变量:Base+P2G 在 P2G 初始化时也接触了目标语言文本,因此"音素级对齐"和"纯文本适应"的贡献未被完全分离,作者承认需要匹配的纯文本对照组。
3. 声调问题未解决:中文用无调拼音计算同音率,导致仅声调不同的近同音字被算作同音,准确恢复声调仍是遗留问题。


6. 关键结论与启发

最重要的 takeaway:在 LLM 语音识别中,用纯文本数据对 LLM 做"音素→文字"初始化,配合音频编码器的"语音→音素"初始化,能让两个组件在端到端训练前就对齐到音素级表示,从而在低资源场景下大幅提升性能——而且这种增益在配对数据越少时越明显,说明文本数据可以部分替代昂贵的语音标注。

对后续研究的启发:
- G2P 质量是关键瓶颈:低资源语言的提升受限主要来自 G2P 工具质量,改进 G2P 或使用更大文本语料是直接方向。
- 正字法类型决定收益:音素-字形差距大的语言(中文、日文汉字)受益最大,正字法本身音素化的语言(鞑靼语、乌尔都语)受益小,这为"何时该用此方法"提供了判据。
- 超越音素级对齐:作者提出未来可探索音素级对齐之外的初始化方式。
- 声调建模:中文声调消歧是明确的遗留问题,值得专门研究。
- 变量隔离:需要设计匹配的纯文本对照组,以厘清"音素对齐"与"语言模型适应"各自的贡献。


说明:论文全文提取完整,以上解读基于提供的完整内容。论文中部分引用(如 Li et al. 2026、Poncelet & Van hamme 2026)标注为 arXiv 预印本,其具体细节以原文引用为准。

#10
eess.AScs.SD
University of Western Australia (QS Top 100)University of New South Wales (UNSW Sydney) (QS Top 100)

CASM: Context-Aware Semi-Markov Post-Processor for Beat Tracking 解读失败跨领域

Zhanhong He, Hanyu Meng, Yaolong Ju
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: Submitted to ICASSP 2027 conference. Code and demo: this https URL
查看摘要
In music beat tracking, model-predicted activations must be decoded into a discrete, musically coherent event sequence. Direct peak picking closely follows local evidence but can retain spurious peaks or miss weak beats. Dynamic Bayesian networks (DBNs), a widely used structured post-processor, improve sequence consistency under predefined global tempo, meter, and transition constraints, but their behavior can depend strongly on these settings. We introduce CASM, a context-aware semi-Markov decoder that instead conditions its temporal constraint on local activation evidence. CASM also accounts for ambiguity among competing periodic interpretations, including half- and double-tempo alternatives. Deterministic safeguards prevent implausible outputs and preserve beat-downbeat consistency. Applied to fixed activations from three neural beat trackers (BeatThis, MSCNN, and TCN), CASM improves temporal continuity while preserving event-level F1 across the GTZAN and SMC datasets, without backbone retraining or dataset-specific retuning. Further analysis shows that CASM is less sensitive than the DBN baseline to the composition of the calibration data.

📖 深度解读

[LLM 解读失败: HTTP 402]

#11
eess.AScs.SD

VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation 解读失败跨领域

Jingqi Sun, Haozhan Tang, Shulin He, Zhong-Qiu Wang
Audio and Speech Processing (eess.AS); Machine Learning (cs.LG); Multimedia (cs.MM); Sound (cs.SD); Signal Processing (eess.SP)
Comments: Submitted to ICASSP 2027 and currently under review
查看摘要
Blind Source Separation(BSS) is a fundamental problem in signal processing, aiming to separate multiple source signals from their mixtures without prior knowledge of the sources or the mixing process. Traditional approaches, such as Independent Vector Analysis (IVA) exploits statistical independence of sources. Recently, diffusion-based approaches have emerged as a promising alternative by leveraging powerful generative priors. Among them, ArrayDPS formulates BSS problem as a posterior sampling problem, and utilizes a pretrained speech diffusion model to guide the recovery of clean source signals. A key factor behind its separation capability is the multi-channel consistency (MC) objective, which enforces the estimated source signals to reconstruct the observed microphone mixtures through the estimated acoustic transfer functions. However, the number of microphones in the array is often limited, which constrains the performance of ArrayDPS. To address this issue, we propose VM-ArrayDPS, a novel method that augments the microphone array with virtual microphones with higher-SNR, these microphones can offer extra MC constraints to enhance the separation performance. Experimental results demonstrate that VM-ArrayDPS significantly outperforms ArrayDPS on both 2-speaker and 3-speaker datasets, showcasing the effectiveness of virtual microphone augmentation in improving BSS performance. We also did ablation studies to show the influence of the number of virtual microphones and weight of the MC objective brought by virtual microphones.

📖 深度解读

[LLM 解读失败: HTTP 402]

#12
eess.AScs.SD

Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech 解读失败跨领域

Hounsu Kim, Joonyong Park, Yuki Saito, Satoru Fukayama, Juhan Nam
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: 5 pages, 1 figure, 3 tables. Under review
查看摘要
Unlike autoregressive models, discrete diffusion-based models for zero-shot text-to-speech generate speech tokens in parallel and can revisit earlier predictions. Mask-and-replace training extends mask-only training by randomly replacing some tokens, and its gains are commonly attributed to self-correction, the ability to revise previously generated tokens. However, exposure to randomly perturbed context during training may itself improve generation, raising the question of whether these gains require inference-time token revision. To investigate this question, we use DeMaR, which combines mask-and-replace training with confidence-ranked mask-only sampling while preserving the total training corruption probability. Trained from scratch on LibriTTS, DeMaR achieves lower word error rates (WER) than autoregressive and mask-only diffusion baselines using the same speech tokenizer. This advantage persists when each token remains unchanged after first being unmasked. Matched training conditions on two heterogeneous speech tokenizers show that both noisy-context augmentation and replacement supervision improve WER under this restriction. These findings demonstrate training-side benefits of replacement beyond enabling inference-time token revision.

📖 深度解读

[LLM 解读失败: HTTP 402]

#13
eess.AScs.SD

Boundary-Free Contextual Biasing: Depth-Adaptive Gating and Reading-Space Matching for Unsegmented Languages 解读失败跨领域

Muhammad Huzaifah, Yu Pan, Zachary Yeo, Ningjie Bai, Guangzhao Yang
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD)
查看摘要
Contextual biasing supplies an ASR system with a list of expected words at inference time, but existing methods rely on word boundaries that Japanese and Chinese do not provide. We present a boundary-free biasing decoder for frozen public CTC models, built on a character-level Aho-Corasick automaton, with no training and no second pass. Two evidence-based mechanisms replace the boundary: a depth-adaptive gate that sets how hard to push from match depth, and reading-space matching for when the audio is right but the characters are wrong. On Aishell-1 NE's hard R1 subset we reach 66.5% recall, above the trained CLAS baseline (64%), transferring to WenetSpeech and to a second architecture without retuning. We release the first open Japanese contextual-biasing benchmark, where biasing lifts rare-word recall by 25 points at precision above 97%, and still by 19 and 22 points against 1,000-word lists.

📖 深度解读

[LLM 解读失败: HTTP 402]

#14
eess.AScs.SD

Mitigating Accent-Language Confusion in Self-Supervised Speech Representations for Language Identification 解读失败跨领域

Minu Kim, Jihwan Lee, David R. Mortensen, Shrikanth Narayanan
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD)
Comments: Submitted to ICASSP 2027
查看摘要
Spoken language identification (LID) aims to recognize the target language regardless of accent. In practice, however, LID models fine-tuned from self-supervised speech representations frequently confuse accents with languages, misclassifying non-native (L2) speech as the speaker's first language (L1). We show that non-native speech representations lie between native target-language and native L1 poles, causing systematic misclassification. To address this, we introduce a geometric projection that estimates an L1-bias direction solely from native speech and removes it before the frozen LID head. Across five MMS-LID models and non-native corpora, this projection substantially improves target language identification for L2-accented speech while preserving predictions for native speech. These results show that accent-induced L1 bias can be corrected directly within the representation space without L2 training data or model adaptation.

📖 深度解读

[LLM 解读失败: HTTP 402]

#15
eess.AS

A Lightweight Hybrid Framework for Acoustic Echo Cancellation and Suppression via Constrained Recursive Training 解读失败

Huawei Zhang, Rilin Chen, Hao Zhang, Meng Yu, Dong Yu
Audio and Speech Processing (eess.AS)
Comments: Accepted by the 19th International Workshop on Acoustic Signal Enhancement (IWAENC 2026)
查看摘要
We propose a hybrid framework that combines deep learning with adaptive filtering for acoustic echo cancellation (AEC) and suppression. It integrates a multi-output neural network to jointly update filter parameters and perform post-processing. A learnable variant of the normalized least mean squares (NLMS) filter is introduced, incorporating a variable step size and a variable transition factor that are recursively updated for dynamic adaptation. In addition, an enhanced near-end speech estimate is generated from the filter output to suppress the residual echoes. To ensure training stability, we propose a constrained recursive training strategy that aligns with the adaptive nature of the filter, where an upper-bound constraint is introduced on the filter output. For efficiency, the framework adopts a lightweight implementation with only 253 k parameters. Experimental results show that the framework outperforms the baselines in terms of both signal quality and speech intelligibility.

📖 深度解读

[LLM 解读失败: HTTP 402]

#16
eess.AS

Direction-preserving Spatial Audio Signal Enhancement in The Spherical Harmonic Domain 解读失败

Huawei Zhang, Jihui Aimee Zhang, Huiyuan June Sun, Prasanga Samarasinghe
Audio and Speech Processing (eess.AS)
Comments: Accepted by the 55th International Congress & Exposition on Noise Control Engineering (INTER-NOISE 2026)
查看摘要
Spatial audio signal enhancement (SASE) aims to separate a desired soundfield from interference and noise while preserving its spatial cues. However, existing algorithms pay limited attention to preserving the spatial cues of the residual interference, which reduces their effectiveness in practical applications. In this paper, we propose a direction-preserving SASE algorithm based on multi-output beamforming. The proposed algorithm extracts the desired soundfield while preserving the spatial cues of both the desired soundfield and the residual interference soundfield. The simulation study shows that the proposed algorithm achieves performance comparable to the existing SASE algorithm in terms of estimation error, while it better preserves the spatial cues of the residual interference within a region of interest, in a noisy and reverberant environment.

📖 深度解读

[LLM 解读失败: HTTP 402]

#17
eess.AS

A Strength-Monotonic Law for Domain Alignment in Frozen-Embedding Bioacoustic Classification 解读失败

Yucheng Gong, Rui Zhou, Binbin Zeng, Qiang Ren, Hongjin Hui
Audio and Speech Processing (eess.AS); Machine Learning (cs.LG)
查看摘要
When does distribution alignment help a frozen foundation-model embedding generalize across acoustic domains? For cross-domain mosquito-species classification we report a strength-monotonic law: the stronger an encoder is on the target task, the more its unseen-domain generalization relies on a distribution-alignment (MMD) term, and the more it is harmed by domain-rebalanced sampling. Across four encoder families and a within-encoder HuBERT layer sweep (n=8), the rebalancing leg orders exactly with encoder strength (Spearman -1.000), while the MMD-benefit leg is monotonic within each stream and -0.857 pooled; fixing architecture and varying only representation strength flips the rebalancing effect from benefit to collapse. The law is actionable: a single MMD term is the sole lever on a strong encoder, so we reduce the field's default recipe to a frozen Perch 2.0 embedding, a lightweight probe, cross-entropy, one MMD, and input augmentation. The reduced recipe stays within seed noise of the full composite (BA_unseen 0.299+/-0.006 vs. 0.307+/-0.014). As boundary conditions of the same law, three community defaults (backbone fine-tuning, multi-modal fusion, and domain rebalancing) each hurt unseen-domain accuracy under a leave-domain protocol, shown with single-variable, multi-seed evidence. We present a mechanism and the recipe it explains, not a leaderboard entry.

📖 深度解读

[LLM 解读失败: HTTP 402]

#18
eess.AScs.SD

Training-Free Instruction TTS Gender Bias Calibration Using Model-Adaptive Steering 解读失败跨领域

Kuan-Yu Chen, Yi-Cheng Lin, Jeng-Lin Li, Jian-Jiun Ding
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: 5 pages, 1 figure, 3 tables, submitted to ICASSP 2027
查看摘要
Instruction text-to-speech (ITTS) systems systematically encode acoustic gender skews from descriptive style prompts, such as occupations or personas, even when demographic attributes are left unspecified. Calibrating the implicit gender distribution of the synthesized voices, across heterogeneous architectures and without model retraining or overriding explicit user prompts, is an open problem. In this work, we propose \textit{model-adaptive steering}, a training-free bias calibration method that steers post-encoder conditioning representations using a group deviation vector paired with a coarse-to-fine strength search on a development set. A deterministic lexical gate bypasses intervention whenever explicit gender keywords are detected, preserving intended prompt semantics. Evaluated on a 12,800-prompt held-out benchmark across four ITTS models (three architectures), the proposed method reduces aggregate calibration error from 11.5--28.1 to 0.8--5.9 percentage points (e.g., shifting Parler-TTS Mini from 78.1\% and PromptTTS++ from 27.3\% female to 50.8--52.4\%). Output rates stay within 0.9 pp across anchor set sizes at fixed operating points, while UTMOS decreases by at most 0.08 and WER degrades by at most 3.3 pp.

📖 深度解读

[LLM 解读失败: HTTP 402]

#19
eess.AScs.SD

Toward Part-Aware Choral Transcription with singing voice assignment 解读失败跨领域

Hanyu Meng, Zhanhong He, Zixun Guo, Yaolong Ju
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: Submitted to ICASSP2027
查看摘要
Single-instrument automatic music transcription (AMT) has advanced substantially, yet choral applications require soprano, alto, tenor, and bass (SATB) to be transcribed as separate parts. Recent note-level choral AMT instead produces a single merged note track, limiting rehearsal, education, and score reconstruction. To address this limitation, we introduce Part-aware Choral Transcription (PawCT), to our knowledge the first end-to-end neural framework that identifies active SATB parts from choral audio and transcribes each into a separate note-level track. PawCT combines part-specific onset, offset, and frame prediction with part-presence estimation, union-level supervision, and structured training targets using a range prior (RP) based on SATB pitch ranges and its ordered-continuity (OC) extension, which adds within-part melodic continuity and cross-part pitch ordering. On YouChorale, PawCT-RP-OC achieves a macro part-aware note F1 of 0.225 at a 50-ms onset tolerance, outperforming an adapted choral baseline (0.165) by 36.4% relative and a two-stage post-hoc assignment pipeline (0.175). Its part-agnostic variant, PagCT, achieves a 50-ms onset F1 of 0.382, compared with 0.237 for the previous state-of-the-art choral AMT model. Cross-dataset evaluations on CSD and Cantoria further assess performance under dataset shift. These results demonstrate the benefit of jointly modeling note transcription and vocal-part assignment. Code and demos are available at this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]

#20
eess.AS

LIFT-SE: Linguistic Inference Followed by Flow Transformation for Generative Speech Enhancement 解读失败

Haoyin Yan, Chengwei Liu, Zheng Xue, Xiaotao Liang, Jifa Cai 等 (7 人)
Audio and Speech Processing (eess.AS)
查看摘要
Generative speech enhancement (SE) is prone to linguistic hallucination when semantic constraint is unreliable under severe noise and reverberation. Moreover, approaches that generate discrete codec tokens are bounded by the quantization error of the codec decoder, regardless of token-prediction accuracy. We propose LIFT-SE, a two-stage generative framework that decouples linguistic inference from acoustic synthesis within QRes-Codec, which exposes a quantized latent and its residual-completed continuous form. The first stage predicts clean codec tokens autoregressively, conditioned on frame-aligned features from a self-supervised front-end distilled toward clean speech. The second stage applies conditional flow matching to transport Gaussian noise to the continuous latent conditioned on the predicted tokens, and the frozen decoder reconstructs the enhanced waveform. Discrete generation provides naturalness, while continuous refinement restores signal fidelity. Experiments on the DNS1 and URGENT benchmarks show that LIFT-SE attains favorable linguistic consistency under reverberant conditions together with competitive perceptual quality, and systematic ablations verify the necessity of both stages. Code will be released in the future.

📖 深度解读

[LLM 解读失败: HTTP 402]

#21
eess.AS

Source-Directed Trajectory Perturbation at First-Order Cost for Domain Generalization in Speech Deepfake Detection 解读失败

Siqing Qin, Kong Aik Lee, Youzhi Tu
Audio and Speech Processing (eess.AS)
查看摘要
Speech deepfake detectors often lose accuracy when the distribution of the test data differs from that of the training data. Meta-learning for domain generalization (MLDG) shows promise by simulating domain shifts with episodic meta-train and meta-test splits. However, the MLDG meta-objective considers only a single clean adaptation trajectory and does not account for the local loss landscape around its endpoints. To address this limitation, we first propose an explicit worst-case MLDG variant, dubbed WC-MLDG-4P, which perturbs both the meta-train and meta-test states but requires four gradient evaluations per episode. We then introduce WC-MLDG-2P, a source-directed alternative to the explicit robust objective. It shifts the clean MLDG endpoint toward a locally higher source-loss state and evaluates the meta-test gradient there, retaining the two-evaluation cost of the first-order MLDG. A first-order expansion relates the resulting gradient change to meta-test directional curvature along the source gradient without explicitly computing a Hessian. Relative to MLDG, WC-MLDG-2P achieves relative mean-EER reductions of 23.4% with XLSR-AASIST and 7.6% with XLSR-Conformer-TCM, respectively.

📖 深度解读

[LLM 解读失败: HTTP 402]

#22
eess.AScs.SD

DuRe-ST: Dual-Relation Spectro-Temporal Modeling for Speech Deepfake Detection 解读失败跨领域

Shaole Li, Siqing Qin, Youzhi Tu, Kong Aik Lee
Audio and Speech Processing (eess.AS); Sound (cs.SD)
查看摘要
Previous speech deepfake detectors can adaptively capture spectro-temporal dependencies through graph attention, yet they largely overlook the co-variation between spectral and temporal representations. To address this gap, we construct a normalized affinity graph from their joint covariance and apply polynomial graph filtering to capture higher-order covariance-induced dependencies. We first develop Cov-ST to isolate the contribution of covariance-based relational modeling. Although it improves detection performance, its sensitivity to the polynomial order suggests limited robustness when covariance relations are modeled alone. We therefore propose DuRe-ST, which jointly exploits covariance-induced and graph-attention-induced relations to capture complementary second-order co-variation and adaptive spectro-temporal dependencies. Experiments show that DuRe-ST achieves an average relative EER reduction of 25.9% over XLSR-AASIST on the ASVspoof benchmarks and 28.4% across four cross-dataset benchmarks with only 4-8k additional trainable back-end parameters. It further outperforms the strongest publicly available comparison models by 2.2-13.2% in relative EER on four benchmarks, while remaining smaller than the publicly available models considered.

📖 深度解读

[LLM 解读失败: HTTP 402]

#23
eess.AS

Benchmarking Adult Addressee Classification Across Child- and Adult-Directed Speech Datasets 解读失败

Sofoklis Kakouros, Daniil Kocharov, Okko Räsänen
Audio and Speech Processing (eess.AS)
Comments: Accepted at SLT 2026
查看摘要
In this work, we present a comprehensive analysis of classification performance for distinguishing child-directed speech (CDS) from adult-directed speech (ADS) using speech data from corpora containing natural in-lab and in-the-wild CDS and ADS. We establish classification benchmarks for these datasets using self-supervised learning (SSL) representations, along with a range of time-pooled representations that go beyond first- and second-order statistics by incorporating cross-channel covariances in high-dimensional embeddings. In addition, we probe these representations to examine how different pooling methods capture prosodic information using linear probes. Overall, SSL-based representations prove particularly effective, achieving the best performance, while different pooling methods offer complementary advantages for the task.

📖 深度解读

[LLM 解读失败: HTTP 402]

#24
eess.AS

Encypher: Shared Agency and Social Presence in Collaborative Music Generation for Dance Cyphers 解读失败跨领域

Zhixing Chen, Cheng-Zhi Anna Huang
Human-Computer Interaction (cs.HC); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: Preprint. Under review at CHI 2027. 11 figures, 1 table. Project page: this https URL
查看摘要
Music and dance are social practices of expression and connection, yet most HCI work in human-AI co-creation centers the solo performer. As generative music matures, we ask not only what AI can compose but what social encounters it can organize around sound. We present Encypher, a collaborative generative music system that translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers. Through five weeks of co-design with local dancers, a user study with unacquainted participants, a public museum event, and a live performance, we found that users developed shared agency, perceiving the music as a response to the room's energy. While newcomers felt uncertain, the system fostered social presence by prompting them to look to each other for cues. By treating sociality as a design concern rather than a downstream effect, we offer a framework and design implications for AI systems for collaborative, embodied expression.

📖 深度解读

[LLM 解读失败: HTTP 402]

#25
eess.AScs.SD

Child ASR Adaptation with Adult Retention: An Empirical Study 解读失败跨领域

Houssam Eddine-Othman Lachemat, Shammur Absar Chowdhury
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: long paper
查看摘要
Automatic Speech Recognition (ASR) systems often underperform for children and non-native speakers, while adapting adult ASR models to child speech can cause adult-speech forgetting. We study child ASR adaptation with adult retention across Arabic and English. We compare full fine-tuning, LoRA, and post-hoc weight-space merging across encoder--decoder, encoder--CTC, and AudioLLM-based ASR systems. Experiments use Arabic native and non-native child speech, English MyST child speech, and adult benchmarks from MGB-2 and LibriSpeech test-clean. We evaluate recognition quality with WER and quantify the adaptation--retention trade-off using Retention Index, Child Adaptation Gain, and Adaptation Recovery. Results show that child adaptation is necessary, especially for non-native Arabic and English child speech, but direct adaptation often reduces adult ASR performance. Bilingual adaptation is more stable than language-specific adaptation. Weight-space merging often improves the trade-off, especially for encoder--CTC, Whisper, and AudioLLM-based ASR, with LERP favoring adult retention and TIES recovering stronger child gains. For the encoder--decoder model, direct bilingual fine-tuning remains strongest in raw WER.\footnote{Code, and models are available at this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]

#26
eess.AScs.SD

When Forgetting Looks Like Improvement: Metric Masking in Streaming Diarizer Adaptation and the Price of Rehearsal 解读失败跨领域

Mo Yu, Yang Liu, Jing Qian
Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: 5 pages, 4 figures
查看摘要
Small-data adaptation can improve speech detection while degrading speaker attribution. We study this discrepancy in a released streaming diarizer adapted on 7.5 h of two-party conversation and evaluated across six corpora. Adaptation substantially improves in-domain diarization performance and transfers to an independent corpus. However, this improvement is not consistent across evaluation scenarios as the additional confusion is mainly associated with impaired temporal identity consistency rather than speaker-count errors. A local-remapping diagnostic reveals different patterns of identity degradation across corpora, indicating that adaptation may alter how streaming models maintain speaker assignments over time. Rehearsal reduces the observed degradation but reduces the cross-domain transfer performance. These results highlight the need to jointly evaluate detection accuracy, identity consistency, and retention behavior when adapting streaming diarization systems.

📖 深度解读

[LLM 解读失败: HTTP 402]

#27
eess.AScs.SD

JoyAI-Voice 2.0: A Full-Continuous Autoregressive Speech Generation Model with Semantic-Acoustic Joint Representation 解读失败跨领域

Yafeng Chen, Boya Dong, Yankun Huang, Hao Li, Jingdong Li 等 (20 人)
Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
查看摘要
We present JoyAI-Voice~2.0, an end-to-end anthropomorphic speech generation model built upon a fully continuous, dual-encoder architecture. Raw speech is encoded into continuous latents and partitioned into patches. Each patch is decomposed by a semantic-acoustic dual encoder into a semantically purified representation and an acoustic representation, which are fused and jointly fed to a causal autoregressive Transformer for planning. The Transformer predicts the conditioning for the next patch, and a local diffusion Transformer renders its full latents for 48\,kHz synthesis. The model is trained with a joint flow-matching and stop-prediction objective, followed by supervised fine-tuning and reinforcement learning with DiffusionNFT to improve model performance. It achieves the lowest average word error rate of 2.51\% on Seed-TTS, a 14.9\% relative reduction over the strongest baseline, and state-of-the-art attribute fidelity on InstructTTSEval in both Chinese and English, leading on 5 of 10 perceptual dimensions with the highest overall score of 0.893 on MDVD-Eval.

📖 深度解读

[LLM 解读失败: HTTP 402]

#28
eess.AScs.SD

Towards AI-Generated Music Plagiarism Detection as a Version Identification Problem 解读失败跨领域

Fotis Koutsikos, Ioannis Prokopiou, Spyridon Kantarelis, Vassilis Lyberatos, Pantelis Vikatos 等 (9 人)
Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
Comments: 5 pages, 3 figures, submitted to 2027 IEEE International Conference on Acoustics, Speech, and Signal Processing
查看摘要
The rapid expansion of text-to-music generative models challenges traditional paradigms of music creation and intellectual property. Plagiarism in this context is rarely an absolute mathematical binary, but an ambiguous threshold negotiated over harmonic structure, melodic contours, or overall perceived stylistic character. In this work, we test the transferability of state-of-the-art music version identification architectures from the human-to-human cover domain to the human-to-AI plagiarism setting. To evaluate this task, we introduce COPYCAT, a benchmark derived from real-world plagiarism cases and extended through generative re-synthesis and digital signal processing obfuscations, yielding 350,654 evaluation pairs. We show that scalar distance thresholding collapses under generative re-synthesis, while a supervised framework leveraging coordinate-wise embedding shifts recovers the dispersed plagiarism signal, raising overall $F_{0.5}$ from $0.612$ to $0.803$.

📖 深度解读

[LLM 解读失败: HTTP 402]

#29
eess.AS

Dialect-Robust Speech Language Models with Synthetic Pseudo-Dialect Augmentation 解读失败跨领域

Shunsuke Mitsumori, Tomoya Mizumoto, Yusuke Fujita
Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
Comments: 7 pages, 1 figure, 7 tables. Accepted to IEEE SLT 2026
查看摘要
Speech Language Model (SLM) performance often degrades on dialects due to data scarcity. Conventional text-to-speech (TTS) augmentation struggles to cover diverse dialects as it requires a certain amount of real dialect speech. We propose synthesizing pseudo-dialect speech by converting LLM-generated dialect text via a standard-language TTS model, requiring zero real dialect speech. Additionally, we introduce intermediate standard-text prediction during training, acting as semantic normalization for downstream tasks. We evaluate dialect understanding via dialect-to-English speech translation across Japanese, German, and Chinese dialects. Compared to synthetic standard speech baselines, pseudo-dialect augmentation improves scores for Japanese (from 25.38 to 26.24) and German (from 31.57 to 32.47). Furthermore, the intermediate standard-text prediction effectively bridges the semantic gap, boosting performance to 28.26 for Japanese and from 11.67 to 16.37 for Chinese. These results suggest that our approach scales to various languages without requiring speech resources specific to each dialect.

📖 深度解读

[LLM 解读失败: HTTP 402]

#30
eess.AScs.SD

Tracing Inputs, Verifying Outputs: Validating Attribution in Music Generation 解读失败跨领域

Taejun Kim, Wonil Kim, Jongmin Jung, Hyeongseok Wi, Sangeun Kum 等 (13 人)
Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
Comments: 15 pages, 4 figures. Audio examples: this https URL
查看摘要
How can we verify whose music contributed to an AI-generated output? This paper demonstrates how input-based attribution can provide verifiable evidence of which audio sources were used in a generation and whether they shaped the output. To do so, we condition the generation solely on audio without any text input, then trace the inputs behind each output, and establish their musical effect. In prompt adherence tests and controlled input swaps, the stems generated by our generator, MixAudio, follow the prompt audio in timbre and the context audio in harmony. Yet these outputs may still reproduce training data not supplied as inputs. We therefore audit memorization with our musical version identification model, musicDNA, and find few reproductions outside the input records. On human-judged cases within the flagged pool, it achieves higher precision and recall than the other tested memorization detectors. The two evaluations suggest that input records and output analysis provide complementary evidence for attribution, on which rights-holder reporting and compensation can draw as the AI music economy takes shape. Audio examples are available at this https URL

📖 深度解读

[LLM 解读失败: HTTP 402]

#31
eess.AScs.SD

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing 解读失败跨领域

William Chen, Prem Seetharaman, Ke Chen, Oriol Nieto, Kevin Duarte 等 (13 人)
Sound (cs.SD); Audio and Speech Processing (eess.AS); Image and Video Processing (eess.IV)
Comments: 27 pages (10 main + 1 limitations); 8 figures
查看摘要
Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]

#32
eess.AScs.SD

Steerspeech: Activation Steering For Emotion Control In Generated Speech 解读失败跨领域

Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin, Salar Rahili
Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
Comments: Under review at IEEE ICASSP 2027. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
查看摘要
Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.

📖 深度解读

[LLM 解读失败: HTTP 402]

#33
eess.AS

WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection 解读失败跨领域

Xi Xuan, Davide Carbone, Wenxin Zhang, Tomi H. Kinnunen
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Signal Processing (eess.SP)
Comments: Submitted to ICASSP 2027
查看摘要
Existing front-ends for speech deepfake detection are primarily categorized into two types. Hand-crafted filterbank features are transparent but limited in capturing higher-level information. SSL features, in turn, lack interpretability and may overlook fine-grained spectral anomalies. We propose WaveScat, a novel family of feature extractors that combines the best of both worlds via the wavelet scattering transform (WST), which cascades wavelet convolutions with modulus nonlinearities to produce deformation-stable, multi-scale features. Experiments on the recent Deepfake-Eval-2024 benchmark, together with cross-dataset evaluations on SpoofCeleb, In-the-Wild, and ASVspoof 5, show that WaveScat outperforms existing front-ends by a wide margin. Our analysis reveals that a small averaging scale combined with high-frequency and directional resolutions is critical for capturing subtle artifacts. This underscores the value of stable and translation-invariant features for speech deepfake detection. The code and supplementary materials are available at this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]

#34
eess.AScs.SD

Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition 解读失败跨领域

Lukuan Dong, Ziwei Li, Saierdaer Yusuyin, Xianyu Zhao, Zhijian Ou
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD)
Comments: NCMMSC 2026 camera-ready version
查看摘要
Phoneme-based ASR factorizes recognition into speech-to-phoneme (S2P) and phoneme-to-grapheme (P2G), enabling cross-lingual acoustic sharing while keeping language-specific orthography in a separate module. While large language models (LLMs) are promising for P2G, multilingual P2G remains challenging due to language-aware generation and severe cross-language data imbalance. We study multilingual LLM-based P2G on the ten-language CV-Lang10 benchmark. We examine robustness strategies that account for S2P uncertainty, including DANP and Simplified SKM (S-SKM). S-SKM is a Monte Carlo approximation that avoids CTC-based S2P probability weighting in P2G training. Robust training and low-resource oversampling reduce the average WER from 10.56% to 7.66%.

📖 深度解读

[LLM 解读失败: HTTP 402]

#35
eess.AS

Do Multimodal Large Language Models Need Reasoning to Classify Dementia from Speech? 解读失败跨领域

Liming Wang, Neguine Rezaii, Bradford C. Dickerson, James Glass
Audio and Speech Processing (eess.AS)
查看摘要
Multimodal large language models (MLLMs) have emerged as a promising approach for improving the accuracy, transferability, and explainability of automatic dementia classification (ADC) systems from voice recordings. Yet it remains unclear whether their reasoning capabilities are beneficial for ADC, and how such capabilities should be leveraged. In this paper, we conduct a careful evaluation of reasoning MLLMs for ADC and show that naive strategies, such as relying on text-based rationales, can lead to hallucinated and inconsistent rationales for diagnosis and yield inferior ADC performance compared with LLM-free baselines. To overcome this limitation, we propose \textbf{De}mentia \textbf{T}hinker with Nonlinear \textbf{A}daptor and Re\textbf{i}nforcement \textbf{L}earning (DeTAiL), an adaptor-based framework that exploits the internal representations of reasoning MLLMs for improved dementia classification. Across two dementia datasets with distinct test formats and label granularities, DeTAiL consistently outperforms strong baselines and methods that rely on text-based rationales. Code and demo will be released upon acceptance.

📖 深度解读

[LLM 解读失败: HTTP 402]

#36
eess.AS

Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR 解读失败跨领域

Gene-Ping Yang, Haibin Wu, Peng Su, Ruizhe Huang, Suwon Shon 等 (13 人)
Audio and Speech Processing (eess.AS)
查看摘要
Modern automatic speech recognition (ASR) systems excel at transcribing lexical content but often omit nonverbal vocalizations (NVs), such as laughter, breaths, coughs, and cries, that carry conversational and affective information. Modeling NVs in ASR is challenging because NV annotations are sparse and highly long-tailed, with frequent categories such as breaths and laughter dominating rarer events such as cries and coughs. We study three data-centric strategies for improving low-resource NV recognition: (1) a two-stage curriculum that first maps all NV events to a generic token and then fine-tunes on target categories; (2) inter-token transfer from high-resource events, such as laughter and breath, to rare events, such as crying; and (3) voice-conversion augmentation with class balancing. Experiments show that shared acoustic structure across vocal events can be exploited to improve rare-category detection while preserving lexical ASR quality.

📖 深度解读

[LLM 解读失败: HTTP 402]

#37
eess.AScs.SD

Speech Emotion Recognition Using CNN and Its Use Case in Digital Healthcare 解读失败跨领域

Nishargo Nigar
Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
Comments: Few technical inconsistencies, needs revision
查看摘要
The process of identifying human emotion and affective states from speech is known as speech emotion recognition (SER). This is based on the observation that tone and pitch in the voice frequently convey underlying emotion. Speech recognition includes the ability to recognize emotions, which is becoming increasingly popular and in high demand. With the help of appropriate factors (such modalities, emotions, intensities, repetitions, etc.) found in the data, my research seeks to use the Convolutional Neural Network (CNN) to distinguish emotions from audio recordings and label them in accordance with the range of different emotions. I have developed a machine learning model to identify emotions from supplied audio files with the aid of machine learning methods. The evaluation is mostly focused on precision, recall, and F1 score, which are common machine learning metrics. To properly set up and train the machine learning framework, the main objective is to investigate the influence and cross-relation of all input and output parameters. To improve the ability to recognize intentions, a key condition for communication, I have evaluated emotions using my specialized machine learning algorithm via voice that would address the emotional state from voice with the help of digital healthcare, bridging the gap between human and artificial intelligence (AI).

📖 深度解读

[PDF 下载失败,无法解读]

#38
eess.AScs.SD

Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker Traits and Speech Properties 解读失败跨领域

Tiantian Feng, Jihwan Lee, Anfeng Xu, Yoonjeong Lee, Thanathai Lertpetchpun 等 (12 人)
Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: Journal of Data-centric Machine Learning Research (DMLR) Accepted
查看摘要
We introduce Vox-Profile, a comprehensive benchmark to characterize rich speaker traits and speech properties using speech foundation models. Unlike prior work that targets a single dimension of speaker traits, Vox-Profile provides holistic and multi-dimensional profiles spanning both static speaker traits (e.g., age, sex, accent, voice quality) and dynamic speech traits (e.g., emotion, speech flow). This benchmark is grounded in speech science and linguistics, developed with domain experts to accurately index speaker and speech characteristics. We report benchmark experiments using over 15 publicly available speech datasets and several widely used speech foundation models that target various static and dynamic speaker and speech properties. In addition to benchmark experiments, we showcase several downstream applications supported by Vox-Profile. First, we show that Vox-Profile can augment existing speech recognition datasets to analyze ASR performance variability. Vox-Profile is also used as a tool to evaluate the performance of speech generation systems. Finally, we assess the quality of our automated profiles through comparison with human evaluation and show convergent validity. Vox-Profile is publicly available at: this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]

#39
eess.AScs.SD

Auditing generative audio calls for known-task audio-llm evaluation 解读失败跨领域

Mengzhe Geng
Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
查看摘要
Speech and audio LLMs are evaluated by comparing waveform predictions with predictions from an automatic speech recognition (ASR) transcript. For fixed closed-set tasks, this conflates acoustic evidence with the need to invoke a generative audio model. We estimate incremental call value with matched selectors sharing pre-call evidence. Each policy may retain the transcript label, use a local encoder, or invoke a generative model; matched control removes generative actions but preserves pre-call evidence and development selection. On VocalSound, transcript-only accuracy is 0.296, while supervised CLAP and WavLM controls reach 0.850 and 0.854 without calls. Full selector reaches 0.925 at 12.5% calls versus 0.921 for matched No-call selector (difference 0.004; 95% CI [-0.025, 0.033]). Thus, results do not show a call gain after transcript and encoder evidence are available. Relevant quantity is incremental accuracy from allowing calls, not the waveform-transcript gap.

📖 深度解读

[LLM 解读失败: HTTP 402]

#40
eess.AScs.SD

When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Test 解读失败跨领域

Mengzhe Geng
Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
查看摘要
Streaming spoken agents may produce the correct final action after acting too early. Final-turn scores do not reveal whether each observed speech prefix supports an exposed action. We introduce the Partial Speech Action Contract for Turn Taking in Speech Language Models (PACT-SLM), a controlled test that assigns the first valid action time and evaluates both action identity and timing. In the primary test, 80 paired contrast groups from four held-out semantic families yield 1,600 prefix predictions across clean and 15 dB noise renderings. Using source-utterance semantic targets rather than counterbalanced branch codes, WavLM Base Plus reaches 26.03% pooled post-onset semantic-label accuracy (95% group-bootstrap interval [22.14%, 29.68%]), exposes an action on 18.99% of pre-onset prefixes, and predicts 5.94% of complete trajectories exactly. Its pooled label score is at the 96th percentile of 100 within-prefix label permutations, below the 97.5th-percentile reference (26.73%). It exceeds matched text, scalar-acoustic, and shuffled-representation probes in pooled post-onset label accuracy. Elapsed time is more onset-exact (36.25% versus 23.13%) but less accurate about action identity (9.92% versus 26.03%). These results motivate separate measurement of action identity and onset timing in partial-speech evaluations.

📖 深度解读

[LLM 解读失败: HTTP 402]

#41
cs.SD

SwinDS-BWE: A Parameter-Efficient Swin-1D Lattice Generator with Decision-Science Discriminators for Speech Bandwidth Extension 解读失败

Tarikul Islam Tamiti, Sajid Fardin Dipto, Luke Baja-Ricketts, David Vergano, Anomadarshi Barua
Sound (cs.SD)
Comments: 5 pages, 1 figure, 4 tables; submitted to the 2027 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2027)
查看摘要
We propose SwinDS-BWE, a decision-science-inspired bandwidth extension (BWE) model with two coupled contributions: (1) Swin Transformers with lattice-style cross-stream interaction yield locality-aware modeling with linear-in-sequence per-window attention cost and reduce the prior AP-BWE generator size 0.5x from 33M to 17M. (2) Three novel lightweight decision-science-inspired discriminators augment AP-BWE's performance: a CVaR discriminator tail-pools activations to emphasize worst high-frequency (HF) segments, a Chance-Constraint HF discriminator penalizes excessive HF energy via a differentiable barrier, and a Multi-Criteria Utility discriminator learns convex style weights over spectral criteria. SwinDS-BWE surpasses prior AP-BWE with a 30x smaller discriminator (42.3M vs. 1.36M) and higher fidelity on English and French datasets. This work shows that decision-science-inspired critics can supervise BWE to reduce discriminator size.

📖 深度解读

[LLM 解读失败: HTTP 402]

#42
cs.SD

BanglaBox: A Phonetically-Balanced Corpus and Data-Efficient Foundation-Model Adaptation for Bangla Text-to-Speech with Zero-Shot Voice Cloning 解读失败

Emtiaz Uddin Ahmed, Araf Mahmud, Sajib Hossain, Tarikul Islam Tamiti, Sajid Fardin Dipto 等 (6 人)
Sound (cs.SD)
Comments: 24 pages, 6 figures, 25 tables, and 1 algorithm. Includes appendices and the ACL Responsible NLP Checklist. Corpus, model weights, code, and evaluation materials: this https URL
查看摘要
We present a recipe for adapting English-pretrained autoregressive TTS foundation models to underrepresented languages, demonstrated on Bangladeshi Bangla. Existing Bangla TTS corpora are small and single-speaker, and to our knowledge no open zero-shot voice-cloning system is available for the Bangladeshi register. We contribute a phonetically- and gender-balanced two-tier Bangladeshi Bangla corpus balanced via a tiered Jensen-Shannon divergence objective over conjunct clusters (juktakkhor), together with three fine-tuning changes: a merge-consistent tokenizer extension, Bangla text normalization, and a prompt-masked dual-loss objective. These changes preserve zero-shot cloning across the language switch. Our BanglaEval protocol applies Wilcoxon signed-rank tests with Bonferroni correction over native-speaker ratings. BanglaBox attains near-natural Naturalness, outperforms commercial and open-source baselines on Naturalness, Speaker Similarity, and Clarity, and reaches speaker similarity comparable to prior few-shot results while using approximately 7x less Bangla fine-tuning audio. We further validate the recipe beyond our own test split using a seven-category stress set of difficult real-world text, the public BnTTS evaluation benchmarks, and naturally occurring Bangla that no language model wrote. All artifacts, including the corpus, weights, code, and complete evaluation materials, are released publicly and unconditionally.

📖 深度解读

[LLM 解读失败: HTTP 402]

#43
cs.SD

Lyapunov-Inspired LyRIC Activation and GLARE Attention in Chaos-Guided State Space Modeling for EMG-To-Speech (ETS) Synthesis 解读失败

Sajid Fardin Dipto, Tarikul Islam Tamiti, Luke Baja-Ricketts, David Vergano, Anomadarshi Barua
Sound (cs.SD); Signal Processing (eess.SP)
Comments: 27 pages, 12 figures, and 20 tables
查看摘要
Electromyography-to-Speech (ETS) synthesis is typically a non-linear, chaotic dynamical system. However, no prior work has studied the chaotic behavior of ETS synthesis to date. Yet, prior works strictly rely on standard reconstruction metrics with parameter-heavy transformers that systematically over-smooth natural acoustic dynamics. To close this gap, for the first time, we propose a chaos-inspired Lyapunov-derived activation function (LyRIC) with two novel chaotic loss functions, Lyapunov Exponent Regularization and Multi-Scale Detrended Fluctuation Analysis, to explicitly capture the deterministic chaos of human phonation. In addition, we introduce a compressed novel encoder, GLAME, which synergizes global Mamba state-space modeling with localized GLARE attention. We comprehensively perform frame-level acoustic evaluation in a multilingual and multi-speaker setup using English and Mandarin datasets. The proposed system outperforms the established baseline with a 4.69x increase in objective intelligibility (STOI: 0.61 vs. 0.13) and a 2.08x improvement in spectral reconstruction (LSD: 1.08 vs. 2.25). Importantly, this improvement is achieved with 73.49% fewer parameters (14.34M vs. 54.10M), establishing a new baseline for ETS synthesis. To the best of our knowledge, this is the first work demonstrating that integrating non-linear chaotic physics into neural networks yields superior yet compact inductive biases for real-time ETS synthesis.

📖 深度解读

[LLM 解读失败: HTTP 402]

#44
cs.SD

A multi-scenario EEG dataset for auditory attention decoding in naturalistic multi-talker environments 解读失败

Shu Peng, Rui Liu, Yufei Zhang, Wenlong You, Zhige Chen 等 (10 人)
Sound (cs.SD)
查看摘要
Understanding how the brain selectively follows relevant speech amid competing voices is a central challenge in auditory neuroscience and a key step toward neuro-steered hearing technologies. However, most open-source Electroencephalography (EEG) datasets for Auditory Attention Decoding (AAD) use idealized single-competing-talker paradigms that oversimplify the acoustic, spatial, and semantic structure of everyday communication. To capture this ecological complexity, we introduce the SoundBubble-EEG dataset: a high-density 128-channel EEG resource comprising more than 25 hours of recordings from 30 participants. The paradigm requires listeners to selectively attend to a dynamic target speaker group, a designated "sound bubble", amid competing multi-speaker distractor bubbles across three realistic scenarios: a restaurant, a home TV viewing, and a meeting discussion. By bridging the gap between constrained laboratory protocols and real-world auditory scenes, this dataset enables investigations of multi-talker speech comprehension, neural speech tracking, and cross-scenario generalization. It also provides a benchmark for AAD algorithms under realistic acoustic and semantic variability and may support auditory neuroscience and the development of neuro-steered hearing technologies.

📖 深度解读

[LLM 解读失败: HTTP 402]

#45
cs.SD

Backdooring Acoustic Foundation Models for Physically Realizable Triggers 解读失败

Zebin Yun, Eyal Ronen, Mahmood Sharif
Sound (cs.SD); Machine Learning (cs.LG)
Comments: Accepted at RAID 2026
查看摘要
Acoustic foundation models (AFMs) have democratized acoustic applications, enabling powerful models for tasks ranging from speech recognition to speaker verification with minimal resources. However, the security of applications based on AFMs remains largely underexplored. Our work addresses this gap by proposing the Foundation Acoustic model Backdoor (FAB) attack, demonstrating that state-of-the-art AFMs are susceptible to backdooring under practical settings. Despite making minimal assumptions about adversary capabilities (e.g., no access to pre-training data), we show that FAB preserves benign performance while inducing backdoors that survive fine-tuning and cause significant degradation across diverse downstream tasks when activated. Notably, FAB utilizes task-agnostic, physically realizable, inconspicuous, and sync-free triggers (e.g., a background siren). We evaluate FAB using two leading AFMs, nine downstream tasks, and four different triggers. We further demonstrate its effectiveness against established defenses and across both digital and physical domains. While extensive end-to-end fine-tuning can mitigate FAB, such a defense is resource-intensive and task-specific. Our work highlights critical risks to AFMs and calls for advanced defenses.

📖 深度解读

[LLM 解读失败: HTTP 402]

#46
cs.SD

A Study on Improving Multi-class Audio Source Separation Via Decoupled CLAP Query Optimization and an Automated Data Engine 解读失败

Amirhossein Hajavi, Hanhee Lee, Pushya Jain, Sky Qiao, Emmanuel Ko 等 (7 人)
Sound (cs.SD)
Comments: Project page: this https URL
查看摘要
Language-queried audio source separation (LASS) enables extracting any sound source using natural language. However, adapting LASS models to application-specific sound classes is challenging due to noisy training data and limited semantic coverage of the CLAP-based control signals. We propose a framework comprised of an automated data engine for training-data curation and a two-stage optimization process for class-specific CLAP control signals. Our objective evaluations across seven sound classes show that data refinement and control signal optimization consistently improve source separation performance. Subjective evaluation with 17 participants further demonstrates perceptual improvements of the model trained with optimized control signals over baseline and similar commercial models.

📖 深度解读

[LLM 解读失败: HTTP 402]

#47
cs.SD

CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound 解读失败

Marcel Gibier, Thomas Thebaud, Olivier Boëffard, Jean-François Bonastre
Sound (cs.SD); Machine Learning (cs.LG)
Comments: Submitted to ICASSP 2027
查看摘要
Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all. Annotators disagree about this, making a ground truth hard to obtain. In this work, we first define the ground truth, then generate the data. We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it. For scale and variety, a controlled set of scenarios fixes the ground truth, and a language model writes the dialogues. The resulting corpus, CARES, contains 10,000 two-speaker scenes. We then benchmark six audio-language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions. We show that these models hear the sounds but miss how the speakers react to them.

📖 深度解读

[LLM 解读失败: HTTP 402]

#48
cs.SD

MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent 解读失败

Zekai Liu, Zhilin Wang, Xuzheng He, Yu Cheng, Yang Yang
Sound (cs.SD); Artificial Intelligence (cs.AI)
查看摘要
Text-to-music systems produce increasingly convincing audio, yet evaluation reveals little about whether the result matches user intent. A global text-audio relevance score can overlook the implicit intent in underspecified prompts and mask failures in specific requirements, such as instrumentation, structure, rhythm, or mood progression. To bridge this gap, we formulate text-to-music intent alignment as satisfying a per-request rubric of independently verifiable items covering both a request's explicit requirements and its implied musical intent. Scoring items individually makes evaluation diagnostic by intent source and musical dimension, rather than a single opaque score. We instantiate this as MuRA-Bench, a benchmark of real-world platform requests curated by music experts. We further propose MIRA (Musical Intent Refinement Agent), a test-time agent that first grounds a request's intent into rubrics, then searches over prompt revisions for a black-box generator under a bounded budget, iteratively generating music, verifying it against the rubrics, and using this feedback to guide a trajectory-aware tree search. Experiments across open-source and commercial backends show that MIRA improves intent alignment, enabling an open-source generator to achieve performance comparable to representative commercial systems (e.g. Suno and Mureka). Project page: this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]

#49
cs.SD

Unsupervised Maneuver-Aware Acoustic Fault Detection for Autonomous Drones 解读失败

Ali M Ali, Ziyi Tang, Nurdaulet Nazarbay, Hashim A. Hashim
Sound (cs.SD); Systems and Control (eess.SY)
查看摘要
This paper presents a maneuver-aware acoustic fault detection framework for autonomous drones that integrates Noise2Noise-inspired deep learning denoising with maneuver-conditioned reconstruction. A key practical constraint motivating this work is that labeled faulty-flight data are difficult and potentially unsafe to collect; the proposed framework therefore follows an unsupervised learning paradigm in which only nominal flight recordings are required for training. In flight environments, acoustic signals acquired from unmanned aerial vehicles are subject to variability arising both from environmental noise and from structured, maneuver-dependent aerodynamic effects. To address these challenges simultaneously, a two-stage learning architecture is developed. In the first stage, a Noise2Noise-inspired denoising model attenuates stochastic acoustic noise while preserving fault-relevant spectral-temporal structures, without requiring clean reference signals. In the second stage, a maneuver-Conditioned Convolutional AutoEncoder (maneuver-CCAE) is trained using maneuver-related labels including drone type and flight direction to model nominal acoustic behavior under varying operating conditions. Fault detection is subsequently performed using reconstruction error as an anomaly score. Experimental results demonstrate that the proposed maneuver-aware conditioning raises the area under the ROC curve (AUC) from $\AUCaeOnly$ (unconditioned baseline) to $\AUCfull$ (full model), validating the critical role of maneuver-dependent modeling. The complete framework is deployed on an NVIDIA Jetson Orin Nano Super embedded platform within a ROS2 pipeline, achieving an end-to-end fault detection latency of approximately $20\,\text{ms}$ per audio segment with a TensorRT half-precision (FP16) backend, confirming real-time viability for onboard UAV health monitoring.

📖 深度解读

[LLM 解读失败: HTTP 402]

#50
cs.SD

Steering Follows Geometry, Not Labels: Emotion Directions in a Full-Duplex Speech Model 解读失败跨领域

Pulak Kuli
Computation and Language (cs.CL); Sound (cs.SD)
Comments: Accepted at the NeurIPS 2026 Workshop on Real-Time Conversational Agents (RTCA), Sydney. OpenReview: this https URL
查看摘要
Full-duplex voice agents need to modulate emotion and delivery during real-time conversations, when de-escalating a complaint, carrying urgency in dispatch, softening a clinical result. Emotion and delivery control is well studied for TTS and turn based models through prompt-conditioned synthesis, reference-conditioned synthesis and activation steering; PersonaPlex controls identity in a duplex model but not affect. We study emotion steering in Moshi, a fully open sourced full-duplex speech language model, across four emotions, using mean-difference activation steering, which costs only a few vector additions per frame and no retraining. We show that emotion is linearly decodable from Moshi's residual stream, but activation steering is only partially achievable, and unevenly so; as happy, angry and surprise steer towards a shared direction while sad is distinctly steerable. We also show that the shared component across the three emotions cannot simply be projected away from all the emotions equally.

📖 深度解读

[LLM 解读失败: HTTP 402]

#51
cs.SD

Itgan at NADI 2026 shared task: Parameter-Efficient Whisper Adaptation for Robust, Mixed-Dialect and Code-Switched Arabic ASR 解读失败跨领域

Ibrahim Almajai
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Sound (cs.SD)
Comments: 12 pages, Arabic NLP 2026 Shared Task
查看摘要
We describe the Itgan systems for the three ASR subtasks of NADI 2026, namely robust country-level ASR (1.1), mixed-dialect ASR (1.2), and Tunisian code-switched ASR (1.3). All three share one recipe, Whisper adapted with LoRA on consumer GPUs, and each was carried by a different addition to it. On 1.1, where the dialect label is given at test time, per-dialect specialists continued from a pooled adapter gave the largest gain, and the submitted system reached 57.1% country-average WER. A post-evaluation linear probe on frozen encoder features routes utterances without the label and recovers 44% of what oracle routing gives. On 1.2 the choice of base model mattered more than adapter capacity, and system combination helped only once we added a decorrelated member, reaching 46.7% WER. On 1.3 our system placed second at 14.49% WER with the lowest CER among the leading submissions, 5.38%. Its last 0.60 WER points came without further training, mostly from an exact weight-space average of independently trained runs, with ROVER voting adding the remainder. Every comparison carries a paired-bootstrap test, and we report eight directions that did not work.

📖 深度解读

[LLM 解读失败: HTTP 402]

#52
cs.SD

TAC: Timestamped Audio Captioning 解读失败跨领域

Sonal Kumar, Prem Seetharaman, Ke Chen, Oriol Nieto, Jiaqi Su 等 (11 人)
Sound (cs.SD)
查看摘要
Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TAC$\rightarrow$LLM and TAC-V$\rightarrow$LLM cascade achieves state-of-the-art scores on benchmarks for both audio (MMAU-Pro, MMSU, MMAR) and audio-visual (DailyOmni, VideoHolmes) understanding and reasoning respectively.

📖 深度解读

[LLM 解读失败: HTTP 402]

#53
cs.SD

ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood 解读失败跨领域

Tiantian Feng, Anfeng Xu, Xuan Shi, Aditya Kommineni, Shakhrul Iman Siam 等 (13 人)
Sound (cs.SD)
Comments: EMNLP 2026 Main Conference (Accepted)
查看摘要
We present ChildVox, a novel benchmark for characterizing the diverse acoustic signals through which children communicate. Specifically, ChildVox follows the full developmental trajectory from birth through school age, covering physiological sounds, non-linguistic vocalizations, canonical syllables, and spoken language. ChildVox integrates more than 20 sub-tasks across 17 child-centered audio and speech datasets, enabling systematic cross-corpus and cross-domain comparison. We evaluate a representative range of audio and speech foundation models, including self-supervised, ASR-oriented, and large audio-language models, on tasks including physiological sound classification, vocalization and canonical syllables modeling, and speech quality assessment and recognition. Benchmark results show that ChildVox provides a suite of high-performance models in recognizing a wide range of acoustic signals from children, supporting downstream applications such as characterizing children's language levels and tracking speech production with age. Our benchmark models on public datasets are released at: this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]

#54
cs.SD

Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language 解读失败跨领域

Lianru Gao, Yujie Guo, Yong Qin
Sound (cs.SD)
查看摘要
Audiobook narration, conversational agents, and audiovisual dubbing require speech that conveys changing emotions and adapts its pacing within a single utterance. But most existing TTS systems typically rely on utterance-level style conditioning, making such fine-grained control difficult to achieve. In light of this, and inspired by the success of post-training in large language models, we propose a unified post-training framework that equips pretrained text-to-speech models with natural-language control over segment-level emotion and duration. Supervised fine-tuning establishes instruction-conditioned speech generation, while reinforcement learning with group relative policy optimization refines control accuracy using emotion and duration rewards alongside content and speaker preservation objectives. By reusing the pretrained architecture, our approach avoids additional inference-time control modules. Experiments demonstrate significantly improved fine-grained controllability while maintaining speech intelligibility and speaker identity, highlighting post-training as a practical approach to extending existing speech synthesis models.

📖 深度解读

[LLM 解读失败: HTTP 402]

#55
cs.SD

Do Language Models Need Music Supervision? Verifiable Rewards for Multi-Constraint Symbolic Music Generation 解读失败跨领域

Haoyue Liu, Xiaoying Tang
Sound (cs.SD); Artificial Intelligence (cs.AI)
查看摘要
Language models now generate symbolic music from text, and research has focused on musicality. However, many applications require a score that meets explicit constraints, which models struggle to satisfy jointly: on MusicConstraintBench, our benchmark of 2,180 items over eight families of programmatically verifiable constraints, Llama-3.1-70B satisfies 0.630 of single-constraint items but only 0.044 of four-constraint ones. As a remedy, we introduce MusicRLVR, which trains a language model with group relative policy optimisation (GRPO) on verifier rewards alone, needing no human annotation, reward model or music-domain supervised fine-tuning. MusicRLVR incorporates (1) a hard validation gate that rejects malformed scores, (2) graded per-family credit that, unlike a binary reward, separates partially correct outputs, and (3) an all-satisfied bonus for meeting every constraint at once. Extensive experiments show that, in under four hours of training, MusicRLVR raises Qwen3-4B-Instruct-2507 from 0.160 to 0.797 on mixed constraints, outperforming Llama-3.1-70B, and generalises to unseen property combinations, out-of-range parameters and more constraints than any training prompt. The recipe transfers to Qwen3-8B, and neither trained model loses significant accuracy on general benchmarks.

📖 深度解读

[LLM 解读失败: HTTP 402]

#56
cs.SD

Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles 解读失败跨领域

Junyoung Koh, Hao-Wen Dong
Sound (cs.SD); Artificial Intelligence (cs.AI)
查看摘要
Multi-pitch estimation in vocal ensembles is challenging because singers occupy overlapping pitch ranges and often sing at closely spaced fundamental frequencies, causing their harmonics to overlap in time-frequency representations. Existing models commonly use harmonic constant-Q transform (HCQT)-based representations to provide frequency-adaptive resolution, at the cost of expensive feature extraction when training mixtures are generated on the fly. We revisit this design and compare HCQT with a linear short-time Fourier transform (STFT), whose frequency bins are directly provided as model inputs. Despite its fixed frequency resolution and the absence of a pitch-aligned input grid, the linear STFT outperforms HCQT while substantially reducing feature-extraction cost. Further analysis shows that a longer analysis window or broader spectral coverage provides no additional improvement, while restricting the input to the predicted pitch range reduces the advantage of the linear STFT. These results suggest that finer frequency resolution does not necessarily improve vocal-ensemble MPE, and that shorter analysis windows can be more effective for time-varying vocal pitches.

📖 深度解读

[LLM 解读失败: HTTP 402]

#57
cs.SD

SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays 解读失败跨领域

Sonal Kumar, Sinan Hersek, Artem Dementyev, Mengzhen Pan, Ishan Chatterjee 等 (9 人)
Sound (cs.SD); Artificial Intelligence (cs.AI)
查看摘要
Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, we introduce a Spatio-temporal Weighted Cross-Entropy Loss. On our evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.

📖 深度解读

[LLM 解读失败: HTTP 402]

#58
cs.SD

ImpactMat: Continuous Material Estimation for Inverse Impact Sound Rendering 解读失败跨领域

Hyebin Cho, Bumsoo Kim, Joon Son Chung
Sound (cs.SD); Machine Learning (cs.LG)
Comments: Preprint
查看摘要
Impact sound rendering synthesizes the sound produced when a 3D object is struck, but practical renderers often rely on fixed material presets such as wood, plastic, or steel. These presets limit the range of impact sounds a renderer can express, while manually adjusting the underlying material parameters remains difficult without expertise in material acoustics. We therefore study inverse impact sound rendering: predicting material parameters from a reference impact sound so that a simulator can recreate a similar material response. To support this task, we introduce ImpactMat, a dataset and benchmark of single and blended material impact sounds paired with ground-truth material parameters. We further propose a feed-forward model that predicts these parameters from one or more recordings, using blended materials to learn smooth transitions between material types. Experiments show that our method outperforms competitive baselines and enables re-rendering from real recordings without manual parameter tuning. The project page is available this https URL .

📖 深度解读

[LLM 解读失败: HTTP 402]