ArxivWatcher

📡 eess.AS / cs.SD
Audio and Speech Processing, Sound
2026年09月09日
LLM: deepseek-v4-flash
73
论文总数
50
跨领域
49
成功解读
24
待处理
#1
eess.AScs.SD

SETEAB: Multiscale approach with Squeeze-and-Excitation Temporal Enhanced Aware Block for Speech Emotion Recognition 跨领域

Duy Vo, Kiet Anh Hoang, Hao Do
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: Accepted to INTERSPEECH 2026
查看摘要
This paper proposes a novel lightweight multiscale architecture for speech emotion recognition (SER) with three key innovations. First, a depthwise convolution-based subsampling module is introduced to reduce model size and computation while preserving salient emotional cues. Second, a Squeeze-and-Excitation block is integrated to enhance channel-wise recalibration and improve representation robustness. Third, a new Temporal Enhanced Aware Block is designed to strengthen temporal dependency modeling and produce more discriminative emotion-aware features. The proposed model is explicitly designed to jointly improve compactness, recognition performance, and generalizability. Experiments on benchmark SER datasets show that our method achieves higher accuracy with reduced computational complexity, while also delivering stronger cross-corpus performance than most recent advanced networks for SER.

📖 深度解读

1. 一句话总结

这篇论文提出了一种名为SETEAB的轻量级语音情感识别模型,通过改进现有的TIM-Net架构,在保持高精度的同时大幅降低了计算量,并提升了跨数据集的情感识别泛化能力。

2. 研究背景与动机

  • 核心问题:如何在资源受限的边缘设备上实现高效且准确的语音情感识别(SER)。
  • 重要性:SER是实现自然人机交互的关键技术,广泛应用于医疗监控、智能客服、车载安全等领域。然而,现有高性能模型(如基于自监督学习的wav2vec 2.0)计算开销巨大,难以在实时或边缘场景部署。
  • 现有方法的不足:论文指出,作为强基线的TIM-Net及其改进版MS-SENet,其核心的Temporal-Aware Block (TAB) 存在三个主要缺陷:
    1. 缺乏显式归一化,可能导致优化不稳定和梯度衰减。
    2. 使用标准卷积,造成特征冗余且对时间分辨率的控制有限。
    3. 对前向和后向时序信息进行简单的直接求和,这种“平等相加”的假设无法准确反映情感动态的复杂性。

3. 核心方法

论文提出的SETEAB框架是对TIM-Net的四个关键改进,旨在构建一个更高效、更鲁棒的时序建模模块。

  • 关键创新点:

    1. 深度可分离卷积下采样 (Depthwise Convolution Subsampling):在模型前端使用类似FastConformer的模块,通过步长为2的卷积快速压缩时间维度的冗余信息,从而显著降低后续计算量和模型大小。
    2. SE-Res2Block:在时序建模前引入一个集成了Squeeze-and-Excitation (SE) 机制的Res2Block,用于增强局部多尺度特征提取,并对特征通道进行重新校准,强调与情感相关的信息。
    3. 新型时序增强感知块 (Temporal Enhanced Aware Block, TEAB):这是核心创新,是对TAB的升级。它引入了预归一化(LayerNorm)来稳定训练,通过通道扩展和深度可分离卷积来提升特征表达能力并减少冗余,最后使用门控残差学习(sigmoid门控)来自适应地强调时间上显著的情感线索。
    4. 加权双向融合 (Weighted Bidirectional Fusion, BiF):摒弃了TIM-Net中简单相加的方式,引入了两个全局共享的可学习参数(a和b)来分别缩放前向和反向时序特征,从而让模型自适应地学习哪个方向的信息更重要。
  • 核心思路直觉解释:
    可以把SETEAB理解为一个高效的“情感信息提炼流水线”。首先,下采样模块像一个“粗筛”,快速去除语音信号中大量不重要的短时变化。接着,SE-Res2Block像一个“精炼器”,从不同尺度(类似用不同大小的放大镜)观察并强化局部情感特征。然后,TEAB模块是核心的“时序分析师”,它分别从前往后和从后往前阅读特征序列,并用一种“门控”机制(类似一个阀门)来决定哪些时刻的情感信息值得被放大和保留。最后,加权双向融合像一个“决策者”,不再认为“过去”和“未来”的信息同等重要,而是通过学习为两者分配不同的权重,综合得出最终的情感判断。

4. 实验与结果

  • 数据集与基准:使用了EmoBox基准协议下的多个公开数据集。语料内实验在EMOVO、IEMOCAP、RAVDESS、MELD和CREMA-D上进行,评估指标为UA和F1。跨语料实验在IEMOCAP、RAVDESS、MELD和SAVEE上进行,评估指标为WA。
  • 对比基线:与wav2vec 2.0 base、TIM-Net和MS-SENet进行了对比。
  • 主要实验结果:
    • 语料内性能:SETEAB的两个变体(R=2和R=4)在平均UA和F1上均优于所有基线。其中,SETEAB (R=2) 取得了最高的平均UA (47.69%),而SETEAB (R=4) 取得了最高的平均F1 (45.23%)。
    • 效率优势:模型参数量仅约0.4-0.5M,计算量仅0.06-0.12 GFLOPs,远小于wav2vec 2.0 base的95M参数和33.53 GFLOPs。特别是SETEAB (R=4),在保持高精度的同时,计算量仅为0.06 GFLOPs,实现了极佳的性能-效率平衡。
    • 跨语料泛化:在12个跨语料测试设置中,SETEAB在7个上取得了最佳结果,平均WA达到37.53%,显著高于wav2vec 2.0 base (27.67%)、TIM-Net (32.22%)和MS-SENet (34.31%)。
  • 消融实验:通过逐步添加各组件(SE-Res2、DW-Sub、Bi-DW-Sub)进行验证,结果表明每个组件都对最终性能有正向贡献,证明了整体设计的有效性。

5. 优势与局限

  • 主要优势:
    1. 极佳的性能-效率平衡:能以极小的模型和计算开销(<0.5M参数,<0.12 GFLOPs)达到超越现有轻量级模型(TIM-Net, MS-SENet)的精度。
    2. 强大的跨语料泛化能力:在跨数据集实验中表现突出,说明其学习到的情感表征更具可迁移性,更贴近真实应用场景。
    3. 设计创新且有效:针对TAB的缺陷提出了系统性的改进方案(TEAB、BiF等),并通过消融实验验证了每个创新的必要性。
  • 局限性:
    1. 绝对精度仍有待提升:虽然优于基线,但在如MELD等复杂数据集上,UA/F1分数仍较低(约20%),距离实际应用可能还有差距。
    2. 跨语料性能方差较大:虽然平均WA最高,但其标准差(±8.20%)也最大,表明其在不同跨域场景下的性能稳定性可能不如某些基线。
    3. 与SOTA自监督模型存在差距:论文主要与轻量级模型对比,虽然效率远超wav2vec 2.0,但在某些数据集上的绝对精度仍低于它(如IEMOCAP的UA),说明在追求极致轻量化的同时,性能天花板可能受限。

6. 关键结论与启发

  • 最重要的takeaway:通过精巧的架构设计(而非堆砌算力),可以在语音情感识别任务上实现精度、效率和泛化能力的“三赢”。这为在资源受限设备上部署情感计算提供了切实可行的路径。
  • 对后续研究的启发:
    1. 时序建模的精细化:TEAB和加权双向融合的思路表明,对时序信息的处理不应是粗粒度的“相加”,而应是“自适应”和“有选择”的。这为其他序列建模任务(如语音事件检测、说话人验证)提供了参考。
    2. 轻量化设计的系统化:SETEAB的成功展示了将下采样、通道注意力、高效卷积等多种轻量化技术进行系统化整合的巨大潜力,而非单一技术的应用。
    3. 跨域泛化的重要性:论文将跨语料性能作为核心评估指标之一,这提醒研究者,模型的“实战能力”(即跨域鲁棒性)与在单一基准上的分数同等重要,是推动技术落地的关键。
👀 推荐值得看
🏷️ 领域副语言与健康 > 语音情感识别
💡 创新轻量级语音情感识别——基于TIM-Net架构,引入深度可分离卷积下采样、SE-Res2Block、门控残差时序增强块和加权双向融合
⭐ 评分7.5
#2
eess.AS

Direction-Preserving Active Noise Control with a Conditional Control-Filter Estimation Network

Ziyi Yang, Zhengding Luo, Boxiang Wang, Libin Zhang, Woon-Seng Gan
Audio and Speech Processing (eess.AS)
Comments: 12 pages, 7 figures, 4 tables
查看摘要
Conventional active noise control (ANC) minimizes the total disturbance at the error microphone without distinguishing desired sound from noise. Direction-preserving ANC (DP-ANC) instead aims to attenuate a noise component arriving from a direction other than the specified desired direction while preserving sound naturally arriving from that direction. Existing approaches typically either require analytical optimization to be repeated for each new observation or estimate and reproduce the desired component through a hear-through secondary-source path. To address these limitations, this paper formulates DP-ANC as a direction-conditioned cancellation-preservation optimization problem. A component-separated objective jointly penalizes residual noise energy and the control response induced by the desired component, with a scalar weighting parameter controlling the cancellation-preservation trade-off. A convolutional network conditioned on the specified desired direction through feature-wise linear modulation (FiLM) is trained using a differentiable secondary-path-aware forward model. At deployment, the network estimates the complete multichannel finite impulse response (FIR) control-filter bank directly from a mixed-reference observation and the specified desired direction in a single forward pass, while retaining the conventional feedforward ANC signal path. Over 3300 evaluation cases, the selected operating point achieves 22.8 dB mean noise reduction with a desired-signal distortion of -11.4 dB. Validation using measured in-ear-device transfer functions further demonstrates consistent performance under measured acoustic configurations.

📖 深度解读

论文解读报告

1. 一句话总结

这篇论文提出了一种方向保持主动噪声控制(DP-ANC)方法,利用条件控制滤波器估计网络,在抑制来自非期望方向噪声的同时,保留来自指定方向的有用声音,且只需一次前向传播即可完成滤波器估计。


2. 研究背景与动机

核心问题

传统主动噪声控制(ANC)系统在降噪时不区分期望声音和噪声,会不加选择地抑制误差麦克风处的所有声音。但在实际场景(如个人聆听设备)中,用户往往希望只消除来自特定方向以外的干扰噪声,同时保留来自某个指定方向的有用声音(如前方对话者的语音)。

为什么重要

在嘈杂环境中进行语音通话、听音乐或使用助听设备时,用户需要听到来自特定方向(如说话者方向)的声音,同时屏蔽来自其他方向的干扰。这种“方向选择性”的降噪需求在耳机、助听器等可穿戴设备中具有广泛的应用价值。

现有方法的不足

方法类别 主要缺陷
信号分量重建法 引入算法延迟、可能破坏双耳线索
波束成形hear-through 需要通过次级源重建期望信号,引入延迟
解析SSANC 每个新观测都需要重新计算矩阵解,计算量大
选择性固定滤波器ANC 无法为未见条件生成新滤波器,缺乏保持目标
GFANC 仅针对噪声衰减,无显式保持项

核心痛点:现有方法要么需要重复的解析优化,要么需要重建期望信号,没有一个方法能同时实现“显式平衡噪声消除与期望信号保持”且无需预设计滤波器组或重复解析优化。


3. 核心方法

方法框架

论文提出一个方向条件化的控制滤波器估计网络:输入短时混合参考观测和指定的期望方向,直接输出完整的多通道FIR控制滤波器组,在传统前馈ANC结构中使用。

关键创新点

① 分量分离的取消-保持优化目标
将误差信号分解为噪声分量和期望分量,分别定义两个损失项:
- 噪声消除项:最小化残余噪声能量
- 期望保持项:惩罚由期望分量引起的控制响应能量

通过标量权重λ连续调节两者平衡,无需预设目标响应或延迟。

② FiLM条件化的方向编码
将期望方向编码为周期表示[cosθ, sinθ],通过两层MLP生成方向嵌入,再通过特征级线性调制(FiLM)注入每个卷积块,使网络能根据指定方向动态调整特征响应。

③ 可微次级路径感知的前向训练
训练时通过可微的次级路径模型评估估计滤波器,梯度可传播至滤波器系数。采用独立信号实现协议(用一组信号估计滤波器,用另一组独立信号评估损失),增强泛化能力。

④ 单次前向传播完成滤波器估计
部署时只需混合参考观测和指定方向,网络直接输出完整滤波器组,无需重复解析优化,计算量仅为解析SSANC的约1/3100。

核心思路的直觉解释

可以把这个问题想象成“声音版的定向消音”:想象你在嘈杂的咖啡馆里,想听清对面朋友说话(期望方向),同时屏蔽背后施工噪音(噪声方向)。传统ANC像“把所有声音都关小”,而DP-ANC像“只关掉背后传来的声音,保留前方声音”。

网络就像一个“智能音频开关”——它通过多个麦克风感知声音从哪里来,根据你指定的“前方”方向,计算出如何产生一个反相声波,恰好抵消来自其他方向的声音,而对来自前方的声音“视而不见”。


4. 实验与结果

数据集与配置

  • 主模拟配置:6个参考麦克风环形阵列,1个次级源,1个误差麦克风
  • 实测验证:Hearpiece入耳式设备数据库(KEMAR测量,48个水平方位角)
  • 信号内容:高斯信号、UrbanSound8K、FSD50K、LibriSpeech语音
  • 评估规模:3300个测试案例,11个平衡内容组

对比基线

  • ANC off(无控制)
  • 传统Wiener ANC
  • 波束成形hear-through
  • 解析SSANC(空间选择性ANC)

主要实验结果

表III:3300个案例的平均性能(预控制SNR = −10 dB)

方法 NR (dB)↑ Ddes (dB)↓ SNRout (dB)↑
ANC off 0.00 – −10.00
传统Wiener 24.86 −0.32 −6.91
波束成形hear-through 15.19 +1.35 1.13
解析SSANC 18.28 −19.14 7.94
DP-ANC (λ=0.1) 22.82 −11.41 12.52

关键发现:
- DP-ANC在匹配期望信号失真水平下,比解析SSANC的噪声消除高0.68–3.06 dB
- DP-ANC的输出SNR最高(12.52 dB),优于所有基线
- 在相同失真水平下,DP-ANC的NR优于解析SSANC

消融与参数研究

  • λ的作用:λ从0增至1时,NR从25.1降至18.8 dB,Ddes从−2.2降至−20.1 dB,实现连续可调的取消-保持平衡
  • 角度分辨率:当期望与噪声方向从0°增至30°时,DP-ANC的NR从21.5增至23.3 dB,Ddes从−0.6降至−9.4 dB
  • 方向敏感性:仅改变指定期望方向即可交换两个物理声源的“保持/消除”角色
  • 实测路径验证:Hearpiece配置下达到16.6 dB宽带NR和−7.7 dB失真

计算效率

方法 计算量 (GMACs)
DP-ANC 0.00946
解析SSANC ≈29.4

DP-ANC的计算量约为解析方法的1/3100。


5. 优势与局限

主要优势

  1. 高效:单次前向传播即可获得滤波器组,计算量比解析方法低三个数量级
  2. 灵活:通过标量λ连续调节取消-保持平衡,无需预设目标响应或延迟
  3. 通用:同一架构适用于不同麦克风配置(模拟阵列和实测Hearpiece路径均验证有效)

局限性

  1. 场景受限:仅评估了单期望源+单噪声源的静态场景,未处理多源、移动源或混响环境
  2. 固定几何:网络针对特定麦克风几何训练,跨设备需要重新训练或适配
  3. 块处理:基于0.5s观测块,非流式处理,不适合实时动态变化环境
  4. 近方向性能受限:当期望与噪声方向接近时,受阵列空间可分离性限制,性能下降

6. 关键结论与启发

最重要的Takeaway

方向保持ANC可以作为一个方向条件化的取消-保持优化问题来求解,通过数据驱动的方式学习从混合观测到控制滤波器的映射,在保持传统前馈ANC信号路径的同时,实现高效的定向降噪与期望信号保持。

对后续研究的启发

  1. 多源扩展:将分量分离目标扩展到多个同时声源,处理更复杂的声学场景
  2. 流式更新:从块处理转向流式处理,适应动态变化的环境
  3. 跨设备适配:开发跨麦克风几何的迁移学习或元学习方法
  4. 混响鲁棒性:在真实混响环境中验证和增强方法性能
  5. 闭环硬件实现:将方法部署到实际硬件,验证实时性能

值得注意的区分

论文声称的“保持”是指控制滤波器对期望分量不产生响应(即不干扰自然到达的期望声音),而非“增强”或“重建”期望信号。Ddes衡量的是期望诱导控制响应的能量,不应理解为感知失真分数。这一区分对于正确理解方法的能力边界至关重要。

👀 推荐值得看
🏷️ 领域语音增强与分离 > 语音增强
💡 创新方向条件化的控制滤波器估计网络——基于主动噪声控制,引入分量分离的取消-保持优化目标和FiLM条件化方向编码,实现方向选择性降噪
⭐ 评分7.5
#3
eess.AScs.SD

Open-Set Vessel Re-Identification from Underwater Ship-Radiated Noise with a Raw-Waveform Selective-Kernel Acoustic Neural Network (SKANN) and a Cross-Passage Evaluation Protocol 跨领域

Sunil Tyagi
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: 15 pages, 5 figures. An Indian provisional patent application (202611107132, filed 6 September 2026) covers aspects of the method described here
查看摘要
Underwater acoustic target recognition has converged on closed-set classification by vessel type, a task that does not answer whether a monitoring system has heard this hull before. We formalise open-set, cross-passage vessel re-identification on public hydrophone data and specify a protocol that removes the two easiest routes to a high score: hull-disjoint splits keyed to MMSI/IMO, galleries and queries from disjoint passages of each hull, source-pure galleries, and an audio-adjudicated transit-deduplication gate. We describe SKANN, a raw-waveform encoder whose front end is a four-scale bank of learned filters fused by selective-kernel attention, trained with an angular-margin objective and an augmentation regime that perturbs recording chain, ambient noise and multipath while preserving the narrowband lines that carry identity. On a 40-hull IARA gallery (96 queries, 98 passage candidates), cross-passage rank-1 is 0.25 for the embedding and 0.26 for an automated narrowband-tonal comparator; the two are statistically indistinguishable at the top of the ranking, the embedding orders the rest of the list more reliably (AUC 0.82 vs 0.76), and their score fusion reaches rank-1 0.35 -- the only contrast that attains nominal significance, presented as evidence of partial complementarity, not as a recommendation. Transit deduplication alone removes a 16-21 point apparent rank-1 advantage, larger than any between-method difference. Two further findings delimit what public data can support: ShipsEar cannot separate hull identity from recording channel under an identity protocol, and cross-network fine-tuning helps vessels seen during fine-tuning but is a null result on unseen ones. The results support analyst triage over a ranked shortlist, not identification. Checkpoint, validation embeddings, transit map and per-query outputs are released under CC-BY-4.0 (doi: https://doi.org/10.5281/zenodo.22160138 ).

📖 深度解读

1. 一句话总结

这篇论文把水下声学目标识别从“判断船型”的封闭集分类任务,重新定义为一个更贴近实战的“跨航次、开集、个体船重识别”问题,并提出了一个严谨的评估协议和一种名为SKANN的原始波形神经网络,最终结论是:在公开数据上,当前方法只能辅助人工筛选,远达不到可靠“验明正身”的程度。

2. 研究背景与动机

  • 核心问题:论文要解决的核心问题是开集、跨航次的个体船只重识别。即,给定一段水下被动声呐录音,系统能否判断出这是“哪一艘”具体的船(而非仅仅是“货船”或“油轮”),或者判断出这是一艘从未见过的船。
  • 重要性:这是海上态势感知的实际需求。监控系统需要识别出“这艘船我之前听过”,以便进行跟踪、预警或取证。这比传统的“分类”任务更具操作价值,是声学目标识别领域的“终极问题”之一。
  • 现有方法的不足:
    1. 任务错位:主流研究集中在封闭集的“船型分类”上,无法回答“是哪条船”的问题。
    2. 评估作弊:许多评估协议存在严重的数据泄漏,例如将同一段录音的一部分用于训练、一部分用于测试,或者将同一次物理航行的不同录音片段同时放入查询集和候选库。这使得模型学到的是“匹配录音”而非“匹配船只”,导致分数虚高。
    3. 数据先天不足:公开数据集(如ShipsEar)本身的结构(单站点、单设备、重复记录时间过近)无法支撑真正的跨航次身份识别研究。

3. 核心方法

论文的核心贡献在于“问题定义+评估协议”和“模型架构”两部分。

  • 提出的方法/框架:

    1. 严谨的评估协议:定义了开集、跨航次的重识别任务,并强制要求:按船身份(MMSI/IMO)划分训练/验证集;查询和候选库来自同一条船的不同物理航次;候选库必须是“同源纯净”的;并设计了一个基于音频的“航次去重门”,用于剔除数据集中重复或近重复的录音。
    2. SKANN模型:一个端到端的原始波形神经网络编码器,用于学习船只的身份嵌入向量。
  • 关键创新点:

    1. 原始波形+选择性核注意力前端:SKANN不使用传统的频谱图,而是直接在原始音频波形上学习。它用一个四尺度(从16毫秒到1秒)的可学习滤波器组来提取特征,并通过“选择性核注意力”机制,让网络根据每个输入片段动态融合不同时间尺度的信息。
    2. 物理感知的数据增强策略:论文明确区分了“应该增强的”和“绝不能增强的”变量。它模拟录音链路变化、环境噪声和多径效应,但刻意排除了多普勒频移和相位随机化,因为这些变换会破坏承载船只身份信息的窄带线谱的绝对频率。
    3. “航次去重”数据完整性门:这是一个评估层面的创新。通过音频分析(而非仅依赖元数据)来识别并合并来自同一次物理航行的重复录音,堵住了评估中最大的一个漏洞。这本身就是一个重要发现,因为去重前后分数差异巨大。
    4. 诚实的负面结果:论文不仅报告了方法的性能,还报告了两个重要的负面发现:ShipsEar数据集无法用于身份识别;跨网络微调对未见过的船只无效。这些发现对领域发展至关重要。
  • 核心思路的直觉解释:
    可以把SKANN理解为一个“声纹识别”系统。它不关心你说了什么(船的类型),只关心“谁在说话”(是哪条船)。它通过一个非常长的“耳朵”(长卷积核)来聆听极低频的机械振动声(如螺旋桨轴频),这些声音就像船的“指纹”。为了防止它作弊(比如通过识别录音设备的品牌来“认人”),训练时会对声音进行各种“伪装”(增强),但绝不改变“指纹”本身的频率。评估时,则严格确保“考试题目”(查询)和“复习资料”(候选库)不是同一份,从而保证分数真实反映其“认人”能力。

4. 实验与结果

  • 数据集:IARA(主验证集)、ShipsEar(验证其不可用性)、ONC(Ocean Networks Canada,用于跨网络泛化测试)。
  • 对比基线:一个全自动的窄带音调比较器(基于传统声呐信号处理),以及两者的分数融合。
  • 主要实验结果(在IARA 40船库上,96个查询):
    • 跨航次Rank-1准确率:SKANN为0.25,音调比较器为0.26,两者在统计上无显著差异。
    • AUC(排序质量):SKANN为0.82,优于音调比较器的0.76,但差异区间的下限触及0,结论不绝对。
    • 分数融合:将两者融合后,Rank-1提升至0.35,这是唯一达到名义显著性的对比,证明了两种方法存在部分互补性。
    • 去重效果:如果不进行航次去重,SKANN的Rank-1会虚高至0.41-0.47,说明数据泄漏能带来16-21个百分点的虚假优势。
  • 消融/诊断实验:
    • ShipsEar实验:在严格的同源纯净协议下,模型在ShipsEar上的表现与随机猜测无异,甚至同一条船的不同录音之间的相似度低于不同船。这表明该数据集无法区分“船的身份”和“录音环境”。
    • 跨网络实验:在ONC数据上微调模型,能显著提升对“微调时见过的船”的识别率,但对“完全没见过的船”毫无帮助,性能依然与随机猜测无异。

5. 优势与局限

  • 主要优势:
    1. 严谨性:论文最大的贡献在于其严谨的评估协议,特别是“航次去重门”,为领域树立了标杆,揭示了以往结果可能被高估的问题。
    2. 诚实性:如实报告了方法的局限和负面结果(如跨网络泛化失败、ShipsEar不可用),这种诚实对领域发展非常宝贵。
    3. 可解释性:通过分析学习到的滤波器,发现网络自发学习到了类似“恒Q变换”的滤波器组和谐波梳状检测器,证明了模型确实在提取与物理机理相关的特征。
  • 局限性:
    1. 绝对性能不足:Rank-1仅0.25-0.35,意味着系统只能作为“辅助筛选”工具,无法作为“身份确认”的最终依据。
    2. 跨网络泛化能力为零:模型无法识别在训练时未见过的水听器网络上的船只,这严重限制了其在实际部署中的应用。
    3. 数据规模与多样性受限:验证集仅含37-40艘船,样本量小,统计功效有限。且未测试不同航速下的鲁棒性,因为公开数据不支持该实验。

6. 关键结论与启发

  • 最重要的Takeaway:在公开数据上,水下个体船只的跨航次重识别是一个远未解决的问题。当前最先进的方法也只能做到“辅助人工分析”,而非“自主识别”。评估协议(尤其是数据去重)对分数的影响,远大于模型架构的选择。
  • 对后续研究的启发:
    1. 数据是瓶颈:论文明确指出,限制发展的不是算法,而是数据。未来应重点收集“同一条船在独立日期、不同传感器上的多次遭遇”的数据,而非单纯增加录音时长或船只数量。
    2. 评估协议应成为标准:未来的研究应采纳本文提出的“跨航次、同源纯净、航次去重”的评估标准,否则报告的数字将失去参考价值。
    3. 跨域适应是未来方向:既然直接泛化到新网络失败,如何通过域适应、自监督学习等技术,利用目标网络的无标签数据来提升性能,是一个值得探索的方向。
    4. 融合是可行路径:学习型特征与传统物理特征(如窄带音调)的融合被证明有效,未来可以探索更系统、更鲁棒的融合策略。
👀 推荐值得看
🏷️ 领域声音事件与场景 > 声学场景分类
💡 创新原始波形选择性核注意力网络(SKANN)——基于原始波形端到端学习与选择性核注意力机制,用于开集跨航次水下船只重识别
⭐ 评分7.5
#4
eess.AS
Imperial College London (QS Top 100)

Beyond Localisation Accuracy: Sensorimotor Effects of HRTF Individualisation

Fulvio Missoni, Katarina C. Poole, Tim Murray-Browne, Andrea Canessa, Lorenzo Picinali
Audio and Speech Processing (eess.AS)
Comments: 18 pages, 5 figures
查看摘要
Everyday listening requires the brain to integrate cues from the body, environment, other senses, and movement, continuously translating auditory information into action. Yet HRTF individualisation is still commonly assessed through localisation accuracy, which may not fully capture its effects on this sensorimotor process. Here, we investigate whether these effects can instead be revealed through behaviour in a more ecologically valid listening task. We used an aurally guided visual search paradigm in which listeners located a visual target using a co-located virtual sound while moving freely, comparing individualised and non-individualised HRTFs under anechoic and reverberant conditions. Performance was assessed through response times and measures of movement organisation. In anechoic conditions, individualised HRTFs produced faster responses than non-individualised HRTFs, with an average reduction of approximately 200ms and the clearest benefit for front-back source locations. This advantage was expressed primarily in movement initiation, whereas overall movement extent was only weakly affected. Under reverberant conditions, HRTF-dependent differences disappeared. These results suggest that HRTF individualisation can influence how listeners plan and initiate orienting actions even when differences in conventional localisation outcomes are limited. Assessing sensorimotor behaviour alongside localisation performance may therefore provide a more sensitive and ecologically relevant account of the perceptual benefits of HRTF individualisation.

📖 深度解读

1. 一句话总结

这篇论文发现,评估个性化头相关传输函数(HRTF)的价值,不能只看传统的定位准确率,通过一个更贴近日常的“听声辨位”任务(听觉引导的视觉搜索),个性化HRTF能让听者行动更快、决策更果断,这种优势在无回声环境下尤为明显。

2. 研究背景与动机

  • 要解决的核心问题:个性化HRTF(根据个人耳廓等形态定制的空间音频技术)到底有没有用?传统的评估方法(静态定位测试)是否遗漏了其真实价值?
  • 为什么重要:HRTF是虚拟现实(VR)和空间音频的核心技术。非个性化的HRTF(如用假人模型KEMAR测量)会导致声音定位不准,尤其是前后混淆。但个性化HRTF测量成本高、耗时长。因此,准确评估其价值,对于是否值得投入资源进行个性化至关重要。
  • 现有方法的不足:
    • 静态定位测试:虽然能测出准确率,但忽略了真实世界中听者会转头、用眼睛看等自然行为,且个体差异大,结果不稳定。
    • 允许自由活动的动态测试:结果不一致,有时个性化优势消失,因为头部运动产生的动态线索可以弥补部分缺陷。
    • 主观音质评价:听者对非个性化音频的容忍度很高,主观评分上区分不出差异。
    • 核心症结:这些方法大多仍将“听”与“动”割裂,忽略了听觉感知与身体动作的持续耦合关系(即“感觉运动”过程)。

3. 核心方法

  • 方法/框架:采用一种更具生态效度的听觉引导的视觉搜索任务。在VR环境中,听者被28个虚拟扬声器环绕,其中一个会发出声音。听者需要仅凭声音快速找到那个发声的扬声器,并报告其上的LED灯数量。任务允许听者自由转动头部,模拟真实世界的定向行为。
  • 实验条件:2(个性化HRTF vs. KEMAR非个性化HRTF)× 2(无回声 vs. 有早期反射的小房间混响)= 4种声学条件。
  • 关键创新点:
    1. 首次将听觉引导的视觉搜索范式用于评估HRTF个性化,将评估焦点从“最终指向是否准确”转向“整个寻找过程是否高效”。
    2. 引入“感觉运动”指标:不仅测量反应时间(RT),还分析头部运动轨迹,提取运动起始时间(MO)和总角运动范围(TAE),以揭示听者的运动规划和探索策略。
    3. 区分“决策”与“执行”:通过分析运动起始(MO)和运动总量(TAE),将个性化的好处定位在“决定往哪转”的早期规划阶段,而非“转动过程”本身。
  • 核心思路直觉解释:想象你在嘈杂的派对上听到有人叫你,个性化HRTF就像一副为你量身定制的“听觉眼镜”,让你能更清晰地判断声音来自哪个方向,从而更快地转头看向正确的人。而非个性化HRTF则像戴了一副别人的眼镜,虽然最终你可能也能找到人,但一开始会犹豫、需要更多试探性的转头来确认方向。

4. 实验与结果

  • 数据集/基准:12名听力正常的参与者。个性化HRTF为每位参与者实测,非个性化HRTF使用KEMAR假人模型。混响条件使用一个真实小会议室的房间脉冲响应。
  • 对比基线:个性化HRTF vs. KEMAR(非个性化)HRTF;无回声 vs. 有混响。
  • 主要实验结果:
    • 任务可行性:平均正确率高达98.54%,说明任务设计合理。
    • 反应时间(RT):在无回声条件下,个性化HRTF比非个性化HRTF的反应时间显著更快约200毫秒。这种优势在前方和后方声源上最明显,而侧方声源无差异。在有混响条件下,这种优势完全消失。
    • 运动规划(MO):个性化HRTF显著提高了听者初始转头方向的正确率(上下方向判别),表明他们能更快地做出正确的“第一反应”。
    • 运动范围(TAE):在水平(Yaw)方向上,对于前方声源,使用非个性化HRTF时听者的头部探索范围显著更大,暗示他们因前后混淆而需要更多搜索。垂直(Tilt)方向上的差异较小且复杂。
  • 消融实验揭示的规律:混响环境会掩盖或消除个性化HRTF带来的行为优势,因为混响破坏了HRTF所依赖的精细频谱线索。

5. 优势与局限

  • 主要优势:
    1. 更高的生态效度:实验任务更贴近真实世界的“听声辨位”行为,结果更能反映实际应用中的表现。
    2. 更敏感的评估指标:通过分析运动起始时间等感觉运动指标,捕捉到了传统定位准确率无法揭示的细微但重要的行为差异。
    3. 提供了新的理论视角:将HRTF个性化的作用解释为提升“初始空间估计的可靠性”,从而加速行动决策,而非仅仅提高最终指向精度。
  • 局限性:
    1. 混响条件渲染不纯净:实验中,只有直达声使用了个性化HRTF,而混响部分(反射声)仍使用非个性化的KEMAR BRIR。因此,混响条件下个性化优势的消失,可能部分归因于这种不匹配的渲染方法,而非真实的声学效应。
    2. 实时渲染误差:VR中头部移动需要实时插值HRTF,这个过程可能引入高频误差,削弱了个性化HRTF的优势。
    3. 实验环境仍受控:虽然允许头部运动,但参与者坐在椅子上,身体移动受限,与完全自由的真实环境仍有差距。此外,未测量眼动,无法全面揭示“眼-头协调”机制。

6. 关键结论与启发

  • 最重要的Takeaway:评估空间音频技术(如HRTF个性化)时,“行动的速度和决策的信心”可能比“最终指向的精度”更敏感、更具生态相关性。个性化HRTF的价值在于让听者能更快地“启动”一个正确的定向动作,而不是让他们最终指得更准。
  • 对后续研究的启发与延伸方向:
    1. 评估范式革新:未来的空间音频评估应更多地采用动态、交互式的任务,并将运动学指标(如反应时间、运动起始、眼动)作为标准评估维度。
    2. 渲染技术改进:在混响环境中,应使用完全个性化的BRIR(即直达声和反射声都经过个性化滤波),以更纯净地测试个性化在真实声学环境中的价值。
    3. 探索感知-运动耦合:研究HRTF不匹配如何具体影响眼-头协调的时间动态,这将为理解空间听觉的感觉运动机制提供更深入的见解。
    4. 个性化技术的目标优化:既然好处主要体现在“初始定向”,未来的HRTF个性化算法或许可以针对性地优化那些对快速决策至关重要的线索(如前后判别),而非追求全空间的高精度。
👀 推荐值得看
🏷️ 领域空间音频 > 双耳渲染
💡 创新听觉引导的视觉搜索任务——基于感觉运动指标(运动起始时间、总角运动范围)评估HRTF个性化效果
⭐ 评分7.5
#5
eess.AScs.SD
Alibaba (World Famous IT Company)

Semantic Refinement of Universal Audio Representations through Audio-Description Alignment 跨领域

Lejun Min, Junyu Dai, Ruichen Zheng, Xinyue Fan, Yang Xiang 等 (10 人)
Audio and Speech Processing (eess.AS); Sound (cs.SD)
查看摘要
Universal audio representations must preserve acoustic detail while making high-level concepts accessible across speech, music, environmental sound, and downstream models of different capacities. We study semantic refinement of an acoustically pretrained encoder by adding audio-description alignment to a foundation of BEST-RQ, reconstruction, and CTC. We compare matched control, shuffled-description, and correctly paired trajectories to distinguish correct correspondence from an extra contrastive objective. Each endpoint is frozen and evaluated with a temporal-mean linear probe and a sequence-aware LLM readout, testing whether the refined information is directly accessible and remains useful to a stronger model. Across three paired seeds, correct alignment improves domain-balanced classification by 4.66 points with the linear probe and 2.59 points with the sequence-aware LLM, with positive changes in every domain. Correct pairing accounts for 87% of the linear-probe gain, while the LLM shows its clearest correspondence-specific benefit in captioning. Dense acoustic objectives provide complementary gains under both readouts. A separate 24-layer continuation remains competitive with leading public encoders under the shared evaluator, supporting the recipe beyond the controlled study.

📖 深度解读

1. 一句话总结

这篇论文研究了如何通过“音频-文本描述对齐”这一语义精炼手段,在保留声学细节的基础上,为通用的音频基础模型补充高层语义信息,并通过严格的对照实验证明了“正确的配对”而非“额外的对比学习目标”才是带来跨域(语音、音乐、环境音)分类性能提升的关键。

2. 研究背景与动机

  • 核心问题:如何在不牺牲声学细节的前提下,为通用的音频表征(覆盖语音、音乐和环境音)注入可迁移的高层语义概念(如场景、风格、乐器)?具体而言,论文要回答一个关键问题:在已有的声学预训练基础上,“音频-描述”对齐带来的性能提升,究竟是因为“正确的语义配对”,还是仅仅因为“增加了一个额外的对比学习任务”?

  • 问题重要性:随着音频系统从特定领域识别走向跨模态理解和生成,一个通用的音频表征需要同时具备两种能力:一是保留精细的时频声学结构(用于还原和识别),二是围绕高层概念组织信息(用于跨任务和跨域迁移)。声学预训练擅长前者,而如何系统性地补充后者(即“语义精炼”)是构建通用音频模型的关键一步。

  • 现有方法不足:

    • 已有的音频-语言模型(如CLAP)虽然证明了语言监督的价值,但其增益是在不同的训练配方、数据混合和下游评估系统下测量的,无法清晰归因。
    • 现有研究缺乏在严格控制变量下的对照实验,无法区分“正确的语义对应关系”与“额外的对比目标”各自贡献了多少。
    • 大多数评估只关注一种下游模型(如简单的线性探针或强大的LLM),不清楚语义信息是“直接线性可读的”,还是“需要依赖一个强大的下游模型才能被挖掘出来”。

3. 核心方法

  • 方法/框架:论文提出了一种多域语义精炼配方。它从一个共享的BEST-RQ-Conformer声学编码器(12层)出发,在保留原有声学目标(BEST-RQ、梅尔/色度重建、CTC语音识别)的基础上,增加了一个“片段级音频-描述对齐”目标。为了验证该目标的有效性,论文设计了三条严格匹配的训练轨迹:

    1. 控制组 (Acoustic+CTC):仅使用声学目标和CTC。
    2. 打乱描述组 (Shuffled):在控制组基础上增加对比学习,但将描述与音频的配对随机打乱。
    3. 正确对齐组 (Correct):在控制组基础上增加对比学习,且描述与音频正确配对。
      这三条轨迹共享初始化、数据顺序、优化器设置等一切变量,唯一区别在于是否加入对比目标以及配对是否正确。
  • 关键创新点:

    1. 严格的因果对照设计:通过“正确对齐 vs. 打乱描述”的对比,首次在匹配的训练条件下,将“正确语义对应”的贡献从“额外对比目标”中分离出来。
    2. 双读头评估协议:同时使用时间均值线性探针(测试信息是否可直接线性访问)和序列感知LLM读头(测试信息是否对高容量模型有用),揭示了不同语义信息对不同下游模型的差异化价值。
    3. 多域、非均衡监督路由:在语音(有转写无描述)、音乐(有歌词和描述)、环境音(仅有描述)三个域上共同训练,并巧妙利用语音域作为“零样本”迁移测试集,检验从音乐和环境音学到的语义对齐是否能泛化到语音。
  • 核心思路直觉:可以把声学预训练比作“学会了声音的‘口音’和‘腔调’”,而语义精炼则是“教它认识‘话里的意思’”。论文的核心操作是,在教“意思”的时候,分了两组:一组是“看图说话”式的正确教学(正确对齐),另一组是“乱点鸳鸯谱”式的错误教学(打乱描述)。通过对比两组的学习效果,就能知道“正确的意思”到底重不重要,而不是“多上一节课”本身就有效。

4. 实验与结果

  • 数据集/基准:

    • 训练数据:内部采集的语音、音乐和环境音(约23.7k小时),其中音乐和环境音配有自动生成的文本描述。
    • 评估基准:采用X-ARES框架下的任务。
      • 任务1 (T1, 分类):覆盖14个数据集,包括语音(CREMA-D, FSC等)、音乐(FMA, GTZAN等)和环境音(ESC-50, FSD50K等)。
      • 任务2 (T2, 生成):ASR(AISHELL-1, LibriSpeech)和音频描述生成(Clotho, SongDescriber等)。
  • 基线方法:主要基线是论文自己设计的控制组 (Acoustic+CTC) 和打乱描述组 (Shuffled)。在规模对比实验中,与WavLM Large、MuQ-large、DaSheng-Base和SPEAR XLarge v2等公开模型比较。

  • 主要实验结果:

    • 正确对齐有效:在3个随机种子下,正确对齐相比控制组,在T1分类任务上,线性探针平均提升4.66分,LLM读头平均提升2.59分,且所有域均有提升。
    • 正确配对是关键:正确对齐相比打乱描述,线性探针T1提升4.07分。这意味着正确配对贡献了从控制组到正确对齐组总增益的87%(4.07 / 4.66)。打乱描述仅带来0.59分的提升,说明“额外的对比目标”本身作用甚微。
    • LLM读头的差异化发现:在LLM读头下,正确配对带来的T1增益为2.10分,其优势在描述生成任务(Captioning)上最明显(比打乱组高2.02分),而在ASR上优势很小(仅0.33分)。
    • 声学目标互补:移除BEST-RQ、梅尔和色度重建目标会损害性能(线性T1 -1.80,LLM T1 -2.02),证明语义目标不能替代声学学习,两者是互补的。
  • 消融实验揭示:

    • 更细粒度的目标(时间/事件描述):在片段级对齐基础上增加时间/事件描述,能进一步提升线性探针T1(+1.31分),但对LLM读头和T2任务影响不大,说明细粒度语义主要增强了信息的“线性可读性”。
    • 描述来源:仅用音乐或仅用环境音描述训练,各自在自己领域提升最大;两者结合能带来最均衡和最高的整体T1性能。

5. 优势与局限

  • 主要优势:

    1. 严谨的因果推断:通过精心设计的对照实验,令人信服地证明了“正确的语义配对”是带来迁移增益的核心原因,而非其他混杂因素。
    2. 全面的评估视角:双读头设计(线性探针 vs. LLM)提供了更细致的视图,揭示了不同形式的语义信息(如片段级 vs. 事件级)如何被不同能力的下游模型所利用。
    3. 实用性强:验证了该精炼配方在扩大模型规模(24层)后,仍能与业界领先的公开模型竞争,证明了其实际应用价值。
  • 局限性:

    1. 依赖自动生成的描述:训练所用的音乐和环境音描述来自内部自动流程,其质量可能不如人工标注,可能限制了语义对齐的上限。
    2. 语音域无描述:语音域没有描述监督,其提升是间接的。虽然证明了迁移能力,但也可能意味着如果加入语音描述,语音域的性能会更高。
    3. 公开对比非严格匹配:与公开模型的对比是“非匹配”的,在架构、数据量、优化细节上均有差异,因此只能作为规模化的参考,不能严格归因于方法本身。

6. 关键结论与启发

  • 最重要的Takeaway:在声学-词汇基础上进行语义精炼时,“正确的音频-描述配对”是产生可迁移语义增益的主要来源,其效果远大于单纯增加一个对比学习目标。 这种增益不仅体现在简单的线性分类器上,也体现在强大的序列感知LLM上,尤其是在描述生成任务中。

  • 对后续研究的启发:

    1. 数据质量 > 目标数量:与其堆砌各种学习目标,不如专注于获取高质量、正确配对的音频-文本数据。这为数据工程指明了方向。
    2. 评估协议需要分层:未来的研究应区分“线性可读的语义”和“需要复杂模型才能提取的语义”,使用不同能力的读头进行评估,能更精确地定位表征学习方法的改进点。
    3. 探索跨域语义迁移:论文证明了从音乐/环境音学到的语义可以迁移到语音。这启发研究者可以探索更多跨域(如从有丰富标注的域到稀缺域)的语义知识迁移方法,以解决数据瓶颈问题。
#6
eess.AS

Spatial Audio Coding Through Relative Room Impulse Response Estimation

Nour Bouayed, Adrien Llave, Jérôme Daniel, Pascal Scalart
Audio and Speech Processing (eess.AS)
Comments: Submitted to International Networked Immersive Audio 2026 (satellite Event of IEEE IS2 2026)
查看摘要
Immersive virtual listening relies on spatial audio technologies such as Higher-Order Ambisonics (HOA), which represent sound scenes as multichannel signals. As the desired spatial resolution increases, so does the number of channels, making efficient compression essential for transmission over bandwidth-limited networks. Moreover, to facilitate deployment by network operators, the target bitrate for immersive audio coding should ideally remain close to the 25 kbps currently allocated to VoLTE audio services. State-of-the-art parametric codecs, such as the recently standardized Immersive Voice and Audio Services (IVAS) codec, achieve compression by transmitting spatial metadata together with a reduced number of transport channels. However, recent studies have shown that IVAS performance degrades on reverberant content, particularly at low bitrates, a limitation that suggests its inability to accurately model room acoustics. In this paper, we propose a novel HOA coding scheme based on the explicit and blind estimation of the Relative Spatial Room Impulse Response (ReSRIR), using a beamformed version of the HOA signal as a reference signal. By exploiting the structure and sparsity of the estimated ReSRIR, we derive an efficient parametric representation for immersive audio coding. Experimental evaluations show that the proposed method achieves higher compression than IVAS in the single-transport-channel regime, while maintaining comparable to slightly better quality.

📖 深度解读

1. 一句话总结

这篇论文提出了一种新的三维音频(Ambisonics)压缩方法,通过“盲估计”房间的声学响应(ReSRIR),并将其高效参数化,从而在比现有标准编码器(IVAS)更低的码率下,达到相近甚至更好的混响环境音质。

2. 研究背景与动机

  • 核心问题:如何在高压缩比(目标约25-32 kbps)下,高效地传输高质量的三维沉浸式音频(特别是Higher-Order Ambisonics, HOA)信号,同时准确还原真实房间的声学效果(尤其是早期反射和混响)。
  • 重要性:随着VR/AR、远程会议等沉浸式通信的发展,需要通过网络传输三维音频。为了在带宽受限的网络上部署,音频码率需接近现有VoLTE服务的25 kbps。HOA格式虽然灵活,但通道数多(3阶需16通道),数据量大,必须进行高效压缩。
  • 现有方法的不足:
    • 最先进的IVAS标准编码器在低码率下处理混响内容时性能下降,因为它对房间声学(特别是早期反射)的建模能力有限。其传输的空间参数只保留了预测系数的实部,导致时域响应是“对称”的(非因果),无法精确表示真实的房间脉冲响应。
    • IVAS主要依赖去相关滤波器来合成混响,这类滤波器更适合模拟后期“噪声状”混响,而不擅长精确重建对感知质量至关重要的早期反射。
    • 虽然存在基于神经网络的SpatialCodec,但它针对的是线性麦克风阵列而非Ambisonics,且其优化目标(解码后波束形成)与提升感知质量不完全一致。

3. 核心方法

论文提出了一种基于“相对空间房间脉冲响应”(ReSRIR)显式估计与编码的HOA编码方案。其核心思路是:不再像IVAS那样在频域传输简化的预测系数,而是回到时域,直接估计并压缩一条连接“参考信号”与“多通道HOA信号”的“房间滤波器”。

关键创新点:

  1. 盲估计ReSRIR:利用“自导向”迭代算法,从HOA信号中盲估计出相对于波束形成信号的ReSRIR。该算法交替估计声源方向(DoA)和ReSRIR,利用波束形成抑制混响,得到一个稀疏、准因果的时域滤波器,这比IVAS的频域参数更物理可解释。
  2. 稀疏+残差的参数化分解:将估计出的ReSRIR分解为两部分:
    • 稀疏部分:包含直达声和主要早期反射的“波包”。通过提取其延迟和幅度,并用“波包”而非单点脉冲来表示,以捕捉墙壁吸收造成的频谱整形。
    • 残差部分:包含剩余的后期混响能量。用降采样的多通道能量包络(方差)来参数化,并在解码端通过“滤波噪声整形”技术合成。
  3. 高效的参数量化与编码:对分解后的参数(延迟差、波包幅度、残差包络)使用学习型向量量化器(VQ)进行压缩。通过将高维波包投影到低维潜在空间再量化,大幅降低了参数码率(约2 kbps)。

直觉解释:想象一个房间的声音传播路径。直达声和几次早期反射就像几个清晰的“回声”(稀疏部分),而后续的混响则像一片模糊的“嗡嗡声”(残差部分)。该方法先通过算法“听”出这些回声和嗡嗡声的特征(ReSRIR),然后只记录这些特征(参数),而不是记录原始声音。解码时,根据这些特征重新“制造”出回声和嗡嗡声,叠加在传输的干声信号上,从而重建出空间感。

4. 实验与结果

  • 数据集:
    • 训练:使用Aalto、IKSLabIR、MOTUS三个公开的真实HOA房间脉冲响应(SRIR)数据库,与DNS-Challenge的语音样本卷积合成3阶Ambisonics信号。
    • 测试:使用两个训练中未见过的SRIR数据库(RSoANU和50-Lebedev)生成测试集。
  • 基线方法:主要对比IVAS @ 32 kbps(单传输通道模式),并参考IVAS @ 64 kbps。为保证公平,所有方法的第一传输通道均使用相同的EVS核心编码器(24.4 kbps)。
  • 主要实验结果:
    • 提出的方法(总码率27 kbps)在QASTANet空间音频质量评分上,与IVAS @ 32 kbps相当或略好,且码率更低(27 vs 32 kbps)。
    • 在训练数据(验证集)上,其得分接近甚至达到IVAS @ 64 kbps的水平。
  • 消融实验揭示:
    • 残差模型:使用简单的对角协方差(能量包络)模型(wp+σ)优于使用完整的协方差矩阵(wp+Σ),后者可能因宽带描述过于粗糙而无法捕捉频率相关的统计特性。
    • 稀疏模型:将“波包”近似为单个“狄拉克脉冲”(δ+σ)会显著降低性能,证实了波包模型对于捕捉早期反射的频谱特性(如墙壁吸收)至关重要。

5. 优势与局限

  • 主要优势:
    1. 更高压缩效率:在达到相近或更好音质的前提下,码率比IVAS @ 32 kbps更低(27 kbps),更接近运营商的目标码率。
    2. 更好的混响建模:通过显式估计和编码时域ReSRIR,能更精确地重建早期反射,解决了IVAS在混响环境下性能下降的问题。
    3. 可解释性与可控性:编码参数(延迟、方向、能量)具有明确的物理意义,便于理解和后续处理。
  • 局限性:
    1. 场景受限:目前方法仅针对单一静态声源的场景,未处理多声源或移动声源的复杂情况。
    2. 评估指标:主要依赖QASTANet这一深度学习客观指标,缺乏全面的主观听音测试验证。
    3. 模型泛化性:虽然测试了未见过的数据集,但训练和测试数据均为合成的(真实SRIR与干声卷积),未在真实录制的HOA内容上进行验证。

6. 关键结论与启发

  • 核心Takeaway:论文有力地证明了“对空间参数(即房间声学)进行精细、物理感知的建模与编码,是提升低码率下沉浸式音频质量的关键”。与其在频域用简化系数近似,不如回到时域,显式地建模和压缩房间的“指纹”(ReSRIR),这能更高效地利用比特率。
  • 对后续研究的启发:
    1. 多声源扩展:将框架扩展至多声源场景,可考虑利用声源的稀疏性或慢时变特性来优化参数分配。
    2. 更精细的残差模型:探索更高效的多频带协方差模型来替代宽带包络,以在不增加过多码率的前提下提升残差合成质量。
    3. 端到端优化:将ReSRIR估计、参数化、量化与核心编码器进行联合训练,可能获得更优的整体率失真性能。
    4. 验证与标准化:进行全面的主观听音测试,并在真实录制的Ambisonics内容上验证,以推动该技术走向实际应用和标准化。

客观备注:论文声称其方法在混响条件下优于IVAS,但实验主要基于客观指标QASTANet,且测试场景为合成的单声源信号。其声称的“可比或略好”的性能,在更复杂、真实场景下的表现仍需进一步验证。

#7
eess.AS

Interpreting Dolphin Vocal Sequences via Multiple Sequence Alignment 解读失败

Daniel Kohlsdorf, Denise Herzing, Thad Starner
Audio and Speech Processing (eess.AS)
查看摘要
Dolphin communication understanding is essential for uncovering the linguistic complexity and social structures of wild pods. We adapt the ClustalW bioinformatics algorithm to analyze continuous acoustic data, treating vocalizations as high-dimensional spectral feature vectors. By replacing discrete scoring with a continuous Gaussian kernel similarity measure, our framework generates Multiple Sequence Alignment (MSA) visualizations that reveal shared structural patterns. These alignments highlight temporal motifs such as synchronized burst pulses in aggressive contexts that are difficult to detect through standard spectrogram inspection.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#8
eess.AScs.SD

Omni Interaction Agent Technical Report 解读失败跨领域

Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu 等 (23 人)
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM); Sound (cs.SD)
Comments: Project Page: this https URL
查看摘要
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#9
eess.AScs.SD

Lead Vocal Separation from Vocal Ensemble Mixtures Using Phoneme Alignment 解读失败跨领域

Yuma Narahata, Tomohiko Nakamura, Yuki Saito, Hiroshi Saruwatari
Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: Accepted for APSIPA Annual Summit and Conference 2026
查看摘要
Contemporary a cappella singing often has a lead-and-accompaniment texture, where the lead vocal (Vo) part carries the main melody and the remaining vocal parts provide accompaniment. Owing to their distinct roles, separating the Vo part from the remaining vocal parts, referred to as Vo separation, enables downstream applications such as lyric recognition and minus-one accompaniment generation for vocal ensemble music. Despite these potential applications, acoustic cues for this task are limited because the target and interfering sources are all singing voices with similar acoustic characteristics and often overlap in time, making Vo separation challenging. In this paper, we propose a Vo separation model that uses phoneme alignment of the Vo part as auxiliary information. The proposed model is based on band-split RoPE Transformer (BS-RoFormer), a state-of-the-art music source separation model, and introduces frame-level phoneme labels into its intermediate representations using feature-wise linear modulation (FiLM). Experimental results show that phoneme-alignment conditioning improves Vo separation performance over an audio-only baseline and yields larger average gains than conditioning only on Vo singing/silence activity. Further analysis suggests that the advantage of phoneme-label information is larger when fewer remaining vocal parts share the same phoneme as Vo.

📖 深度解读

[PDF 下载失败,无法解读]

#10
eess.AScs.SD

ConversationalVoice: Full-Duplex Speech Data from Real Conversations through Source-Faithful Reconstruction and Conversation-Grounded Expansion 解读失败跨领域

Richard Yucheng He, Baodong Cao, Chen Xu, Yihang Liu, Tairan Chen
Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: 12 pages, 3 figures, 1 table, preprint
查看摘要
Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordings. We present Conversational Voice, a pipeline that converts real two-speaker excerpts into three complementary training-data artifacts. (1) Separation recovers speaker-specific tracks with stable speaker assignments, a canonical transcript, and naturally observed interaction timing. (2) Reconstruction generates speech in matched voices from a fixed source transcript, reconstructs the source turn order, pauses, and overlaps, and adds word-level alignment and delivery instructions. (3) Expansion generates new dialogue constrained by the source context, speakers, and observed interaction pattern. Automatic speaker-verification metrics remain strong across stages, with same-speaker similarity of 0.983-0.991 and positive discrimination margins of 0.199-0.209. Predicted speech quality (NISQA MOS) is 3.56 for separation, 4.41 for reconstruction, and 4.61 for expansion. A Gemini-based automatic evaluator assigns expansion mean scores of 4.94/5 for contextual coherence and 4.80/5 for dialogue naturalness. Expansion and reconstruction exhibit broadly similar interaction profiles; expansion's turn, overlap-event, backchannel, and interruption rates are 4.6%, 8.0%, 13.2%, and 16.0% lower, respectively. We evaluate data properties only; downstream gains in full-duplex model training remain for future work.

📖 深度解读

[PDF 下载失败,无法解读]

#11
eess.AScs.SD

Rescuing Performance from the Demo: Co-Designing Drum Gesture Mappings with a Percussionist 解读失败跨领域

Jordie Shier, Teresa Pelinski, Charalampos Saitis, Andrew Robertson, Andrew McPherson
Human-Computer Interaction (cs.HC); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: Accepted for publication at AIMC 2026
查看摘要
Augmenting instruments with sensors and neural network mappings is a well-explored digital musical instrument design approach. While augmentations can create new expressive opportunities, they also exert aesthetic influence and can constrain musicians' gestural language, which, if left unchecked, can lead to technological capture. To examine this, we conducted a study with a professional percussionist, co-developing a gesture mapping toolkit and recording a ten-track album. Drawing on the concept of productive dissonance, our study aimed to hold the musician's aesthetic in tension with technological constraints. This, along with a practice-based reflective approach, supported the development of a continuous gesture recognition method for percussive mapping and surfaced insights into the design process. We identify knowing-when as a form of tacit knowledge that supported productive dissonance, and raise an open question: absent a musician's broader social context, how do we know whether a technology's influence is genuinely supporting their practice?

📖 深度解读

[PDF 下载失败,无法解读]

#12
eess.AScs.SD

From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection 解读失败跨领域

Mengzhe Geng, Yujia Lu, Patrick Littell, Manuela Kunz, Xie Chen
Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
查看摘要
Speech deepfakes can mimic a speaker's voice convincingly enough to deceive listeners and automated systems. This has driven strong progress in speech deepfake detection, but most detectors still end with one score per utterance. That score is useful for ranking systems, yet it says little about why a borderline item should be trusted, deferred, or reviewed. Two utterances can fall in the same score band for different reasons, for example because passive and retrieval evidence disagree or because the keyed probe is unavailable. We ask whether the final decision can remain scalar without discarding that provenance. We answer this question with an auditable decision record that carries four aligned cues into a late calibration step: a passive detector score, a conditional keyed-probe score on a marked derivative, retrieval support, and a speaker-profile margin, together with explicit disagreement coordinates. On the 4,080-example ASVspoof 5 Track 1 matched subset, the fixed retrieval-augmented rule improves on retrieval-only evidence, from 15.84 percent to 11.91 percent EER, and late calibration over the full record reaches 8.43 percent EER. At a 33.75 percent review budget, the exposed cue union covers 82.85 percent of the calibrated model's errors. The best passive WavLM run still reaches 6.71 percent EER, so we do not present the decision record as a stronger standalone detector. Its contribution is to preserve the evidence behind each surfaced utterance while still producing one operating score for thresholding and review.

📖 深度解读

[PDF 下载失败,无法解读]

#13
eess.AScs.SD

An Investigation of Reprogramming for Cross-Language Adaptation in Speaker Verification Systems 解读失败跨领域

Jingyu Li, Aemon Yat Fei Chiu, Tan Lee
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: Accepted to the 14th International Symposium on Chinese Spoken Language Processing (ISCSLP 2024)
查看摘要
Language mismatch is among the most common and challenging domain mismatches in deploying speaker verification (SV) systems. Adversarial reprogramming has shown promising results in cross-language adaptation for SV. The reprogramming is implemented by padding learnable parameters on the two sides of input speech signals. In this paper, we investigate the relationship between the number of padded parameters and the performance of the reprogrammed models. Sufficient experiments are conducted with different scales of SV models and datasets. The results demonstrate that reprogramming consistently improves the performance of cross-language SV, while the improvement is saturated or even degraded when using larger padding lengths. The performance is mainly determined by the capacity of the original SV models instead of the number of padded parameters. The SV models with larger scales have higher upper bounds in performance and can endure longer padding without performance degradation.

📖 深度解读

[PDF 下载失败,无法解读]

#14
eess.AScs.SD

A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations 解读失败跨领域

Aemon Yat Fei Chiu, Kei Ching Fung, Roger Tsz Yeung Li, Jingyu Li, Tan Lee
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: Accepted to the 15th International Symposium on Chinese Spoken Language Processing (ISCSLP 2026)
查看摘要
Enhancing explainability in speech self-supervised learning (SSL) is important for understanding and effectively utilising speech SSL representations. This study conducts a large-scale layer-wise probing analysis of 11 speech SSL models, examining speaker identity together with acoustic, prosodic, and paralinguistic attributes. The results confirm a general hierarchy wherein initial layers encode fundamental acoustics and middle layers synthesise abstract traits. The consensus that final layers purely abstract linguistic content is challenged, as larger models unexpectedly recover speaker-discriminative information in their deep layers. Furthermore, the intermediate representations of speech SSL models are found to provide stronger probing performance for several attributes than specialised speaker embeddings. These insights give a systematic empirical characterisation of speaker-related information in SSL models, providing guidelines for selecting representations for downstream tasks.

📖 深度解读

[PDF 下载失败,无法解读]

#15
eess.AScs.SD

CUHK-EE Systems for the vTAD Challenge at NCMMSC 2025 解读失败跨领域

Aemon Yat Fei Chiu, Jingyu Li, Yusheng Tian, Guangyan Zhang, Tan Lee
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: Accepted to the 20th National Conference on Man-Machine Speech Communication (NCMMSC 2025)
查看摘要
This paper presents the Voice Timbre Attribute Detection (vTAD) systems developed by the Digital Signal Processing & Speech Technology Laboratory (DSP&STL) of the Department of Electronic Engineering (EE) at The Chinese University of Hong Kong (CUHK) for the 20th National Conference on Human-Computer Speech Communication (NCMMSC 2025) vTAD Challenge. The proposed systems leverage WavLM-Large embeddings with attentive statistical pooling (ASTP) to extract robust speaker representations, followed by two variants of Diff-Net, i.e., Feed-Forward Neural Network (FFN) and Squeeze-and-Excitation-enhanced Residual FFN (SE-ResFFN), to compare timbre attribute intensities between utterance pairs. Experimental results demonstrate that the WavLM-Large+FFN system generalises better to unseen speakers, achieving 77.96% accuracy and 21.79% equal error rate (EER), while the WavLM-Large+SE-ResFFN model excels in the 'Seen' setting with 94.42% accuracy and 5.49% EER. These findings highlight a trade-off between model complexity and generalisation, and underscore the importance of architectural choices in fine-grained speaker modelling. Our analysis also reveals the impact of speaker identity, annotation subjectivity, and data imbalance on system performance, pointing to future directions for improving robustness and fairness in timbre attribute detection.

📖 深度解读

[PDF 下载失败,无法解读]

#16
eess.AScs.SD

BFA: Real-time Multilingual Text-to-speech Forced Alignment 解读失败跨领域

Abdul Rehman, Jingyao Cai, Jian-Jun Zhang, Xiaosong Yang
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: Updated algorithm and results will be published later
查看摘要
We present Bournemouth Forced Aligner (BFA), a system that combines a Contextless Universal Phoneme Encoder (CUPE) with a connectionist temporal classification (CTC)based decoder. BFA introduces explicit modelling of inter-phoneme gaps and silences and hierarchical decoding strategies, enabling fine-grained boundary prediction. Evaluations on TIMIT and Buckeye corpora show that BFA achieves competitive recall relative to Montreal Forced Aligner at relaxed tolerance levels, while predicting both onset and offset boundaries for richer temporal structure. BFA processes speech up to 240x faster than MFA, enabling faster than real-time alignment. This combination of speed and silence-aware alignment opens opportunities for interactive speech applications previously constrained by slow aligners.

📖 深度解读

[PDF 下载失败,无法解读]

#17
eess.AScs.SD

AudioFuse: Unified Spectral-Temporal Learning via a Hybrid ViT-1D CNN Architecture for Robust Phonocardiogram Classification 解读失败跨领域

Md. Saiful Bari Siddiqui, Utsab Saha
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Sound (cs.SD); Signal Processing (eess.SP)
Comments: Published in the Proceedings of 2026 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) main conference. Find the published version at: this https URL
查看摘要
Biomedical audio signals, such as phonocardiograms (PCG), are inherently rhythmic and contain diagnostic information in both their spectral (tonal) and temporal domains. Standard 2D spectrograms provide rich spectral features but compromise the phase information and temporal precision of the 1D waveform. We propose AudioFuse, an architecture that simultaneously learns from both complementary representations to classify PCGs. To mitigate the overfitting risk common in fusion models, we integrate a custom, wide-and-shallow Vision Transformer (ViT) for spectrograms with a shallow 1D CNN for raw waveforms. On the PhysioNet 2016 dataset, AudioFuse achieves a state-of-the-art competitive ROC-AUC of 0.8608 when trained from scratch, outperforming its spectrogram (0.8066) and waveform (0.8223) baselines. Moreover, it demonstrates superior robustness to domain shift on the challenging PASCAL dataset, maintaining an ROC-AUC of 0.7181 while the spectrogram baseline collapses (0.4873). Fusing complementary representations thus provides a strong inductive bias, enabling the creation of efficient, generalizable classifiers without requiring large-scale pre-training. The complete source code for our baseline and AudioFuse models is publicly available on GitHub at: this https URL .

📖 深度解读

[PDF 下载失败,无法解读]

#18
eess.AScs.SD
Xinjiang University (211)Tsinghua University (QS Top 100, 985, 211)

CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment 跨领域

Hanwen Liu, Saierdaer Yusuyin, Hao Huang, Zhijian Ou
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Sound (cs.SD)
Comments: Fix two typos in Figure 2 in INTERSPEECH 2026 version
查看摘要
Large-language-model (LLM)-based text-to-speech (TTS) systems can generate natural speech, but most are not designed for low-latency dual-streaming synthesis. High-quality dual-streaming TTS depends on accurate text--speech alignment and well-designed training sequences that balance synthesis quality and latency. Prior work often relies on GMM-HMM based forced-alignment toolkits (e.g., MFA), which are pipeline-heavy and less flexible than neural aligners; fixed-ratio interleaving of text and speech tokens struggles to capture text--speech alignment regularities. We propose CTC-TTS, which replaces MFA with a CTC based aligner and introduces a bi-word based interleaving strategy. Two variants are designed: CTC-TTS-L (token concatenation along the sequence length) for higher quality and CTC-TTS-F (embedding stacking along the feature dimension) for lower latency. Experiments show that CTC-TTS outperforms fixed-ratio interleaving and MFA-based baselines on streaming synthesis and zero-shot tasks.

📖 深度解读

1. 一句话总结

这篇论文提出了 CTC-TTS,一种基于大语言模型(LLM)的流式语音合成方法,用轻量级的 CTC 对齐 替代了笨重的 MFA 强制对齐,并设计了 “双词(bi-word)”交错序列,在保证合成质量的同时显著降低了流式合成的延迟。


2. 研究背景与动机

  • 核心问题:如何让基于 LLM 的 TTS 系统支持 低延迟的“双流式”合成(即文本边输入、语音边输出),同时保证合成语音的质量和自然度。
  • 为什么重要:传统的 LLM-TTS 需要等整句文本输入完毕才开始合成,首包延迟(FPL)高,无法满足实时交互场景(如语音助手、实时翻译)的需求。双流式合成是解决该问题的关键。
  • 现有方法的不足:
    • 早期方法(如 VALL-E)将全部文本和语音拼接,延迟高。
    • 现有流式方法(如 IST-LM、CosyVoice 2)使用 固定比例交错 文本和语音,难以捕捉文本与语音之间的真实对齐规律,模型学习困难。
    • 高质量的对齐式交错方法(如 ELLA-V、SyncSpeech)依赖 MFA(GMM-HMM 强制对齐),这种工具链复杂、灵活性差,且需要帧级别的精确音素边界,对模型学习而言并非最优。

3. 核心方法

  • 方法/框架:CTC-TTS 框架包含五个组件:G2P 模型(文本转音素)、CTC 模型(语音-音素对齐)、解码器-only Transformer(建模交错序列)、神经音频编解码器(NAC)和对齐交错模块。
  • 关键创新点:

    1. CTC 对齐替代 MFA:利用 CTC 模型(如 Whistle)的 Viterbi 解码路径,将语音帧与音素序列进行“结构级”对齐,无需帧级精确边界。这比 MFA 更轻量、更灵活。
    2. 双词(bi-word)交错策略:将“当前词音素 + 分隔符 + 下一个词音素”作为一个文本块,后接“当前词对应的语音 token”,构成一个训练块。这提供了紧凑的“前瞻”信息,帮助模型理解上下文,同时不引入过多延迟。
    3. 两种变体实现质量-延迟权衡:
      • CTC-TTS-L:将文本和语音 token 在序列长度维度上拼接,追求更高的合成质量。
      • CTC-TTS-F:将文本和语音 embedding 在特征维度上堆叠,允许模型在读到第一个音素后立即开始合成,显著降低首包延迟。
  • 核心思路直觉:

    • 传统 MFA 像是一个“逐帧精确标注”的严苛老师,要求每个音素的起止时间都精确到帧。而 CTC 对齐更像是一个“按块划分”的宽松老师,只要求知道“哪些音素对应哪些语音片段”,这种粗粒度的对齐信息对自回归模型来说已经足够,且更容易学习。
    • “双词”交错可以理解为:模型在合成当前词的声音时,不仅知道当前词的发音,还“偷看”了下一个词的发音,以便更好地处理协同发音等上下文效应,但又不会像看整句话那样引入大量延迟。

4. 实验与结果

  • 数据集:
    • 单说话人流式实验:VoiceAssistant400K 数据集(1750 小时训练)。
    • 多说话人零样本实验:LibriSpeech 960 小时训练集,测试集为 test-clean(续说任务)和 Seed-TTS test-en(跨说话人任务)。
  • 基线方法:
    • 单说话人:LLMVox(固定比例交错 + 特征堆叠)。
    • 多说话人:复现的 ELLA-V(MFA 对齐 + ELLA-V 序列组织),并做了消融对比(CTC+ELLA-V、MFA+bi-word)。
  • 主要实验结果:
    • 单说话人流式:CTC-TTS-F 相比 LLMVox,WER 从 2.40% 降至 1.80%,CER 从 1.36% 降至 1.04%,且 FPL-A 更低(159ms vs 167ms)。CTC-TTS-L 进一步将 WER 降至 1.50%,CER 降至 0.79%,但 FPL-A 略高(210ms)。
    • 多说话人零样本(续说任务):CTC-TTS-L 在 WER(4.82%)、CER(2.47%)和 MOS(4.33)上全面优于所有基线,接近 Ground Truth 水平。CTC-TTS-F 也优于 MFA+bi-word。
    • 多说话人零样本(跨说话人任务):CTC-TTS-L 依然表现最佳(WER 6.33%,MOS 4.23)。CTC-TTS-F 略逊于 MFA+bi-word,但作者指出两者不可直接比较(特征堆叠 vs 长度拼接)。
  • 消融实验揭示:
    • CTC 对齐 vs MFA 对齐:在相同 bi-word 序列下,CTC 对齐(CTC-TTS-L)优于 MFA 对齐(MFA+bi-word),尤其在跨说话人任务中优势明显,说明 CTC 对齐泛化性更好。
    • bi-word 序列 vs ELLA-V 序列:在相同 CTC 对齐下,bi-word 序列(CTC-TTS-L)远优于 ELLA-V 序列(CTC+ELLA-V),说明 bi-word 的交错设计更有利于模型学习。

5. 优势与局限

  • 主要优势:
    1. 轻量且灵活:CTC 对齐摆脱了 MFA 的复杂工具链,且不需要帧级精确边界,降低了系统复杂度和训练难度。
    2. 质量与延迟兼顾:通过 L/F 两种变体,提供了清晰的质量-延迟权衡选项,用户可根据场景需求选择。
    3. 泛化能力强:在跨说话人、跨领域的零样本任务中,CTC 对齐表现出比 MFA 更好的泛化性能。
  • 局限性:
    1. 依赖单码本 NAC:当前设计基于单码本神经音频编解码器(WavTokenizer),对于多码本 NAC(如 SoundStream、EnCodec)的扩展性未验证。
    2. CTC 对齐的精度上限:CTC 对齐不提供帧级精确边界,虽然对当前任务足够,但在需要精确时长控制的场景下可能受限。
    3. 实验规模有限:主观评测(MOS/SMOS)仅基于 30 个样本和 20 位听者,样本量较小,统计功效有限。

6. 关键结论与启发

  • 最重要的 takeaway:“结构级对齐 + 紧凑前瞻交错” 是 LLM-TTS 实现高质量低延迟流式合成的关键。CTC 对齐提供的粗粒度对应关系,配合 bi-word 序列的局部上下文,足以让自回归模型学习到文本到语音的映射,且比 MFA 更高效、更易泛化。
  • 对后续研究的启发:
    1. 对齐器的选择:CTC 对齐有望成为 MFA 的轻量替代品,未来可探索将 CTC 对齐扩展到其他语言、其他对齐单元(如字素、子词)。
    2. 序列设计空间:bi-word 交错策略展示了“前瞻窗口”大小对质量-延迟权衡的影响,未来可探索更灵活的动态前瞻机制。
    3. 模型架构扩展:将 CTC-TTS 的思路扩展到多码本 NAC、多说话人联合训练等更复杂的场景,是自然的延伸方向。
    4. 端到端化:论文提到未来可引入神经 G2P 和更精确的神经强制对齐,有望进一步简化流水线,实现更端到端的训练。

说明:论文全文信息完整,以上解读覆盖了所有要求的部分。实验部分的关键数字均来自论文中的表格,解读时区分了论文声称的结论与实际展示的数据。

#19
eess.AScs.SD
Tsinghua University (QS Top 100, 985, 211)

Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition 跨领域

Lukuang Dong, Ziwei Li, Saierdaer Yusuyin, Xianyu Zhao, Zhijian Ou
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD)
Comments: NCMMSC 2026 accepted version
查看摘要
Phoneme-based ASR factorizes recognition into speech-to-phoneme (S2P) and phoneme-to-grapheme (P2G), enabling cross-lingual acoustic sharing while keeping language-specific orthography in a separate module. While large language models (LLMs) are promising for P2G, multilingual P2G remains challenging due to language-aware generation and severe cross-language data imbalance. We study multilingual LLM-based P2G on the ten-language CV-Lang10 benchmark. We examine robustness strategies that account for S2P uncertainty, including DANP and Simplified SKM (S-SKM). S-SKM is a Monte Carlo approximation that avoids CTC-based S2P probability weighting in P2G training. Robust training and low-resource oversampling reduce the average WER from 10.56% to 7.66%.

📖 深度解读

学术论文解读报告

1. 一句话总结

这篇论文研究如何让大语言模型(LLM)在多语言语音识别中更好地完成“音素转文字”任务,通过引入鲁棒性训练和数据平衡策略,将10种语言的平均词错误率从10.56%降到了7.61%。

2. 研究背景与动机

核心问题: 在多语言环境下,如何让基于大语言模型的“音素到文字”(Phoneme-to-Grapheme, P2G)转换模块更准确、更鲁棒?

为什么重要: 基于音素的语音识别(ASR)系统将识别过程分解为“语音→音素”和“音素→文字”两个阶段。这种设计允许不同语言共享声学建模能力,对低资源语言尤其有价值。传统的P2G解码依赖加权有限状态转换器(WFST),需要为每种语言单独构建词典和解码图,扩展性差。用LLM替代WFST可以摆脱对语言特定资源的依赖,但多语言场景下存在独特挑战。

现有方法的不足:
- WFST方法需要语言特定的词典和解码图,难以扩展到多语言和快速变化的词汇
- 已有的LLM-P2G方法(如DANP、TKM、SKM)主要在单语言场景下验证,多语言场景下的表现未知
- 多语言训练面临两个核心难题:①模型需要根据音素输入生成特定语言的文字(语言感知生成);②不同语言的数据量严重不均衡(如英语有2227小时训练数据,而鞑靼语仅20.8小时)

3. 核心方法

方法框架: 论文采用两阶段解耦架构——Whistle-large模型负责“语音→音素”(S2P),Qwen3系列LLM(1.7B/4B参数)负责“音素→文字”(P2G)。P2G模型通过LoRA微调,输入格式为“音素序列”,输出为“语言标识符+文字序列”。

关键创新点:

  1. 统一的多语言LLM-P2G框架:使用单一模型处理10种语言的P2G转换,通过输出语言标识符(如<lid>)实现语言感知生成,无需为每种语言单独构建解码图。

  2. S-SKM(简化采样K边缘化)策略:这是对已有SKM方法的简化。SKM需要计算每个采样音素假设的CTC概率作为权重(需要运行前向算法),而S-SKM直接对所有采样假设取平均(蒙特卡洛估计),省去了概率计算开销。类比来说,SKM像是对多个候选答案按“可信度”加权投票,而S-SKM则让每个候选答案拥有同等投票权——实验表明这在实践中几乎不影响效果。

  3. 低资源语言过采样策略:以意大利语(271.5小时)为参照,对每个低资源语言按整数倍重复采样,使其训练数据量至少达到意大利语水平。这类似于让“少数群体”在训练中获得更多“发言机会”。

  4. 系统对比DANP与S-SKM两种鲁棒性策略:DANP通过向训练数据注入S2P模型的噪声音素假设来增强鲁棒性;S-SKM则通过从CTC分布采样音素路径来模拟S2P不确定性。

4. 实验与结果

数据集: CV-Lang10基准——包含10种语言的Common Voice子集(英语、西班牙语、法语、意大利语、吉尔吉斯语、瑞典语、荷兰语、俄语、土耳其语、鞑靼语)。单语言消融实验使用波兰语和德语。

基线对比:
- E1:直接多语言微调(无鲁棒性处理)
- E2/E4:分别加DANP或S-SKM鲁棒性训练
- E3/E5:鲁棒性训练+过采样
- E6:WFST传统基线

主要实验结果:

模型 平均WER 训练时长加权WER
E1(直接微调) 10.56% 8.46%
E2(+DANP) 8.45% 8.11%
E3(+DANP+过采样) 7.61% 7.93%
E4(+S-SKM) 9.64% 8.41%
E5(+S-SKM+过采样) 7.66% 8.22%
E6(WFST) 6.56% 9.20%

关键发现:
- 单语言消融实验中,S-SKM与SKM性能几乎一致(差距在0.15%以内),验证了简化方法的有效性
- DANP在多语言场景下略优于S-SKM(8.45% vs 9.64%)
- 过采样显著提升低资源语言表现:如吉尔吉斯语从10.07%降至3.97%(DANP),鞑靼语从23.25%降至13.25%
- WFST在最低资源语言上仍占优(如吉尔吉斯语0.94% vs 3.97%),但LLM-P2G在训练数据充足的多数语言上已能匹敌或超越WFST
- 所有模型的语言识别准确率都很高(98.8%-99.0%),说明错误主要来自文字生成而非语言识别

5. 优势与局限

优势:
1. 免图解码:不需要为每种语言构建WFST解码图,单一统一模型即可处理多语言,扩展性好
2. 鲁棒性提升显著:结合DANP和过采样,平均WER从10.56%降至7.61%,相对改善约28%
3. S-SKM简化有效:在几乎不损失性能的情况下省去了CTC概率计算,降低了训练复杂度

局限:
1. 低资源语言仍不及WFST:在数据极少的语言(如吉尔吉斯语、鞑靼语)上,WFST凭借语言特定词典仍明显占优
2. S2P错误仍是瓶颈:对于S2P音素错误率高的语言(如瑞典语11.33%),即使加了鲁棒性训练和过采样,最终WER仍然偏高(10.59%)
3. 过采样策略简单粗暴:按整数倍重复采样可能不是最优方案,且论文未深入探讨过采样倍数与性能的权衡

6. 关键结论与启发

最重要的takeaway: 多语言音素型ASR的性能瓶颈因语言而异——对S2P错误率适中的低资源语言(如吉尔吉斯语),主要限制是P2G训练数据不足,过采样非常有效;对S2P错误率高的语言(如瑞典语),即使P2G再优化,上游错误仍然制约整体性能。因此,S2P质量和P2G鲁棒性需要联合优化。

对后续研究的启发:
1. S2P与P2G联合训练:论文提到免发音词典的端到端联合训练(S2P+P2G)是值得探索的方向,可能突破当前两阶段解耦的性能天花板
2. 更智能的数据平衡:简单的整数倍过采样已有显著效果,更精细的采样策略(如根据语言难度或S2P错误率动态调整)可能带来更大提升
3. 鲁棒性方法的语言适配:DANP和S-SKM在不同语言上表现各有优劣,未来可探索按语言特性自适应选择鲁棒性策略
4. LLM规模的影响:论文主要使用1.7B和4B模型,更大规模的LLM是否能在低资源语言上进一步缩小与WFST的差距,值得验证

#20
eess.AS

Towards Fine-Grained Multi-Dimensional Speech Understanding: Data Pipeline, Benchmark, and Model 跨领域

Guojian Li, Zhixian Zhao, Zhennan Lin, Jingbin Hu, Qirui Zhan 等 (12 人)
Audio and Speech Processing (eess.AS)
查看摘要
While speech Large Language Models (LLMs) excel at conventional tasks like basic speech recognition, they lack fine-grained, multi-dimensional perception. This deficiency is evident in their struggle to disentangle complex features like micro-acoustic cues, acoustic scenes, and paralinguistic signals. This resulting incomplete comprehension of real-world speech fundamentally bottlenecks the development of perceptive and empathetic next-generation speech systems. At its core, this persistent perceptual limitation primarily stems from three interacting factors: scarce high-quality expressive data, absent fine-grained modeling for multi-dimensional attributes, and reliance on restricted coverage, coarse-grained benchmarks. We address these challenges through three pillars: First, our robust data curation pipeline resolves complex acoustic environments and long-audio timestamp alignment challenges to extract a high-quality spontaneous speech corpus from audiovisual sources. Second, we construct FMSU-Bench, a pioneering benchmark covering 14 speech attribute dimensions to rigorously assess the fine-grained, multi-dimensional speech understanding capabilities of current models. Third, empowered by our curated corpus, we introduce FM-Speech. Driven by a decoupled attribute modeling and progressive curriculum fine-tuning framework, it substantially elevates fine-grained, multi-dimensional acoustic perception. Extensive evaluations on FMSU-Bench reveal that current speech LLMs still require significant improvement in multi-dimensional, fine-grained understanding. In contrast, FM-Speech substantially outperforms current open-source models, establishing a robust paradigm for real-world speech understanding.

📖 深度解读

1. 一句话总结

这篇论文针对当前语音大模型在“细粒度、多维度”语音理解上的不足,通过构建高质量的数据管线、发布全新的基准测试集(FMSU-Bench)并提出渐进式课程微调框架,训练出了一个在多项任务上超越现有开源模型的语音理解模型(FM-Speech)。

2. 研究背景与动机

  • 核心问题:当前的语音大模型虽然能很好地完成语音识别等常规任务,但对语音中蕴含的“细粒度”和“多维度”信息(如细微的声学特征、环境声、副语言信号等)感知能力很弱,无法像人类一样全面、深入地理解一段真实世界的语音。
  • 问题重要性:这种感知能力的缺失是开发更具“感知力”和“共情力”的下一代语音AI(如情感陪伴、智能助手)的根本瓶颈。如果模型只能听懂“说了什么”,却听不懂“怎么说”和“在什么环境下说”,就无法实现真正智能的交互。
  • 现有方法的不足:
    • 数据层面:现有语料库要么是缺乏表现力的朗读语音(如LibriSpeech),要么是缺乏统一、细粒度、多维度标注的表现力语音(如Emilia)。
    • 评估层面:现有基准(如AIR-bench, MMAU)维度覆盖不全,且多采用粗粒度的整体标签,无法精细评估模型对细微声学特征的感知能力。
    • 模型层面:现有语音LLM(如Qwen3-Omni)对语音属性的建模是受限或纠缠的,容易产生“文本幻觉”(即过度依赖文本先验而忽略真实声学证据)。

3. 核心方法

论文提出了一个“三位一体”的解决方案,包含数据、基准和模型三个部分。

  • 数据管线:一个由LLM驱动、多专家交叉验证的鲁棒数据管线。它从电影、电视剧等“野生”音视频中提取高质量、带细粒度标注的自发言语语料。核心是解决复杂声学环境和长音频时间戳对齐的难题。
  • 基准测试集 (FMSU-Bench):一个开创性的、包含超过2万个中英双语实例的基准,系统覆盖了14个语音维度,并组织成一个5层分类体系(说话人属性、声学韵律、情感语义、声学场景、语言-副语言整合)。
  • 模型 (FM-Speech):基于Qwen3-Omni架构,采用渐进式课程微调框架进行训练。

关键创新点:
1. 鲁棒的数据管线:提出了“安全分块”策略(Safe-Chunking),通过VAD和ASR双重时间戳融合,在5-6分钟的窗口内精确地在静音处切分音频,避免截断语音。并设计了“宏观到微观”的两阶段渐进式标注策略,利用LLM先标注依赖上下文的宏观属性,再细化到微观属性。
2. 多专家交叉验证机制:为了克服单一LLM(Gemini)的幻觉和偏见,引入了多个专家模型(如Qwen3-ASR、emotion2vec、WavLM等)对转写、情感、声学特征、说话人属性等进行交叉验证和过滤,保证了数据质量。
3. 渐进式课程微调框架:将训练过程分解为三个阶段(热身、能力提升、最终对齐),并动态混合三种数据类型(单维MCQ、单维开放QA、全维度JSON生成)。这避免了直接让模型学习复杂结构化输出导致的信息过载和模态差距,使其能逐步从“感知”过渡到“生成”。
4. FMSU-Bench基准设计:设计了包含“细粒度声学干扰项”和“语义陷阱干扰项”的多选题,能有效区分模型是真正听懂了声音,还是在根据文本语义猜测。

4. 实验与结果

  • 数据集/基准:主要在作者自己构建的 FMSU-Bench 上进行评估,该基准包含中英双语、14个任务维度。
  • 对比基线:与11个先进的语音LLM对比,包括8个开源模型(如Qwen3-Omni, Kimi-Audio, Audio Flamingo 3等)和3个专有模型(Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3.1 Pro)。
  • 主要实验结果:
    • 在FMSU-Bench上,FM-Speech 取得了72.8%的平均分,全面超越了所有评估的开源模型(最好的Qwen3-Omni为69.4%),甚至超过了专有的Gemini 3 Flash (71.9%),逼近最强的Gemini 3.1 Pro (74.0%)。
    • 在多个维度(如性别、年龄、口音、语气、背景声、副语言事件等)上,FM-Speech取得了最佳或次优成绩。
    • 实验也揭示了现有模型的普遍弱点:在需要微观声学感知(如音高、音色)和语言-副语言整合的任务上,所有模型(包括最强的Gemini 3.1 Pro)的表现都显著下降,说明这是当前技术的共同瓶颈。
  • 消融实验揭示:
    • 渐进式课程的必要性:如果跳过课程,直接用全维度JSON数据微调(Single Stage),模型性能反而会下降(Qwen3-Omni从69.4%降至67.8%),证明了渐进式训练对于避免“信息过载”至关重要。
    • 框架的通用性:将该训练框架应用于另一个模型Qwen2.5-Omni,同样带来了显著的性能提升(从59.7%提升至63.9%),证明了该框架具有跨架构的泛化能力。

5. 优势与局限

  • 主要优势:
    1. 系统性:论文不是只提出了一个模型,而是构建了从数据到评估再到训练的完整闭环,解决了该领域数据、基准、方法三方面的问题。
    2. 数据质量高:通过多专家交叉验证和人工审核,确保了训练和测试数据的质量,为模型性能提供了坚实基础。
    3. 方法有效且可迁移:渐进式课程微调框架被证明是有效且架构无关的,可以应用于其他语音LLM,为社区提供了有价值的训练范式。
  • 局限性:
    1. 依赖专有LLM:数据标注和基准构建过程严重依赖Gemini 2.5 Pro等专有模型,这可能导致数据存在潜在的偏见,且无法完全复现。
    2. 性能天花板:尽管FM-Speech表现出色,但在某些微观声学任务(如PIT, VT)上的绝对准确率仍不高(约60%左右),说明细粒度感知仍是巨大挑战。
    3. 基准的“应试”性质:FMSU-Bench主要采用多选题形式,虽然便于客观评估,但可能无法完全反映模型在开放、真实场景下的理解能力。

6. 关键结论与启发

  • 最重要的takeaway:当前语音LLM的“语义理解”能力已接近饱和,真正的瓶颈在于“声学感知”能力。通过高质量的数据和精心设计的训练策略,可以显著提升模型的细粒度、多维度语音理解能力,开源模型完全有能力与专有模型竞争。
  • 对后续研究的启发:
    1. 数据是燃料:构建高质量、细粒度标注的数据是推动该领域进步的关键,本文提供了一套可行的自动化管线。
    2. 训练策略很重要:渐进式课程学习是训练复杂多模态模型的有效策略,可以推广到其他多模态理解任务中。
    3. 评估需要新维度:未来的语音理解评估需要超越传统的ASR和粗粒度情感分类,设计能衡量“真听”而非“瞎猜”的评测任务(如本文的语义陷阱题)。
    4. 未来方向:如何进一步提升模型对微观声学特征(如音色、韵律)的感知,以及如何将这种细粒度理解能力应用于更复杂的交互式任务(如情感陪伴、心理辅导),是值得探索的方向。
#21
eess.AS

Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding 解读失败跨领域

Zhiyuan Zhu, Yixuan Chen, Yiwen Shao, Wenxiang Guo, Changhao Pan 等 (16 人)
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI)
查看摘要
Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding. We propose Spatial-Omni, a lightweight method that implements SO-Encoder to inject First-Order Ambisonics (FOA) spatial audio into existing Omni LLMs as an independent modality, without modifying their original audio encoders. SO-Encoder provides spatial tokens with limited additional context cost and improves spatial audio understanding through efficient staged training. To support training and evaluation, we construct SO-Dataset, SO-QA, and SO-Bench from open-source data, real recordings, and simulations, containing 400K FOA spatial audio clips and 2.1M spatial question answering pairs. SO-Bench covers 16 spatial audio understanding subtasks, including basic detection and location estimation, spatial relation understanding, and complex spatial reasoning. Experiments show that Spatial-Omni outperforms existing open-source Large Audio-Language Models (LALMs) and Omni LLM models on spatial audio understanding tasks while retaining a reasonable level of general audio understanding. Code and data are available at this https URL .

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#22
eess.AS

Goodbye Equal Error Rate, Hello Local Information Disclosure: Evaluating Voice Anonymisation against 1-to-N Linkage Threats 解读失败跨领域

Dāvis Šterns, Konstantinos Drossos, Natasha Fernandes, Tom Bäckström, Catuscia Palamidessi
Audio and Speech Processing (eess.AS)
Comments: Accepted for publication at the Symposium on Security and Privacy in Speech Communication (SPSC 2026)
查看摘要
Voice anonymisation aims to protect speaker identity. Currently, its empirical privacy evaluation heavily relies on the Equal Error Rate (EER). Originally designed for biometric verification, EER aggregates scores globally, implicitly assuming an attacker is only trying to verify if two specific voice samples match (a 1-to-1 comparison). This introduces a threat model mismatch with real-world database linkage attacks, where an attacker searches across a fixed set of N enrolled identities (a 1-to-N closed-set search), allowing global averages to obscure localised privacy failures. While recent 1-to-N metrics address this aggregation issue, they abstract away the magnitude of the biometric evidence. In this paper, we propose a modular, information-theoretic evaluation framework explicitly designed for the 1-to-N linkage threat model. Within this framework, our core metric, Local Information Disclosure (LID), provides a principled estimate of the identity information disclosed by a single trial utterance in bits by mapping raw similarity scores to an estimated posterior distribution over the enrolled identities. Demonstrating this framework on VoicePrivacy 2024 Challenge systems reveals how global metrics can obscure privacy vulnerabilities. Even for top-performing systems where standard evaluations report near-perfect EERs (48%), our metric exposes that attackers gain a statistical advantage in at least 63% of trials with maximum observed disclosures reaching 1 bit (effectively doubling the attacker's confidence in the target speaker after observing a single trial utterance). We conclude that shifting toward explainable metrics and appropriate threat models is a practical step toward identifying worst-case vulnerabilities and aligning with strict privacy regulations.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#23
eess.AScs.SD

Extracting Voice Styles from Frozen TTS Models via Gradient-Based Inverse Optimization 解读失败跨领域

Gyeongmin Kim
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: 5 pages, 2 figures, 2 tables. Code and demo: this https URL
查看摘要
Some text-to-speech systems ship a synthesis model and preset style vectors but withhold the reference encoder that turns a recording into a style vector, so a user cannot obtain a style for a new voice. We recover that vector without the encoder by inverting the released pipeline with gradient descent: all weights stay frozen and only the style vector is optimized, against time-pooled WavLM statistics of one recording of the target. The objective discards the time axis, so no transcript is needed. Two constraints from the release make the result a drop-in style: every row of the timbre style stays on the unit sphere where the presets lie, and the small duration style, which a time-pooled loss cannot see, is fitted to the recording's speaking rate. On SupertonicTTS, over 147 speakers and 100 held-out sentences each, ECAPA-TDNN similarity rises from 0.129 to 0.419, every recovered style starts from a preset and ends closer to its target than that preset was, and the pooled word error rate stays below that of the presets. A verifier at its equal-error point accepts 52% of the recovered voices as the target speaker, against 1% of the presets.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#24
eess.AScs.SD

The ISCSLP 2026 Real-World Audio-Visual Speech Enhancement Challenge 解读失败跨领域

Kai Li, Wenze Ren, Junjie Li, Cheng Yu, Peijun Yang 等 (14 人)
Audio and Speech Processing (eess.AS); Sound (cs.SD)
Comments: The First Real-World Audio-Visual Speech Enhancement (AVSE) Challenge
查看摘要
Audio-visual speech enhancement (AVSE) uses visual-speech cues from a target speaker to recover that speaker's speech from noisy or overlapping speech. Many widely used protocols construct mixed signals from separately recorded audio sources and assume reliable video, leaving their performance under natural overlap and visual failure insufficiently characterized. The Real-World AVSE Challenge evaluates two related settings. Track~1 comprises two scenarios: real-world mixtures recorded with two speakers speaking simultaneously, without a corresponding clean reference signal, and synthetic remixes obtained by manually mixing the separately recorded speech of two speakers, with a clean reference signal available; Track~2 reuses audio but pairs it with a degraded target video and contains additional 3-m far-field recordings. The speakers in the development and test sets are disjoint. Evaluation metrics include clean-waveform fidelity, learned quality estimates, transcription accuracy, and speaker identification. In the remix task on the development set, the baseline model achieved an SI-SDR of $-4.069$~dB and an STOI of $0.388$ on Track~1, and an SI-SDR of $-2.851$~dB and an STOI of $0.470$ on Track~2. We release the AV-ConvTasNet checkpoints, the offline evaluator, and the official baseline results on the development and test sets.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#25
eess.AS

Array-Agnostic Ambisonics Encoding via Diffusion Posterior Sampling 解读失败跨领域

Amit Milstein, Nir Shlezinger, Boaz Rafaely
Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)
查看摘要
Spatial audio enhances user immersion by reproducing 3D sound fields, with Ambisonics being a widely adopted representation. While Ambisonics is theoretically independent of the recording setup, practical microphone arrays introduce hardware-dependent encoding artifacts. Moreover, existing data-driven solutions lack flexibility, as they are typically restricted to fixed array geometries. To overcome these limitations, we propose ADEPS, a generative framework that explicitly embeds the physical acquisition model into the inference process. By leveraging this formulation, ADEPS effectively compensates for array-specific distortions while enabling zero-shot encoding across arbitrary array topologies. We train the underlying generative prior in an unsupervised manner solely on target Ambisonic representations. Extensive evaluations across diverse simulated and real microphone arrays demonstrate that ADEPS consistently outperforms both traditional linear and parametric baselines in spatial fidelity and spectral quality.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#26
eess.AScs.SD

ToolDF: Tool-Integrated Reasoning for Mixed-Authenticity Audio Deepfake Detection 解读失败跨领域

Taewoo Kim, Young Han Lee, Nam In Park, Chanwoo Kim
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Sound (cs.SD)
Comments: To appear in Findings of the Association for Computational Linguistics: EMNLP 2026
查看摘要
Audio deepfake detection is commonly formulated as clip-level binary classification of single-domain audio. However, real-world manipulated audio can exhibit mixed authenticity, where genuine and manipulated cues coexist across temporal transitions, overlapping sources, or both. This setting requires not only detecting manipulated audio but also localizing the components that provide evidence for the decision. We propose ToolDF, a tool-integrated reasoning framework for mixed-authenticity audio deepfake detection. ToolDF employs an audio large language model as an orchestrator trained with supervised tool-use trajectories. It adaptively analyzes the audio scene, selectively performs source separation, routes components to domain-specific experts, and aggregates their evidence into an interpretable verdict. We further introduce a mixed-authenticity ADD benchmark covering temporal transitions, acoustic overlaps, and hybrid mixtures. Experimental results show that ToolDF achieves the best overall performance on composite-type detection, achieving macro-F1 gains of 3.72 and 14.39 points over the strongest monolithic baseline and a fixed pipeline, respectively, while providing interpretable evidence localized to temporal regions and acoustic sources. Our source code and dataset are publicly available online.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#27
eess.AScs.SD

Positive Sample Propagation along the Audio-Visual Event Line 解读失败跨领域

Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, Meng Wang
Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: Accepted to CVPR 2021. Code is available at this https URL
查看摘要
Visual and audio signals often coexist in natural environments, forming audio-visual events (AVEs). Given a video, we aim to localize video segments containing an AVE and identify its category. In order to learn discriminative features for a classifier, it is pivotal to identify the helpful (or positive) audio-visual segment pairs while filtering out the irrelevant ones, regardless whether they are synchronized or not. To this end, we propose a new positive sample propagation (PSP) module to discover and exploit the closely related audio-visual pairs by evaluating the relationship within every possible pair. It can be done by constructing an all-pair similarity map between each audio and visual segment, and only aggregating the features from the pairs with high similarity scores. To encourage the network to extract high correlated features for positive samples, a new audio-visual pair similarity loss is proposed. We also propose a new weighting branch to better exploit the temporal correlations in weakly supervised setting. We perform extensive experiments on the public AVE dataset and achieve new state-of-the-art accuracy in both fully and weakly supervised settings, thus verifying the effectiveness of our method.

📖 深度解读

[PDF 下载失败,无法解读]

#28
eess.AScs.SD

Efficient and Microphone-Fault-Tolerant 3D Sound Source Localization 解读失败跨领域

Yiyuan Yang, Shitong Xu, Niki Trigoni, Andrew Markham
Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
Comments: Accepted by Interspeech 2025 Conference
查看摘要
Sound source localization (SSL) is a critical technology for determining the position of sound sources in complex environments. However, existing methods face challenges such as high computational costs and precise calibration requirements, limiting their deployment in dynamic or resource-constrained environments. This paper introduces a novel 3D SSL framework, which uses sparse cross-attention, pretraining, and adaptive signal coherence metrics, to achieve accurate and computationally efficient localization with fewer input microphones. The framework also supports operational microphones at unknown positions: their recordings remain available and are used to estimate both source and microphone positions. Preliminary experiments demonstrate its scalability for multi-source localization without requiring additional hardware. This work advances SSL by balancing the model's performance and efficiency and improving its robustness for real-world scenarios.

📖 深度解读

[PDF 下载失败,无法解读]

#29
eess.AScs.SD

HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding 解读失败跨领域

Bohan Li, Shi Lian, Hankun Wang, Yiwei Guo, Yu Xi 等 (9 人)
Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
Comments: 14 pages, 2 figures, 8 tables; Accepted by EMNLP 2026 Main Conference
查看摘要
Unified speech foundation models require a holistic tokenization space that is both learnable by language models and decodable into high-quality waveforms. Existing speech tokenizers, however, often fail to satisfy these requirements simultaneously, leading to increased architectural complexity and more involved training designs. We propose HoliTok, a continuous Holistic speech Tokenization model designed for unified generation-understanding modeling. HoliTok encodes 48~kHz speech into a compact 25~Hz sequence of 128-dimensional latents. It is trained with a progressive strategy that jointly preserves signal-level fidelity, incorporates semantic information, and maintains strong latent learnability. Based on this tokenization, we build a unified AR+DiT model for speech synthesis and recognition, where the same latent sequence supports both generation-specific and unified generation-understanding tasks. Experiments show that HoliTok achieves competitive reconstruction fidelity, improves generative learnability for high-quality and controllable synthesis, and, among the evaluated representations, is the only one that operates robustly in our unified generation-understanding architecture without additional optimization tricks. These results suggest that HoliTok serves as an effective speech tokenizer and a foundational representation interface for unified spoken language modeling. The code is available at: this https URL .

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#30
eess.AScs.SD

Interleaved Speech Language Models Latently Work In Text 解读失败跨领域

Talia Sternberg, Gallil Maimon, Yossi Adi
Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Comments: Preprint. 23 pages, 20 figures, 5 tables
查看摘要
Speech language models (SLMs) increasingly combine speech and text, often by interleaving their tokens within a single sequence. Yet how these two modalities interact in the model's latent space remains unclear. In this work, we analyze interleaved speech--text LMs from different model families and training configurations using three complementary methods. We reveal that these models pass through an implicit latent transcription phase in which the text token matching the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. This phenomenon occurs in diverse, natural speech, and intermediate representations also encode likely text continuations. We further show that implicit transcription emerges most when combining text-LM pretraining and speech--text interleaving, and that its prevalence is positively associated with spoken factual-knowledge retrieval. Our analysis sheds light on the internal interaction between speech and text modalities in interleaved SLMs.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#31
eess.AS

X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction 解读失败跨领域

Kaiqi Fu, Rime Wen, Altman Lin, Shawn Qin, Roy Gan 等 (7 人)
Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
查看摘要
Accurate and responsive turn-taking is essential for spoken dialogue systems, which must distinguish in real time between user interruptions, backchannels that should be ignored, and the completion of an utterance. Prior modular approaches typically optimize turn state prediction at the utterance or fixed-chunk level, creating a mismatch with the continuous turn state estimate, and often depend on an auxiliary ASR model, which limits responsiveness and increases overall system complexity. Therefore, we present X2-Turn, a frame-synchronous turn state prediction method via delayed-stream modeling. Specifically, building on the pretrained Voxtral Realtime model, we introduce a frame-synchronous turn state head that operates in parallel with the ASR head on shared streaming representations, jointly predicting ASR tokens and fine-grained turn states at the frame level. Experiments on bilingual EasyTurn and Full-Duplex-Bench demonstrate that the proposed method achieves an effective trade-off between turn state accuracy and decision latency.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#32
eess.AS

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue 解读失败跨领域

Yen-Ju Lu, Yuzhe Wang, Yaohan Guan, Xiluo He, Jiarui Hai 等 (11 人)
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
Comments: 24 pages, 4 figures
查看摘要
Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#33
eess.AScs.SD

CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations 解读失败跨领域

Gabriel Meseguer-Brocal, Yuexuan Kong, Romain Hennequin
Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)
查看摘要
Joint-Embedding Predictive Architecture (JEPA) has shown strong performance in learning rich representations through self-supervised prediction in latent space. However, it typically relies on teacher--student architecture with an EMA to stabilise training, and can tend to yield uninformative representations. Contrastive learning is stable to train and produces strong global representations, but remains limited on local tasks by the global nature of its objective. In this work, we combine both into CoJEPA: a single shared backbone jointly trained with a JEPA objective on masked sequence tokens and a contrastive objective on the class token. The contrastive gradient provides stability, removing the need for an EMA teacher entirely, while JEPA enriches the sequence tokens via local predictions that contrastive learning alone cannot provide. Crucially, no extra parameters are added to the backbone: the same model is guided towards richer representations purely through the design of its training signal. CoJEPA takes the best of both worlds, outperforming or matching both individual methods across global and local MIR tasks, with a particularly strong advantage on tonal and harmonic understanding, and without any task-specific architectural changes. CoJEPA shows that combining objectives with complementary inductive biases can substitute for scale, encouraging future work to invest in smarter training objectives over ever-larger models.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#34
eess.AScs.SD

VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis 解读失败跨领域

Mengzhe Geng
Sound (cs.SD); Computation and Language (cs.CL); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
查看摘要
Expressive speech systems have to decide how an utterance is delivered before any waveform is rendered. In dialogue agents, narration, and role-conditioned TTS, that planning step sets affect, pitch, energy, rate, pause, emphasis, and stance, yet standard audio metrics rarely show whether those choices were actually licensed by the source record. This leaves a practical evaluation gap: a system may sound plausible while relying on a memorized script instead of the cue that governs delivery. VoxReason casts this pre-synthesis step as a listener-free task for source-grounded speech planning. Systems output a source-cited speaking plan, and a deterministic verifier checks citation legality, slot agreement, unsupported state, schema validity, and one-cue counterfactual locality. On 1,440 checked source-label cases, shortcut controls show why slot accuracy alone is unsafe: a key-lookup oracle reaches 1.000 plan-slot accuracy on seen keys, while an emotion prior still reaches 0.958 slot accuracy on source-key-disjoint cases without citing intensity or identity. In a separate 100-case learned source-key-disjoint comparison, a 7B locality SFT+CF repair improves plan-slot accuracy/locality from 0.684/0.141 to 0.919/1.000, and removing source records lowers citation-required grounded score by 0.488. The resulting benchmark isolates whether planned delivery is warranted by the record before waveform evaluation or listener studies are used.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#35
cs.SD

What Did I Just Say? Self-Listening for Full-Duplex Speech Models 解读失败

Xuanning Zhou, Junyi Ao, Xiaotong Liu, Tom Ko, Benyou Wang 等 (6 人)
Sound (cs.SD)
查看摘要
Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model's realized speech as anchor interruption. To address this problem, we propose Self-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken. AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped with self-listening mechanism achieve better anchoring performance.

📖 深度解读

[PDF 下载失败,无法解读]

#36
cs.SD

Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition 解读失败

Tingyi Lin, Wen-Ren Yang, Kuanwei Chen
Sound (cs.SD); Computation and Language (cs.CL)
Comments: 18 pages, 8 figures. Submitted to Speech Communication. Code: this https URL (archived: doi: https://doi.org/10.5281/zenodo.21904810 )
查看摘要
Human emotion is graded and frequently mixed, yet most multimodal recognizers collapse it onto a single hard label. We argue the recognizer should instead expose a distribution over affective space. Our text+speech system, alongside its categorical decision, emits a $9\times9$ probability matrix over the Valence-Arousal plane, trained with a two-dimensional Gaussian soft target under a Kullback-Leibler/cross-entropy objective, aimed at counseling support. Evaluation is strict: speaker-independent 5-fold leave-one-session-out IEMOCAP with rotating-session inner validation, headline metrics only on the held-out session. Within one fixed encoder-fusion-head pipeline we compare Transformer and state-space (Mamba-1/2/3) backbones at matched depth and width, at two operating points ($T\approx550$, $T\approx2750$). The featured dual-head system reaches 73.0% $\pm$ 0.3 unweighted accuracy over three seeds (separate rerun: 72.1%), exceeding the Transformer fusion baseline by 3.0 UA points (95% session-bootstrap CI [1.0,4.7]; significant under paired t-test and session-level bootstrap), with no latency or memory advantage at these lengths; swapping the ~1M trainable front-end for frozen WavLM-Large features (learnable layer weights) lifts the same architecture to 76.6% $\pm$ 1.3. Pre-specified controls scope the claims honestly: simpler valence-arousal auxiliaries reproduce the classification lift within noise, and a dedicated regression head tracks the continuous ratings slightly better, so the head's specific value is the normalized affect distribution itself. That distribution recovers the circumplex: its center of mass tracks valence and arousal (CCC 0.66/0.66; predominantly between-class structure, weaker within-class tracking), and its entropy is weakly but consistently linked to categorical rater ambiguity, not dimensional spread.

📖 深度解读

[PDF 下载失败,无法解读]

#37
cs.SD

Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs 解读失败

Song-ha Jo, Sehyun Lee, Soyoon Kim, Jaesik Choi, Sanghyuk Choi
Sound (cs.SD); Artificial Intelligence (cs.AI)
Comments: EMNLP2026 Findings
查看摘要
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.

📖 深度解读

[PDF 下载失败,无法解读]

#38
cs.SD

BinauralVAE: Spatial Audio Reconstruction For World Models 解读失败

Luis Vitor Zerkowski, Luiz Velho
Sound (cs.SD); Machine Learning (cs.LG)
Comments: 17 pages, 7 figures
查看摘要
Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline ( this https URL ) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.

📖 深度解读

[PDF 下载失败,无法解读]

#39
cs.SD

Frequency Estimation Based on SNR-adaptive Frequency Estimator Under Wide SNR Range 解读失败

Hee-Yang Jung, Dong-Hee Paek, Woo-Jin Jung, Seung-Hyun Kong
Sound (cs.SD); Artificial Intelligence (cs.AI)
Comments: 10 pages, 6 figures
查看摘要
Frequency estimation is the problem of estimating individual tone frequencies from noisy multi-tone sinusoidal signals. Existing frequency estimation methods have difficulty accurately estimating both the number of tone frequencies and the individual tone frequencies in low signal-to-noise ratio (SNR) environments, because weak tone frequency components are buried in noise. In addition, existing methods generally exhibit a trade-off between robustness at low SNR and frequency estimation precision at high SNR, making it difficult to achieve consistently superior frequency estimation performance over a wide SNR range. To overcome these limitations, this paper proposes an SNR-adaptive frequency estimator (SAFE). SAFE consists of a time-frequency image neural network (TFINet), which enhances weak tone frequency components at low SNR, and an SNR-based frequency selector (SFS), which selects an appropriate frequency estimator according to the SNR of the estimated tone frequencies. TFINet enhances tone frequency components even in the low-SNR range, while SFS estimates the SNR of each tone frequency and selects either a robust frequency estimator or a super-resolution frequency estimator according to the estimated SNR. This enables SAFE to achieve robustness at low SNR while preserving high precision at high SNR. Simulation results show that SAFE achieves an False Negative Rate (FNR) of 13.00% over the SNR range from -10 dB to 0 dB, corresponding to an 13.04% improvement over the state-of-the-art method. In addition, SAFE reduces the Nearest Neighbor-Root Mean Squared Error (NN-RMSE) by 56.67% compared with the state-of-the-art method, demonstrating that SAFE performs more accurate frequency estimation. Furthermore, experiments using real-world data demonstrate that SAFE provides robust frequency estimation performance even in practical environments with clutter.

📖 深度解读

[PDF 下载失败,无法解读]

#40
cs.SD

Comparing Self-Supervised and Domain-Invariant Features for Cross-Domain Voice Phishing Detection 解读失败

Jeongmin Lee, Seung Yun, Minkyu Lee, Ran Han, Yoonkyu Woo 等 (6 人)
Sound (cs.SD); Computation and Language (cs.CL)
Comments: 5 pages, 2 figures. Accepted at INTERSPEECH 2026
查看摘要
Voice phishing detection faces three critical challenges: real criminal recordings are unavailable due to privacy constraints; when available, only a handful of samples exist, insufficient for fine-tuning; and lightweight acoustic-only detection is needed as an alternative to large self-supervised models. We compare domain-invariant prosodic features and self-supervised representations (HuBERT, wav2vec2.0) through cross-domain evaluation-training on scenario-based actor recordings and testing on authentic criminal calls. Domain-invariant prosodic features achieve 69.5% F1 zero-shot and 71.0% with 5-shot learning. HuBERT achieves highest performance (94.2% F1, 5-shot), while wav2vec2.0 exhibits a precision-oriented detection profile (90.2% F1 with 99.4% precision, 5-shot). These findings reveal fundamental trade-offs: domain-invariant features enable zero-shot deployment when no real data exists, while SSL methods achieve higher performance but require real samples and compute.

📖 深度解读

[PDF 下载失败,无法解读]

#41
cs.SD

KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction 解读失败

Ryuichiro Higashinaka, Shinnosuke Takamichi, Tetsuji Ogawa
Sound (cs.SD); Computation and Language (cs.CL)
Comments: Accepted at APSIPA ASC 2026
查看摘要
Realizing full-duplex spoken dialogue requires large amounts of two-channel, one-speaker-per-channel conversational speech data. Although conversational text-to-speech (TTS) engines have been developed, they are not necessarily robust to two-party simultaneous phenomena such as backchannels, interruptions, and overlaps that occur while the interlocutor is speaking. In this work, aiming at conversational speech synthesis that reproduces human-like overlap, we propose KABURI-TTS. KABURI-TTS takes a per-speaker phoneme raster as input and renders the speech of the two speakers on separate channels, conditioned on the per-frame phonemes and the voice activity derived from them. Because the phoneme raster is supplied by a separate module, the proposed method enables controllable generation of one-speaker-per-channel, two-party spoken dialogue. A user evaluation shows that, compared with strong baselines, the proposed method attains higher naturalness at both the utterance and the interaction level. Furthermore, an analysis of voice activity confirms that the proposed method produces more overlap and more frequent turn-taking.

📖 深度解读

[PDF 下载失败,无法解读]

#42
cs.SD

Iterative Audio Separation with Mixture Consistency via MIMO Model Extension 解读失败

Yukara Ikemiya, WeiHsiang Liao, Yuki Mitsufuji
Sound (cs.SD); Machine Learning (cs.LG)
查看摘要
This paper proposes a general framework for stable and effective iterative audio separation with mixture consistency by extending source separation models to a multi-input multi-output (MIMO) configuration. In the field of audio separation, mixture consistency is an essential property for many applications that require accurate phase and timbral information of target sources. While iterative approaches such as diffusion models achieve perceptually superior results in speech enhancement or user-guided target source separation tasks, most existing methods focus on single-step separation with a single-input single-output (SISO) or single-input multi-output (SIMO) configuration through architectural improvements, since mixture-consistent audio separation is generally regarded as a regression problem that admits a unique solution. By extending these architectures to a MIMO configuration, we introduce iterative prediction without compromising the architectural advantages or the characteristics of mixture consistency. We conduct a comprehensive ablation study of combining the framework with discriminators and extending it to a generative model. Experimental results demonstrate significant performance improvements when applying the proposed framework to state-of-the-art separation models.

📖 深度解读

[PDF 下载失败,无法解读]

#43
cs.SD

TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models 解读失败

Heyu Chang, Nianwen Si, Hao Zhang, Wenlin Zhang, Dan Qu
Sound (cs.SD); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
Comments: Accepted to Interspeech 2026
查看摘要
Large audio-language models (LALMs) can hallucinate audio objects, answering "yes" to absent sound events, thus undermining reliability in audio question answering. We propose Token-Adaptive Decoding (TAD), a training-free strategy for hallucination mitigation that grounds the initial yes/no decision by contrasting logits under real audio with a matched silent reference. TAD introduces a token-adaptive, confidence-guided gate that is decision-critical at the first decoding step and class-conditional on affirmative tokens, using the audio-silent margin to avoid overcorrection when evidence is weak or already sufficient. Experiments on AudioCaps-Hallucination show that, relative to Audio-Aware Decoding (AAD), a contrastive baseline with fixed contrast strength, TAD improves F1 for Qwen2 by 0.059 to 0.117 across Popular, Adversarial, and Random splits, and for Gemma by 0.025 to 0.064, while on Clotho-AQA it raises F1 from 0.810 to 0.816 on Qwen2 and remains comparable to AAD on Gemma.

📖 深度解读

[PDF 下载失败,无法解读]

#44
cs.SD

Silent Metronome: Rhythmic Grounding for Live Music Accompaniment 解读失败

Kevin Bretz, Derya Soydaner, Aske Plaat
Sound (cs.SD)
Comments: 5 pages, 2 figures, 1 table
查看摘要
Live accompaniment models generate music for an incoming audio stream, committing to each output frame before hearing what comes next. In this strictly causal setting the model must infer tempo, meter, and metrical phase from its own imperfect past, whereby compounding errors quickly become audible as rhythmic drift. Put simply, the model has ears but no temporal reference, so when the ears hear imperfect, ambiguous music, the model will produce a flawed output. We propose Silent Metronome (SiMe), which gives it the temporal reference, encoding the phase within the beat and within the bar as periodic functions, pairing them with tempo and time signature, and supplying the result as a separate conditioning channel. Because this reference is independent of the generated audio, it cannot drift. Complementary auxiliary heads shape the latent representation, including a novel head that predicts the model's own future tokens. With the metrical signal taken from ground-truth annotations, beat alignment improves by a factor of 3.2 over the strictly causal baseline and surpasses a non-causal reference granted a full second of look-ahead. Coherence between input and accompaniment stays within a single point of that reference. These results suggest that streaming accompaniment systems should treat rhythm as a signal to be shared, as human ensembles do, rather than inferred.

📖 深度解读

[PDF 下载失败,无法解读]

#45
cs.SD

Clean Accuracy Does Not Guarantee Provenance Robustness: A Prospective Codec-Stress Evaluation of Audio Attribution 解读失败

Gang Shi
Sound (cs.SD); Machine Learning (cs.LG)
Comments: 13 pages, 5 figures
查看摘要
Audio provenance attribution - which system produced a synthetic utterance - is reported at near-ceiling accuracy on clean benchmarks, yet audio reaching an analyst has usually been transcoded. We report a prospectively registered measurement of closed-set attribution after single-stage codec transport, with the analysis region fixed from fidelity metadata before any attribution model was trained. On two corpora, in-support losses reach 53.5 [43.5, 63.6] and 70.3 [63.0, 77.5] Macro-F1 points for WavLM-Base+, and 61.0 [56.8, 65.1] and 49.8 [41.6, 57.9] for W2V2-BERT 2.0, under simultaneous component-level bands. Degradation is strongly condition- and representation-dependent: within one in-support grid WavLM losses run from -0.4 to +53.5 points, and the two encoders differ beyond a prespecified +/-5-point margin at six of twelve conditions. A clean-qualified ECAPA-TDNN and a Proxy-Anchor head degrade comparably, so the effect is not confined to one representation family or a weak linear head. The registered matched-fidelity comparison was not estimable on this grid, and waveform and perceptual measures order the conditions differently: MP3 at 8 kbit/s ranks mid-grid on SI-SDR but last on PESQ-WB while causing the largest loss. For the tested tasks, corpora, representations and codec grid, a clean accuracy figure does not by itself characterise deployment robustness.

📖 深度解读

[PDF 下载失败,无法解读]

#46
cs.SD
University of Oxford (QS Top 100)

Geometry-Informed Distributed Acoustic Scene Understanding

Yiyuan Yang, Shitong Xu, Niki Trigoni, Andrew Markham
Sound (cs.SD)
Comments: Accepted by Interspeech 2026 Conference
查看摘要
Acoustic scene understanding in multi-room environments is a difficult task. Most existing systems use a single centralized microphone array, and they often fail because walls and doors block sound signals. To address this challenge, we propose a geometry-informed distributed acoustic scene understanding framework. Our system leverages distributed microphones and uses an audio spectrogram transformer and a topology-aware graph neural network to fuse spatio-temporal acoustic features. Then, these features are decoded into discrete semantic triplets. Finally, a frozen large language model combines these symbolic observations with the environmental geometry. This allows the system to perform spatial understanding, infer plausible missing transitions, and generate a physically consistent narrative of the scene. Experiments on a custom multi-room simulator demonstrate that our framework outperforms centralized baselines and improves spatial consistency under simulated occlusion.

📖 深度解读

论文解读报告

1. 一句话总结

这篇论文提出了一套“几何信息引导的分布式声学场景理解框架”,通过在多房间环境中部署分布式麦克风,结合图神经网络融合空间声学特征,并利用大语言模型根据房间布局的物理约束来“脑补”被墙壁遮挡的声音事件,从而生成逻辑连贯、物理合理的室内活动叙述。

2. 研究背景与动机

核心问题

在多房间室内环境中,如何利用分布式麦克风网络实现完整的声学场景理解——不仅要识别孤立的声音事件,还要生成一个符合物理空间逻辑的、连贯的“故事叙述”(例如:谁在什么时间从哪个房间走到了哪个房间)。

为什么重要

  • 声学感知相比视觉感知具有隐私保护优势,且能在黑暗中工作,适合老年看护、室内安防等长期监控场景。
  • 实际应用需要系统输出连贯的场景描述而非零散的声音标签,例如在老人跌倒时不仅检测到“跌倒声”,还要理解整个事件的空间演变过程。

现有方法的不足

  1. 集中式感知的物理局限:大多数现有系统假设单一收听位置,但墙壁和门会阻挡声音,导致非视距区域完全“听不到”,产生碎片化的场景理解。
  2. 忽视环境拓扑结构:分布式麦克风方法虽然扩大了覆盖范围,但通常将每个检测到的声音视为孤立数据点,没有利用房间之间的物理连通性来判断哪个传感器的信息更可靠。
  3. 缺乏逻辑接地:真实物理环境中人的活动必须遵循空间规则(如不能从卧室直接瞬移到厨房而不经过走廊),现有系统在声学信号缺失时无法推断合理的中间过渡,输出物理上不可能的跳跃。

3. 核心方法

方法框架总览

论文提出一个三阶段流水线:分布式声学特征提取与融合 → 语义三元组符号化 → LLM叙述生成。

关键创新点

创新点1:几何信息引导的拓扑图构建
将多房间环境建模为拓扑图,节点是麦克风,边编码房间连通性和声学传播特性。边权重根据物理距离和墙壁衰减系数计算,使图结构能反映非视距特性和跨房间的声音泄漏。这相当于给系统一张“建筑地图”,让它知道哪些房间相邻、声音穿过墙壁会衰减多少。

创新点2:空间-时间图融合架构
采用分层处理:先用共享的Audio Spectrogram Transformer(AST)对每个麦克风的音频编码,再通过图神经网络进行空间消息传递(让相邻房间的麦克风信息相互补充),最后用GRU建模时间动态。这解决了“单个麦克风听不全,多个麦克风怎么融合”的问题——通过图结构让系统知道该信任哪个传感器。

创新点3:语义三元组符号化(动态声学场景图)
将连续声学特征转化为离散的语义三元组⟨主体,关系,对象⟩(如⟨Person, walking, Corridor⟩),作为连接感知和推理的“语义瓶颈”。采用查询式Transformer解码器,并引入层次化条件预测(先预测主体,再基于主体预测关系,最后基于两者预测对象),防止逻辑上不可能的配对。

创新点4:几何感知的LLM零样本推理
使用冻结的Llama-3-8B模型,将环境几何(房间邻接关系)和检测到的语义三元组线性化为提示词,让LLM根据物理约束“脑补”缺失的过渡事件。例如,如果检测到客厅有脚步声、走廊有开门声,LLM能推断出人从客厅走到了走廊并打开了门,即使中间某些声音被遮挡了。

核心思路的直觉解释

可以把这套系统想象成一个“分布式听觉网络+会推理的叙述者”:多个麦克风像分布在各个房间的“耳朵”,图神经网络像“大脑皮层”负责整合不同耳朵听到的信息(知道哪个耳朵更可信),语义三元组像“速记笔记”把关键事件提炼成结构化符号,最后大语言模型像一个“了解建筑布局的故事讲述者”,根据速记笔记和建筑地图,把碎片化的事件串成一个逻辑通顺的完整故事,自动填补那些没被听到但物理上必然发生的过渡环节。

4. 实验与结果

数据集与基准

  • 数据集:基于pyroomacoustics构建的自定义多房间声学模拟器,模拟2-4个房间,混响时间0.2-0.6秒,6个分布式麦克风节点。声学事件来自LibriSpeech(语音)和ESC-50(环境音),信噪比20-40dB。
  • 基线对比:
  • Centralized-Single:单一阵列集中式感知
  • Distributed Acoustic-Only:分布式感知但无几何信息
  • Template-Based:将三元组直接映射为文本,无LLM推理

主要实验结果

系统变体 Triplet F1 BLEU-4 ROUGE-L BERTScore SCS(%)
Centralized-Single 0.51 0.15 0.22 0.45 35.0
Distributed Acoustic-Only 0.74 0.41 0.47 0.65 61.8
Template-Based 0.86 0.39 0.46 0.61 79.5
Ours(完整框架) 0.87 0.55 0.62 0.77 88.2

关键发现:
- 分布式感知相比集中式大幅提升检测精度(F1从0.51→0.74),说明空间多样性对克服遮挡至关重要。
- 完整框架相比无几何信息的分布式基线,空间一致性从61.8%提升到88.2%,说明几何信息对物理合理性贡献显著。
- 相比Template-Based方法,完整框架在语言质量指标上大幅领先(BLEU-4从0.39→0.55),证明LLM推理不仅提升了叙述流畅度,还帮助填补了检测间隙。

消融实验揭示的规律

  • 去掉几何先验Ω:SCS从88.2%骤降至66.5%,是所有消融中影响最大的——说明仅靠声学证据无法保证空间逻辑正确性。
  • 去掉图融合:Triplet F1从0.87降至0.81,SCS降至80.9%——说明空间消息传递对构建全局一致表征至关重要。

5. 优势与局限

主要优势

  1. 系统性解决多房间声学遮挡问题:通过分布式感知+拓扑图融合+几何感知推理的完整链路,在模拟环境中显著优于集中式基线。
  2. 物理约束与LLM推理的有效结合:将环境几何作为提示词注入LLM,让模型在生成叙述时自动遵循空间逻辑,填补声学检测的空白,这是纯数据驱动方法难以实现的。
  3. 模块化设计:感知、符号化和推理各阶段解耦,便于独立改进和替换组件(如更换更强的音频编码器或LLM)。

局限性

  1. 模拟到现实的鸿沟:实验完全基于仿真,假设已知房间布局、麦克风位置和门状态。真实环境存在设备失配、时钟不同步、家具变动、非平稳噪声等问题。
  2. 场景复杂度有限:数据相对结构化,未覆盖同一房间内多个声源重叠的“鸡尾酒会”场景;且错误的三元组可能让LLM自信地生成错误叙述,缺乏不确定性传播机制。
  3. 静态拓扑假设:框架假设环境几何Ω是已知且静态的,无法处理动态门或临时隔断等拓扑变化。

6. 关键结论与启发

最重要的Takeaway

在多房间声学场景理解中,“在哪里听”和“知道房间怎么连”和“听到什么”同样重要。 分布式感知解决了覆盖问题,但只有将环境几何显式注入感知和推理的各个环节,系统才能输出物理上可信的场景叙述。几何先验的消融实验(SCS从88.2%降至66.5%)是最有力的证据。

对后续研究的启发与延伸方向

  1. 不确定性感知的推理:论文指出错误三元组可能导致LLM“自信地胡说”,未来可研究如何传播预测不确定性,让LLM区分“直接检测到的事件”和“推断的过渡事件”。
  2. 动态拓扑适应:从静态已知几何扩展到动态环境(门开关、临时隔断),需要在线更新拓扑图和传播先验。
  3. 真实世界验证:从仿真走向真实多房间测试平台,解决设备失配、时钟同步等工程挑战。
  4. 更复杂的声学场景:探索密集多声源重叠场景下的分布式感知与推理,这可能需要在符号化阶段引入更强大的分离能力。

注:本论文为arXiv预印本(arXiv:2609.08026v1),标注日期为2026年9月,实验全部基于模拟器进行,尚未经过真实环境验证。论文声称的“有效性”应理解为“在受控模拟条件下的有效性”。

#47
cs.SD
Beijing University of Posts and Telecommunications (211)

Stabilizing Instruction Supervision for Instruct-TTS via Controllable Diversification and Drift Filtering

Yizhong Geng, Kecan Mao, Qifei Li, Cong Wang, Yingming Gao 等 (9 人)
Sound (cs.SD)
Comments: 5 pages, 2 figures, 5 tables. Audio demos: this https URL
查看摘要
Instruct-TTS systems expand structured style labels into natural-language training instructions through LLM rewriting, yet we find that over 40% of unconstrained rewrites contain semantic drift that corrupts supervision and weakens generalization. We formalize this problem as instruction supervision instability and propose a data-centric stabilization recipe that jointly improves coverage and fidelity through three mechanisms: controllable instruction diversification for systematic expansion, LLM-based drift filtering for quality control, and attribute-aligned supervision that grounds prosody control in acoustic perturbations. On the Chinese split of InstructTTSEval, our recipe raises instruction-following from 34.5% without fine-tuning and 51.0% with naive fine-tuning to 56.4%, while constrained rewriting reduces drift from 40.4% to 15.4%. Ablations confirm the three mechanisms are complementary, and the drift taxonomy may generalize to instruction-driven generation beyond TTS.

📖 深度解读

1. 一句话总结

这篇论文发现,在 Instruct-TTS(用自然语言指令控制语音合成)系统中,用大语言模型把结构化标签改写成训练指令时,超过 40% 的改写会产生“语义漂移”(即指令含义被悄悄改变),从而污染训练数据;作者提出了一套“数据为中心”的稳定化方案(可控多样化 + 漂移过滤 + 属性对齐监督),将指令遵循准确率从 51.0% 提升到 56.4%。

2. 研究背景与动机

核心问题: Instruct-TTS 系统依赖“指令-语音”配对数据进行训练,但这些数据难以大规模人工收集。常见的替代方案是用 LLM 将结构化标签(如情感、口音、风格)改写成自然语言指令,然而这一过程存在两个耦合的缺陷:覆盖不足(生成指令的措辞和视角单一)和语义漂移(改写后的指令悄悄改变了原本的控制含义)。作者将这两个问题统一为“指令监督不稳定性”。

为什么重要: Instruct-TTS 是语音合成从“固定标签控制”走向“自由文本控制”的关键技术方向,广泛应用于有声书、助手、内容创作等场景。如果训练数据本身不可靠,模型学到的“指令跟随”能力就是虚假的——这解释了为什么在噪声数据上朴素微调的效果甚至可能不如不微调。

现有方法的不足:
- NLP 领域的 Self-Instruct、Evol-Instruct 等方法假设 LLM 输出质量可以通过标准提示词管理,但 TTS 场景中 LLM 改写存在领域特有的漂移模式(如把“指令”写成“执行台词”)。
- 现有 Instruct-TTS 工作(PromptTTS、VoxInstruct 等)主要关注模型架构和训练目标,把上游指令数据当作“给定的”,没有系统性地审视数据质量。
- 单纯扩大多样性不保证保真度,单纯过滤不扩大覆盖度——两者必须联合解决,但此前缺乏这样的系统性方案。

3. 核心方法

论文提出一个数据为中心的稳定化配方(data-centric stabilization recipe),包含三个互补机制:

① 可控指令多样化(Controllable Instruction Diversification)
- 核心思路:不是让 LLM 自由发挥,而是通过三个硬约束来系统化地生成语义等价的指令变体:
- 人设(persona):指定说话者视角(如“父亲”“北京主持人”),多样化语用意图;
- 句法模式(syntactic pattern):约束表面形式,避免模板化;
- 属性槽(attribute slots):要求所有目标属性必须出现,防止遗漏导致的实体/标签漂移。
- 直觉类比:这就像让不同性格的老师用不同方式说同一句话,但要求每个老师都必须把关键词说到位。

② 漂移过滤(Drift Filtering)
- 核心思路:即使约束改写将漂移率从 40.4% 降到 15.4%,残余漂移仍然有害。因此引入一个与生成器不同模型家族的 LLM 验证器(GPT-4o 验证 DeepSeek-R1 生成的指令),基于三类漂移分类法进行评分和投票过滤。
- 三类漂移:指令转执行(把“请用北京口音叙述”写成“嘿各位,让我告诉你们……”)、角色代入(以角色身份说话而非发出指令)、实体/标签漂移(悄悄改变种子属性)。
- 直觉类比:这就像请一位与出题老师不同的阅卷老师来批改试卷,只有得分高且多次批改都通过的题目才被保留。

③ 属性对齐监督(Attribute-Aligned Supervision)
- 核心思路:标签到指令的改写对低层韵律控制(音高、语速、音量)覆盖不可靠,因此直接对语音片段施加参数化的声学扰动(如音高 ±1/2/3 半音、语速 ×0.8~1.3、音量 ±3/6/9 dB),并将扰动设置映射为自然语言属性提示,形成高保真的(扰动语音,属性提示)配对。
- 直觉类比:这就像用“物理实验”来教学生理解“大声”和“小声”的区别——不是靠文字描述,而是直接调节音量旋钮让学生听到差异。

4. 实验与结果

数据集/基准:
- 训练数据:约 90 小时中文语音(12k 片段,每段 <30 秒),覆盖 8 种口音和 10 种说话风格;每片段生成 17 个扰动变体。
- 评估基准:InstructTTSEval(中文分集),包含三个任务——属性控制发音与风格(APS)、对话场景描述(DSD)、角色扮演(RP)。
- 评估方式:Gemini 3 Pro 作为自动裁判(听音频+读指令判断是否遵循),另有 20 名母语者进行人工评分。

基线方法:
- VoxInstruct(外部基线,官方 checkpoint)
- Base(无 SFT,即不微调的 CosyVoice 2 骨干)
- Naive SFT(在未过滤的 LLM 改写数据上朴素微调)

主要实验结果:

设置 平均指令遵循准确率 NMOS CMOS
VoxInstruct 47.5% 3.18 3.12
Base(无 SFT) 34.5% 3.30 3.17
Naive SFT 51.0% 3.46 3.22
Full Recipe 56.4% 4.16 4.16
  • 完整配方比朴素微调提升 5.4 个百分点,人工自然度评分提升 0.7 以上。
  • 约束改写将漂移率从 40.4% 降至 15.4%(三类漂移均约减半)。

消融实验揭示:
- 漂移过滤贡献最大:移除后平均准确率从 56.4% 降至 48.9%(尽管训练集更大),说明增益来自去除漂移样本而非数据量增加。
- 可控多样化贡献次之(移除后降至 51.8%),说明系统化覆盖扩展超出过滤单独能提供的价值。
- 属性对齐监督对平均值影响中等(53.1%),但对 APS 任务影响显著(49.5%→42.7%),印证其对低层属性控制的针对性作用。
- 三个机制互补:过滤保保真度、多样化扩覆盖、属性对齐强化韵律控制。

过滤策略对比: 阈值+投票联合策略效果最好(DSD 65.4%),但数据保留率最低(61.5%),体现了覆盖-保真度的内在权衡。

5. 优势与局限

主要优势:
1. 问题定义清晰:首次将 Instruct-TTS 中的指令监督不稳定形式化为三类漂移分类法,为后续研究提供了可复用的分析框架。
2. 数据为中心的方案具有通用性:不改变模型架构,纯靠数据质量提升获得显著增益,适用于任何 Instruct-TTS 骨干模型。
3. 多模型家族交叉验证:生成(DeepSeek-R1)、验证(GPT-4o)、评估(Gemini)使用不同模型家族,降低了单一 LLM 的系统性偏差。

局限性:
1. 依赖多个商业 LLM API:DeepSeek-R1、GPT-4o、Gemini 的使用意味着成本较高且存在 API 不可用风险,实际部署门槛不低。
2. 仅在中文单一语言上验证:实验局限于 InstructTTSEval 中文分集,漂移分类法是否适用于英文或其他语言尚待验证。
3. 过滤导致数据量显著减少:联合过滤策略仅保留 61.5% 的候选数据,在数据稀缺场景下可能不适用;且过滤阈值需要人工调优。

6. 关键结论与启发

最重要的 takeaway: Instruct-TTS 的指令跟随能力瓶颈主要不在模型架构,而在训练数据的“覆盖度”和“保真度”必须同时提升——单靠扩大数据量(朴素微调)或单靠过滤都不够,需要系统性的数据工程方案。论文用清晰的消融实验证明了这一点:过滤是最大贡献者,但多样化提供了过滤无法替代的覆盖增益。

对后续研究的启发与延伸方向:
1. 漂移分类法可迁移:论文提出的三类漂移(指令转执行、角色代入、实体/标签漂移)可能不仅适用于 TTS,也可能适用于其他指令驱动的生成任务(如文生图、文生视频),值得跨领域验证。
2. 数据质量评估自动化:当前依赖 LLM 验证器进行过滤,未来可以探索更轻量级的自动评估方法(如小模型蒸馏大模型的判断能力)。
3. 覆盖-保真度权衡的可调性:论文展示了阈值和投票强度可以调节,这为不同数据规模和应用场景提供了灵活的旋钮,值得进一步研究如何自适应地确定最优过滤强度。
4. 属性对齐监督的扩展:目前只覆盖了音高、语速、音量三个维度,未来可以扩展到更细粒度的声学属性(如呼吸声、颤音、共鸣等),进一步缩小自然语言控制与声学操作之间的鸿沟。

#48
cs.SD
KAIST - Korea Advanced Institute of Science & Technology (QS Top 100)

Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models

Bella Godiva, Yeonju Kim, Yong Man Ro
Sound (cs.SD); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
Comments: Accepted to EMNLP 2026
查看摘要
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.

📖 深度解读

1. 一句话总结

这篇论文提出了一个名为 AV-STE 的即插即用模块,它利用说话者的唇部视频来修复被噪声破坏的语音语义令牌,从而在不重新训练底层对话模型的前提下,显著提升全双工语音对话系统在嘈杂环境下的响应连贯性。

2. 研究背景与动机

  • 核心问题:全双工语音对话系统(如 Moshi)在真实世界中,当用户语音被背景噪声或他人说话声干扰时,其音频感知能力会严重下降,导致系统产生不连贯或答非所问的回应。
  • 问题重要性:全双工对话是实现自然、实时人机交互的关键,允许打断和插话等行为。然而,这种交互必须发生在真实环境中,而真实环境充满噪声。如果系统在噪声下无法理解用户,其实用性将大打折扣。
  • 现有方法的不足:
    • 纯音频方法:当前主流全双工模型仅依赖音频,在噪声下性能脆弱。
    • 音频-视觉语音识别(AVSR)方法:虽然能利用视觉信息提升鲁棒性,但 AVSR 输出的是文本,需要额外的文本到语音令牌的转换阶段,增加了延迟和复杂性,不适合直接的语音到语音对话。
    • 端到端音视觉对话模型(如 AV-Dialog):虽然直接将视觉信息融入对话模型,但需要对大型语音对话模型进行昂贵的多模态重新训练,且可能损害其原有的对话能力。

3. 核心方法

论文提出的 AV-STE(Audio-Visual Speech Token Enhancer) 是一个模块化的流式前端,核心思路是“在问题发生前解决它”。它不修改对话模型,而是在嘈杂的语音令牌进入对话模型(如 Moshi)之前,先利用视觉信息将其“清洗干净”。

  • 方法框架:
    1. 流式音视觉编码器:从带噪语音和同步的唇部视频中提取鲁棒的视觉语音特征。该编码器基于 AV-HuBERT 改造,通过因果卷积和有限前瞻掩码实现低延迟流式处理。
    2. 噪声自适应融合模块:这是核心创新点。它将“视觉特征”与“带噪的语音令牌”智能地融合起来。
  • 关键创新点:
    1. 冻结下游模型,即插即用:AV-STE 作为一个独立的前端模块,可以无缝集成到任何现有的、使用语义令牌的语音对话模型中,而无需对该模型进行任何训练,保留了其预训练的对话能力。
    2. 软令牌交叉注意力:不是直接使用可能出错的“硬”语音令牌(即 argmax 后的结果),而是使用包含不确定性信息的“软”令牌分布(对所有可能令牌的概率加权嵌入)进行交叉注意力计算,让视觉特征能更鲁棒地从嘈杂的语音表征中提取信息。
    3. 噪声自适应调制(NAM):设计了一个门控机制,能根据当前帧语音令牌的可靠性(通过熵衡量)动态调整融合权重。当音频清晰时,模型更信任音频令牌;当音频嘈杂时,模型则更依赖视觉特征。这避免了在清晰环境下视觉信息(可能产生唇音歧义)干扰音频判断。
    4. 流式低延迟设计:通过因果卷积和有限前瞻(160ms)的注意力掩码,确保整个增强过程可以实时流式进行,满足全双工对话的低延迟要求。

4. 实验与结果

  • 数据集:
    • 训练/域内评估:LRS3(音视觉语音数据集),使用 AudioSet 添加噪声和干扰语音。
    • 域外评估:Seamless Interaction(双人对话视频语料库),模型从未见过该数据。
  • 基线方法:
    • Moshi:冻结的全双工语音对话模型(主基线)。
    • PersonaPlex:另一个全双工对话模型。
    • SE (PASE) + Moshi:一个非流式的、纯音频的生成式语音增强模型,作为参考基线。
  • 主要实验结果:
    • 语义令牌恢复:在 LRS3 域内,面对非语音噪声和说话人干扰时,AV-STE 将平均语义令牌准确率从约 27% 大幅提升至约 68%。在更具挑战性的同数据集多说话人干扰下,准确率从 12.61% 提升至 65.60%。
    • 对话响应连贯性:在将 AV-STE 与冻结的 Moshi 集成后,在-10dB 极端噪声下,GPT-4o 评判的响应连贯性得分从 1.25 提升至 2.00(非语音噪声),从 1.22 提升至 1.94(说话人干扰)。
    • 同数据集干扰对比:在最具挑战性的 LRS3 同数据集说话人干扰下,AV-STE+Moshi 的平均连贯性得分(1.91)显著优于 Moshi(1.42)、PersonaPlex(1.17)和 SE(PASE)+Moshi(1.67)。
    • 域外泛化:在 Seamless Interaction 数据集上,AV-STE 依然能提升响应连贯性(如非语音噪声下从 1.58 提升至 2.10),证明了其泛化能力。
  • 消融实验揭示:
    • 视觉信息有效:音频-视觉(A+V)输入显著优于纯音频(A)输入,证实了视觉唇语信息的价值。
    • 软令牌优于硬令牌:使用软令牌交叉注意力(Soft-CA)比硬令牌(Hard-CA)效果更好。
    • 噪声自适应调制有效:在交叉注意力基础上加入 NAM 模块能进一步提升性能。
    • 有限前瞻的平衡:模型性能在 4 帧(160ms)前瞻时达到峰值,之后提升有限,表明这是准确率和延迟的最佳平衡点。

5. 优势与局限

  • 主要优势:
    1. 即插即用,成本低:无需重新训练昂贵的对话大模型,可以轻松集成到现有系统中,实用性强。
    2. 显著提升噪声鲁棒性:在多种噪声和干扰场景下,尤其是在极低信噪比下,都能大幅提升语义理解和响应连贯性。
    3. 保持对话行为:在提升连贯性的同时,没有破坏原模型(如 Moshi)的轮次转换(turn-taking)行为(TOR 和延迟基本不变)。
  • 局限性:
    1. 依赖视觉输入质量:论文假设视觉输入是可靠的,未探索在摄像头遮挡、画面模糊或目标说话人不在画面内等情况下的性能。
    2. 域偏移敏感性:在域外数据集(Seamless Interaction)上,虽然噪声下性能有提升,但干净语音下的令牌准确率从 76.78% 降至 43.79%,表明模型对数据分布变化敏感。
    3. 干净语音下的表征失配:AV-STE 在干净语音下预测的语义令牌与原生 Mimi 令牌不完全一致,可能引入微小的表征偏差。

6. 关键结论与启发

  • 最重要的 Takeaway:视觉唇语信息可以作为强大的辅助信号,在令牌级别修复被噪声破坏的语音语义,而无需触及庞大的对话模型本身。这为提升全双工语音对话系统的鲁棒性提供了一条高效、模块化的新路径。
  • 对后续研究的启发与延伸方向:
    1. 鲁棒性扩展:未来工作可以探索如何让 AV-STE 对视觉噪声(如遮挡、模糊)也具有鲁棒性,例如通过视觉-音频联合降噪。
    2. 跨域泛化改进:使用更多样化的音视觉对话数据进行训练,以增强模型对域偏移的适应能力,是提升其实用性的关键。
    3. 更广泛的模型适配:虽然论文验证了 Mimi/Moshi 组合,但其“语义令牌增强”的理念可以推广到其他采用语义令牌表征的语音模型中,具有广阔的适用前景。
#49
cs.SD

X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR

Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin 等 (8 人)
Sound (cs.SD); Artificial Intelligence (cs.AI)
查看摘要
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across AISHELL-1/2/3 and WenetSpeech, X2Streaming-ASR achieves a mean character-level commit latency of 27-84 ms relative to forced-aligned character endpoints, compared with 409-585 ms for the evaluated streaming baselines. It achieves the best streaming CER among the evaluated systems on AISHELL-1 and AISHELL-3 with substantially lower latency.

📖 深度解读

1. 一句话总结

这篇论文提出了一种名为 X2Streaming-ASR 的流式语音识别方法,它让模型自己决定“何时该听、何时该说”,从而在保证识别准确率的同时,将每个字的输出延迟从几百毫秒大幅降低到几十毫秒。

2. 研究背景与动机

  • 核心问题:流式语音识别(Streaming ASR)需要在说话人讲话的同时实时输出文字,其核心挑战是如何在“等待更多音频以提升准确率”和“尽快输出以降低延迟”之间做出最优决策。现有方法无法根据每个字的具体情况灵活调整等待时间。
  • 重要性:流式ASR是实时字幕、语音助手和全双工对话系统的基础。低延迟和高准确率直接决定了下游服务(如大模型对话)能否尽早且正确地启动,影响用户体验。
  • 现有方法的不足:
    • 固定配置类(如固定chunk、固定look-ahead):对所有位置一视同仁,无法根据上下文和声学信息动态调整等待时长。
    • 训练目标类(如FastEmit、minLT):要么一味鼓励早输出,要么强制在声学边界输出,都忽略了某些“难字”需要更多未来上下文才能正确识别这一事实。
    • 修订类(如Uni-ASR):允许先输出再修正,不是单次硬提交(hard-commit),不适合需要严格一次性输出的场景。

3. 核心方法

  • 方法/框架:X2Streaming-ASR 将流式识别分解为两个决策:何时提交(when to commit) 和 提交什么(what to commit)。模型在“听”(编码音频)和“说”(解码文本)两种状态间交替,由语言模型在每个音频帧上做出二元决策:w(等待,继续听)或 e(触发解码,输出文本直到结束符)。
  • 关键创新点:
    1. 位置相关的自适应提交策略:用强化学习直接优化“何时提交”的决策,取代了全局固定的延迟参数,让模型学会在不确定时多等,在确定时立即输出。
    2. 三阶段训练框架:
      • 阶段一:用监督学习训练一个基础的流式识别模型。
      • 阶段二:用阶段一的模型自动探测每个字的合理提交时间,作为“预热”标签,训练初始提交策略。
      • 阶段三:使用GRPO(组相对策略优化)直接优化提交策略,奖励函数结合了识别错误率和延迟,并采用字符级、分段分配的信用分配机制,精准强化或抑制特定的等待/提交行为。
    3. 字符级分段信用分配:不同于将整句奖励广播给所有决策,该方法将同一参考字符的不同轨迹进行组内比较,并将优势值(advantage)回传给产生该字符的特定“提交段”(一个e及其前面的连续w),从而避免模型学到“全程无脑等待”的次优策略。
  • 核心思路直觉:可以把模型想象成一个“边听边记”的速记员。传统方法是规定他每隔固定时间(如80ms)必须抬头看一眼,或者必须等一个词说完才能记。X2Streaming-ASR 则训练这个速记员自己判断:听到熟悉的词(如“上海”)就立刻记下;听到模糊的音(如“天气”中的“天”),如果后面还有声音能帮助确认,他就多听一会儿再记,而不是盲目地早记或晚记。

4. 实验与结果

  • 数据集:训练集包括 AISHELL-1/2/3、AliMeeting 和 WenetSpeech。测试集为 AISHELL-1/2/3 的测试集以及 WenetSpeech 的 meeting 和 net 测试集。
  • 基线方法:对比了开源的 Zipformer 和 Paraformer-online,以及论文中引用的 Uni-ASR 和 MoCha-ASR(后两者无开源代码,仅引用其论文数据)。
  • 主要实验结果:
    • 延迟:X2Streaming-ASR 在所有测试集上的平均字符级延迟仅为 27–84ms,而 Zipformer 和 Paraformer 的延迟在 409–585ms 之间,延迟降低了 79–95%。P95 延迟也远低于基线。
    • 准确率:在 AISHELL-1 和 AISHELL-3 测试集上,X2Streaming-ASR 取得了所有流式系统中最优的 CER(1.96% 和 2.68%),同时延迟最低。在 AISHELL-2 和 WenetSpeech 上,其准确率略逊于最强的基线(如 Uni-ASR),但仍保持数量级的延迟优势。
  • 消融实验揭示:
    • 阶段二 vs 阶段三:仅靠监督学习(阶段二)的模型,其性能点落在“强制延迟”曲线上,说明监督学习倾向于学习全局性的早提交策略。而经过强化学习(阶段三)后,模型性能点显著优于强制延迟曲线,证明了其学到了“按需等待”的自适应策略。
    • 奖励分配方式:使用句子级奖励会导致模型为了追求整句准确率而“全程等待”(延迟高达383ms),而本文提出的字符级分段信用分配则能有效避免这一问题,在保持低延迟(27ms)的同时提升准确率。

5. 优势与局限

  • 主要优势:
    1. 极低的提交延迟:相比现有流式基线,延迟降低了一个数量级,达到几十毫秒级别,非常接近声学边界。
    2. 准确率与延迟的帕累托改进:在多个数据集上,能以更低的延迟达到或超越现有流式基线的准确率,证明了自适应等待策略的有效性。
    3. 单次硬提交:每个字只输出一次,无需二次修正,适合对输出一致性要求高的场景。
  • 局限性:
    1. 模型上限受限:其离线(非流式)识别准确率弱于某些最强基线(如 Uni-ASR),这限制了其流式识别的准确率上限。在 AISHELL-2 和 WenetSpeech 上准确率并非最优。
    2. 训练流程复杂:三阶段训练流程(特别是阶段二的探测和阶段三的RL)较为复杂,对工程实现和调参要求高。
    3. 数据标注成本:阶段二需要为训练数据生成提交时间标签,虽然用模型自动探测,但全量标注 WenetSpeech 成本过高,目前仅在部分子集上进行。

6. 关键结论与启发

  • 最重要的 takeaway:流式ASR的延迟问题不应被视为一个全局的、位置无关的超参数,而应被建模为一个位置相关的序列决策问题。通过强化学习直接优化“何时提交”的策略,可以打破“低延迟”与“高准确率”之间的强权衡,实现“不确定时等待,确定时立即输出”的理想行为。
  • 对后续研究的启发:
    1. 更优的识别骨干网络:将本文的自适应提交策略应用于更强的离线/流式ASR模型(如Uni-ASR),有望在保持极低延迟的同时,进一步提升准确率上限。
    2. 跨语言与多场景泛化:论文仅在中文普通话数据集上验证,将该方法扩展到其他语言(如英语)和更复杂的真实场景(如噪声环境、多说话人)是重要的未来方向。
    3. 与端到端语音对话模型结合:该工作与论文中提到的 X2-Turn 等全双工对话建模高度相关,其“何时提交”的决策机制可直接服务于对话系统中的“何时该说话”(turn-taking)决策,实现更自然的人机交互。
#50
cs.SD

TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context

Fritz Cremer, Jonathan Cremer
Sound (cs.SD); Computation and Language (cs.CL); Machine Learning (cs.LG)
Comments: 13 pages, 2 figures, 2 tables. Both authors contributed equally
查看摘要
Text-to-speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality typically comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU. Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches approximately 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. On our LLM-as-a-judge audiobook-reading benchmark, TontaubeV1 matches ElevenLabs Flash v2.5 and outperforms Fish Audio S2 Pro, the April 2026 Gradium API, and Cartesia Sonic 3 on prosody. The model weights are released on Hugging Face under the Tontaube Community Model License 1.0.

📖 深度解读

1. 一句话总结

TontaubeV1 是一个总参数量仅 2.9B 的流式文本转语音(TTS)系统,通过将语音分解为“语义主干 + 三层声学细节”的层级化编码,并用四个大小不一的 Transformer 分别预测,在单张消费级 GPU 上实现了媲美 ElevenLabs 的自然韵律和实时流式合成。

2. 研究背景与动机

  • 核心问题:如何在低计算成本、低延迟(流式)的前提下,生成具有自然韵律(如重音、语调、停顿)的高质量语音。传统 TTS 常面临“质量越高,成本/延迟越高”的权衡。
  • 重要性:低成本、高质量的流式 TTS 对于实时交互(如语音助手)、大规模内容生成(如有声书)等应用至关重要,能显著降低部署门槛和运营成本。
  • 现有方法不足:
  • 大型模型(如更大规模的 LLM-based TTS)韵律好,但 GPU 占用高、推理慢、首包延迟大。
  • 小型模型速度快,但韵律平淡(“机器感”强),长文本中韵律一致性差。
  • 现有基于 codec 的模型(如 Qwen3-TTS)通常将声学细节的预测合并到一个共享的小模块中,容量分配不够灵活,且流式生成通常需要训练专门的因果 codec。

3. 核心方法

  • 方法/框架:采用“层级化 Codec 建模”的生成框架。将语音编码为 DualCodec 的 4 个流(1 个语义流 + 3 个声学流),每个流由一个独立的、大小递减的 Qwen3 衍生 Transformer 模型(CB0-CB3)自回归生成。
  • 关键创新点:
    1. 容量按信息重要性分配:假设韵律主要由语义流决定,因此将最大的模型(1.7B)分配给语义流(CB0),而将三个较小的模型(0.6B 变体)分配给声学细节流(CB1-CB3),实现了计算资源的有效利用。
    2. 字符级文本分词:将文本按字符而非子词(subword)进行分词,使文本长度与语音帧率的关系更稳定,简化了长文本的上下文窗口管理,并让发音学习更鲁棒。
    3. “时间对齐”的位置编码方案:为文本和音频 token 分配基于“发生时间”的逻辑位置(而非物理序列位置),使文本与音频、不同音频流之间在时间上对齐,有助于模型学习韵律。
    4. 有界上下文的长文本生成与流式重建:通过成对的文本/音频边界标记和滚动窗口,实现任意长度文本的有界上下文生成。针对 DualCodec 非因果解码器无法直接流式的问题,提出将重叠的 DualCodec 重建结果映射到 VibeVoice 的因果声学潜在空间进行解码,从而在不重新训练 codec 的情况下实现流式输出。

4. 实验与结果

  • 数据集/基准:
  • 主基准:自建的 400 段英文有声书朗读基准(取自 PG-19 测试集),使用 LLM-as-a-judge(Gemini 3.1 Pro)进行成对比较。
  • 辅助基准:Seed-TTS 评估集的 1088 个英文零样本例子,用于计算 WER。
  • 对比基线:ElevenLabs Flash v2.5、Fish Audio S2 Pro、Gradium API (Apr. 2026)、Cartesia Sonic 3。
  • 主要实验结果:
  • 韵律质量:TontaubeV1 与 ElevenLabs Flash v2.5 持平(偏好率 50.1%,统计上无显著差异),显著优于 Fish Audio S2 Pro(82.1%)、Gradium API(86.2%)和 Cartesia Sonic 3(82.3%)。
  • 文本正确性:与 ElevenLabs 持平(48.9%),优于 Gradium 和 Cartesia,与 Fish Audio 持平。
  • 性能:单张 RTX 5090 上,首音频延迟约 200ms,单输入 RTF 为 0.08(约 12.5 倍实时),8 路并发时聚合 RTF 为 0.02(约 50 倍实时)。
  • 消融实验:论文未报告传统意义上的消融实验。但提及早期实验发现“为每个 codebook 分配独立模型比共享模型收敛更好”,这直接支撑了其核心架构设计选择。

5. 优势与局限

  • 主要优势:
    1. 高质量与高效率的平衡:在韵律上达到 ElevenLabs 水平,但模型小(2.9B),可在消费级 GPU 上实现流式生成,成本低。
    2. 模块化设计:四个独立模型可独立调度,便于实现并发声学细化(chunk-local acoustic refinement),提升服务吞吐量。
    3. 创新的流式方案:通过“双 codec”映射,巧妙地绕过了非因果 codec 无法流式的问题,且无需训练新 codec。
  • 局限性:
    1. 评估范围有限:主要基准仅针对英文有声书朗读,未充分验证德语及其他多语言质量、声音相似度、对话场景等。
    2. 级联延迟:四阶段串行生成(CB0→CB3)会引入额外延迟,尽管通过流式技术掩盖了部分延迟。
    3. 依赖文本预处理:模型要求输入“口语化文本”(spoken-form),数字、日期等需预先通过额外的 verbalizer 转换,增加了使用复杂性和潜在错误源。

6. 关键结论与启发

  • 最重要的 takeaway:在 TTS 中,通过将“语义/韵律”与“声学细节”分离,并采用“大模型管语义,小模型管细节”的层级化建模策略,可以在不牺牲韵律质量的前提下,显著降低模型规模和推理成本,实现高效流式合成。
  • 对后续研究的启发:
    1. 容量分配的指导原则:为 TTS 中的不同信息流(语义 vs. 声学)分配不同规模的模型容量,是一种高效的设计范式。
    2. 流式生成的新思路:利用“潜在空间拼接”而非“波形拼接”来解决非因果编解码器的流式问题,为其他类似架构提供了借鉴。
    3. 位置编码的语义化:将位置编码与“时间”而非“序列顺序”绑定,是一种值得探索的建模思路,可能对长序列建模有普适意义。
    4. 评估方法:采用 LLM-as-a-judge 进行大规模、可重复的韵律评估,是未来 TTS 评测的一个可行方向,但其有效性仍需更多与人类听感的对比验证。
#51
cs.SD

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing 解读失败

Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan 等 (33 人)
Sound (cs.SD); Computation and Language (cs.CL); Multimedia (cs.MM)
Comments: Open-source at this https URL
查看摘要
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#52
cs.SD

Disentangled Global-Local Feature Learning with E-Branchformer for Audio Deepfake Detection 解读失败

Phuong Tuan Dat, Ho Bao Thu, Nguyen Tran Trung, Pham Viet Hoang, Nguyen Thi Thu Trang
Sound (cs.SD)
Comments: Accepted to SALMA Workshop @ EMNLP 2026
查看摘要
The rapid advancement of voice synthesis technologies such as text-to-speech and voice conversion poses significant threats to speech-based authentication systems, necessitating robust deepfake detection methods. In this work, we propose a novel E-Branchformer-based architecture that effectively leverages self-supervised speech representations for audio deepfake detection. Our model employs parallel branches to simultaneously capture global contextual dependencies through multi-head self-attention and local temporal patterns through convolutional processing. To enhance discriminative capability, we integrate depthwise convolution and Squeeze-and-Excitation modules that enrich the classification token with refined patch token information after feature merging. Extensive experiments on ASVspoof 2021 LA, DF, and In-the-Wild datasets demonstrate state-of-the-art performance with equal error rates of 0.88%, 1.85%, and 6.30% respectively, substantially outperforming existing methods. Comprehensive ablation studies validate that the dual-branch architecture provides complementary discriminative information, Squeeze-and-Excitation Aggregation significantly improves SSL feature integration, and the combination of DWConv and SE modules is critical for effective class token enhancement. The superior performance on real-world scenarios demonstrates strong generalization capability to diverse acoustic conditions and unseen spoofing attacks.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#53
cs.SD

TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction 解读失败

Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu, Jie Lin, Yu-Kuan Fu 等 (10 人)
Sound (cs.SD)
查看摘要
Full-duplex voice interaction requires more than utterance-level conversion. It must process streaming speech, manage turn-taking and interruptions, while preserving pretrained linguistic competence and acoustic paralinguistic cues. We ask whether TASTE (Text-Aligned Speech Tokenization and Embedding) provides a viable path toward this goal. We present TASTE2, which transforms utterance-level TASTE into an incremental dialogue stack. A shared text-token vocabulary removes word-level averaging, while modality-aligned dialogue training predicts one continuous audio latent per text token without interleaving heterogeneous token streams. An incremental Speech Detokenizer enables streaming synthesis through CosyVoice2. After speech and dialogue training, TASTE2 (Merge) reaches 56.3% on LLaMA-Questions against a 57.3% Qwen2.5-7B Instruct text-only reference (98.2% accuracy retention), and TASTE2 (Direct) reaches 53.0% (92.4% retention). We build TASTE2 VoiceBot, which processes user speech incrementally, streams synthesized audio, and stops generation on barge-in. On Full-Duplex-Bench v1.0, TASTE2 and TASTE2 VoiceBot handle interruptions well while maintaining high conversational coherence. Natural conversation remains challenging, and deployed mean time to first audio is 2.701 s on two NVIDIA RTX A6000 after TensorRT acceleration. Finally, to our knowledge, we provide the first systematic characterization of explicit paralinguistic control in a TASTE based model. Fast speaking rate serves as a cross-strategy proof of concept after dialogue SFT, while emotion control is strategy dependent and the remaining attributes stay weak. Together, these results establish TASTE based modeling as a practical route toward full-duplex systems while identifying natural conversation robustness, speech generation latency, and feature general paralinguistic control as open challenges. Explore TASTE2 online.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#54
cs.SD

LipCoder: Voice-Enabled Coding Toolkit 解读失败跨领域

Hayoon Kim, Sungho Lee, Juhwi Kim, Bongwon Suh, Kyogu Lee
Human-Computer Interaction (cs.HC); Sound (cs.SD)
Comments: Accepted to the Posters and Demos Track at the 28th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2026)
查看摘要
AI-assisted programming environments have accelerated software development, giving rise to new paradigms like vibe coding. However, their benefits remain largely inaccessible to visually impaired programmers, as existing screen readers and assistive tools offer limited support for these emerging workflows. We introduce LipCoder, a voice-centric programming toolkit designed to deliver editor-level functionality through auditory and speech-based interfaces. LipCoder offers features comprising speech feedback and earcon cues for comprehension and validation, as well as natural language input for navigation and modification. In an exploratory evaluation, 5 visually impaired programmers performed a series of coding tasks comparing LipCoder with a baseline of VSCode, Copilot, and VoiceOver. Quantitative trends and qualitative feedback point to directions for auditory-first design that may broaden accessibility in speech-driven coding environments.

📖 深度解读

[PDF 下载失败,无法解读]

#55
cs.SD

ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction 解读失败跨领域

Zhuoqiang Cai, Yujie Sun, Chaoyue Niu, Hongyun Yu, Zhiwen Chen 等 (7 人)
Graphics (cs.GR); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD)
Comments: 10 pages, 4 figures, 4 tables. Accepted to ACM Multimedia 2026 (MM '26) as a poster presentation
查看摘要
We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting, formulating the problem as an asymmetric task involving speech-constrained articulation and one-to-many listener reactions. To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure and a stochastic residual that models the remaining one-to-many interaction dynamics. On top of this backbone, Motion Memory acts as a training-only regularizer during brief late-stage fine-tuning to provide local priors for weakly conditioned listening windows, while semantic-group scaling controls residual injection across expression, jaw, and neck. This design balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process. Results from unified, state-wise, and ablation evaluations show that conversational 3D motion benefits from decomposing stable and uncertain components rather than applying stochasticity uniformly. ECHO provides a practical formulation and technical basis for deployable conversational digital humans under strict audio-only conditions.

📖 深度解读

[PDF 下载失败,无法解读]

#56
cs.SD

TamilEOT: A Dataset and Model for Semantic End-of-Turn Detection in Tamil Telephone Speech 解读失败跨领域

Santhoshkumar V
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Sound (cs.SD)
Comments: 13 pages, 7 tables. Code, data and models: this https URL
查看摘要
A voice agent has to decide, at every pause, whether the user has finished speaking. Without a model of the language that decision falls back to a fixed silence timeout: set it short and the agent interrupts, set it long and every turn pays the full wait. Open semantic end-of-turn detectors exist, but to our knowledge none covers a South Indian language. We release TamilEOT: 18,485 labelled turn boundaries cut from 116 real Tamil telephone conversations, and two audio-only detectors fine-tuned from Smart Turn v3. On a held-out split of 4,168 clips from 30 unseen calls, accuracy rises from 70.30% zero-shot to 83.71% (8.7 MB) and 86.13% (21 MB); ROC-AUC rises from 0.751 to 0.921. Both models run in under 150 ms single-threaded on a laptop CPU. We also report what building it cost. Rule-derived labels, checked against a blind human listening pass, were right 95.9% of the time on the positive class and 44.4% on the negative class, which is below chance, because the rule answered a different question than the model is asked. Replacing them with an audio-LLM labeller measured at 97.5% human agreement cost US$5.69. Of every training lever we measured, only encoder capacity moved the result; three runs at identical config and seed span 0.87 accuracy points, which is the floor below which none of our other deltas mean anything. Replaying the same labelled boundaries through the production VAD and streaming adapter costs a further 2.60 points, and 7.8% of boundaries are never surfaced to the model at all. Data, weights, code and every negative result are public.

📖 深度解读

[PDF 下载失败,无法解读]

#57
cs.SD

Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts 解读失败跨领域

Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan 等 (7 人)
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD)
Comments: EMNLP 2026 Findings
查看摘要
Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two distinct forms of evidence within a single modality: perceptual signals (e.g., a photograph or recording of a dog) and propositional signals (e.g., the declarative claim "this is a dog"), such that any measured modality bias is inherently confounded with evidence-form bias, precluding clean attribution to either source. To address this, we introduce Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form. Evaluating five OLLMs, we find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias: models exhibit a stronger bias toward perceptual signal in vision but propositional in audio. Further analyses via layer-wise linear probing and contrastive decoding reveal that modality bias is already linearly decodable from early representation layers and can only be partially mitigated, calling for mitigation strategies beyond surface-level interventions.

📖 深度解读

[PDF 下载失败,无法解读]

#58
cs.SD

MVWeaver: A Hierarchical Music Video Generation Agent with a Learned Song-to-Visual Bridge 解读失败跨领域

Sifei Li, Minyan Luo, Xu Li, Guodong Qi, Xincan Wang 等 (10 人)
Multimedia (cs.MM); Sound (cs.SD)
Comments: 5 pages, 2 figures
查看摘要
Music videos are an important form of audiovisual expression in contemporary culture. They translate and extend the expressive content of songs through deliberate visual design. Existing automatic music video (MV) generation systems can generate visually plausible shots, yet often struggle with long-form coherence and song-grounded visual development. We present MVWeaver, a music video generation agent that integrates hierarchical planning with a learned song-to-visual bridge that translates song understanding into executable shot plans. The MVWeaver architecture comprises a comprehensive song analysis module, a visual planner that constructs hierarchical plans, and downstream image and video generation models that render the planned content. To equip a general-purpose LLM with MV-specific song-to-visual knowledge, we learn a bridge between song analysis and visual planning from real-MV-derived supervision and curate 1,861 real-world song--MV pairs with structured song-side, MV-side, and teacher-inferred song-to-visual rationale annotations. Using these annotations, we perform LoRA-based supervised fine-tuning (SFT) of a large language model to predict song-to-visual bridges that guide hierarchical visual planning. Our experiments demonstrate stronger song-grounded visual translation, richer visual development, and greater conceptual and shot-to-shot coherence, while ablations support the benefits of learned bridge conditioning.

📖 深度解读

[PDF 下载失败,无法解读]

#59
cs.SD

When Speech Meets Lips: Interpretable Audio-Visual Synchronization for L2 Pronunciation Assessment 解读失败跨领域

Bowen Yu, Mingyu Huang, Yishen Liu, Yue Zhao
Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD)
查看摘要
Automatic Pronunciation Assessment (APA) systems have achieved strong performance with transformer-based models and self-supervised speech representations. However, most methods rely only on acoustic signals and overlook temporal synchronization between speech and articulatory movements, limiting diagnostic feedback on timing mismatches important for L2 pronunciation training. We propose an interpretable audio-visual synchronization framework that explicitly models speech-lip temporal alignment through feature encoding, cross-attention fusion, lag estimation, stability quantification, and visualization. The framework introduces frame-level lag trajectories and a Lag Stability Index (LSI) to quantify synchronization robustness. We also interviewed 30 participants, including 10 instructors and 20 students with diverse first-language backgrounds, to assess its effectiveness. By transforming implicit alignment into interpretable representations, the framework connects automatic scoring with actionable Computer-Aided Pronunciation Training feedback. Datasets and supplemental materials are available at this https URL .

📖 深度解读

[PDF 下载失败,无法解读]

#60
cs.SD

AdoDAS: A Privacy-Preserving Multimodal Challenge for Adolescent Depression, Anxiety, and Stress Assessment 解读失败跨领域

Zhaojie Luo, Junkun Wang, Tianhua Qi, Yuxuan Wu, Xin Zhao 等 (14 人)
Multimedia (cs.MM); Sound (cs.SD)
Comments: 5 pages, 1 figure, 3 tables. To appear in the Proceedings of the 34th ACM International Conference on Multimedia (MM '26), November 10-14, 2026, Rio de Janeiro, Brazil. Zhaojie Luo and Junkun Wang contributed equally
查看摘要
Adolescent depression, anxiety, and stress (D/A/S) call for scalable tools that complement, rather than replace, professional evaluation. Under a privacy-preserving policy, the AdoDAS Grand Challenge withholds minors' raw recordings and distributes anonymized audio-visual representations and ASR-derived text. Its 6,000 participants provide 24,000 segments across one scripted-reading and three open-response sessions. Two tracks assess multi-task binary D/A/S screening and ordinal prediction of 21 DASS-21 item responses. From 191 registrations, the final leaderboards included 95 eligible screening teams and 64 item-prediction teams. Audio-visual baselines achieved 0.4604 mean F1 and 0.2675 mean Quadratic Weighted Kappa; leading submissions reached 0.5921 and 0.2776. Representative systems emphasize cross-session modelling, temporal multimodal fusion, psychometric structure, and task-aware calibration.

📖 深度解读

[PDF 下载失败,无法解读]

#61
cs.SD

RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems 解读失败跨领域

Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong, Vien Nguyen Thi, Viet-Anh Nguyen 等 (6 人)
Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM); Sound (cs.SD)
Comments: 6 pages, 4 figures, 3 tables. Accepted at the 2026 IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS 2026). (c) 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses
查看摘要
Recent multimodal Speech Emotion Recognition (SER) systems achieve high accuracy through interaction-heavy cross-modal transformers, but their computational cost limits deployment in latency-sensitive and resource-constrained surveillance systems. To address this challenge, we propose RAFM_SER++, a lightweight multimodal SER framework featuring an asymmetric Residual Attention Fusion Mechanism (RAFM). Rather than relying on computationally expensive bidirectional interactions, RAFM injects affective speech cues into semantic text representations through a one-directional residual attention pathway. Combined with a BYOL-inspired cross-modal alignment objective and attention-guided pooling, the proposed framework improves multimodal representation learning while maintaining low computational overhead. Experiments on the IEMOCAP and ESD benchmarks demonstrate that RAFM_SER++ consistently outperforms the HuBERT-Base baseline and achieves a superior accuracy-efficiency trade-off compared with the state-of-the-art MemoCMT. Specifically, RAFM_SER++ reduces trainable parameters by more than 60%, achieves faster inference (79.60 it/s), and attains BACC scores of 81.10% on IEMOCAP and 95.39% on ESD. These results indicate that lightweight asymmetric multimodal fusion is an effective alternative to interaction-heavy cross-modal transformers for real-time surveillance applications.

📖 深度解读

[PDF 下载失败,无法解读]

#62
cs.SD

Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning 解读失败跨领域

Kuan-Yi Lee, Tsung-En Lin, Hung-Yi Lee
Sound (cs.SD); Artificial Intelligence (cs.AI)
Comments: Accepted in ISCSLP 2026
查看摘要
Recent advancements in large multimodal models (LMMs) have shown strong capabilities in audio understanding. However, most systems rely solely on end-to-end reasoning, limiting interpretability and accuracy for tasks that require structured knowledge or specialized signal analysis. In this work, we present Audio-Maestro -- a tool-augmented audio reasoning framework that enables audio-language models to autonomously call external tools and integrate their timestamped outputs into the reasoning process. This design allows the model to analyze, transform, and interpret audio signals through specialized tools rather than relying solely on end-to-end inference. Experiments show that Audio-Maestro consistently improves general audio reasoning performance: Gemini-2.5-flash's average accuracy on MMAU-Test rises from 67.4% to 72.1%, DeSTA-2.5 from 58.3% to 62.8%, and GPT-4o from 60.8% to 63.9%. To our knowledge, Audio-Maestro is the first framework to integrate structured tool output into the large audio language model reasoning process.

📖 深度解读

[PDF 下载失败,无法解读]

#63
cs.SD

Ensemble-Guided Distillation for Compact and Robust Acoustic Scene Classification on Edge Devices 解读失败跨领域

Hossein Sharifi, Behnam Raoufi, Mahdy Ramezani, Khosrow Hajsadeghi, Saeed Bagheri Shouraki
Sound (cs.SD)
查看摘要
We present a compact, quantization-ready acoustic scene classification (ASC) framework that couples an efficient student network with a learned teacher ensemble and knowledge distillation. The student backbone uses stacked depthwise-separable "expand-depthwise-project" blocks with global response normalization to stabilize training and improve robustness to device and noise variability, while a global pooling head yields class logits for efficient edge inference. To inject richer inductive bias, we assemble a diverse set of teacher models and learn two complementary fusion heads: z1, which predicts per-teacher mixture weights using a student-style backbone, and z2, a lightweight MLP that performs per-class logit fusion. The student is distilled from the ensemble via temperature-scaled soft targets combined with hard labels, enabling it to approximate the ensemble's decision geometry with a single compact model. Evaluated on the TAU Urban Acoustic Scenes 2022 Mobile benchmark, our approach achieves state-of-the-art (SOTA) results on the TAU dataset under matched edge-deployment constraints, demonstrating strong performance and practicality for mobile ASC.

📖 深度解读

[PDF 下载失败,无法解读]

#64
cs.SD

Decoder-Side Semantic Conditioning for Low-Bitrate Neural Speech Compression 解读失败跨领域

Liuyang Bai, Weiyi Lu, Li Guo
Sound (cs.SD); Computation and Language (cs.CL); Machine Learning (cs.LG)
Comments: Accepted to APSIPA ASC 2026
查看摘要
Speech codecs are usually optimized for waveform fidelity, allocating bits to acoustic detail that can be inferred from linguistic structure. This leads to inefficient compression and degraded recognition performance. We propose SemDAC, a semantic-aware neural speech codec that adds hierarchical semantic conditioning to residual vector quantization (RVQ). The first RVQ quantizer is distilled from HuBERT features to produce semantic tokens capturing phonetic content, while later quantizers encode residual acoustics. The decoder is conditioned on semantic tokens via feature-wise linear modulation (FiLM), steering reconstruction toward information not explained by semantic abstraction. At 0.95 kbps, SemDAC matches or surpasses a 2.5 kbps DAC baseline on PESQ, STOI, SI-SNR, and Whisper WER, with comparable ViSQOL, and achieves higher subjective MOS than higher-bitrate DAC baselines. Results show that explicit semantic conditioning, rather than token disentanglement or increased model size alone, improves compression efficiency and recognition robustness.

📖 深度解读

[PDF 下载失败,无法解读]

#65
cs.SD

Scaling Audio Models Efficiently: A Joint Study of Compute Constraints and Optimization Behavior 解读失败跨领域

Vyom Agarwal, Mokshda Gangrade, Siddharth Pal, Jerry Wu
Sound (cs.SD); Artificial Intelligence (cs.AI)
查看摘要
Large automatic speech recognition (ASR) models such as Whisper must be deployed across hardware with widely varying memory and inference-speed constraints. We present a compression framework that jointly parametrizes Whisper deployment along \emph{six} dimensions: model size $x_N$, temporal resolution $x_T$, encoder token stride $x_V$, low-rank adaptation capacity $x_R$, weight precision $x_Q$ and sparsity pattern $x_P$. All axes are jointly optimized against three deployment objectives (word error rate, inference FLOPs, and memory footprint) using a non-dominated sorting genetic evolutionary search (NSGA). Across 50 of the 1,680 candidate configurations evaluated, we measure the marginal effect of each axis on the three objectives and identify compression combinations that dominate naive single-axis scaling, and report a consistent negative result: 1:4 structured sparsity fails to recover acceptable accuracy under any tested recovery budget. We report real, measured memory and accuracy figures for genuinely quantized deployment artifacts, and provide a lookup table mapping deployment scenarios (cloud, server, edge, ultra-constrained) to specific axis configurations with their measured accuracy/memory/compute trade-offs

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#66
cs.SD

TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation 解读失败跨领域

Qinzhe Hu, Chenda Li, Wangyou Zhang, Shujie Liu, Yan Lu 等 (6 人)
Sound (cs.SD); Artificial Intelligence (cs.AI)
Comments: Accepted to Interspeech 2026
查看摘要
Recent advances in speech separation (SS) have led to compact front-end models with small parameter sizes, yet their high computational cost remains a major barrier for deployment on edge devices. To address this, we propose TF-MoE, a sparse Mixture-of-Experts (MoE) framework that enhances model capacity with almost no increase in inference cost. Our method introduces dynamic expert specialization in time and frequency dimensions through alternating time-wise and frequency-wise MoE modules, each dynamically selecting experts per frame or mel band. Built upon a mel-band-splitting Conformer backbone, TF-MoE achieves strong performance on SS tasks under low-compute settings. Experimental results demonstrate that TF-MoE consistently improves separation performance under computation cost constraints, outperforming BSRNN by +3.8 dB SDR on Libri2Mix with comparable 4.1 GMACs/s inference cost. This positions TF-MoE as a promising candidate for edge-device deployment.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#67
cs.SD

Probing Speaker Identity Sensitivity in Audio Deepfake Detectors 解读失败跨领域

Daniyal Kabir Dar, Arun Ross
Sound (cs.SD); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
Comments: Accepted at IEEE/IAPR International Joint Conference on Biometrics (IJCB) 2026. 8 pages, 3 figures, 7 tables
查看摘要
Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks. Yet the same detector that achieves less than 1% error on one dataset can see its error rate increase twentyfold when evaluated on a different dataset. We argue that one contributing factor is speaker-identity reliance: standard training corpora correlate speaker identity with the genuine/synthetic label, allowing detectors to partially rely on speaker-related cues rather than synthesis artifacts alone. We propose the Identity Sensitivity Score (ISS), a per-utterance diagnostic that quantifies how much a detector's output changes across different speaker identity contexts. ISS requires no ground-truth labels at inference time and can be computed from the detector score and a pool of reference speaker examples. Across two detectors and two datasets, incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified utterances, and ISS alone predicts misclassification with area-under-curve (AUC) up to 0.954. To test whether ISS actually captures identity-sensitive behavior rather than serving only as a proxy for prediction confidence, we apply voice conversion to 500 utterances and measure the resulting detector-score shift. Utterances flagged as identity-sensitive by ISS respond 19 to 30 times more strongly to this manipulation than utterances flagged as stable. These results position ISS as a practical inference-time diagnostic for speaker-dependent failure analysis in audio deepfake detection.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#68
cs.SD

Do EEG Foundation Models Transfer to Speech? A Benchmark on Overt and Imagined Speech Decoding 解读失败跨领域

Owais Mujtaba Khanday, Mohamed Baha Ben Ticha, Sanae Belfrouh, Marc Ouellet, Jose A.Gonzalez-Lopez
Sound (cs.SD)
Comments: 6 pages, 1 figure, 3 tables, submitted to IberSPEECH 2026
查看摘要
EEG foundation models pretrained on thousands of hours have shown large gains over task-specific networks for motor imagery, seizure detection, sleep staging, and emotion recognition, but their transfer to speech decoding - arguably the most demanding non-invasive BCI application - remains untested. We present the first systematic benchmark of EEG foundation models against strong convolutional baselines for speech decoding, using two corpora: UGR-MINDVOICE (overt and covert Iberian Spanish) and BCI Competition 2020 Track 3 (imagined speech). We compare two foundation models (LaBraM, EEGMamba) against three established baselines (EEGNet, ShallowFBCSPNet, EEGConformer) under a unified preprocessing and fine-tuning protocol. Large-scale EEG pretraining yields no consistent advantage over a 16K-parameter CNN on speech tasks, indicating that current general-purpose EEG pretraining does not yet transfer to speech production and motivating speech-specific foundation models.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#69
cs.SD

Soft Posterior Speaker Injection for Multi-Talker Speech Recognition 解读失败跨领域

Jian Zhu, Cheng Luo
Sound (cs.SD); Multimedia (cs.MM)
Comments: This paper is submitted to ICASSP2027
查看摘要
Multi-talker automatic speech recognition (MT-ASR) remains challenging in the presence of overlapping speech. Hard segmentation introduces irreversible errors, whereas serialized output training (SOT) avoids explicit segmentation but does not condition a pretrained encoder on speaker activity. We propose Soft Posterior Speaker Injection (SPSI). A Soft Posterior Head predicts per-frame speaker posteriors $\hat{\mathbf{P}}$ and injects them into Whisper through Multi-layer Feature-wise Linear Modulation (MFLM) and Speaker Memory Prompts (SMP). The benefit of SPSI concentrates where overlap is hardest and under domain transfer. On controlled two-speaker LibriSpeech overlap, SPSI reduces concatenated minimum-permutation word error rate (cpWER) from $61.5\%$ to $60.0\%$ in the high-overlap bin, with a smaller $51.9\%$ to $51.0\%$ change on the full set relative to SOT. Same-backbone speaker-auxiliary objectives and voice activity detection (VAD) pipelines do not outperform SOT. Zero-shot LibriCSS is comparable, but freeze-posterior overlap-heavy adaptation reduces held-out LibriCSS cpWER from $42.3\%$ to $36.8\%$ on sessions $8$--$9$, a $5.5$-point gain over SOT. Ablations indicate complementary roles of MFLM and SMP.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#70
cs.SD

Local Chord Corruption Is Not Recognizer Replay: Structure-Matched Calibration for Chord-Conditioned Generation 解读失败跨领域

Weiwen Huang
Sound (cs.SD); Multimedia (cs.MM)
Comments: 5 pages, 2 figures, 2 tables. Code: this https URL
查看摘要
Synthetic chord substitutions offer controlled tests of music generation, but their effects can differ from those of a complete recognized chord sequence. We propose structure-matched calibration, which constructs synthetic chord sequences that preserve the locations and harmonic relations of recognizer-induced changes. Paired generation measures how closely these sequences reproduce the response to complete recognizer replay. On 29 of 30 MUSDB18-HQ songs, central four-second tritone corruption produces a larger target response than complete recognizer replay. On 24 held-out MoisesDB songs, structure matching reduces response distance to replay by 81% for MIDI-SAG and 77% for MusicGen-Chord. Distance to replay decreases on every song in both models. Joint matching also brings output chord sequences closer to replay than either temporal or relational matching alone. Calibration extends to AccoMontage's native beat-based interface, improving 23 of 24 songs. These results establish a method for making synthetic chord tests representative of recognized harmony, while distinguishing response magnitude from the harmonic structure of generated music.

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#71
cs.SD

One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing 解读失败跨领域

Ke Lei, Chenyuhao Wen, Yu Zhang, Wenxiang Guo, Changhao Pan 等 (12 人)
Sound (cs.SD)
查看摘要
Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at this https URL , code can be found at: this https URL .

📖 深度解读

[LLM 连接失败: https://api.deepseek.com/v1/chat/completions]

#72
cs.SD

Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction 解读失败跨领域

Xiang Li, Jiabao Gao, Sipei Lin, Xuan Zhou, Chi Zhang 等 (8 人)
Artificial Intelligence (cs.AI); Sound (cs.SD)
Comments: Accepted by ICLR 2026 Conference
查看摘要
The pursuit of human-like conversational agents has long been guided by the Turing test. For modern speech-to-speech (S2S) systems, a critical yet unanswered question is whether they can converse like humans. To tackle this, we conduct the first Turing test for S2S systems, collecting 2,968 human judgments on dialogues between 9 state-of-the-art S2S systems and 28 human participants. Our results deliver a clear finding: no existing evaluated S2S system passes the test, revealing a significant gap in human-likeness. To diagnose this failure, we develop a fine-grained taxonomy of 18 human-likeness dimensions and crowd-annotate our collected dialogues accordingly. Our analysis shows that the bottleneck is not semantic understanding but stems from paralinguistic features, emotional expressivity, and conversational persona. Furthermore, we find that off-the-shelf AI models perform unreliably as Turing test judges. In response, we propose an interpretable model that leverages the fine-grained human-likeness ratings and delivers accurate and transparent human-vs-machine discrimination, offering a powerful tool for automatic human-likeness evaluation. Our work establishes the first human-likeness evaluation for S2S systems and moves beyond binary outcomes to enable detailed diagnostic insights, paving the way for human-like improvements in conversational AI systems.

📖 深度解读

[PDF 下载失败,无法解读]

#73
cs.SD

Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition 解读失败跨领域

Yuxiang Mei, Delai Qiu, Shengping Liu, Jiaen Liang, Yanhua Long
Computation and Language (cs.CL); Sound (cs.SD)
Comments: Published in IEEE Transactions on Audio, Speech and Language Processing, 2026
查看摘要
Speech Large Language Models (Speech-LLMs) have emerged as a powerful approach for automatic speech recognition (ASR) by aligning speech encoders with large language models. However, adapting these systems to multilingual settings with imbalanced data distributions remains challenging. In such scenarios, a stability-plasticity dilemma often arises: fully shared Parameter-Efficient Fine-Tuning (PEFT) can cause negative inter-lingual interference for under-represented languages, while fully language-specific tuning limits the cross-lingual beneficial knowledge transfer needed for low-resource tasks. To address this, we propose Zipper-LoRA, a novel rank-level decoupling framework with three variants (Static, Hard, and Soft) that dynamically synthesizes LoRA updates from shared and language-specific subspaces. By using a lightweight language-conditioned router, Zipper-LoRA dynamically controls the contribution of each subspace at the LoRA rank level, enabling fine-grained sharing where languages are compatible and strict decoupling when conflicts occur. To further stabilize optimization under imbalanced data, we propose a two-stage training strategy with an Initial-B warm start that significantly accelerates convergence. Experiments on a 12-language mixed-resource setting show that Zipper-LoRA consistently outperforms both fully shared and independent baselines, particularly in extremely low-resource scenarios. Moreover, we demonstrate that these gains are robust across both chunked and non-chunked encoder configurations, confirming the framework's reliability for practical, large-scale multilingual ASR. Our code and data will be available at this https URL for reproducibility.

📖 深度解读

[PDF 下载失败,无法解读]