Can an LLM directly edit the weights of audio models to improve performance without backward passes, scale 100× from 318M to 32.4B parameters, and compress weights in a single forward pass?
Inspired by recent work on Weight-Space Programming (WSP) [Takehi et al., 2026], we investigate three questions for audio models:
Wav2Vec2-XLS-R-300M + AASIST) to a 32.4B-parameter Speech Recognition model (NVIDIA-Nemotron-3.5-Transcribe-30B-A4B)?
How WSP Discovered the 5 Building Blocks (and Why We Built the Harness): We gave an LLM agent a Python sandbox to iteratively inspect model errors on practice audio, write weight-editing code, and check the result in a forward pass (iterating through 23 rounds p01 → p23 on Deepfake Detection and 8 rounds v1 → v8 on Speech Recognition). Across both tasks, the agent repeatedly converged on the exact same five building blocks. We packaged these five blocks into our open-source audiowsp harness so any agent or researcher can apply them directly:
An Automatic Speech Recognition (ASR) model's weights should respond to what is spoken (and a deepfake detector's weights should respond to whether a voice is real or synthetic)—not whether the audio was uploaded to a social platform or sent over a telephone call.
Consider the concrete speech recognition example below. All three audio players contain the exact same 4.8-second recording of a speaker saying:
"HURSTWOOD WALKED THE FLOOR MENTALLY ARRANGING THE CHIEF POINTS OF HIS SITUATION"
Click Play on all three files, then drag the WSP Weight Edit slider right below them from 0% (Unedited Base Model \(W_{\text{orig}}\)) to 100% (WSP Edited Model \(W_{\text{WSP}}\)). Watch how the unedited model fails on the 8 kHz telephone call (75.0% Word Error Rate) and how editing the weights fixes the transcript and lowers error rates across both Speech Recognition (Chart A) and Deepfake Detection (Chart B):
.WAV)8.3% → 0.0%"HURSTWOOD WALKED THE FLOOR MENTALLY ARRANGING THE CHIEF POINTS OF HIS SITUATION"
8 kbps .MP3) — Live Output8 kHz) — Live Output
Figure 1: Top: Single-utterance live demo on LibriSpeech dev-clean Utterance 2 across Studio .WAV, 8 kbps .MP3, and 8 kHz Telephone audio. Bottom: Performance across complete public evaluation datasets as weights interpolate from \(W_{\text{orig}}\) (0%, faint pink ghost bars) to \(W_{\text{WSP}}\) (100%, solid bars): (A) Speech Recognition Word Error Rate (WER %) on Common Voice (5,445 utts), URGENT 2024 (1,000 utts), AMI (2,602 utts), and the 10-Dataset Leaderboard Mean (25,929 total utts); (B) Deepfake Equal Error Rate (EER %) on ASVspoof 2019 LA (71,237 trials), ASVspoof 2021 DF (533,928 trials), ASVspoof 2021 LA (148,176 trials), and In-The-Wild (31,779 trials). All calibration data and evaluation scripts are saved at ishiki-labs/audiowsp-asr-calibration-data and ishiki-labs/ishiki-deepfake-0.5.
How Block 1 Benchmarks Sensitivity Across Augmentations: For any practice audio clip \(x_{\text{studio}}\), we generate paired augmented copies \(\tilde{x}_{\text{aug}}\) with the exact same speaker, words, and timing, and record the difference in hidden-layer activations \(\Delta H_t = H_t(\tilde{x}_{\text{aug}}) - H_t(x_{\text{studio}})\). In addition to MP3 and 8 kHz telephone filtering shown above, the audiowsp harness benchmarks sensitivity across neural codecs (Encodec), room reverberation (RIR), additive background noise (AWGN / babble), and volume/gain shifts.
Block 1 of WSP benchmarked how the hidden neurons change when an audio file is compressed (\(\Delta H_t\)). How do we edit the layer's weight matrix \(W\) so it ignores \(\Delta H_t\) without any drop in task performance?
First, let's define what a "direction" in a hidden layer means:
Wav2Vec2 Layer 20, each audio frame is represented by a list of 1,024 numbers (a vector in \(\mathbb{R}^{1024}\)).SVD). This unit vector \(\vec{v}_{\text{aug}}\) points in the direction that telephone/MP3 compression pushes the neurons.
Why Naive Removal Hurts Performance: When we take the dot product between \(\vec{v}_{\text{aug}}\) and \(\vec{v}_{\text{task}}\) in Layer 20, it is \(\vec{v}_{\text{aug}} \cdot \vec{v}_{\text{task}} = 0.47\)—meaning the two directions share 22.1% of their energy. If we naively delete \(\vec{v}_{\text{aug}}\) from the weights, we also erase 22.1% of the task direction, making clean studio deepfake error 4× worse (0.22% → 0.89% EER).
By subtracting that overlap first (\(\vec{u}_{\perp} = \vec{v}_{\text{aug}} - 0.47\,\vec{v}_{\text{task}}\)), our edit direction \(\hat{u}_{\perp}\) is at an exact 90° angle to \(\vec{v}_{\text{task}}\). Editing the weights via \(W_{\text{new}} = W(I - \gamma \hat{u}_{\perp}\hat{u}_{\perp}^T)\) removes compression sensitivity with 0.0% loss of task signal:
| Weight Edit Formula | W (I − γ u_perp u_perp^T) [90° Perpendicular] |
|---|---|
| Task Signal Retained | 100.0% (Zero Task Loss!) |
| ASVspoof 2019 Studio EER | 0.12% EER (Better than Base!) |
| In-The-Wild Real-World EER | 2.51% EER (4.5× Lower Error!) |
| LibriSpeech Clean ASR WER | 2.02% WER (Zero Clean Drop!) |
Figure 2: Comparison of (Step 1) Unedited Base Weights, (Step 2) Naive Removal \(W(I - \vec{v}_{\text{aug}}\vec{v}_{\text{aug}}^T)\) (which erases 22.1% of \(\vec{v}_{\text{task}}\) because \(\vec{v}_{\text{aug}} \cdot \vec{v}_{\text{task}} = 0.47\)), and (Step 3) 90° Perpendicular Weight Surgery \(W(I - \gamma \hat{u}_{\perp}\hat{u}_{\perp}^T)\). Evaluated on complete public test sets: ASVspoof 2019 LA (71,237 trials), In-The-Wild (31,779 trials), and LibriSpeech test-other (2,939 utterances).
In our 32.4B Speech Recognition model (Nemotron-3.5), the Conformer encoder outputs 1,280 channels per frame. When background noise or room echo is present, some channels carry strong clean speech while other channels carry mostly noise.
We sort all 1,280 channels by the ratio of Clean Speech Power (Green Bars) to Background Noise Power (Red Bars). As shown in the interactive chart below, only the top 96 channels have more speech than noise (Green > Red). In the remaining 1,184 channels, noise is louder than speech (Red > Green)! Keeping only the top 96 channels when fitting the encoder-to-LLM bridge drops 4-dataset noisy Word Error Rate from 8.09% (Unedited) and 7.81% (Keep All 1,280) down to 6.87%:
Figure 3: Generalized Eigenvalue Decomposition (GEVD) channel spectrum across all 1,280 Conformer dimensions in Nemotron-3.5, calibrated on paired clean vs. noisy/reverberant utterances from LibriSpeech dev-clean + dev-other and evaluated across the complete 4-dataset locked noisy test suite (LibriSpeech test-clean [2,620], test-other [2,939], TED-LIUM test [1,469], and MLS en test [3,769]; 10,797 total utterances).
In a standard audio model, the output layer (last layer) only reads the final transformer block (Layer 24) at a single frame \(h_t\). Why is reading only Layer 24 not enough?
Look at the measured layer-by-layer chart below from Wav2Vec2-XLS-R (Layers 0 to 24):
Figure 4: Normalized linear probe separability across Transformer Layers 0 to 24 of Wav2Vec2-XLS-R-300M for fine-grained vocoder pitch/phase artifacts (green curve, peaking at Layers 8–12) versus high-level phonetic/lexical context (blue curve, peaking at Layers 20–24).
In addition to connecting earlier layers into the output layer, we also connect adjacent frames \([h_{t-1}, h_t, h_{t+1}]\) so the output layer can measure frame-to-frame speed \(\frac{1}{2}(h_{t+1} - h_{t-1})\)—which spikes during rapid stop consonants (such as the crisp "p", "t", and "k" sounds in speech, denoted /p/, /t/, /k/)—and we score anomalies on one side only via \(\log(1 + \max(0, z)^2)\) so real human voices on the negative tail (\(z \le 0\)) receive zero false-alarm penalty.
Check boxes 1, 2, and 3 in order below to connect each tap into the output layer and see the measured drop in error rate:
Figure 5: Interactive multi-tap ablation circuit showing how connecting (1) earlier transformer layers, (2) adjacent time frames \([h_{t-1}, h_{t+1}]\) for frame-to-frame speed, and (3) one-sided squared anomaly scoring \(\log(1+\max(0,z)^2)\) into the output layer \(W\) reduces error across complete public test sets: ASVspoof 2021 DF (533,928 trials), In-The-Wild (31,779 trials), and the 4-dataset noisy ASR suite (10,797 utterances).
Why does training with 25 epochs of gradient descent (SGD or AdamW) learn the wrong weights on audio datasets, and how does our One-Forward-Pass Optimizer (OFP) find the optimal shortcut-free weights in a single pass?
Click through Steps 1, 2, and 3 in the interactive 2D feature-space plot below:
Blue Circles = Clean Real Voice | Red X = Fake Voice. Click Step 2 to see how 25-epoch SGD tilts the boundary.
Figure 6: 2D feature-space visualization of shortcut learning versus One-Forward-Pass Optimization (OFP), evaluated across 4 unseen test datasets (ASVspoof 2021 LA, ASVspoof 2021 DF, In-The-Wild, and DECRO; 751,197 total trials).
AdamW).
Can we also compress an audio model's weight matrices in a single forward pass without losing accuracy?
When compressing a weight matrix \(W\) using standard Singular Value Decomposition (SVD), the algorithm keeps directions with the largest average static energy \(\text{Cov}(Z_t)\). In audio, steady vowels and background hum have large static energy, whereas stop consonants (such as the crisp "p", "t", and "k" sounds in speech, denoted /p/, /t/, /k/) last only \(10\text{–}20\text{ ms}\)—so they have small static energy but large frame-to-frame change \((Z_t - Z_{t-1})\).
Standard SVD discards those fast-changing directions, cutting stop-consonant burst peaks by -42% (red shaded area below). By compressing in the combined metric \(\Sigma_{\text{joint}} = \text{Cov}(Z_t) + 2\,\text{Cov}(Z_t - Z_{t-1})\), we preserve 99.1% of the consonant peaks while shrinking our 318M deepfake detector to 86.8M (3.66× smaller) and shrinking Wav2Vec2-Large ASR from 315M to 142M (2.23× smaller):
Figure 7: Frame-to-frame transition velocity \(|Y_t - Y_{t-1}|\) across three stop-consonant bursts ("p", "t", "k") for the full 315M Wav2Vec2-Large teacher (black dashed curve) versus 2.23× compressed 142M models using Standard SVD/ASVD (0%, red) and Transition-Preserving SVD (100%, green), evaluated across 10,797 locked-test utterances (saved verification JSON).
All checkpoints, forward-pass datasets, and the unified audiowsp harness are publicly available:
github.com/ishikilabsinc/audiowsp318M Full & 86.8M Compact + 20.8K Forward-Pass Dataset): huggingface.co/ishiki-labs/ishiki-deepfake-0.5huggingface.co/ishiki-labs/ishiki-asr-0.5| Dataset & Paper | Split (Trials) | Base (W2V2-AASIST, 318M) EER / AUC | Published / Leaderboard SOTA (Method, Params, EER / AUC) | Did SOTA Train on This Dataset? | What Our Forward-Pass Set Used (Dataset) | Ours (318M Full) EER / AUC | Ours (86.8M Compact) EER / AUC |
|---|---|---|---|---|---|---|---|
| ASVspoof 2019 LA · [Paper] | Eval (71.2K) |
0.22% / 0.9997 |
LaST-Net (317M): 0.09% · Nes2NetX (318M): 0.12% |
Yes (25.4K train + 24.8K dev) | 3.0K train + 600 dev (0 eval) |
0.12% / 0.99999 |
0.16% / 0.99996 |
| ASVspoof 2021 LA · [Paper] | Eval (148.2K) |
8.11% / 0.9779 |
XLS-R Conformer: 0.69% |
Trained on ASV19 LA train | 0 clips (ASV19 + PSTN probes) |
1.42% / 0.9964 |
1.51% / 0.9959 |
| ASVspoof 2021 DF · [Paper] | Eval (533.9K) |
6.63% / 8.32% |
VeriSpoof-Base (2.85B): 0.97% · XLSR-Mamba (319M): 1.88% |
Multi-corpus (8 corpora) | 0 clips (ASV19/5 + codec probes) |
1.98% / 0.9983 |
2.19% / 0.9980 |
| In-The-Wild (ITW) · [Paper] | Full (31.8K) |
11.22% / 0.9572 |
UTCN-XLSR2B (433M): 0.37% · DF-Arena-1B (1B): 0.91% |
7-corpus fit + ITW selection | 0 clips in forward-pass set |
2.51% / 0.9958 |
2.72% / 0.9951 |
| DECRO (EN + ZH) · [Paper] | Eval (37.3K) |
9.57% / 0.9583 |
Spectra-AASIST (316M): 0.17% |
Undisclosed (Arena) | 0 DECRO clips (1.28K CFAD) |
1.30% / 0.9986 |
1.41% / 0.9980 |
| ASVspoof 5 (Track 1) · [Paper] | Eval (680.8K) |
16.25% / 0.9003 |
VELMA-2 (316M, Prop.): 0.38% · NSGA-II Fusion (6.6B): 2.37% |
Yes (ASV5 train + dev) | 5.0K train + 3.44K dev (0 eval) |
5.73% / 0.9856 |
5.80% / 0.9857 |
| DFADD · [Paper] | Test (3.8K) |
11.88% / 0.9436 |
VeriSpoof-Base (2.85B): 0.00% · XLSR+SLS (340M): 7.54% |
Yes (full DFADD train) | 704 train + 350 dev aliases |
0.27% / 0.99995 |
0.53% / 0.99991 |
| TIMIT-TTS (Clean) · [Paper] | Full (20.2K) |
2.09% / 0.9954 |
Standard MoE: 0.93% (0.9512 AUC) |
No (unseen test) | 0 clips (unseen) |
0.01% / 0.999998 |
0.03% / 0.999997 |
| TIMIT-TTS (DTW) · [Paper] | Full (20.2K) |
10.51% / 0.9535 |
RawNet2 Baseline (17.6M): 0.7800 AUC |
No (ASV19-trained) | 0 clips (unseen) |
0.69% / 0.9995 |
0.93% / 0.9997 |
| TIMIT-TTS (Augmented) · [Paper] | Orig Real (20.2K) |
2.14% / 0.9946 |
RawNet2 Baseline (17.6M): 0.9300 AUC |
No (ASV19-trained) | 0 clips (unseen) |
0.46% / 0.9996 |
1.17% / 0.9994 |
| TIMIT-TTS (DTW + Aug) · [Paper] | Orig Real (20.2K) |
12.84% / 0.9381 |
XLS-R 2B + LogReg (2B): 3.60% |
No (ASV19 train + dev) | 0 clips (unseen) |
1.85% / 0.9972 |
1.86% / 0.9983 |
| MULTISpoof · [Paper] | Test (43.3K) |
14.62% / 0.9214 |
Asilomar 2026 (no public test table) | N/A | 1.33K val clips (0 test) |
2.94% / 0.9964 |
3.32% / 0.9955 |
| Fake-or-Real (for-norm) · [Paper] | Testing (4.6K) |
7.42% / 0.9816 |
VeriSpoof-Plus (2.85B): 0.09% · XLSR+SLS (340M): 5.08% |
Yes (full for-norm train) | 0 clips (unseen) |
3.13% / 0.9953 |
4.29% / 0.9912 |
| MLAAD · [Paper] | Eval (64.0K) |
15.40% / 0.9120 |
AASIST Voiced/Unvoiced: 5.82% · WavLM-LDA: 10.00% |
Yes (custom MLAAD splits) | 664 train + 200 dev aliases |
6.84% / 0.9785 |
7.51% / 0.9730 |
| DiffSSD · [Paper] | Test (54.6K) |
18.90% / 0.8840 |
Wiener-Hopf 2D CNN (0.42M): 2.95% · Wav2Vec2 Baseline: 3.00% |
Yes (full ~70K DiffSSD train) | 755 train + 249 val aliases |
8.92% / 0.9742 |
9.77% / 0.9691 |
All 13 evaluation splits below are public Open ASR Leaderboard datasets (`hf-audio/esb-datasets-test-only-sorted`, `hf-audio/open-asr-leaderboard`, and `voice-arena-leaderboard/Monsoon-en-IN-test`). Our forward-pass calibration used strictly the saved public development splits (ishiki-labs/audiowsp-asr-calibration-data: LibriSpeech dev-clean [2,703 utts] + dev-other [2,864 utts] + MLS dev [3,807 utts] + TED-LIUM dev [591 utts], with 0 evaluation test utterances):
| Dataset & Official Paper / Repo | Eval Split (Utterances) | Base (Nemotron-3.5-30B) WER | Previous Open ASR Leaderboard #1 (Model & WER) | What Our Forward-Pass Set Used (Saved Data) | Ours (Single-Model Edited Weights) WER | Ours (ishiki-asr-0.5 Full) WER |
|---|---|---|---|---|---|---|
| AMI-Cleaned · [Paper] | Test (2,602) |
6.14% |
Cohere-Transcribe-03-2026: 5.66% |
0 AMI clips (LibriSpeech dev) |
5.97% |
5.14% |
| Earnings22-Cleaned · [Paper] | Test (549) |
4.19% |
NVIDIA-Nemotron-3.5-30B: 4.19% |
0 Earnings22 clips (LibriSpeech dev) |
4.07% |
3.54% |
| GigaSpeech-Cleaned · [Paper] | Test (2,346) |
4.34% |
NVIDIA-Nemotron-3.5-30B: 4.34% |
0 GigaSpeech clips (LibriSpeech dev) |
4.28% |
3.55% |
| SPGISpeech · [Paper] | Test (1,036) |
1.45% |
Canary-Qwen-2.5B / Parakeet-TDT: 1.42% |
0 SPGISpeech clips (LibriSpeech dev) |
1.41% |
1.21% |
| URGENT 2024 · [Paper] | Test (1,000) |
10.97% |
StagesLabs-Whisper-Large-v3: 9.45% |
0 URGENT24 clips (LibriSpeech dev) |
10.72% |
8.66% |
| URGENT 2024 (Clean) · [Paper] | Test (1,000) |
6.37% |
FireRedASR2-AED: 4.68% |
0 URGENT24 clips (LibriSpeech dev) |
5.99% |
4.02% |
| Voice Arena Monsoon (en-IN) · [Repo] | Test (527) |
3.32% |
NVIDIA-Nemotron-3.5-30B: 3.32% |
0 Monsoon clips (LibriSpeech dev) |
3.26% |
2.34% |
| VoxPopuli-AA-Cleaned · [Paper] | Test (628) |
1.28% |
NVIDIA-Nemotron-3.5-30B: 1.28% |
0 VoxPopuli clips (LibriSpeech dev) |
1.23% |
0.90% |
| LibriSpeech (test-clean) · [Paper] | Test (2,620) |
1.12% |
MusicAudio-Qwen2-7B: 1.08% |
dev-clean (2,703) (0 test) |
1.10% |
0.58% |
| LibriSpeech (test-other) · [Paper] | Test (2,939) |
2.07% |
NVIDIA-Nemotron-3.5-30B: 2.07% |
dev-other (2,864) (0 test) |
2.02% |
1.90% |
| TED-LIUM (test) · [Paper] | Test (1,469) |
2.45% |
Cohere-Transcribe-03-2026: 2.28% |
tedlium-dev (591) (0 test) |
2.41% |
1.83% |
| Multilingual LibriSpeech (en) · [Paper] | Test (3,769) |
3.82% |
Cohere-Transcribe-03-2026: 3.64% |
mls-dev (3,807) (0 test) |
3.82% |
3.10% |
| Common Voice (en) · [Paper] | Test (5,445) |
8.61% |
StagesLabs-Whisper-Large-v3: 6.45% |
0 Common Voice clips (LibriSpeech dev) |
5.95% |
3.75% |
| Mean WER (10 Public Leaderboard Sets) | 25,929 utts |
4.59% |
NVIDIA-Nemotron-3.5-30B: 4.59% (#1) |
9,965 public dev utts (0 test) |
4.27% |
3.18% |
| Mean WER (8 Main Short-Form Sets) | 14,605 utts |
4.76% |
NVIDIA-Nemotron-3.5-30B: 4.76% (#1) |
9,965 public dev utts (0 test) |
4.62% |
3.67% |
If you use our models, forward-pass datasets, or the audiowsp harness in your work, please cite:
Amit Kumar Singh Yadav and Robert Xu. "Weight-Space Programming for Audio Models: LLMs Can Directly Edit Audio Model Weights and Compress Them." Ishiki Labs, November 2026.
@misc{yadav2026audiowsp,
title = {{Weight-Space Programming for Audio Models: LLMs Can Directly Edit Audio Model Weights and Compress Them}},
author = {Yadav, Amit Kumar Singh and Xu, Robert},
year = {2026},
month = nov,
organization = {Ishiki Labs},
howpublished = {\url{https://github.com/ishikilabsinc/audiowsp}},
note = {Research Blog Post and Open-Weights Release}
}