Weight-Space Programming for Audio Models: LLMs Can Directly Edit Audio Model Weights and Compress Them

Can an LLM directly edit the weights of audio models to improve performance without backward passes, scale 100× from 318M to 32.4B parameters, and compress weights in a single forward pass?

Authors

Amit Kumar Singh Yadav

Robert Xu

Affiliation

Ishiki Labs

Published & Artifacts

November 18, 2026

Deepfake-0.5 · ASR-0.5 · Harness

TL;DR

Inspired by recent work on Weight-Space Programming (WSP) [Takehi et al., 2026], we investigate three questions for audio models:

  1. Can an LLM directly edit the weights of audio models to improve performance without backward passes?
  2. Understand scalability of Weight-Space Programming (WSP) as model size grows 100×—from a 318M-parameter Audio Deepfake Detector (Wav2Vec2-XLS-R-300M + AASIST) to a 32.4B-parameter Speech Recognition model (NVIDIA-Nemotron-3.5-Transcribe-30B-A4B)?
  3. Can an LLM directly compress audio model weights in a single forward pass without losing accuracy?

How WSP Discovered the 5 Building Blocks (and Why We Built the Harness): We gave an LLM agent a Python sandbox to iteratively inspect model errors on practice audio, write weight-editing code, and check the result in a forward pass (iterating through 23 rounds p01 → p23 on Deepfake Detection and 8 rounds v1 → v8 on Speech Recognition). Across both tasks, the agent repeatedly converged on the exact same five building blocks. We packaged these five blocks into our open-source audiowsp harness so any agent or researcher can apply them directly:

Block 1: Benchmarking Weight Sensitivity to Audio Augmentations

An Automatic Speech Recognition (ASR) model's weights should respond to what is spoken (and a deepfake detector's weights should respond to whether a voice is real or synthetic)—not whether the audio was uploaded to a social platform or sent over a telephone call.

Consider the concrete speech recognition example below. All three audio players contain the exact same 4.8-second recording of a speaker saying:
"HURSTWOOD WALKED THE FLOOR MENTALLY ARRANGING THE CHIEF POINTS OF HIS SITUATION"

Click Play on all three files, then drag the WSP Weight Edit slider right below them from 0% (Unedited Base Model \(W_{\text{orig}}\)) to 100% (WSP Edited Model \(W_{\text{WSP}}\)). Watch how the unedited model fails on the 8 kHz telephone call (75.0% Word Error Rate) and how editing the weights fixes the transcript and lowers error rates across both Speech Recognition (Chart A) and Deepfake Detection (Chart B):

1. Original Studio Audio (.WAV)
Word Error Rate (WER): 8.3% → 0.0%
ASR Transcript:
"HURSTWOOD WALKED THE FLOOR MENTALLY ARRANGING THE CHIEF POINTS OF HIS SITUATION"
2. Social Platform Upload (8 kbps .MP3) — Live Output
Word Error Rate: 66.7% WER (8 / 12 words wrong)
Live ASR Transcript:
"HURSHWOOD WALKED BEFORE MENTALLY ERRING TING THE DEAP POINT OF TE TI PELATION"
3. Telephone Call (8 kHz) — Live Output
Word Error Rate: 75.0% WER (12 / 16 words wrong)
Live ASR Transcript:
"ET WHO HAD WALKED BEFORE MIOY ERNENE TI FINE TE TE TEDERD"
0% (Unedited Base Model W_orig)

Figure 1: Top: Single-utterance live demo on LibriSpeech dev-clean Utterance 2 across Studio .WAV, 8 kbps .MP3, and 8 kHz Telephone audio. Bottom: Performance across complete public evaluation datasets as weights interpolate from \(W_{\text{orig}}\) (0%, faint pink ghost bars) to \(W_{\text{WSP}}\) (100%, solid bars): (A) Speech Recognition Word Error Rate (WER %) on Common Voice (5,445 utts), URGENT 2024 (1,000 utts), AMI (2,602 utts), and the 10-Dataset Leaderboard Mean (25,929 total utts); (B) Deepfake Equal Error Rate (EER %) on ASVspoof 2019 LA (71,237 trials), ASVspoof 2021 DF (533,928 trials), ASVspoof 2021 LA (148,176 trials), and In-The-Wild (31,779 trials). All calibration data and evaluation scripts are saved at ishiki-labs/audiowsp-asr-calibration-data and ishiki-labs/ishiki-deepfake-0.5.

How Block 1 Benchmarks Sensitivity Across Augmentations: For any practice audio clip \(x_{\text{studio}}\), we generate paired augmented copies \(\tilde{x}_{\text{aug}}\) with the exact same speaker, words, and timing, and record the difference in hidden-layer activations \(\Delta H_t = H_t(\tilde{x}_{\text{aug}}) - H_t(x_{\text{studio}})\). In addition to MP3 and 8 kHz telephone filtering shown above, the audiowsp harness benchmarks sensitivity across neural codecs (Encodec), room reverberation (RIR), additive background noise (AWGN / babble), and volume/gain shifts.

Block 2: Removing Augmentation Sensitivity Without Drop in Performance

Block 1 of WSP benchmarked how the hidden neurons change when an audio file is compressed (\(\Delta H_t\)). How do we edit the layer's weight matrix \(W\) so it ignores \(\Delta H_t\) without any drop in task performance?

First, let's define what a "direction" in a hidden layer means:

Why Naive Removal Hurts Performance: When we take the dot product between \(\vec{v}_{\text{aug}}\) and \(\vec{v}_{\text{task}}\) in Layer 20, it is \(\vec{v}_{\text{aug}} \cdot \vec{v}_{\text{task}} = 0.47\)—meaning the two directions share 22.1% of their energy. If we naively delete \(\vec{v}_{\text{aug}}\) from the weights, we also erase 22.1% of the task direction, making clean studio deepfake error 4× worse (0.22% → 0.89% EER).

By subtracting that overlap first (\(\vec{u}_{\perp} = \vec{v}_{\text{aug}} - 0.47\,\vec{v}_{\text{task}}\)), our edit direction \(\hat{u}_{\perp}\) is at an exact 90° angle to \(\vec{v}_{\text{task}}\). Editing the weights via \(W_{\text{new}} = W(I - \gamma \hat{u}_{\perp}\hat{u}_{\perp}^T)\) removes compression sensitivity with 0.0% loss of task signal:

Weight Edit Formula W (I − γ u_perp u_perp^T) [90° Perpendicular]
Task Signal Retained 100.0% (Zero Task Loss!)
ASVspoof 2019 Studio EER 0.12% EER (Better than Base!)
In-The-Wild Real-World EER 2.51% EER (4.5× Lower Error!)
LibriSpeech Clean ASR WER 2.02% WER (Zero Clean Drop!)

Figure 2: Comparison of (Step 1) Unedited Base Weights, (Step 2) Naive Removal \(W(I - \vec{v}_{\text{aug}}\vec{v}_{\text{aug}}^T)\) (which erases 22.1% of \(\vec{v}_{\text{task}}\) because \(\vec{v}_{\text{aug}} \cdot \vec{v}_{\text{task}} = 0.47\)), and (Step 3) 90° Perpendicular Weight Surgery \(W(I - \gamma \hat{u}_{\perp}\hat{u}_{\perp}^T)\). Evaluated on complete public test sets: ASVspoof 2019 LA (71,237 trials), In-The-Wild (31,779 trials), and LibriSpeech test-other (2,939 utterances).

Visualizing Signal-vs-Noise Channel Filtering Across 1,280 Channels

In our 32.4B Speech Recognition model (Nemotron-3.5), the Conformer encoder outputs 1,280 channels per frame. When background noise or room echo is present, some channels carry strong clean speech while other channels carry mostly noise.

We sort all 1,280 channels by the ratio of Clean Speech Power (Green Bars) to Background Noise Power (Red Bars). As shown in the interactive chart below, only the top 96 channels have more speech than noise (Green > Red). In the remaining 1,184 channels, noise is louder than speech (Red > Green)! Keeping only the top 96 channels when fitting the encoder-to-LLM bridge drops 4-dataset noisy Word Error Rate from 8.09% (Unedited) and 7.81% (Keep All 1,280) down to 6.87%:

Keep Top 96 Channels (Speech > Noise Cutoff) → 6.87% Noisy WER (Best!)

Figure 3: Generalized Eigenvalue Decomposition (GEVD) channel spectrum across all 1,280 Conformer dimensions in Nemotron-3.5, calibrated on paired clean vs. noisy/reverberant utterances from LibriSpeech dev-clean + dev-other and evaluated across the complete 4-dataset locked noisy test suite (LibriSpeech test-clean [2,620], test-other [2,939], TED-LIUM test [1,469], and MLS en test [3,769]; 10,797 total utterances).

Block 3: Increasing Context Across Layers (Multi-Tapping)

In a standard audio model, the output layer (last layer) only reads the final transformer block (Layer 24) at a single frame \(h_t\). Why is reading only Layer 24 not enough?

Look at the measured layer-by-layer chart below from Wav2Vec2-XLS-R (Layers 0 to 24):

Figure 4: Normalized linear probe separability across Transformer Layers 0 to 24 of Wav2Vec2-XLS-R-300M for fine-grained vocoder pitch/phase artifacts (green curve, peaking at Layers 8–12) versus high-level phonetic/lexical context (blue curve, peaking at Layers 20–24).

In addition to connecting earlier layers into the output layer, we also connect adjacent frames \([h_{t-1}, h_t, h_{t+1}]\) so the output layer can measure frame-to-frame speed \(\frac{1}{2}(h_{t+1} - h_{t-1})\)—which spikes during rapid stop consonants (such as the crisp "p", "t", and "k" sounds in speech, denoted /p/, /t/, /k/)—and we score anomalies on one side only via \(\log(1 + \max(0, z)^2)\) so real human voices on the negative tail (\(z \le 0\)) receive zero false-alarm penalty.

Check boxes 1, 2, and 3 in order below to connect each tap into the output layer and see the measured drop in error rate:

Layer 8 Layer 16 Layer 24 1. Multi-Layer Tap (Early Pitch/Phase + Mid Formants) 2. Frame Speed Tap: 0.5 · (h_{t+1} − h_{t−1}) 3. One-Sided Anomaly Tap: log(1 + max(0, z)²) Output Layer W Measured Benchmark Error: ASVspoof 2021 DF: 1.98% EER In-The-Wild Deepfake: 2.51% EER Noisy ASR: 6.87% WER

Figure 5: Interactive multi-tap ablation circuit showing how connecting (1) earlier transformer layers, (2) adjacent time frames \([h_{t-1}, h_{t+1}]\) for frame-to-frame speed, and (3) one-sided squared anomaly scoring \(\log(1+\max(0,z)^2)\) into the output layer \(W\) reduces error across complete public test sets: ASVspoof 2021 DF (533,928 trials), In-The-Wild (31,779 trials), and the 4-dataset noisy ASR suite (10,797 utterances).

Block 4: Finding the Optimal Weights in One Forward Pass

Why does training with 25 epochs of gradient descent (SGD or AdamW) learn the wrong weights on audio datasets, and how does our One-Forward-Pass Optimizer (OFP) find the optimal shortcut-free weights in a single pass?

Click through Steps 1, 2, and 3 in the interactive 2D feature-space plot below:

Blue Circles = Clean Real Voice | Red X = Fake Voice. Click Step 2 to see how 25-epoch SGD tilts the boundary.

Figure 6: 2D feature-space visualization of shortcut learning versus One-Forward-Pass Optimization (OFP), evaluated across 4 unseen test datasets (ASVspoof 2021 LA, ASVspoof 2021 DF, In-The-Wild, and DECRO; 751,197 total trials).

Why 1 Forward Pass Is Enough: In a single forward pass (`torch.inference_mode()`), we collect the feature correlation matrix \(A = \Phi^T \Pi \Phi + \lambda I\), the task separation vector \(\vec{d}_{\text{task}}\), and the vertical compression shift \(C_{\text{aug}} = \mathbb{E}[\Phi(\text{MP3}) - \Phi(\text{WAV})]\). Solving for the weights \(W\) with sensitivity to \(C_{\text{aug}}\) locked to zero (\(C_{\text{aug}}^T W = 0\)) has an exact closed-form solution: $$W_{\text{OFP}} = A^{-1}\vec{d}_{\text{task}} - A^{-1}C_{\text{aug}}\left(C_{\text{aug}}^T A^{-1} C_{\text{aug}}\right)^{-1} C_{\text{aug}}^T A^{-1}\vec{d}_{\text{task}}$$ This single pass reduces sensitivity to compression shortcuts to 0.0 and achieves 4.12% Equal Error Rate across 4 unseen deepfake datasets (vs. 6.95% for 25-epoch AdamW).

Block 5: Compressing Audio Model Weights

Can we also compress an audio model's weight matrices in a single forward pass without losing accuracy?

When compressing a weight matrix \(W\) using standard Singular Value Decomposition (SVD), the algorithm keeps directions with the largest average static energy \(\text{Cov}(Z_t)\). In audio, steady vowels and background hum have large static energy, whereas stop consonants (such as the crisp "p", "t", and "k" sounds in speech, denoted /p/, /t/, /k/) last only \(10\text{–}20\text{ ms}\)—so they have small static energy but large frame-to-frame change \((Z_t - Z_{t-1})\).

Standard SVD discards those fast-changing directions, cutting stop-consonant burst peaks by -42% (red shaded area below). By compressing in the combined metric \(\Sigma_{\text{joint}} = \text{Cov}(Z_t) + 2\,\text{Cov}(Z_t - Z_{t-1})\), we preserve 99.1% of the consonant peaks while shrinking our 318M deepfake detector to 86.8M (3.66× smaller) and shrinking Wav2Vec2-Large ASR from 315M to 142M (2.23× smaller):

Our Transition-Preserving SVD (Static Energy + 2× Frame Change — Keeps 99.1% of Spikes)

Figure 7: Frame-to-frame transition velocity \(|Y_t - Y_{t-1}|\) across three stop-consonant bursts ("p", "t", "k") for the full 315M Wav2Vec2-Large teacher (black dashed curve) versus 2.23× compressed 142M models using Standard SVD/ASVD (0%, red) and Transition-Preserving SVD (100%, green), evaluated across 10,797 locked-test utterances (saved verification JSON).

Complete Benchmark Results & Open-Source Releases

All checkpoints, forward-pass datasets, and the unified audiowsp harness are publicly available:

Table 1: Audio Deepfake Detection Across 15 Benchmarks (2.08M Trials)

Dataset & Paper Split (Trials) Base (W2V2-AASIST, 318M) EER / AUC Published / Leaderboard SOTA (Method, Params, EER / AUC) Did SOTA Train on This Dataset? What Our Forward-Pass Set Used (Dataset) Ours (318M Full) EER / AUC Ours (86.8M Compact) EER / AUC
ASVspoof 2019 LA · [Paper] Eval (71.2K) 0.22% / 0.9997 LaST-Net (317M): 0.09% · Nes2NetX (318M): 0.12% Yes (25.4K train + 24.8K dev) 3.0K train + 600 dev (0 eval) 0.12% / 0.99999 0.16% / 0.99996
ASVspoof 2021 LA · [Paper] Eval (148.2K) 8.11% / 0.9779 XLS-R Conformer: 0.69% Trained on ASV19 LA train 0 clips (ASV19 + PSTN probes) 1.42% / 0.9964 1.51% / 0.9959
ASVspoof 2021 DF · [Paper] Eval (533.9K) 6.63% / 8.32% VeriSpoof-Base (2.85B): 0.97% · XLSR-Mamba (319M): 1.88% Multi-corpus (8 corpora) 0 clips (ASV19/5 + codec probes) 1.98% / 0.9983 2.19% / 0.9980
In-The-Wild (ITW) · [Paper] Full (31.8K) 11.22% / 0.9572 UTCN-XLSR2B (433M): 0.37% · DF-Arena-1B (1B): 0.91% 7-corpus fit + ITW selection 0 clips in forward-pass set 2.51% / 0.9958 2.72% / 0.9951
DECRO (EN + ZH) · [Paper] Eval (37.3K) 9.57% / 0.9583 Spectra-AASIST (316M): 0.17% Undisclosed (Arena) 0 DECRO clips (1.28K CFAD) 1.30% / 0.9986 1.41% / 0.9980
ASVspoof 5 (Track 1) · [Paper] Eval (680.8K) 16.25% / 0.9003 VELMA-2 (316M, Prop.): 0.38% · NSGA-II Fusion (6.6B): 2.37% Yes (ASV5 train + dev) 5.0K train + 3.44K dev (0 eval) 5.73% / 0.9856 5.80% / 0.9857
DFADD · [Paper] Test (3.8K) 11.88% / 0.9436 VeriSpoof-Base (2.85B): 0.00% · XLSR+SLS (340M): 7.54% Yes (full DFADD train) 704 train + 350 dev aliases 0.27% / 0.99995 0.53% / 0.99991
TIMIT-TTS (Clean) · [Paper] Full (20.2K) 2.09% / 0.9954 Standard MoE: 0.93% (0.9512 AUC) No (unseen test) 0 clips (unseen) 0.01% / 0.999998 0.03% / 0.999997
TIMIT-TTS (DTW) · [Paper] Full (20.2K) 10.51% / 0.9535 RawNet2 Baseline (17.6M): 0.7800 AUC No (ASV19-trained) 0 clips (unseen) 0.69% / 0.9995 0.93% / 0.9997
TIMIT-TTS (Augmented) · [Paper] Orig Real (20.2K) 2.14% / 0.9946 RawNet2 Baseline (17.6M): 0.9300 AUC No (ASV19-trained) 0 clips (unseen) 0.46% / 0.9996 1.17% / 0.9994
TIMIT-TTS (DTW + Aug) · [Paper] Orig Real (20.2K) 12.84% / 0.9381 XLS-R 2B + LogReg (2B): 3.60% No (ASV19 train + dev) 0 clips (unseen) 1.85% / 0.9972 1.86% / 0.9983
MULTISpoof · [Paper] Test (43.3K) 14.62% / 0.9214 Asilomar 2026 (no public test table) N/A 1.33K val clips (0 test) 2.94% / 0.9964 3.32% / 0.9955
Fake-or-Real (for-norm) · [Paper] Testing (4.6K) 7.42% / 0.9816 VeriSpoof-Plus (2.85B): 0.09% · XLSR+SLS (340M): 5.08% Yes (full for-norm train) 0 clips (unseen) 3.13% / 0.9953 4.29% / 0.9912
MLAAD · [Paper] Eval (64.0K) 15.40% / 0.9120 AASIST Voiced/Unvoiced: 5.82% · WavLM-LDA: 10.00% Yes (custom MLAAD splits) 664 train + 200 dev aliases 6.84% / 0.9785 7.51% / 0.9730
DiffSSD · [Paper] Test (54.6K) 18.90% / 0.8840 Wiener-Hopf 2D CNN (0.42M): 2.95% · Wav2Vec2 Baseline: 3.00% Yes (full ~70K DiffSSD train) 755 train + 249 val aliases 8.92% / 0.9742 9.77% / 0.9691

Table 2: Automatic Speech Recognition Across All 13 Open ASR Leaderboard Benchmarks (25,929 Utterances)

All 13 evaluation splits below are public Open ASR Leaderboard datasets (`hf-audio/esb-datasets-test-only-sorted`, `hf-audio/open-asr-leaderboard`, and `voice-arena-leaderboard/Monsoon-en-IN-test`). Our forward-pass calibration used strictly the saved public development splits (ishiki-labs/audiowsp-asr-calibration-data: LibriSpeech dev-clean [2,703 utts] + dev-other [2,864 utts] + MLS dev [3,807 utts] + TED-LIUM dev [591 utts], with 0 evaluation test utterances):

Dataset & Official Paper / Repo Eval Split (Utterances) Base (Nemotron-3.5-30B) WER Previous Open ASR Leaderboard #1 (Model & WER) What Our Forward-Pass Set Used (Saved Data) Ours (Single-Model Edited Weights) WER Ours (ishiki-asr-0.5 Full) WER
AMI-Cleaned · [Paper] Test (2,602) 6.14% Cohere-Transcribe-03-2026: 5.66% 0 AMI clips (LibriSpeech dev) 5.97% 5.14%
Earnings22-Cleaned · [Paper] Test (549) 4.19% NVIDIA-Nemotron-3.5-30B: 4.19% 0 Earnings22 clips (LibriSpeech dev) 4.07% 3.54%
GigaSpeech-Cleaned · [Paper] Test (2,346) 4.34% NVIDIA-Nemotron-3.5-30B: 4.34% 0 GigaSpeech clips (LibriSpeech dev) 4.28% 3.55%
SPGISpeech · [Paper] Test (1,036) 1.45% Canary-Qwen-2.5B / Parakeet-TDT: 1.42% 0 SPGISpeech clips (LibriSpeech dev) 1.41% 1.21%
URGENT 2024 · [Paper] Test (1,000) 10.97% StagesLabs-Whisper-Large-v3: 9.45% 0 URGENT24 clips (LibriSpeech dev) 10.72% 8.66%
URGENT 2024 (Clean) · [Paper] Test (1,000) 6.37% FireRedASR2-AED: 4.68% 0 URGENT24 clips (LibriSpeech dev) 5.99% 4.02%
Voice Arena Monsoon (en-IN) · [Repo] Test (527) 3.32% NVIDIA-Nemotron-3.5-30B: 3.32% 0 Monsoon clips (LibriSpeech dev) 3.26% 2.34%
VoxPopuli-AA-Cleaned · [Paper] Test (628) 1.28% NVIDIA-Nemotron-3.5-30B: 1.28% 0 VoxPopuli clips (LibriSpeech dev) 1.23% 0.90%
LibriSpeech (test-clean) · [Paper] Test (2,620) 1.12% MusicAudio-Qwen2-7B: 1.08% dev-clean (2,703) (0 test) 1.10% 0.58%
LibriSpeech (test-other) · [Paper] Test (2,939) 2.07% NVIDIA-Nemotron-3.5-30B: 2.07% dev-other (2,864) (0 test) 2.02% 1.90%
TED-LIUM (test) · [Paper] Test (1,469) 2.45% Cohere-Transcribe-03-2026: 2.28% tedlium-dev (591) (0 test) 2.41% 1.83%
Multilingual LibriSpeech (en) · [Paper] Test (3,769) 3.82% Cohere-Transcribe-03-2026: 3.64% mls-dev (3,807) (0 test) 3.82% 3.10%
Common Voice (en) · [Paper] Test (5,445) 8.61% StagesLabs-Whisper-Large-v3: 6.45% 0 Common Voice clips (LibriSpeech dev) 5.95% 3.75%
Mean WER (10 Public Leaderboard Sets) 25,929 utts 4.59% NVIDIA-Nemotron-3.5-30B: 4.59% (#1) 9,965 public dev utts (0 test) 4.27% 3.18%
Mean WER (8 Main Short-Form Sets) 14,605 utts 4.76% NVIDIA-Nemotron-3.5-30B: 4.76% (#1) 9,965 public dev utts (0 test) 4.62% 3.67%

Citation

If you use our models, forward-pass datasets, or the audiowsp harness in your work, please cite:

Amit Kumar Singh Yadav and Robert Xu. "Weight-Space Programming for Audio Models: LLMs Can Directly Edit Audio Model Weights and Compress Them." Ishiki Labs, November 2026.
@misc{yadav2026audiowsp,
  title        = {{Weight-Space Programming for Audio Models: LLMs Can Directly Edit Audio Model Weights and Compress Them}},
  author       = {Yadav, Amit Kumar Singh and Xu, Robert},
  year         = {2026},
  month        = nov,
  organization = {Ishiki Labs},
  howpublished = {\url{https://github.com/ishikilabsinc/audiowsp}},
  note         = {Research Blog Post and Open-Weights Release}
}