Menu

[R] Taking EmbeddingGemma-2 (577M) from Rank #26 to #1 on the MAEB audio benchmark by filling its 256d null space with PCA-reduced intermediate Conformer layers (zero training) [R]

Reading 0:00
15s threshold

Model & weights: BrunoHays/embeddinggemma-2-nullspace-audio

I wanted to share an experiment on google/embeddinggemma-2 (577M) for MAEB, the 30-task audio benchmark on the MTEB leaderboard . By projecting intermediate Conformer layers into the unused null space of the model's final projection matrix using only PCA, the model's macro score jumps from 49.3% to 54.4% (+5.1 points) , moving from Rank #26 to #1 on MAEB audio-only (and #16 to #1 on full MAEB ), ahead of 8B-9B multimodal models.

1. Last-layer filtering in text-aligned audio encoders

Across the MAEB leaderboard, LLM-backed Omni embedders score 95-99% on multilingual text-to-audio retrieval, but drop to 1%-6% on unsupervised audio clustering and ~20% on vocal emotion ( CREMA-D ). Aligning an audio encoder to a text LLM filters the final layers: lexical content ( what was said ) is kept, while paralinguistic and acoustic traits ( timbre, pitch, emotion ) are discarded.

2. Probing the 12 Conformer layers

Layer-wise speech studies ( Pasad et al.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.