foundation · first model
fusion embedding 1
the first fusion embedding: a frozen VLM base extended with an audio pathway, proving one shared space could hold a new modality.
fusion embedding 1 is where the family starts. it grafted an audio pathway onto a frozen qwen3-vl-embedding-2b, and in doing so put text, image, video, and audio into one 2048-dimensional space for the first time in the family.
it is the proof of the whole thesis: that a frozen vision-language base can be extended with an entirely new modality without rebuilding the space it already had. it is superseded by fusion embedding 2, which replaced the resampler with gated deep adapters, but it remains the foundation everything since is built on.
how it works
audio is injected into the base, not bolted beside it. sound passes through a frozen qwen2.5-omni audio tower; a trained resampler maps those features into the base's own token space; and the frozen qwen3-vl base then produces the embedding just as it does for text, image, and video.
that is the visual difference from fusion embedding 2: fe1 feeds audio through a resampler into the base, while fe2 attaches modality-gated deep adapters. same shared-space goal, two different ways to reach it.
training data
trained on a roughly 484k-clip audio corpus with native-resolution targets and a train-time embedding dimension of 384. only the resampler learns; both the qwen3-vl base and the qwen2.5-omni audio tower stay frozen throughout.
the model shipped iteratively across v0.1, v0.2, and v0.3, each a retrieval-tuned step on the last. the line is final at v0.3.
results
v0.3 is the final checkpoint of the line and the first fusion model to clear its own base on audio: audiocaps audio→text reaches R@10 0.741 and R@1 0.332, produced by a further audiocaps fine-tune on v0.2. clotho zero-shot sits around 0.448, a real gap to specialist audio retrievers, and exactly the gap fe2's adapter design was built to close.
audio retrieval (v0.3 flagship), held-out recall
| benchmark | score |
|---|---|
| audiocaps audio→text R@10 | 0.741 |
| audiocaps audio→text R@1 | 0.332 |
| clotho zero-shot text→audio | ≈0.448 |
honest limitations
- —superseded by fusion embedding 2; kept as the family's foundation and archival reference, not the recommended model.
- —audio is injected through a resampler into a frozen base, which bounds the audio path; clotho zero-shot shows a real gap to specialist audio retrievers.
- —the base's text, image, and video behaviour is inherited unchanged; fe1's contribution is the audio modality, not new vision-language performance.
usage
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
"EximiusLabs/fusion-embedding-1-2b-preview",
revision="v0.3",
trust_remote_code=True,
).eval()
# text, image, video, and audio all read out into one 2048-d space
text = model.embed_text("rain on a tin roof")
audio = model.embed_audio("clip.wav")
# L2-normalized vectors -> cosine similarity is a dot product
similarity = (text * audio).sum()citation
@software{eximius_fusion_embedding_1_2026,
author = {Eximius Labs},
title = {Fusion Embedding 1: extending a frozen vision-language
base with an audio pathway},
year = {2026},
url = {https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview},
}how it fits the family
fusion embedding 1 is the origin: the first time the family held text, image, video, and audio in one shared space, and the checkpoint that proved the frozen-base approach works.
fusion embedding 2 is its direct successor: it keeps the shared-space thesis and swaps the resampler graft for gated deep adapters, which is why fe2 is a new family rather than a point release. every sense pack and engram itself descend from what fe1 established.
related models