flagship · multimodal embedding
fusion embedding 2
one ~2B open-weight model that puts text, image, video, and audio in a single embedding space.
fusion embedding 2 is a roughly two-billion-parameter multimodal embedding model. it reads text, images, video, and audio and returns them all in one shared 2048-dimensional space, so a raw dot product between any two of them is a valid cross-modal similarity.
it is the base every other model in the family plugs into. the sense packs, the motion head, and the engram memory layer all read and write this same space. against its own frozen base it adds a whole modality, audio, without moving a single weight on the paths that already worked.
how it works
the base is a frozen qwen3-vl-embedding-2b, which already handles text, image, and video natively. audio is grafted on: a frozen qwen2.5-omni-7b audio tower feeds a trained connector that projects sound into the same space, so the model gains a modality without retraining the ones it already had.
on top of that sit modality-gated deep adapters. the adapters are audio-gated: they only engage on the audio path, and the text, image, and video paths are kept bitwise-identical to the base. adding the new modality costs the existing ones nothing, and this adapter design is what separates fe2 from the original resampler graft in fusion embedding 1.
training data
the heavy towers stay frozen. training touches only the audio connector and the gated adapters, learning to fold sound into the space the vision-language base already established rather than rebuilding that space.
keeping the non-audio paths bitwise-preserved means the model's native text, image, and video behaviour is exactly the base model's. the family's guarantee is that a new sense never degrades an old one.
results
the figures below are retrieval recall on held-out audio-text sets. the audio path is strong and, on the robot-memory split, leads a field of standard vision-language encoders, but these are specific benchmarks, not a claim of general state-of-the-art across every audio or video task.
retrieval, recall on held-out sets
| benchmark | score |
|---|---|
| audiocaps audio→text R@10 | 0.743 |
| audiocaps text→audio R@10 | 0.775 |
| clotho text→audio R@10 | 0.482 |
| vggsound audio↔text R@10 | ≈0.665 / 0.681 |
| model | R@1 | R@5 | R@10 |
|---|---|---|---|
| fusion embedding 2 (ours) | 0.043 | 0.153 | 0.215 |
| siglip l/16-384 | 0.033 | 0.110 | 0.178 |
| clip vit-l/14 | 0.028 | 0.106 | 0.185 |
| imagebind-huge | 0.028 | 0.104 | 0.158 |
| chance | 0.010 | 0.051 | 0.101 |
Rewind is our robot-memory retrieval benchmark, still in development. Track V is its Ego4D split: a frozen 60-clip / 634-query set where every model embeds identical frames. Fusion Embedding 2 leads every cell (about 3× chance at R@5). The margin is real but absolute numbers are low by design (one frame per 5s window, short answer spans) and narrow at R@10.
honest limitations
- —audio is grafted through a frozen 7b tower and a connector rather than natively co-trained end to end, which bounds how far the audio path can be pushed.
- —the numbers here are retrieval recall on specific benchmarks; they are not a claim of general state-of-the-art across every audio or video task.
- —on the robot-memory split the lead is consistent but the absolute recall is low by design, and the margin narrows at R@10.
usage
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
"EximiusLabs/fusion-embedding-2-2b-preview",
revision="v0.1-preview",
trust_remote_code=True,
).eval()
# every modality lands in the same 2048-d space (the canonical readout)
text = model.embed_text("a dog barking in the distance")
audio = model.embed_audio("clip.wav")
# vectors are L2-normalized, so cosine similarity is a plain dot product
similarity = (text * audio).sum()citation
@software{eximius_fusion_embedding_2_2026,
author = {Eximius Labs},
title = {Fusion Embedding 2: a unified multimodal embedding
space with modality-gated adapters},
year = {2026},
url = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview},
}how it fits the family
fusion embedding 2 is the anchor. because the space is shared and the base paths are frozen, new senses can be added as composable adapters without touching what already works. that is exactly how ember and k3 vision attach.
it is the successor to fusion embedding 1: same shared-space thesis, but the resampler graft is replaced by gated deep adapters, which is why fe2 is a new family rather than a revision.
engram indexes a robot's video, audio, and motion into this one space, which is what lets a plain-language question reach across modalities.
related models