Matryoshka Embeddings: Shorter Vectors in text-embedding-3
OpenAI's text-embedding-3 models, released on 25 January 2024, support shortening vectors via a dimensions parameter. We explain the technique behind it — Matryoshka Representation Learning — verify the benchmark numbers, calculate storage savings at scale, and show why migrating from ada-002 still means re-embedding the entire corpus.
Why Embedding Dimensions Became a Cost Problem
Every retrieval system built on embeddings stores one vector per chunk of text. The dimension of that vector is fixed by the model. text-embedding-ada-002, OpenAI's standard since December 2022, produces 1536 dimensions — 6,144 bytes per vector in float32. At one hundred million chunks that is 614 GB of raw vectors, before any index structure. HNSW-style indexes keep vectors in RAM, so dimension count drives memory cost, query latency, and hardware sizing in roughly linear fashion.
Until January the dimension was not negotiable. Post-hoc reduction such as PCA exists, but it adds a fitted transformation that must be versioned and applied consistently to every query, and it degrades retrieval quality in ways that are hard to bound in advance. Most teams simply paid for 1536 dimensions whether their workload needed them or not. The cost surfaced elsewhere: larger indexes, more RAM, slower distance computations.
What OpenAI Shipped on 25 January
On 25 January 2024 OpenAI released two embedding models. text-embedding-3-small produces 1536 dimensions and costs $0.00002 per 1,000 tokens — five times cheaper than ada-002 at $0.0001. Its MTEB average rises from 61.0 to 62.3, and the multilingual MIRACL average from 31.4 to 44.0. text-embedding-3-large produces up to 3072 dimensions at $0.00013 per 1,000 tokens, scoring 64.6 on MTEB and 54.9 on MIRACL.
The structurally interesting feature is not the benchmark delta but the new dimensions API parameter. Both models can return shortened vectors, and OpenAI states they were trained with a technique that lets developers trade off performance against cost. The announcement barely names it. The method behind it is Matryoshka Representation Learning, published in May 2022 — long before the launch.
Matryoshka Representation Learning in Brief
MRL was published by Kusupati et al. (arXiv:2205.13147) and presented at NeurIPS 2022. The problem it solves: conventional training diffuses information across all dimensions of an embedding, so truncating a vector destroys its geometry. MRL adds training losses on nested prefixes of the vector — typically halving steps such as 64, 128, 256 up to the full dimension — so that the first m dimensions already form a usable embedding on their own.
The paper reports embeddings up to 14 times smaller at the same ImageNet-1K classification accuracy, with no additional cost at inference time. Information also interpolates: dimensions between the explicitly trained sizes remain meaningful. The structure is coarse-to-fine, like the nested dolls the method is named after — early dimensions carry the most general semantics, later dimensions add successively finer detail.
Shortening Vectors With Minimal Loss
With text-embedding-3 you either pass the dimensions parameter in the API call or truncate the full vector yourself. If you truncate, re-normalize the result to unit length: OpenAI's vectors ship normalized, and indexes that use the inner product as a cosine substitute rely on that property. According to OpenAI's published figures, text-embedding-3-large shortened to 256 dimensions still scores 62.0 on MTEB — above the full 1536-dimensional ada-002 at 61.0 and with twelve times fewer floats per vector.
Be clear about what this does not do. Token pricing is unchanged: you pay per input token regardless of output size. Shortening does not make different model spaces comparable. And the loss is small, not zero: 64.6 at 3072 dimensions versus 62.0 at 256. Whether that gap matters is a property of your retrieval task, not of the benchmark.
The Storage Math at Scale
Float32 storage costs 4 bytes per dimension, and at corpus scale the arithmetic is unforgiving. The table combines OpenAI's published MTEB averages with raw vector storage for a corpus of 100 million chunks. Index overhead, replicas, and backups come on top of these numbers in every case.
Memory is usually the binding constraint, not disk. An HNSW index over 3072-dimensional float32 vectors needs the full 1.23 TB in RAM plus graph overhead; at 256 dimensions the same corpus fits in 102 GB. The MRL paper also demonstrates adaptive retrieval: shortlist candidates with a short prefix, then re-rank the shortlist with full vectors — reporting up to 14x wall-clock speedups at comparable accuracy.
| Model | Dimensions | MTEB avg | Bytes per vector (float32) | Storage for 100M vectors |
|---|---|---|---|---|
| text-embedding-ada-002 | 1536 | 61.0 | 6,144 | 614 GB |
| text-embedding-3-small | 1536 | 62.3 | 6,144 | 614 GB |
| text-embedding-3-small | 512 | 61.6 | 2,048 | 205 GB |
| text-embedding-3-large | 3072 | 64.6 | 12,288 | 1.23 TB |
| text-embedding-3-large | 1024 | 64.1 | 4,096 | 410 GB |
| text-embedding-3-large | 256 | 62.0 | 1,024 | 102 GB |
Migration Means Full Re-Indexing
Embedding spaces of different models are mutually incompatible. A query embedded with text-embedding-3-small cannot search documents embedded with ada-002, and shortened vectors of the new models do not become comparable to old ones either; there is no conversion function between spaces. Adopting the new models therefore means re-embedding every chunk in the corpus and rebuilding every index. There is no shortcut — and the two new models span different spaces even from each other.
The API bill is the small part: one billion tokens cost $20 with 3-small and $130 with 3-large. The real cost is operational — batch pipelines running against rate limits, a dual-write phase if the system must stay online, and a retrieval evaluation on your own data before switching over. One practical consequence of MRL: store the full 3072 dimensions once, and you can derive shorter indexes later by truncation without calling the API again.
Measure before you migrate. A few hundred annotated query-document pairs from your own domain say more than any leaderboard. Run them against the old index and the new one, at full and shortened dimensions, and let the deltas decide the configuration. In our projects this evaluation set is the most reusable artifact of a migration.
Outlook From February 2024
We expect Matryoshka training to become a standard property of embedding models rather than an OpenAI differentiator. The open-source side moved within three weeks: Nomic released nomic-embed-text-v1.5 on 14 February 2024, trained with MRL and resizable from 768 down to 64 dimensions, with open weights and open training data. At 256 dimensions it scores 61.04 on MTEB — less than a point below its full size.
Two developments seem likely from here. Vector databases will add native support for multi-resolution indexes and truncation at query time. And MRL will be combined with scalar and binary quantization, which attacks the 4 bytes per dimension rather than the dimension count — the savings multiply. Storage will stop dictating model choice. What remains is the part no launch post can do for you: evaluating retrieval quality on your own corpus.
Sources
- OpenAI: New embedding models and API updates (25 January 2024)
- Kusupati et al.: Matryoshka Representation Learning, arXiv:2205.13147, NeurIPS 2022 (26 May 2022)
- Muennighoff et al.: MTEB — Massive Text Embedding Benchmark, arXiv:2210.07316 (13 October 2022)
- Nussbaum et al.: Nomic Embed — Training a Reproducible Long Context Text Embedder, arXiv:2402.01613 (2 February 2024)
- Nomic AI: nomic-embed-text-v1.5 model card (14 February 2024)