What Makes Synthetic Hard Negatives Work
in Vision-Language Pretraining?

Overview

TL;DR

Synthetic hard negatives generated directly in the representation space work well for unimodal self-supervised learning, but naively transferring them to CLIP-style vision-language pretraining fails: cross-modal blends fall into the modality gap, and intra-modal blends that touch the positive leak its signal into the denominator. SNAP fixes both at once by generating intra-modal negatives that never involve the positive from either modality, delivering consistent gains on top of CLIP and FLIP with under 10% training overhead.

Abstract

Motivation

Representation-space negatives, two new ways to fail

CLIP-style pretraining samples image–text pairs randomly and pushes apart mismatched pairs via a batch-level InfoNCE loss, without controlling how hard the in-batch negatives actually are. In unimodal self-supervised learning, synthesizing hard negatives directly in the representation space is a fast, lightweight fix for exactly this problem, so the natural question is whether the same trick transfers to vision-language pretraining. It does not, and understanding why reveals constraints that have no unimodal analogue.

Representation space vs. input space. Existing methods for synthesizing hard vision-language data operate in the input space, i.e., LLMs rewriting captions or diffusion models generating images, incurring heavy compute and risking data leakage. SNAP instead synthesizes negatives directly from in-batch embeddings, symmetrically for both modalities.

Geometric intuition. (a) Blending image and text embeddings creates samples that are trivially rejected. (b) Including the positive pair in synthesis creates negatives that contain the actual signal, causing gradient conflict. (c) Generating intra-modal negatives while filtering out the positive anchor respects the modality geometry on both counts.

Our approach

Guided by this geometric analysis, we propose SNAP (Synthetically Negative Augmented Pretraining): a model-agnostic method that generates synthetic hard negatives directly in the representation space for both image and text modalities, applicable on top of any InfoNCE-based framework such as CLIP or FLIP. We characterize the two failure modes with representation-level diagnostics, identify a secondary temperature-saturation effect, and use both to derive the three design choices that define SNAP.

Method

Intra-modal, positive-free, fixed temperature

We assume an image encoder \(f_\theta\) and text encoder \(g_\phi\) producing \(\ell_2\)-normalized embeddings \(\mathbf{v}_i, \mathbf{t}_i\) in a shared \(d\)-dimensional space. For a batch \(\mathcal{B}\), the standard bidirectional CLIP objective is:

Augmenting the denominator

For each image query \(\mathbf{v}_i\), we generate a set of synthetic text negatives \(\mathcal{S}_{\text{txt}}^i\) and add them to the image-to-text denominator (and symmetrically, synthetic image negatives \(\mathcal{S}_{\text{img}}^i\) to the text-to-image denominator):

Architecture, image-to-text direction. Standard CLIP computes the similarity matrix \(\mathbf{V}^\top\mathbf{T}\) using only batch negatives (white cells) and the positive pair (green). SNAP concatenates synthetic text negatives \(\tilde{\mathbf{T}}\) (purple cells) onto the batch, forming an augmented similarity matrix at negligible extra cost. The text-to-image direction follows the analogous procedure.

Six candidate generation strategies

Hardness is defined by cosine similarity to the query: given image query \(\mathbf{v}_i\), we select the top-\(N\) batch text negatives with highest similarity to \(\mathbf{v}_i\) as a hard-negative pool, then synthesize from it:

Cross-modal vs. intra-modal, all strategies vs. the survivors

Applying all six strategies cross-modally degrades well below the CLIP baseline. Converting to intra-modal mode helps but still underperforms, because strategies \(s{=}1,2\) now leak the positive instead. Restricting to \(s{=}3,4\) in intra-modal mode nearly recovers the baseline outright, and surpasses it once the temperature is fixed (next).

All ablations on CC3M, ViT-B/16, with a learnable temperature, the remaining gap closes once \(\tau\) is fixed (below).

Pairwise strategy complementarity. Across every pairing, \(s{=}3\) and \(s{=}4\) consistently outperform any combination that involves cross-modal blending or the positive pair.

Geometric diagnostics

We quantify the failure modes directly: positive leakage as cosine similarity to the true positive (random baseline \(0.19\)), and trivially-separable rate (TSR) as the fraction of synthetics easier than the hardest 10% of real negatives.

Only \(s{=}3,4\) avoid both cross-modal mixing and explicit positive inclusion: leakage at or below the random baseline, low TSR, and positive relative hardness.

Temperature divergence

Adding synthetic negatives changes the contrastive denominator's interaction with a learnable temperature. With the positive-free intra-modal strategies active, CLIP+SNAP's logit scale saturates the OpenCLIP clamp (100) within half an epoch, over-sharpening the softmax and suppressing gradient from exactly the hard negatives we added. Fixing \(\tau = 0.07\) removes this failure mode entirely.

Logit-scale trajectories (ViT-B/32, CC3M, epochs 5–10). CLIP+SNAP with learnable \(\tau\) saturates the clamp almost immediately; both fixed-\(\tau\) runs stay stable.

SNAP: three design choices

• s=3,4 Intra-modal, positive-free strategies only, avoiding both cross-modal geometric collapse and intra-modal positive leakage.
• τ=0.07 Fixed temperature rather than learned, preventing the logit-scale saturation that synthetic hard negatives otherwise trigger.
• i2t + t2i Bidirectional synthesis for both image and text queries, which outperforms either direction alone or repeating/reweighting existing negatives as a control.

Computational overhead

Since SNAP generates negatives via vector additions, scalar multiplications, and normalization on already-computed embeddings, it adds no extra forward or backward pass through the encoders.

Results

Consistent gains across backbones, scales, and tasks

We train CLIP and FLIP baselines and their SNAP variants under matched data and training budgets, across three backbones (ViT-B/16, ViT-B/32, RN-50), two pretraining scales (CC3M, CC12M), and three downstream families: zero-shot retrieval (Flickr30k, MSCOCO), zero-shot classification (ImageNet and 10 transfer datasets), and linear probing. For CC3M we pretrain three seeds and report the mean.

Representative results

A small sample of the full results across Flickr30k/MSCOCO retrieval, 11-way zero-shot classification, and linear probing, see the paper for the complete tables. Gains appear in the large majority of settings for both CLIP and FLIP, holding across all three backbones; the few exceptions are isolated cells unchanged or lower by at most a few tenths of a point.

Discussion & Limitations

A dual-modality constraint, not a CLIP-specific one

The failure modes we identify arise from the dual-modality structure of contrastive learning rather than from any CLIP-specific architectural choice, so we expect SNAP's design principles to be relevant to other dual-encoder frameworks that use representation-space synthetic negatives. Our evaluation is limited to CC3M and CC12M due to computational constraints; scaling to larger datasets such as LAION-400M remains future work.

Citation

BibTeX

@misc{giakoumoglou2026snap,
      title={{What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?}},
      author={Nikos Giakoumoglou and Paschalis Giakoumoglou and Andreas Floros and Kleanthis Marios Papadopoulos and Tania Stathaki},
      year={2026},
      eprint={2610.09700},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.09700},
}