Hyper3-CLIP: hierarchy-conditioned hyperbolic vision-language training
Query-conditioned visual pooling and hierarchical training in hyperbolic space.
An image rarely has just one useful description. A scene can be described as a whole, as an object, or as a part. The corresponding text may be a caption, a sentence fragment, or a short phrase. Most vision-language dual encoders still compress the image and text into one global vector each.
Hyper3-CLIP asks whether those levels can be made explicit during training without making inference more expensive.
The paper was accepted as an oral presentation at the ECCV 2026 Beyond Euclidean workshop, scheduled for September 8, 2026, in Malmö.
Read the paper (PDF) · Code · Model
What the model changes
Hyper3-CLIP builds a lightweight hierarchy from each training caption: the full caption, sentence fragments, localized descriptions, and extracted phrases. Each text query attends to the image's patch tokens and produces its own query-conditioned visual node.
The visual and text nodes are then trained together in hyperbolic space. The objective represents image-text alignment alongside part-whole and parent-child relations. Query-conditioned pooling is used only during training, so inference keeps the cost and interface of a standard dual encoder.
What we found
In the ViT-B experiments reported in the paper, Hyper3-CLIP improves all eight COCO and Flickr R@5/R@10 retrieval measurements over the inherited UNCHA setup. Under a shared evaluator, multi-label mAP increases from 75.51 to 78.20 on VOC and from 50.93 to 54.28 on COCO.
The controlled ablation is the more useful result. The full query-conditioned objective raises caption-hierarchy average precision by 3.64 points while leaving average retrieval essentially unchanged. Mean-pooled visual nodes, reversed visual ordering, and shuffled text-parent links remove much of that gain.
The results are not a universal win. ImageNet hierarchy metrics remain competitive with, rather than better than, UNCHA. Zero-shot classification is also unusually sensitive to prompt wording on Food-101, CUB, and Flowers. The paper reports that failure directly.
What is new
The contribution is the combination of two existing ideas: query-conditioned visual pooling and hyperbolic entailment training. Hyper3-CLIP gives each text query its own visual node and connects those nodes through explicit hierarchy relations. It does not introduce a new objective family, and the paper separates the added mechanism from the parts inherited from UNCHA, HyCoCLIP, and related work.
Authors
Matin Mahmood; Antonio Rueda-Toicen; Mohamed ElBassat; Seifeldin Elkerdany; Weixing Wang; Gerard de Melo.
Matin is affiliated with hyper³labs. The other affiliations are listed in the paper.