Hyperbolic Semantic Binding

How current text conditioning for image generation works:

• Step 1: Runs a prompt “a dog with hat and cat with sunglasses” through CLIP to get a matrix of text embeddings with size = number of words x embedding dimension per word

• Step 2: Run cross-attention between that text matrix with matrix of spatial patches to obtain which word each patch should express.

The problem is, inter-word correlation is weakly expressed. For example, “a Dog with Hat and Cat with Sunglasses”, implies that the Hat should be worn by Dog, and Sunglasses worn by Cat. But that binding is often lost.

You get stuff like, A Dog with Sunglasses and a Cat with Hat (swapped), or both get hats, or the hat doesn’t exist at all, or any combination. It’s quite stochastic.

People have found that the cross-attention map dictates what gets expressed, so they try to “fix” the cross attention mapping. And it works, a little, but it’s treating the symptom, not the cause.

We can instead look upstream. You would find that binding is poorly encoded in the matrix of text embeddings is weak. This happens before cross attention even happens.

The reason binding is weak, is because CLIP operates in Euclidean space, and that space is not suitable for expressing hierarchies in sentences (like hat *belongs* to dog).

We know that embedding space encodes meaning as directions: the famous vector addition (king - man + woman = queen). So, the natural solution is to try adding “dog” to “hat” -> dog+hat, and “cat” to “sunglasses” -> cat+sunglasses. And that gets you somewhere, but somehow it’s not good.

So, why isn’t it good?

In euclidean space, you simply cannot fit a increasing hierarchical without either compressing parent-child distance or sibling-sibling distance. (because trees grow exponentially, euclidean space only linearly). (I formalize this argument in Lemma 1 and Lemma 2 in paper)

We can visualize this with various prompts, and the embeddings are not well separated. Which means, distance is distorted. Which mean, the meaning of addition (dog+hat) is not well preserved.

What we can do is a transformation that “decompresses”. Like blowing up a wrinkly balloon.

In literature, hyperbolic space does something very close to that. It expands out Euclidean embeddings such that distance is “undistorted”. Other people have thus trained embeddings in Hyperbolic space and found it to preserve such distances better. I find (much to my surprise) that even without training, simply with a transformation post-training, is enough to “decompress” distance distortions, albeit not perfect.

And with that insight, we build the system:

We can transform CLIP embeddings of “a dog with hat and cat with sunglasses” into Hyperbolic Space. Then, We perform three tricks in that space.

• First, möbius addition —> (dog + hat in hyperbolic space)

• Second, contrast learning —> pull dog and hat closer, push cat and hat further, but using hyperbolic distances.

• Third, one round of entropy minimization —> improves saliency of already salient features, further locks in geometry from our previous steps

Basically, do all the same tricks, but in hyperbolic space.

Experimental results are in the paper (beating benchmarks, various case studies, etc.); the usual suspects. (paper to be linked later)

And interestingly, the longer the prompt, the more distorted the output, the better my mechanism works to salvage the situation.