Analysis Overview

This page collects all experimental results, figures, and structural diagnostics from our two-phase study. Phase 1 places embeddings directly in the Poincare ball; Phase 2 keeps embeddings Euclidean and applies geometry only at the output layer. The Poincare GloVe section provides the reference picture of what successful hyperbolic embeddings look like.

Phase 1 - Full Hyperbolic Embedding

Following Nickel & Kiela (2017): embeddings live in the Poincare ball, optimized via RiemannianAdam with a 500-step burn-in phase. Embeddings are mapped to tangent space before the LSTM via logmap0.

Perplexity by Token Frequency Bin

We stratify validation perplexity by token frequency to test the hypothesis that hyperbolic geometry helps with rare tokens. If the curved space encoded a frequency hierarchy, rare tokens near the ball boundary should have better representations.

Raw Numbers

Blue = Euclidean, Pink = Hyperbolic. Bars scaled relative to max (3279).

Embedding Norm Diagnosis

The crucial diagnostic: do embedding norms correlate with token frequency? In Nickel & Kiela's framework, a well-trained hyperbolic embedding places general tokens near the origin (small norm) and specific rare tokens near the boundary (large norm).

Phase 2 - Output-Layer Geometry

Motivated by Moreira et al. (2023): embeddings and LSTM remain fully Euclidean; only the output classification layer uses geometry. This avoids boundary saturation while still testing whether hyperbolic distance structure aids classification. All three variants (Euclidean, Hyperbolic, Spherical) use identical Adam optimizers -- no manifold learning rates or burn-in phases.

Phase 2 Perplexity Results

Phase 2 Architecture Detail

Why Phase 1 Fails: Boundary Saturation

Moreira et al. (2023) give a precise theoretical explanation for the Phase 1 failure, and our diagnostics confirm it exactly.

The Theoretical Argument

In d-dimensional hyperbolic space, the ratio of ball volume to ball surface area is bounded by r/d. As d grows this ratio approaches 0, exactly as in Euclidean space. All volume concentrates at the boundary. Given that cross-entropy loss is unbounded below as embeddings approach the unit ball boundary, the optimizer follows the gradient to the boundary.

Once all embeddings lie at radius r_eff = (1-epsilon)/sqrt(-k), the Poincare distance between any two points u, v with ||u||=||v||=r_eff reduces to a function of the angle between them only. The space is then isometric to a Euclidean sphere -- the radial hierarchy-encoding property is lost entirely.

# From our diagnostics (Phase 1, 15k steps, dim=256):
euclidean_mean_norm  = 2.68   # broad distribution
hyperbolic_mean_norm = 3.20   # saturated at boundary
hyperbolic_std_norm  = 0.05   # extremely tight clustering

# Poincare ball boundary = 1.0 (normalized), but with curvature k=1
# r_eff ~ (1-epsilon) / sqrt(k) ~ 1.0 in Poincare parameterization
# Our embedding norms show saturation in geoopt's non-unit-ball parameterization

# Norm-frequency correlation:
r_euclidean = 0.335   # clear hierarchical structure
r_hyperbolic = 0.045  # no structure

Why Phase 2 Avoids This

Phase 2 never places the embedding table in the Poincare ball. The LSTM hidden state (a 256-dim Euclidean vector) is projected to the ball only at the final classification step, where it is used as a query point for hyperbolic softmax. There is no pressure for this single projection to saturate, since the hidden state itself is regularized by the language modeling task and dropout.

Reference: Poincare GloVe Analysis

From Tifrea et al. (2019). These plots show what successful hyperbolic embeddings look like on full-word vocabulary (190k words, 20D). This is the gold standard our BPE token embeddings should eventually approach.

Norm vs Frequency Rank - Full Words

Distance Structure - Poincare vs Euclidean

Nearest Neighbor Quality (from notebook)

Nearest neighbors in Poincare distance are semantically coherent and capture analogical structure. This is what well-trained hyperbolic embeddings can produce. At the subword BPE level, analogous quality would mean tokens like -ing clustering near general morphological roots, with rare compositional tokens near the boundary.

Center vs Border of the Ball

All Runs

Complete table of all wandb runs with metrics and configurations.

Open Questions and Future Work

1. Is the Token Co-occurrence Graph Hyperbolic?

Tifrea et al. measure the Gromov delta-hyperbolicity of word co-occurrence graphs and find 2*delta_avg/d_avg ratios as low as 0.0034, confirming that word co-occurrence data is genuinely tree-like. This measurement has not been performed for BPE subword tokens. BPE tokens are defined by frequency-based compression, not semantic content, and their co-occurrence structure may differ fundamentally from full-word graphs.

2. Do Phase 2 Embeddings Show Radial Structure?

The Phase 2 models have not yet been subjected to the norm-vs-frequency diagnostic. Since embeddings remain Euclidean, the interesting question is whether the LSTM hidden state, when projected to the Poincare ball, develops radial structure tied to the frequency of the next token being predicted.

3. Overfitting in Phase 2 Hyperbolic

Hyperbolic MLR train PPL (57.7) vs val PPL (117.4) shows a larger train/val gap than Euclidean (88.6 / 120.1). The hyperbolic output layer may have more capacity due to the richer distance structure, but this capacity is not yet being regularized effectively. Investigating dropout rates and weight decay specifically for the hyperbolic output parameters is the immediate next experiment.

4. Low-Dimension Regime

Theory predicts hyperbolic space should show the clearest advantages at low embedding dimensions (e.g., dim=8 or 16), where Euclidean space genuinely struggles to represent hierarchical structure. Our current experiments all use dim=256. Running a dimension sweep (8, 16, 32, 64, 128, 256) with Phase 2 architecture is the most direct test of the theoretical prediction.