The Soft Atlas

The map

Confidence made visible

Same UMAP coordinates as the Atlas, colored by EM best-fit theme — but opacity is membership confidence: solid dots are certain, faint dots are the in-between companies. Hover for each company's actual mixture, or filter to just the companies that cross categories.

Do they trade together?

Test a membership against the tape

A soft membership is a hypothesis: these companies are versions of the same business. The market gets a vote. Pick any company, choose peers from its membership clusters, and chart them — indexed prices plus the daily return correlation of each peer to your company, both raw and with the S&P 500's move regressed out. Residual correlation near zero means the shared label is descriptive, not tradeable. We ran this idea systematically as a five-year, 100-name pairs backtest: Trading the Co-Movement →

The method

K-means with the confidence kept

Spherical k-means is the infinite-concentration limit of a von Mises-Fisher mixture — the same model with every membership probability forced to 0 or 1. Running actual EM relaxes exactly that:

# k-means is this model with kappa forced to infinity (hard boxes).
# EM keeps the confidence: mixture of von Mises-Fisher on the unit sphere.
for step in range(200):                       # converges in seconds, CPU only
    # E: posterior membership of every company in every theme
    R = softmax(X @ (kappa[:,None] * mu).T + log C(kappa) + log pi)
    # M: refit each theme from ALL companies, weighted by membership
    mu    = normalize(R.T @ X)
    kappa = rbar * (d - rbar**2) / (1 - rbar**2)   # fitted tightness
    pi    = R.mean(axis=0)
# honest footnote: fitted in a 64-dim PCA subspace - in the raw 1024 dims the
# concentrations explode and every posterior collapses to 0/1 (k-means again)

Three things fall out that hard clustering cannot produce: per-company membership mixtures, per-cluster fitted tightness (κ), and a principled measure of which assignments were never confident to begin with.

Finding 1

Most companies really are one thing

The median company's membership is essentially certain (median entropy 0.00) and 81% of companies keep their Atlas cluster under EM. The hard clustering wasn't wrong for the typical company — it was wrong precisely for the 855 companies that move and the handful below that were always coin-flips. EM's value is knowing which is which.

Finding 2

Cluster tightness, measured

κ is each theme's fitted concentration: how tightly its members hug the center. The extremes tell the story — SPAC Vehicles (κ=158.6) is the tightest family on the map because its members are nearly identical businesses, while Maritime Freight Transporters (κ=39.0) is the loosest: less a family than a neighborhood.

Finding 3

The companies that live between themes

The highest-entropy names — the ones any single label misfiles. These are the conglomerates, the transition stories, and the businesses the market itself struggles to categorize (and, we'd argue, to price):

Finding 4

Who moves under EM

855 companies get a different best-fit theme once tightness is fitted — mostly members of loose clusters being claimed by tighter neighbors. The most confident moves:

Mixture of von Mises-Fisher distributions fitted by EM in a 64-dim PCA subspace of the bge-m3 embeddings (raw 1024-dim fits collapse to hard assignments - the honest reason most published "soft" clusterings of text embeddings aren't). Initialized from the Atlas partition so components inherit its LLM-written labels. Educational; not investment advice.