How Linguabase Was Built

70+ reference sources. 2.3M supercomputer hours. 130M LLM calls for generation, validation, and ranking. Professional lexicography. Production-tested since 2011.

The Pipeline: Expand, Audit, Contract

Every data layer follows the same pattern:

We evaluate millions of candidate relationships across 400K words, producing over 100 million validated connections.

Development Timeline

Professional Lexicography

Orin Hargraves (professional lexicographer, contributor to major dictionaries) wrote 2,000+ custom definitions and 4,400+ sense-grouped word associations. His work focused on the words that need the most human judgment: interjections, prepositions, the highest-frequency words, and words with so many dictionary senses that their entries are virtually unreadable.

Linguistics grad students and post-docs created 5,000+ thematic word lists over several years:

• Parts of a watch, types of gargoyles, painting supplies
• Cross-domain connections and specialized vocabulary
• Edge cases automated approaches miss: archaic tool names, regional food terms, domain-specific jargon

All contributors are credited on the about page.

Reference Sources

We integrated 70+ linguistic resources—professional thesauri like the NASA Thesaurus, public domain lexicons like WordNet and Roget’s (we explain why these aren’t enough on their own), Library of Congress subject headings, and specialized vocabularies spanning scientific, governmental, artistic, and medical domains. Each source required custom parsing and integration; what works for word games differs from what works for dictionary lookup, and each source required different extraction logic.

Pre-LLM Computational Linguistics

Before LLMs existed, we built the initial network using classical NLP:

• Wikipedia extraction—Link analysis, disambiguation pages, definition summaries
• Latent Dirichlet Allocation—2.3M supercomputer hours (NSF XSEDE grant, 2013–14) for topic clustering across massive corpora
• Word2Vec models—Statistical word embeddings trained on billions of tokens

These methods produced millions of candidate relationships that human curation and later LLM validation refined into what we ship today.

Library of Congress Expansion

We methodically processed all 648,000 Library of Congress subject classifications—capturing the themes of millions of books humans wanted to write in English. Librarians organized these into topics like “orange horticulture” or “Indus Valley civilization.” By analyzing pools of words across these classifications, we discovered semantic clusters that no dictionary or thesaurus would surface. These serve as “idea seeds” for expanding our association network into domains that traditional lexicography misses.

LLM Validation

LLMs generated the foundational data—semantic graphs, sense labels, word families, categories—but none of it was usable as-is. Every output got re-scored, amalgamated with other sources, and re-prompted for judgment calls. The key insight is that LLMs are better validators than generators: an LLM can confirm that “key → reef” is a valid association (it’ll say yes), but it won’t generate that association reliably on its own (see where LLM generation plateaus). So the pipeline generates broadly, then uses LLM scoring to rank, filter, and audit.

False Cognate Removal

False cognates are words that look related but aren’t—they share spelling patterns by coincidence, not common origin. String similarity filters can’t detect them, and LLMs sometimes wrongly confirm they’re related:

Capitalization Intelligence

Since lowercase words are capitalized at the beginning of sentences, LLMs often ignore the difference, mixing up “polish” (verb) with “Polish” (nationality). For semantic networks, this contaminates association lists.

We evaluated capitalization variants for ambiguous terms. Results: 3,509 words have two valid forms, 86 words have three. Each variant gets its own decontaminated association list.

Common Word Coverage

Traditional thesauruses treat function words as “stopwords”—ignored entirely. We put extra effort into building associations for hundreds of the most common words that other sources systematically skip:

Gestalt Enrichment

One source of richness is gestalt associations—sensory, emotional, and cultural connections that taxonomic approaches miss entirely:

These are not synonyms. They’re how humans actually experience concepts. A thesaurus won’t tell you that “crisis” evokes “siren”—but your players know this instantly.

Production Refinement

Linguabase has powered “In Other Words” since 2011. Edge cases have emerged through real gameplay that automated testing alone wouldn’t catch:

• Words that look related but aren’t (false cognates discovered through player confusion)
• Cultural associations that shift over time
• Difficulty calibration based on actual player success rates
• Content filter refinements from user reports

Ongoing Curation

Beyond automated processing, the pipeline incorporates ~80,000 lines of human-authored curation—sense-grouped associations by a professional lexicographer, thematic word lists built by linguistics grad students, hand-written definitions, antonym pairs, and content filters—plus ~670,000 lines of algorithmic corrections and reference data (corpus frequencies, automated morphology, pronunciation data). Each pipeline run incorporates feedback from the previous one.

Why 2M Internally, 400K Deployed?

Internally, we maintain a ranked corpus of 1.8 million terms with 267 million semantic connections. The 400K deployment threshold captures every word a well-read adult English speaker would recognize. Below that line, quality degrades gradually—there’s no clean cutoff, just a long tail where legitimate specialized terms mix with noise.

The bottom of the corpus includes real English that simply isn’t suitable for gameplay: Wadati-Benioff zones (seismology), monomethylarsonic acid (chemistry), q-Pochhammer symbol (mathematics), Struve functions (analysis), Zaydi jurisprudence (Islamic law). These are valid terms with real semantic connections, but no player would recognize them. Mixed in with these are legitimate proper nouns too obscure for games—Pavel Vilikovsky (Slovak novelist), Ghulam Ahmad Mahjoor (Kashmiri poet), Petru Cercel (Wallachian prince)—and sentence fragments, OCR artifacts, and outright gibberish: law of diminishing, same molecular formula, loogie hock, wkurxjk, fkrrvh.

The deeper data still does essential work behind the scenes.

When generating puzzles, every category must be mutually exclusive—no word should plausibly fit two groups. Clearing that conceptual space requires knowing how the full vocabulary interconnects, not just the game-facing slice. “Retinochrome” and “teratosaurid” will never appear in a puzzle—they rank around 1.5 millionth. But the photosensitive pigment connects to light, rhodopsin, retinal; the extinct reptile connects to archosaurs, fossils, carnivores. Those connections inform how we segment semantic space when building categories from the top 400K. The deeper corpus provides the resolution for differentiating concepts, even when most of it never faces the player.

The same depth serves ranking. When a word correlates with thousands of other terms throughout the full corpus—appearing in their association lists, category memberships, sense graphs—that pattern is a familiarity signal. Words that show up everywhere are common; words that cluster in narrow technical domains are specialized. The 400K threshold works because 1.8 million terms were ranked to find it.

Small World Property

Analysis of the Linguabase network shows that 76% of English word pairs connect in 7 hops or fewer. Average path length is 6.43 steps.

Of the 2M headwords, about 870K (57%) are reachable through the top-40 associations of other words. The semantic space is more connected than it looks.

Continuous Improvement

Linguabase is actively maintained, not a static dataset:

• New vocabulary—Words enter mainstream usage constantly (“doomscrolling,” “brain rot,” “situationship”). We add them as they stabilize.
• Quality refinement—Each pipeline run improves grades based on new validation data
• Sense updates—Meanings shift over time; we track semantic drift
• False cognate auditing—Ongoing review catches edge cases

The People Behind It

Fifteen years of work by lexicographers, linguists, data architects, and dozens of vocabulary contributors. See the full credits →