How Linguabase Works

The Architecture

We maintain a full 1.5M-word backend and deploy a curated 400K-word subset for production — every word players actually want, without the noise.

Semantic closeness is impossible to precisely quantify — but the connection weights serve as an effective proxy. Lower weights indicate either lower confidence or more oblique/distant relationships. Higher weights indicate stronger, more immediate associations.

Why Not Just Use an LLM?

You could prompt ChatGPT for word associations. For a single query, it works fine. But at scale, you’ll hit problems:

• Flat experience in aggregate — LLMs cluster on expected associations from training data. Ask for 1,000 different words and you’ll get similar patterns: the dominant sense, the obvious connections. Your users will notice the sameness.
• Sense imbalance — “Bank” returns 45 financial terms and 5 river terms. The pool table sense? Aviation banking? Rarely represented.
• No graded difficulty — LLMs don’t distinguish “elephant → trunk” (strong, obvious) from “elephant → Ganesh” (weaker, interesting). Every connection looks the same.
• Edge cases everywhere — Capitalization (polish vs Polish), false cognates (dig → digress), superconnector words that dominate everything, function words with no associations. Each requires special handling.
• No curation layer — 750K+ manual overrides addressing awkward phrasings, wrong connections, and special cases. You’d need to build this yourself.

Linguabase isn’t a replacement for LLMs — we use them extensively for ranking and validation. But LLMs are better editors than authors. Our data provides the diverse, sense-balanced, edge-case-handled foundation that makes LLM-based applications feel polished rather than generic.

Vocabulary Depth: Why 400K?

How many words are in English? It’s nearly impossible to count, even with clear policies on British vs. American spelling, proper nouns, accent variants, and punctuation. Every dictionary draws the line differently.

For comparison:

• NASPA Word List (Scrabble, US): 176K words
• Collins Scrabble Words (international): 267K words
• Linguabase deployed: ~400K words
• Linguabase full: 1.5M words

But what counts as “one word”? Consider these cases where reasonable people disagree:

Linguabase includes all common variants as separate entries. This is arguably more useful for a semantic graph — each form can have its own association cloud reflecting how it’s actually used — but it means our word counts are higher than dictionaries that merge variants under a single headword.

Why maintain 1.5M but deploy 400K? Two reasons. First, we’re continually improving — new words enter English constantly, and the pipeline keeps refining. Second, words form a network. Even terms at rank 1.2M contribute signal that helps rank everything else, the same way PageRank uses links from obscure pages to determine which pages matter.

But measuring “usefulness” for ranking is harder than it sounds. Objective measures like pure frequency would rank “the” and “of” highest while missing that “red panda” is easily understood by most English speakers. PageRank-style algorithms surface superconnectors rather than familiar terms.

The Linguabase word ranking is an effective proxy for what words are known to a broad user base, including English language learners. The highest ranks align closely with vocabulary standards like the Oxford 3000 and CEFR levels (A1–B2) — but enhanced with common derivatives. Where Oxford 3000 lists “actor,” our ranking includes both “actor” and “actors.”

Here’s what you find at different ranks:

For “In Other Words,” we thresholded at ~400K. This includes every common word any player might want, plus enough depth for interesting discoveries. Different applications might threshold differently — a medical app might want terms at rank 300K that a word game would skip.

Superconnector Demotion

Some terms appear too often. “Heritage” appears 110,000+ times across our sources. “Unique” appears 168,000 times. Without intervention, these “superconnector” words would dominate every association list.

Our solution applies inverse-frequency weighting — a cousin of TF-IDF:

If “heritage” ranks #3 in a word’s association list, after Tier 8 demotion it becomes #15 — pushed out of top results. Demotions make sure superconnectors don’t dominate.

The Data Pipeline

This isn’t “GPT said these words are related.” Linguabase is built on a decade of pre-LLM work, now enhanced with LLM validation. The pipeline runs 28 hours end-to-end.

Phase 1: Human Sources (Pre-LLM)

Years of aggregating professional and curated linguistic resources:

• Leading dictionaries — We manually consulted unprotectable facts from Oxford, Merriam-Webster, Collins, Chambers, and Macquarie to identify consistency patterns (e.g., when multiple sources agreed on the same associations)
• Professional thesauri — NASA Thesaurus, Getty (Art & Architecture, Geographic Names, Artist Names), Library of Congress Subject Headings, AGROVOC, NLM Metathesaurus, DTIC Defense Thesaurus
• Public domain — WordNet, Moby Thesaurus, GNU GCIDE, Roget’s, Wiktionary
• Semi-curated — Graduate student linguistic work, freelancer-aggregated glossaries from hundreds of specialized domains

Total: 70+ reference sources cross-referenced to validate factual word relationships.

Phase 2: In-House Custom Lists (Pre-LLM)

5,000+ thematic word lists created in-house by lexicographer Orin Hargraves:

• Parts of a watch, types of gargoyles, architectural terms
• Regional vocabulary, specialized jargon, cross-domain connections
• Years of professional lexicographic curation

Phase 3: Machine-Processed Sources (Pre-LLM)

Computational linguistics before LLMs existed:

• Wikipedia extraction — Link analysis, disambiguation, definition summaries
• Latent Dirichlet Allocation — 2.3M supercomputer hours (NSF XSEDE grant, 2013-14) for topic clustering
• Word2Vec models — Statistical word embeddings
• PPDB — Paraphrase Database
• Crossword databases — 6M+ clue-answer pairs from xd.saul.pw
• VerbNet 3.2 — Largest English verb lexicon

Phase 4: LLM-Era Enhancement (2023-present)

All the above was amalgamated, then enhanced with modern LLMs:

• Library of Congress expansion — 648K classifications from 17M books as “idea seeds”
• SWOW human associations — Real experimental data from 12K headwords
• Etymological extraction — 1.36M GPT-extracted word origins
• Gestalt relations — Sensory/experiential associations (crisis → siren)
• LLM validation — 130M API calls for ranking and auditing, not generation

The key insight: LLMs are better at recognizing valid relationships than generating them. We provide candidates from a decade of diverse sources, then use LLMs to evaluate, rank, and audit.

Manual Curation Layer

Beyond automated processing, 750,000+ hand-curated override lines address what algorithms miss:

• Special-case relationship corrections
• Definition refinements for awkward or robotic phrasing
• Pronunciation and syllable data
• Word family edge cases

This is the work that separates production-quality data from raw output.

Common Word Coverage

Traditional thesauruses treat function words as “stopwords” — ignored entirely. Linguabase does the opposite: we put extra effort into building associations for hundreds of the most common words that other sources systematically skip.

These words are the glue of language. When your game or AI needs to understand how “and” connects to other concepts, we have answers that no traditional thesaurus provides.

Gestalt Enrichment

One source of richness in Linguabase is gestalt relations — experiential and sensory associations that taxonomic approaches miss entirely. These aren’t delivered as a separate data layer; they’re folded into the core associations, enriching what you get for each word.

These are NOT synonyms. They’re how humans actually experience concepts. A thesaurus won’t tell you that “crisis” evokes “siren” — but humans know this instantly. By incorporating these into the association pipeline, Linguabase captures connections that feel natural to users even when they’re invisible to purely lexical approaches.

False Cognate Removal

False cognates are words that look related but aren’t — they share spelling patterns by coincidence, not common origin. String similarity filters can’t detect them:

Our solution: Opus 4.5 audited ALL 385K headwords for false cognates.

Capitalization Intelligence

Capitalization creates genuine meaning distinctions that most systems ignore:

LLMs often treat text case-insensitively, conflating “polish” (verb) with “Polish” (adjective/noun for nationality). For semantic graphs, this contaminates association lists — “Poland” shouldn’t appear in the associations for shoe polish.

Our solution: GPT-4.1 evaluated all 2ⁿ capitalization variants for 26K ambiguous terms, attempting natural sentences with each form. Results:

• 3,509 words have two valid capitalization variants (like turkey/Turkey)
• 86 words have three variants (like swat/Swat/SWAT)
• Each variant gets its own decontaminated association list

Variant vs Semantic Distinction

Word families serve two different purposes:

Why this matters:

• “butterfly → butterflies” = variant swap (same concept, just plural)
• “butterfly → fly” = semantic hop (different concept via shared morpheme)

But not everything that looks splittable should be split. “Mushroom” contains “room” and “mush,” but it’s not a compound of those words — it comes from French mousseron. String patterns alone can’t distinguish real morphological relationships from coincidental letter sequences. This is why the etymological audit matters.

Results: 891K variant words, 2.75M semantic words — properly separated.

Example: “elephant”

Core Associations

Sense Labels

Sense Clouds

Word Family

Graph Operations

Because the data is structured as a weighted graph, you can perform operations that flat synonym lists cannot support:

Pathfinding

Find how “sugar” connects to “peace”:

Semantic Distance

How close are two concepts?

Convergence

Given a target word, which intermediate words get you closer?

Content Blocklists

For applications with content policies, Linguabase provides two levels of filtering:

Hard Block (Offensive Terms)

Colorful derivations of sexual and racial slurs that most applications will want to exclude entirely — particularly for user-facing features like sharing, leaderboards, or public display. These words exist in dictionaries but have no legitimate use in puzzles or public contexts.

Soft Block (Suggestive Terms)

Words that are technically inoffensive but may be inappropriate for automated puzzle generation:

• Contaminated by sound — “niggardly” (meaning stingy, from Old Norse) is unrelated to any slur, but its phonetic similarity makes it unusable in modern contexts
• Multiple senses — “knockers,” “cock,” “balls” have innocent meanings (door hardware, rooster, spheres) but also sexual ones
• Context-dependent — “anal,” “threesome” may be fine in medical or numerical contexts but problematic in games

Developers can use soft-block lists to exclude these words from automated puzzle generation while still allowing them as valid user-entered answers. The distinction: a puzzle shouldn’t randomly present “cock” as a target word, but if a player types it as a valid answer for “rooster,” the game should accept it.

Continuous Improvement

Linguabase is actively maintained, not a static dataset:

• New vocabulary — Words enter mainstream usage constantly (“doomscrolling,” “finsta,” “situationship”). We add them as they stabilize.
• Quality refinement — Each pipeline run improves grades based on new validation data
• Sense updates — Meanings shift over time; we track semantic drift
• False cognate auditing — Ongoing review catches edge cases

License terms include current data with update arrangements available.

Small World Property

Analysis of the graph shows that 76% of English word pairs connect in 7 hops or fewer. Average path length is 6.43 steps.

The hop-distance distribution peaks at 5-6 hops:

Of the 1.5M headwords, about 870K (57%) are reachable through the top-40 associations of other words. The remaining 43% are rare or isolated terms that don’t appear in other words’ association lists.

This is why word association games work. The semantic space is more connected than people realize.