LINGUABASE — BUILD A DICTIONARY & THESAURUS (machine-readable method file)
==========================================================================
Companion to https://www.linguabase.org/ — a free, public-domain (CC0)
semantic dataset of English. This file is self-contained: tables, method,
measured results, and a complete build script. Written to be read by an
LLM asked to adapt the method to a different dictionary-like feature.

THE DATA
--------
Download (parquet or TSV, identical on three mirrors, linked from
linguabase.org/download.html). Tables used here, 6 of 13:

  vocabulary          word, core_rank           core_rank 1 = most familiar
  definitions         word, definition          one readable paragraph per word
  senses              word, sense_labels        the named senses (a list)
  sense_associations  word, sense, associates   one related-word cloud PER SENSE
  associations        word, associates          one ranked cloud per word, senses mixed
  word_families       word, relatives           morphological/etymological kin

The core files are the ranking cut at the recommended 400,000-rank
threshold; the full files carry the whole ranking (1,983,183 vocabulary
rows). Everything below is measured against the core files.

Multi-word entries are first-class rows: "ice cream", "cold shoulder"
and "hot dog" carry ranks, definitions, senses and families exactly the
way single words do, and this method keeps them as headwords.

Notes that matter:
- The shipped files are not sorted by rank, and the TSVs carry a header
  row — sort on core_rank before taking prefixes.
- Accented rows are canonical: café has the row, cafe does not. Fold
  accents when looking words up.
- Definitions mark italics two ways: {curly braces} (345,164 of 363,916
  rows) and literal <i> tags (2,120 rows). Render both or strip both.
- senses and sense_associations were built independently. Join them BY
  NAME: for each label in senses.sense_labels, take the row of
  sense_associations with the same (word, sense). Measured over the
  25,789 two-plus-sense entries below, 131,905 of 131,928 labels find
  a same-named cloud.
  Do NOT take every sense_associations row for a word — the table holds
  many more rows per word (key has 690, covering compound contexts like
  key lime and key deer); the labels are the curated entry points.

THE METHOD
----------
A dictionary is a headword list plus a bundle per headword:

  1  sort the vocabulary by core_rank; cut at your depth
  2  drop capitalized entries, then keep entries spelled with letters,
     spaces and hyphens — café and ice cream stay; don't, a.m. and
     24/7 go
  3  fold derived forms into their base: a word folds when its family
     holds a better-ranked base it derives from by suffix
     (-s -es -d -ed -ing -er -ers -est -ly, with e-drop, y->i, and
     doubled-consonant spellings). DOGS folds into DOG, EASIER into
     EASY. Phrase forms fold the same way whenever the family lists
     their base: ICE CREAMS folds into ICE CREAM. Folded words become
     cross-references, not entries.
  4  per entry: the definition paragraph, one section per sense label
     (words from the same-named sense_associations cloud), the family.
  5  family-filter every related-word list: drop a related word when it
     is the headword, in the headword's family, or a phrase containing
     the headword — BARK never lists BARKING or "dog bark". Use a
     token-sequence containment check so phrase headwords are caught
     inside longer phrases too.

Every number below was produced this way — measured, never estimated.

MEASURED RESULTS (June 2026 core release, ranked by core_rank)
--------------------------------------------------------------
Funnel at depth 50,000:
  50,000 -> 48,822 uncapitalized -> 47,956 letters/spaces/hyphens
  -> 36,195 heads after the fold -> 25,789 with 2+ senses
  = 36,195 entries + 11,761 cross-references (773 of them phrase
    forms: HOT DOGS -> HOT DOG); a definition paragraph for
    35,383 of the 36,195. 141 entries have one sense; 10,265 have
    none — mostly conversational phrases (good morning, by the
    way) — and ship as paragraph and family only. After the
    label-to-cloud join and the family filter, 25,785 entries
    show at least one sense section and 25,784 show two or more.

Depth sensitivity (same pipeline, depth varied):
  depth   entries(2+ senses)  one-sense  no-senses  cross-refs  paragraphs
  10,000       6,671              22        522        2,605     7,213/7,215
  25,000      14,671              66      3,173        6,482    17,816/17,910
  50,000      25,789             141     10,265       11,761    35,383/36,195
  100,000     43,349             276     30,161       19,960    68,445/73,786
  400,000    122,361           1,071    177,501       53,772   266,185/300,933

Content holes: 5,341 heads in the top 100,000 have neither a paragraph
nor senses — 5,223 conversational phrases (you have to, of course not),
116 hyphenated compounds (jet-lag), and exactly two plain words:
pipecleaner (rank 84,766) and hmu (96,595). Require a paragraph if bare
entries shouldn't ship.

Bundle weight, measured on twelve real entries (key, elephant, bark,
crane, hiccup, anchor, bridge, mole, café, ice cream, cold shoulder,
petrichor): 22,269 bytes total — sense rows 67.2%, paragraph 22.7%,
family the rest. The full dictionary.json at depth 50,000 is ~75 MB.

THE BUILD SCRIPT (complete, runnable)
-------------------------------------
# linguabase-dictionary-bundle.py — build per-word dictionary bundles
# needs the six *.core.parquet tables in the working directory
CAP = 50_000        # familiarity depth: entries come from the top CAP
MIN_SENSES = 0      # 0 = every head gets a bundle; 2 = sense-split only

import pandas as pd, json

v    = pd.read_parquet('vocabulary.core.parquet').sort_values('core_rank')
defs = pd.read_parquet('definitions.core.parquet').set_index('word').definition
sen  = pd.read_parquet('senses.core.parquet').set_index('word').sense_labels
fam  = pd.read_parquet('word_families.core.parquet').set_index('word').relatives
sa   = pd.read_parquet('sense_associations.core.parquet')

rank = dict(zip(v.word, v.core_rank))
kin  = lambda w: list(fam.get(w, []))

words = v[v.core_rank <= CAP]
words = words[words.word == words.word.str.lower()]     # no capitals
words = words[words.word.str.fullmatch(                 # letters, spaces,
    r"([^\W\d_]|[ -])+")]                               # hyphens

SFX = 's es d ed ing er ers est ly'.split()
def derived(w, b):                    # is w = b + a suffix?
    if not b: return False
    forms = [b, b + b[-1]]
    if b[-1] == 'e': forms.append(b[:-1])
    if b[-1] == 'y': forms.append(b[:-1] + 'i')
    return any(w == f + s for f in forms for s in SFX)

def fold_base(w):                     # best-ranked base, or None
    cands = [b for b in kin(w)
             if rank.get(b, 9e9) < rank[w] and derived(w, b)]
    return min(cands, key=rank.__getitem__) if cands else None

base = {w: fold_base(w) for w in words.word}
heads = [w for w in words.word if base[w] is None]

sa_idx = sa.set_index(['word', 'sense']).associates

def leaks(w, a):
    toks = a.replace('-', ' ').replace('_', ' ').lower().split()
    return a == w or a in kin(w) or f' {w} ' in f' {" ".join(toks)} '

def sections(w):
    out = []
    for label in list(sen.get(w, [])):
        try: cloud = list(sa_idx.at[(w, label)])
        except KeyError: continue     # label without a cloud: skip
        kept = [a.replace('_', ' ') for a in cloud if not leaks(w, a)]
        if kept: out.append([label, kept])
    return out

bundles = {}
for w in heads:
    s = sections(w)
    if len(s) < MIN_SENSES: continue
    bundles[w] = {'w': w, 'r': int(rank[w]),
                  'd': defs.get(w, ''),      # '' when no paragraph yet
                  's': s, 'f': kin(w)}

with open('dictionary.json', 'w') as f:
    json.dump(bundles, f, ensure_ascii=False, indent=1)
with open('crossrefs.tsv', 'w') as f:
    f.write('folded_form\theadword\n')
    for w in words.word:
        if base[w]: f.write(f'{w}\t{base[w]}\n')

AT LOOKUP TIME
--------------
Load dictionary.json and crossrefs.tsv once. A lookup is three checks:
  exact hit in bundles -> render the bundle
  hit in crossrefs     -> "DOGS folds into DOG", render the base
  accent-folded hit    -> "cafe's row is café", render it
Render {braces} and <i> spans in the definition as italics.

WHAT COUNTING DOES NOT SETTLE
-----------------------------
Whether depth 50,000 matches your users' vocabulary (usage logs, not
counting). The family filter can eat a true synonym — bridge's family
lists viaduct and drawbridge, so its structure sense loses both; loosen
the filter to exact inflections if that costs too much. Whether bare
heads (no paragraph, no senses) belong in your product — this build
keeps them; require a paragraph to drop them. Capitalized entries
(Sunday, December) were dropped by choice here; the tables carry them.
Related words are not pruned to the core file — join back to
vocabulary if every related word must resolve to an entry.

YOUR APP
--------
To adapt: state what one entry should show, then (1) write your
headword rules as filters over the ranked vocabulary, reusing the
filters above; (2) count entries at three depths; (3) render three
sample entries at each. The tables are CC0 — no permission needed,
commercial or otherwise.