# Word dataset provenance & licensing — Random Word Generator

The Random Word Generator ships **local, offline word datasets**. There is no
backend and no runtime download: every word list is committed to this
repository as a generated module in `src/words/datasets/` and bundled
lazily (one chunk per language) by Vite/Astro.

## Source

All bundled datasets are derived from:

- **hermitdave/FrequencyWords** — <https://github.com/hermitdave/FrequencyWords>
  - Files: `content/2018/<lang>/<lang>_50k.txt` (English, Italian, Spanish,
    French, German), frequency-ranked word lists ("word occurrences" lines).
  - License: **CC-BY-SA 4.0** for content (MIT for the project's code, which
    we do not use).

- **Upstream corpus:** the frequency lists were generated from the
  **OPUS OpenSubtitles 2018** subtitle corpus, which is released under
  **CC-BY-SA** (see <http://opus.nlpl.eu/OpenSubtitles2018.php>).

## Derivation pipeline (reproducible)

`scripts/build-word-datasets.mjs` regenerates every dataset module:

1. Read the frequency-ranked lines in file order (descending frequency —
   the rank is the curated cut: only the most frequent words are kept).
2. Keep tokens that are pure words in the language's native lowercase
   alphabet (per-language letter sets in the script), 2–15 letters long,
   lowercased. Numbers, punctuation, hyphenated/apostrophized tokens and
   tokens with foreign letters are dropped.
3. Deduplicate (first occurrence = highest frequency wins).
4. Keep the first 6,000 entries (all five languages reach the cap).

The tool at runtime never modifies or extends these lists.

## License of the bundled derived lists

Because the source content is CC-BY-SA 4.0, the derived word lists bundled
in `src/words/datasets/*.ts` are distributed under **CC-BY-SA 4.0** with
attribution preserved:

- every generated module carries a header naming the source file, its URL,
  and both license layers (FrequencyWords content + OPUS OpenSubtitles
  upstream);
- this document records the provenance for the repository as a whole.

## Known limitations (documented, not hidden)

- The lists are subtitle-derived: they skew toward spoken, colloquial
  language and **include proper nouns and loanwords** (proper nouns cannot
  be detected reliably in a lowercased frequency list). The generator is a
  creativity/brainstorming tool, not a dictionary or spell checker.
- **No part-of-speech metadata exists** in this data. The tool therefore
  offers **no part-of-speech filter** — fabricating POS tags would be
  dishonest, and the tool fails closed rather than inventing metadata.

## Fail-closed language policy

A language is offered in the UI **only** if a real, licensed dataset module
exists for it (see `src/words/registry.ts`). Requesting any other language
code raises an error instead of producing invented words. If a future
language's source turns out to be unavailable or unsuitably licensed, it
must not be added to the registry.

## Adding or refreshing a dataset

1. Verify the upstream source's license permits redistribution with
   attribution (and note any share-alike obligations here).
2. Add/keep the language in `scripts/build-word-datasets.mjs` with its
   native alphabet filter.
3. Run `node scripts/build-word-datasets.mjs` (requires network) and commit
   the regenerated `src/words/datasets/<code>.ts`.
4. Register the language in `src/words/registry.ts`.
5. Update the dataset-integrity unit tests if limits change.