diff --git a/.changeset/clear-inlang-sdk-positioning.md b/.changeset/clear-inlang-sdk-positioning.md new file mode 100644 index 0000000000..086948c3bc --- /dev/null +++ b/.changeset/clear-inlang-sdk-positioning.md @@ -0,0 +1,5 @@ +--- +"@inlang/sdk": patch +--- + +Clarify the SDK README and generated project README positioning for `.inlang` as the canonical localization file format with version control via lix. diff --git a/README.md b/README.md index d730fc52fa..6c507a05ea 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@

inlang icon inlang

- The open file format for localizing software (i18n). + The open project file format for localization (i18n).

@@ -40,24 +40,45 @@ --- -Inlang is an open project format and SDK for localization tooling. +Inlang is an open project file format for localization. -It is not a new message syntax or a SaaS translation backend. It gives editors, CLIs, IDE extensions, and runtimes a shared, queryable source of truth for localization data. +An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://github.com/opral/lix). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. -You can keep using your existing translation files and message syntax. Plugins connect inlang to formats like JSON, ICU MessageFormat v1, i18next, and XLIFF. +It is not another i18n library, message syntax, translation app, or SaaS backend. It gives editors, CLIs, IDE extensions, runtimes, and coding agents one shared place to read and write localization data. + +The `@inlang/sdk` is the reference implementation for reading and writing `.inlang` projects. + +`.inlang` is the canonical project format. Plugins import and export formats like JSON, ICU MessageFormat v1, i18next, and XLIFF for compatibility with existing translation files and runtimes. Version control via lix adds file-level history, merging, and change proposals to `.inlang` projects. + +Messages, variants, and locale data live in the `.inlang` database. External translation files such as `messages/en.json` are compatibility files outside `project.inlang/`, connected through plugins. + +```text +project.inlang # canonical single binary file +``` + +For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. + +```text +project.inlang/ +├── settings.json # locales, plugins, file patterns; kept in Git +├── .gitignore # generated; ignores everything except settings.json +├── README.md # generated; explains this folder to coding agents +├── .meta.json # generated SDK metadata +└── cache/ # plugin cache, created when plugins are loaded +``` ## The problem -Common translation files like JSON, YAML, ICU, or XLIFF are good at serializing messages. But they are not databases. +Common translation files like JSON, YAML, ICU, or XLIFF are good at storing messages. But they do not describe the whole localization project. -Once multiple tools need to read and write the same project, missing database semantics become the bottleneck: +Once multiple tools need to read and write the same project, plain translation files start to miss important information: -- Structured CRUD operations instead of ad-hoc parsing -- Queries across locales, variants, and metadata -- Transactions, history, merging, and collaboration -- One source of truth that editors, CI, and runtimes can all share +- CRUD operations instead of custom parsing +- Search and reports across locales, variants, and metadata +- Version control via [lix](https://github.com/opral/lix) +- One shared file that editors, CI, and runtimes can all use -Without a common substrate, every tool invents its own format, sync, and collaboration model. +Without one shared format, every tool invents its own file structure, sync logic, and collaboration workflow. Even basic import/export for translation file formats gets duplicated across tools instead of being shared. @@ -78,12 +99,18 @@ Every tool has its own format, its own sync, its own collaboration layer. Cross- ## The solution -Inlang adds those database semantics in a shared project format while keeping your external file formats. It provides: +Inlang puts the missing project information into a shared file format while keeping your external translation files. It provides: -- A message-first data model and SDK for structured reads and writes -- Queryable storage for translations, settings, and edits +- A message-first structure and SDK for CRUD operations +- Storage that tools can search, update, and report on - Plugins to import/export formats like JSON, ICU1, i18next, and XLIFF so that file-format support can be shared instead of reimplemented in every tool -- Versioning and collaboration primitives via [lix](https://github.com/opral/lix) +- Version control via [lix](https://github.com/opral/lix) + +Core data model: + +- **Bundle** — one translatable unit across locales +- **Message** — locale-specific translation for a bundle +- **Variant** — text pattern plus selector matches ``` ┌──────────┐ ┌───────────┐ ┌────────────┐ @@ -102,29 +129,57 @@ The result: - Switch tools without migrations — they all use the same file - Cross-team work without hand-offs — developers, translators, and designers all edit the same source -- Automation just works — one source of truth, no glue code +- Automation just works — the same data, no glue code - Keep your preferred message format — plugins handle import/export -If you only need an app runtime and a couple of translation files, your current setup may already be enough. Inlang becomes useful when multiple tools need to operate on the same localization source of truth. +If you only need an app runtime and a couple of translation files, JSON or your current i18n setup may already be enough. Use inlang when localization becomes shared work: multiple tools, teams, automations, or agents need to use the same localization data. -## Popular tools +## For coding agents and tool builders -- [Paraglide](https://inlang.com/m/gerre34r/library-inlang-paraglideJs) — i18n library for JS/TS with fully translated, typesafe & fast apps in minutes -- [Fink](https://inlang.com/m/tdozzpar/app-inlang-finkLocalizationEditor) — translation editor in the browser, invite collaborators to help -- [Sherlock](https://inlang.com/m/r7kp499g/app-inlang-ideExtension) — VS Code extension to translate right in your editor -- [Parrot](https://inlang.com/m/gkrpgoir/app-parrot-figmaPlugin) — see translations directly in Figma -- [CLI](https://inlang.com/m/2qj2w8pu/app-inlang-cli) — lint messages, machine translate, quality control in CI/CD +If you are building i18n tooling or generating localization code, target the `.inlang` file format instead of inventing your own JSON schema. -## Build your own +Use `@inlang/sdk` to read and write `.inlang` projects. Your output will then work with tools that read and write the inlang file format: runtimes, editors, plugins, CI checks, and translation workflows. + +Why target `.inlang` instead of JSON? + +- One file gives every tool the same data model for bundles, messages, variants, locales, metadata, and settings. +- CRUD operations are available through the SDK instead of custom parsing and rewriting. +- Version control via [lix](https://github.com/opral/lix) keeps localization changes reviewable and mergeable. +- Plugins handle JSON, ICU, i18next, XLIFF, and other formats for compatibility. + +Do not: + +- Invent a custom `locales/en.json` schema as the project format. +- Create a new localization config when a `.inlang` project already exists. +- Edit generated or cached files inside an unpacked `project.inlang/` directory. +- Bypass `@inlang/sdk` when reading or writing `.inlang`. + +## Tools that read/write the inlang file format + +These are independent tools that read and write `.inlang`; they are not inlang itself. + +- [Paraglide](https://inlang.com/m/gerre34r/library-inlang-paraglideJs) — JS/TS runtime that compiles messages from `.inlang` +- [Fink](https://inlang.com/m/tdozzpar/app-inlang-finkLocalizationEditor) — web editor for `.inlang` projects +- [Sherlock](https://inlang.com/m/r7kp499g/app-inlang-ideExtension) — VS Code extension that reads `.inlang` projects +- [Parrot](https://inlang.com/m/gkrpgoir/app-parrot-figmaPlugin) — Figma plugin that connects design text to `.inlang` +- [CLI](https://inlang.com/m/2qj2w8pu/app-inlang-cli) — linting, machine translation, and CI workflows for `.inlang` + +## Read and write `.inlang` ```ts -import { loadProjectFromDirectory } from "@inlang/sdk"; +import { loadProjectFromDirectory, loadProjectInMemory } from "@inlang/sdk"; +import fs from "node:fs/promises"; + +const packedProject = await loadProjectInMemory({ + blob: await fs.readFile("./project.inlang"), +}); -const project = await loadProjectFromDirectory({ +// Loads the Git-friendly unpacked representation. +const unpackedProject = await loadProjectFromDirectory({ path: "./project.inlang", }); -const messages = await project.db.selectFrom("message").selectAll().execute(); +const messages = await packedProject.db.selectFrom("message").selectAll().execute(); ``` [Read the docs →](https://inlang.com/docs) diff --git a/blog/update-on-inlang-v2/index.md b/blog/update-on-inlang-v2/index.md index 106a3e14ae..7871f4b1ca 100644 --- a/blog/update-on-inlang-v2/index.md +++ b/blog/update-on-inlang-v2/index.md @@ -5,7 +5,7 @@ og:description: "Accelerating both inlang and lix by prioritizing lix." Dear inlang community, -The release of the inlang SDK v2 (variant support) is blocked until we have a 1.0 release of the lix version control system. We are prioritizing [lix](https://lix.dev/) so we can unblock v2. Which means: +The release of the inlang SDK v2 (variant support) is blocked until we have a 1.0 release with version control via lix. We are prioritizing [lix](https://lix.dev/) so we can unblock v2. Which means: - The release of the inlang SDK v2 is likely postponed until Jan/Feb next year. @@ -23,13 +23,13 @@ As a sneak peek of what’s coming with v2, check out the Fink v2 demo [https:// ## What is lix? -Unbeknown to many of you, inlang has been built on the lix version control system over the past two years. You probably ask yourself right now: “What is lix?”. +Unbeknown to many of you, inlang has been built on version control via lix over the past two years. You probably ask yourself right now: “What is lix?”. Lix is a version control system, a new technology that allows controlling changes in various file formats, such as .csv, .inlang, music, video, architecture, .cad, and more. Controlling changes refers to workflows like change tracking, automation pipelines (CI/CD), or review systems. You can try out a CSV file demo of lix [here](https://csv.lix.opral.com/). -## Inlang needs lix version control to succeed +## Inlang needs version control via lix to succeed What makes globalization of software complicated is the required coordination effort. Designers need to know that translators updated translations to adjust their UIs, developers need to redeploy the app if translations change, auditors need to know that a message has changed, … the list goes on. diff --git a/docs/architecture.md b/docs/architecture.md index b4e082c5bf..479c1acb1c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,10 +1,10 @@ # Architecture -Inlang's architecture has three layers: storage, data model, and plugins. +Inlang's file format has three layers: storage, data model, and plugins. ``` ┌─────────────────────────────────────────────┐ -│ Storage (SQLite + Lix) │ +│ Storage (SQLite + version control via Lix) │ ├─────────────────────────────────────────────┤ │ Data Model (Bundle, Message, Variant) │ ├─────────────────────────────────────────────┤ @@ -14,13 +14,17 @@ Inlang's architecture has three layers: storage, data model, and plugins. ## Storage -An `.inlang` file is a SQLite database with built-in version control via [Lix](https://lix.dev). One portable file containing all your translations, settings, and change history. +An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. -SQLite was chosen because: +For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. + +Version control via lix adds file-level history, merging, and change proposals to `.inlang` projects. + +The storage layer is designed to be: - **Queryable** — Filter, join, and aggregate translations with SQL -- **Portable** — Single file, no server, works in browser via WASM -- **Proven** — Battle-tested, used everywhere +- **Portable** — Single file, no server +- **Git-friendly when unpacked** — Store a directory representation in repos for reviewable changes ## Data Model @@ -44,22 +48,22 @@ See [Data Model](/docs/data-model) for details. ## Plugins -Plugins handle the transformation between external file formats (JSON, i18next, XLIFF) and inlang's internal data model. +Plugins handle the transformation between external translation files (JSON, i18next, XLIFF) and inlang's internal data model. ``` ┌─────────────────┐ ┌─────────┐ ┌──────────────────┐ │ .inlang file │◄─────►│ Plugins │◄─────►│ Translation files│ -│ (SQLite) │ │ │ │ (JSON, XLIFF) │ +│ │ │ │ │ (JSON, XLIFF) │ └─────────────────┘ └─────────┘ └──────────────────┘ ``` -Plugins only do import/export — they don't touch the database directly. This keeps the core simple and makes format support extensible. +`.inlang` is the canonical project format. External translation files are compatibility files that plugins import and export. Plugins only do import/export — they don't write the `.inlang` project directly. This keeps the core simple and makes format support extensible. See [Plugin API](/docs/plugin-api) for the reference or [Writing a Plugin](/docs/write-plugin) to build your own. ## Message-first design -Traditional i18n tools are file-first: you load `en.json`, `de.json`, `fr.json` as separate resources and iterate through files to find translations. +Traditional i18n tools are translation-file-first: you load `en.json`, `de.json`, `fr.json` as separate resources and iterate through files to find translations. Inlang is message-first: you query messages directly from the database. @@ -80,6 +84,7 @@ const messages = await project.db Why this matters: - **Tools don't care about files** — They care about messages. Files are an import/export detail. +- **CRUD operations** — Tools can read and write messages through the SDK instead of custom parsing. - **Query across locales** — Find missing translations, compare locales, aggregate stats. - **Future-proof** — The data model works regardless of where translations come from (files, APIs, databases). @@ -88,4 +93,3 @@ Why this matters: - [CRUD API](/docs/crud-api) — Query and modify translations - [Writing a Tool](/docs/write-tool) — Build a tool using the SDK - [Writing a Plugin](/docs/write-plugin) — Support a custom file format - diff --git a/docs/crud-api.md b/docs/crud-api.md index 4d5cf8d936..72387ccb53 100644 --- a/docs/crud-api.md +++ b/docs/crud-api.md @@ -6,6 +6,7 @@ Inlang uses [Kysely](https://kysely.dev) for type-safe database queries. Access ```typescript import { loadProjectFromDirectory } from "@inlang/sdk"; +import fs from "node:fs"; const project = await loadProjectFromDirectory({ path: "./project.inlang", @@ -13,12 +14,13 @@ const project = await loadProjectFromDirectory({ }); // project.db is a Kysely instance -const bundles = await project.db - .selectFrom("bundle") - .selectAll() - .execute(); +const bundles = await project.db.selectFrom("bundle").selectAll().execute(); ``` +> **Saving changes:** CRUD operations update the in-memory `.inlang` database. To save a packed `.inlang` file, call `project.toBlob()`. To save an unpacked `project.inlang/` directory, `saveProjectToDirectory()` needs an import/export plugin; without one, bundles, messages, and variants have no file path to export to. + +If the project uses plugins, check `await project.errors.get()` after loading. Close the project with `await project.close()` in one-off scripts. + ## Create ### Insert a bundle @@ -28,7 +30,7 @@ await project.db .insertInto("bundle") .values({ id: "greeting", - declarations: [] + declarations: [], }) .execute(); ``` @@ -36,15 +38,16 @@ await project.db ### Insert a message ```typescript -await project.db +const message = await project.db .insertInto("message") .values({ id: crypto.randomUUID(), bundleId: "greeting", locale: "en", - selectors: [] + selectors: [], }) - .execute(); + .returning("id") + .executeTakeFirstOrThrow(); ``` ### Insert a variant @@ -54,9 +57,9 @@ await project.db .insertInto("variant") .values({ id: crypto.randomUUID(), - messageId: messageId, + messageId: message.id, matches: [], - pattern: [{ type: "text", value: "Hello world!" }] + pattern: [{ type: "text", value: "Hello world!" }], }) .execute(); ``` @@ -66,25 +69,27 @@ await project.db ```typescript import { insertBundleNested } from "@inlang/sdk"; +const messageId = crypto.randomUUID(); + await insertBundleNested(project.db, { id: "greeting", declarations: [], messages: [ { - id: crypto.randomUUID(), + id: messageId, bundleId: "greeting", locale: "en", selectors: [], variants: [ { id: crypto.randomUUID(), - messageId: messageId, + messageId, matches: [], - pattern: [{ type: "text", value: "Hello!" }] - } - ] - } - ] + pattern: [{ type: "text", value: "Hello!" }], + }, + ], + }, + ], }); ``` @@ -93,10 +98,7 @@ await insertBundleNested(project.db, { ### Get all bundles ```typescript -const bundles = await project.db - .selectFrom("bundle") - .selectAll() - .execute(); +const bundles = await project.db.selectFrom("bundle").selectAll().execute(); ``` ### Get bundle by ID @@ -175,19 +177,19 @@ const results = await project.db ### Find missing translations ```typescript -const missingGerman = await project.db - .selectFrom("bundle") - .where((eb) => - eb.not( - eb.exists( - eb.selectFrom("message") - .where("message.bundleId", "=", eb.ref("bundle.id")) - .where("message.locale", "=", "de") - ) - ) - ) - .selectAll() +const bundles = await project.db.selectFrom("bundle").selectAll().execute(); +const germanMessages = await project.db + .selectFrom("message") + .select("bundleId") + .where("locale", "=", "de") .execute(); + +const translatedBundleIds = new Set( + germanMessages.map((message) => message.bundleId), +); +const missingGerman = bundles.filter( + (bundle) => translatedBundleIds.has(bundle.id) === false, +); ``` ## Update @@ -198,7 +200,7 @@ const missingGerman = await project.db await project.db .updateTable("bundle") .set({ - declarations: [{ type: "input-variable", name: "count" }] + declarations: [{ type: "input-variable", name: "count" }], }) .where("id", "=", "greeting") .execute(); @@ -210,7 +212,7 @@ await project.db await project.db .updateTable("variant") .set({ - pattern: [{ type: "text", value: "Updated text" }] + pattern: [{ type: "text", value: "Updated text" }], }) .where("id", "=", variantId) .execute(); @@ -233,11 +235,11 @@ await updateBundleNested(project.db, { { id: variantId, matches: [], - pattern: [{ type: "text", value: "Updated!" }] - } - ] - } - ] + pattern: [{ type: "text", value: "Updated!" }], + }, + ], + }, + ], }); ``` @@ -246,10 +248,7 @@ await updateBundleNested(project.db, { ### Delete a bundle ```typescript -await project.db - .deleteFrom("bundle") - .where("id", "=", "greeting") - .execute(); +await project.db.deleteFrom("bundle").where("id", "=", "greeting").execute(); // Cascades: all messages and variants are deleted ``` @@ -257,10 +256,7 @@ await project.db ### Delete a message ```typescript -await project.db - .deleteFrom("message") - .where("id", "=", messageId) - .execute(); +await project.db.deleteFrom("message").where("id", "=", messageId).execute(); // Cascades: all variants are deleted ``` @@ -268,10 +264,7 @@ await project.db ### Delete a variant ```typescript -await project.db - .deleteFrom("variant") - .where("id", "=", variantId) - .execute(); +await project.db.deleteFrom("variant").where("id", "=", variantId).execute(); ``` ## Upsert @@ -285,12 +278,12 @@ await project.db .insertInto("bundle") .values({ id: "greeting", - declarations: [] + declarations: [], }) .onConflict((oc) => oc.column("id").doUpdateSet({ - declarations: [] - }) + declarations: [], + }), ) .execute(); ``` @@ -312,4 +305,3 @@ await upsertBundleNested(project.db, { - [Data Model](/docs/data-model) — Understand bundles, messages, and variants - [Writing a Tool](/docs/write-tool) — Build a complete tool using CRUD operations - [Unpacked Project](/docs/unpacked-project) — Load projects from disk - diff --git a/docs/data-model.md b/docs/data-model.md index 8d095cbb0a..445b44e688 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -20,24 +20,26 @@ A bundle groups translations by key. One bundle = one translatable unit across a ```typescript type Bundle = { - id: string; // e.g., "greeting", "error_404" + id: string; // e.g., "greeting", "error_404" declarations: Declaration[]; -} +}; ``` The `id` is your translation key — what you reference in code. The id is assumed to be stable; changing it would break all references. Declarations define variables available to all messages in the bundle. +See [Message Shapes](/docs/message-shapes) for concrete JSON examples of declarations, selectors, matches, and pattern parts. + ## Message A message is a locale-specific translation. One message per locale per bundle. ```typescript type Message = { - id: string; // auto-generated UUID - bundleId: string; // references Bundle.id - locale: string; // e.g., "en", "de", "fr" + id: string; // auto-generated UUID + bundleId: string; // references Bundle.id + locale: string; // e.g., "en", "de", "fr" selectors: VariableReference[]; -} +}; ``` Selectors are used for conditional matching (plurals, gender, etc.). If your message has no conditions, selectors is empty. @@ -48,11 +50,11 @@ A variant is the actual text pattern. Most messages have one variant, but plural ```typescript type Variant = { - id: string; // auto-generated UUID - messageId: string; // references Message.id - matches: Match[]; // conditions for this variant - pattern: Pattern; // the text content -} + id: string; // auto-generated UUID + messageId: string; // references Message.id + matches: Match[]; // conditions for this variant + pattern: Pattern; // the text content +}; ``` ### Simple example @@ -126,24 +128,25 @@ const bundles = await project.db .execute(); // Find missing translations -const missing = await project.db - .selectFrom("bundle") - .where((eb) => - eb.not( - eb.exists( - eb.selectFrom("message") - .where("message.bundleId", "=", eb.ref("bundle.id")) - .where("message.locale", "=", "de") - ) - ) - ) - .selectAll() +const allBundles = await project.db.selectFrom("bundle").selectAll().execute(); +const germanMessages = await project.db + .selectFrom("message") + .select("bundleId") + .where("locale", "=", "de") .execute(); + +const translatedBundleIds = new Set( + germanMessages.map((message) => message.bundleId), +); +const missing = allBundles.filter( + (bundle) => translatedBundleIds.has(bundle.id) === false, +); ``` ## Next steps - [CRUD API](/docs/crud-api) — Full reference for query operations +- [Message Shapes](/docs/message-shapes) — Concrete JSON shapes for patterns, matches, and declarations - [Architecture](/docs/architecture) — See how the data model fits in - [Writing a Tool](/docs/write-tool) — Build a tool that queries messages - [Plugin API](/docs/plugin-api) — Import types for plugins diff --git a/docs/getting-started.md b/docs/getting-started.md index 1a995dc916..9d7d06e4cf 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,16 +1,107 @@ # Getting Started -There are two ways to use inlang: +Use inlang when localization data needs to be shared across tools, teams, automations, or coding agents. If you only need an app runtime with a couple of translation files, your current i18n setup may already be enough. -## Use an existing tool +An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). For Git repositories, the file can be unpacked into a directory of plain files. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. -Browse the ecosystem and adopt a tool that fits your workflow. Most users start here. +## Install + +Use Node.js 20 or newer and an ESM project (`"type": "module"` in `package.json`). + +```bash +npm install @inlang/sdk +``` + +## Which path should I use? + +| Situation | Use | +| ------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| A `project.inlang/` directory already exists in the repository | `loadProjectFromDirectory()` and work through `project.db` | +| You are creating localization data from scratch | `newProject()` → `loadProjectInMemory()` → `project.toBlob()` | +| You need to write JSON, i18next, ICU, XLIFF, or another translation file format | Configure an import/export plugin, then use the SDK CRUD API | +| You only need runtime translation lookup for one app | Your existing i18n library may be enough | + +Do not create a second localization config when a `.inlang` project already exists. Use `@inlang/sdk` to read and write the project. + +## Quickstart: create and save a canonical `.inlang` file + +This example creates a project, inserts one message, saves the packed `.inlang` file, and reloads it. + +```typescript +import { + insertBundleNested, + loadProjectInMemory, + newProject, + selectBundleNested, +} from "@inlang/sdk"; +import fs from "node:fs/promises"; + +const initialBlob = await newProject({ + settings: { + baseLocale: "en", + locales: ["en"], + }, +}); + +const project = await loadProjectInMemory({ blob: initialBlob }); + +await insertBundleNested(project.db, { + id: "greeting", + declarations: [], + messages: [ + { + id: "greeting_en", + bundleId: "greeting", + locale: "en", + selectors: [], + variants: [ + { + id: "greeting_en_default", + messageId: "greeting_en", + matches: [], + pattern: [{ type: "text", value: "Hello world!" }], + }, + ], + }, + ], +}); + +const savedBlob = await project.toBlob(); +await fs.writeFile( + "project.inlang", + new Uint8Array(await savedBlob.arrayBuffer()), +); + +const reloadedProject = await loadProjectInMemory({ + blob: new Blob([new Uint8Array(await fs.readFile("project.inlang"))]), +}); + +const bundles = await selectBundleNested(reloadedProject.db).execute(); +console.log(bundles[0]?.messages[0]?.variants[0]?.pattern); + +await project.close(); +await reloadedProject.close(); +``` + +Use `project.toBlob()` for the packed `.inlang` file. Use `saveProjectToDirectory()` only for the unpacked Git representation, and only when an import/export plugin can write your translation files. + +## Use a tool built on inlang + +Browse tools that read and write the `.inlang` project file format. Most app developers start here. [Browse Tools →](/c/tools) ## Build your own -Use the SDK to build i18n tools — linters, editors, CLI tools, IDE extensions — that work with any translation format. +If you are building i18n tooling or generating localization code, target the `.inlang` file format instead of inventing your own JSON schema. -[Write a Tool →](/docs/write-tool) +Use `@inlang/sdk` to build linters, editors, CLI tools, IDE extensions, and coding agents that work with any translation format through the shared `.inlang` message structure. + +Why target `.inlang`? +- CRUD operations instead of custom parsing +- Version control via [lix](https://lix.dev) +- Plugins for JSON, ICU, i18next, XLIFF, and other formats +- One data model that tools can share + +[Write a Tool →](/docs/write-tool) diff --git a/docs/install-plugin.md b/docs/install-plugin.md index 32df2cc0c8..bf97243d6b 100644 --- a/docs/install-plugin.md +++ b/docs/install-plugin.md @@ -16,7 +16,7 @@ Add the plugin URL to the `modules` array in `project.inlang/settings.json`: "baseLocale": "en", "locales": ["en", "de", "fr"], + "modules": [ -+ "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@3/dist/index.js" ++ "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@6.1.4/dist/index.js" + ] } ``` @@ -33,12 +33,12 @@ Plugins are loaded via [jsdelivr](https://www.jsdelivr.com/), a free CDN for npm Version pinning examples: ``` -@inlang/plugin-i18next@3 # Major version (3.x.x) -@inlang/plugin-i18next@3.5 # Minor version (3.5.x) -@inlang/plugin-i18next@3.5.2 # Exact version +@inlang/plugin-i18next@6 # Major version (6.x.x) +@inlang/plugin-i18next@6.1 # Minor version (6.1.x) +@inlang/plugin-i18next@6.1.4 # Exact version ``` -Pin to at least a major version to avoid breaking changes. +Pin an exact version for CI and agents. Use a major version only when you intentionally accept compatible updates. ## Configuring a plugin @@ -49,7 +49,7 @@ Most plugins require configuration. Add settings using the `plugin.` "baseLocale": "en", "locales": ["en", "de", "fr"], "modules": [ - "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@3/dist/index.js" + "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@6.1.4/dist/index.js" ], "plugin.inlang.i18next": { "pathPattern": "./locales/{locale}.json" @@ -59,6 +59,27 @@ Most plugins require configuration. Add settings using the `plugin.` Each plugin documents its available settings on its marketplace page. +`pathPattern` is resolved relative to the directory that contains `project.inlang/`. For example, `./locales/{locale}.json` with `path: "./project.inlang"` reads and writes `./locales/en.json`, not `./project.inlang/locales/en.json`. + +After loading a project with plugins, check `project.errors.get()` before trusting the imported data: + +```typescript +import { loadProjectFromDirectory } from "@inlang/sdk"; +import fs from "node:fs"; + +const project = await loadProjectFromDirectory({ + path: "./project.inlang", + fs, +}); + +const errors = await project.errors.get(); +if (errors.length > 0) { + throw new AggregateError(errors, "Could not load inlang project"); +} + +await project.close(); +``` + ## Using multiple plugins You can install multiple plugins. Each plugin handles different files or provides different functionality: @@ -68,7 +89,7 @@ You can install multiple plugins. Each plugin handles different files or provide "baseLocale": "en", "locales": ["en", "de"], "modules": [ - "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@3/dist/index.js", + "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@6.1.4/dist/index.js", "https://cdn.jsdelivr.net/npm/@inlang/plugin-t-function-matcher@3/dist/index.js" ], "plugin.inlang.i18next": { @@ -98,11 +119,11 @@ Then reference it from `node_modules`: ```json { - "modules": ["../node_modules/@inlang/plugin-i18next/dist/index.js"] + "modules": ["./node_modules/@inlang/plugin-i18next/dist/index.js"] } ``` -The path is relative to the `project.inlang` directory. +The path is relative to the directory that contains `project.inlang/`. ### Custom plugins @@ -134,14 +155,17 @@ Browse available plugins at [inlang.com/c/plugins](https://inlang.com/c/plugins) **Plugin not loading** - Verify the URL is correct and ends with `.js` +- Verify the plugin version supports the current `@inlang/sdk` import/export API - Check that the CDN is accessible - For local plugins, ensure the path is relative to `project.inlang` +- Inspect `await project.errors.get()` after `loadProjectFromDirectory()` **Files not importing** - Check that `pathPattern` matches your file structure - Verify `{locale}` placeholder is in the correct position - Ensure locale codes in filenames match your `locales` array +- Remember that `pathPattern` is relative to the directory containing `project.inlang/` **Settings validation errors** diff --git a/docs/introduction.md b/docs/introduction.md index f7ff783b69..ec7bdf339f 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -2,29 +2,37 @@ ## What is inlang? -Inlang is an open project format and SDK for localization tooling. +Inlang is an open project file format for localization. -It is not a new message syntax or a SaaS translation backend. Instead, it gives editors, CLIs, IDE extensions, and runtimes a shared, queryable source of truth for localization data. +An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. -You can keep using your existing translation files and message syntax. Plugins connect inlang to formats like JSON, ICU MessageFormat v1, i18next, and XLIFF. +It is not another i18n library, message syntax, translation app, or SaaS backend. Instead, it gives editors, CLIs, IDE extensions, runtimes, and coding agents one shared place to read and write localization data. + +The `@inlang/sdk` is the reference implementation for reading and writing `.inlang` projects. + +`.inlang` is the canonical project format. Plugins import and export formats like JSON, ICU MessageFormat v1, i18next, and XLIFF for compatibility with existing translation files and runtimes. Version control via lix adds file-level history, merging, and change proposals to `.inlang` projects. + +Messages, variants, and locale data live in the `.inlang` database. External translation files such as `messages/en.json` are compatibility files outside `project.inlang/`, connected through plugins. + +For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. The SDK has two main parts: -- **Storage + data model** for translations, settings, and structured edits +- **Storage + message structure** for translations, settings, and structured edits - **An API** for loading, querying, and modifying that data programmatically ## Why inlang? -Common translation files like JSON, YAML, ICU, or XLIFF are good at serializing messages. But they are not databases. +Common translation files like JSON, YAML, ICU, or XLIFF are good at storing messages. But they do not describe the whole localization project. -Once multiple tools need to read and write the same project, missing database semantics become the bottleneck: +Once multiple tools need to read and write the same project, plain translation files start to miss important information: -- Structured CRUD operations instead of ad-hoc parsing -- Queries across locales, variants, and metadata -- Transactions, history, merging, and collaboration -- One source of truth that editors, CI, and runtimes can all share +- CRUD operations instead of custom parsing +- Search and reports across locales, variants, and metadata +- Version control via [lix](https://lix.dev) +- One shared file that editors, CI, and runtimes can all use -Without a common substrate, every tool invents its own format, sync, and collaboration model. +Without one shared format, every tool invents its own file structure, sync logic, and collaboration workflow. The result is fragmented tooling: @@ -39,7 +47,7 @@ The result is fragmented tooling: └──────────┘ └───────────┘ └──────────┘ ``` -Inlang follows a simple idea: **one shared project format for localization tools, while keeping your external file formats**. +Inlang follows a simple idea: **one shared project file format for localization tools, while keeping your external translation files**. ``` ┌──────────┐ ┌───────────┐ ┌────────────┐ @@ -58,30 +66,51 @@ Inlang follows a simple idea: **one shared project format for localization tools - Switch tools without migrations — they all use the same file - Cross-team work without hand-offs — developers, translators, and designers all edit the same source -- Automation just works — one source of truth, no glue code +- Automation just works — the same data, no glue code - Keep your preferred message format — plugins handle import/export ## How it works -Under the hood, an inlang project stores localization data in SQLite and uses a message-first data model. +Under the hood, an inlang project uses a message-first data model. + +Core data model: + +- **Bundle** — one translatable unit across locales +- **Message** — locale-specific translation for a bundle +- **Variant** — text pattern plus selector matches -Lix adds history and sync semantics on top, and plugins map that data to the files you already use. +Version control via lix is built in, and plugins map that data to the files you already use. ``` ┌─────────────────┐ ┌─────────┐ ┌──────────────────┐ │ .inlang file │◄─────►│ Plugins │◄─────►│ Translation files│ -│ (SQLite) │ │ │ │ (JSON, XLIFF) │ +│ │ │ │ │ (JSON, XLIFF) │ └─────────────────┘ └─────────┘ └──────────────────┘ ``` - **Plugins** import and export your translation files (`JSON`, `ICU1`, `i18next`, `XLIFF`, etc.) -- **inlang** stores the data in a queryable project format -- **Lix** provides versioning and collaboration primitives for distributed changes +- **inlang** stores the data in a project format that tools can query +- **Version control via lix** handles distributed changes -If you only need an app runtime and a couple of translation files, your current setup may already be enough. Inlang becomes useful when multiple tools need to operate on the same localization source of truth. +If you only need an app runtime and a couple of translation files, JSON or your current i18n setup may already be enough. Use inlang when localization becomes shared work: multiple tools, teams, automations, or agents need to use the same localization data. To store an inlang project in git, you can use the **unpacked format** — a directory instead of a single file. See [Unpacked Project](/docs/unpacked-project) for details. +## For coding agents and tool builders + +If you are building i18n tooling or generating localization code, target the `.inlang` file format instead of inventing your own JSON schema. + +Use `@inlang/sdk` to read and write `.inlang` projects. Your output will then work with tools that read and write the inlang file format: runtimes, editors, plugins, CI checks, and translation workflows. + +Why target `.inlang` instead of JSON? + +- One file gives every tool the same data model for bundles, messages, variants, locales, metadata, and settings. +- CRUD operations are available through the SDK instead of custom parsing and rewriting. +- Version control via [lix](https://lix.dev) keeps localization changes reviewable and mergeable. +- Plugins handle JSON, ICU, i18next, XLIFF, and other formats for compatibility. + +Do not invent custom translation-file schemas, create a second localization config when `.inlang` already exists, edit generated/cache files in unpacked projects, or bypass `@inlang/sdk`. + ## Next steps - [Getting Started](/docs/getting-started) — Set up your first project @@ -91,4 +120,4 @@ To store an inlang project in git, you can use the **unpacked format** — a dir ## Credits -Inlang builds on [Lix](https://lix.dev) for version control and [Kysely](https://kysely.dev) for the query API. +Inlang uses version control via [lix](https://lix.dev) and [Kysely](https://kysely.dev) for the query API. diff --git a/docs/message-shapes.md b/docs/message-shapes.md new file mode 100644 index 0000000000..6948a2af60 --- /dev/null +++ b/docs/message-shapes.md @@ -0,0 +1,273 @@ +# Message Shapes + +This page shows the concrete JSON shapes used by bundles, messages, variants, declarations, selectors, matches, and patterns. + +Use these shapes when inserting data through `project.db`, `insertBundleNested()`, or a plugin's `importFiles()` return value. + +## Minimal Message + +```typescript +const messageId = crypto.randomUUID(); +const variantId = crypto.randomUUID(); + +await insertBundleNested(project.db, { + id: "greeting", + declarations: [], + messages: [ + { + id: messageId, + bundleId: "greeting", + locale: "en", + selectors: [], + variants: [ + { + id: variantId, + messageId, + matches: [], + pattern: [{ type: "text", value: "Hello world!" }], + }, + ], + }, + ], +}); +``` + +## Pattern + +A `pattern` is an array. It can mix text, expressions, and markup. + +### Text + +```typescript +[{ type: "text", value: "Hello world!" }]; +``` + +### Interpolation + +Use an `input-variable` declaration for variables provided by the caller, then reference it with an expression. + +```typescript +const messageId = crypto.randomUUID(); +const variantId = crypto.randomUUID(); + +{ + id: "greeting", + declarations: [{ type: "input-variable", name: "name" }], + messages: [ + { + id: messageId, + bundleId: "greeting", + locale: "en", + selectors: [], + variants: [ + { + id: variantId, + messageId, + matches: [], + pattern: [ + { type: "text", value: "Hello " }, + { + type: "expression", + arg: { type: "variable-reference", name: "name" }, + }, + { type: "text", value: "!" }, + ], + }, + ], + }, + ], +} +``` + +### Expression With Annotation + +Annotations describe formatting functions. Plugins decide which annotations they can import or export. + +```typescript +{ + type: "expression", + arg: { type: "variable-reference", name: "count" }, + annotation: { + type: "function-reference", + name: "number", + options: [], + }, +} +``` + +Options can use literals or variable references: + +```typescript +{ + type: "function-reference", + name: "number", + options: [ + { + name: "style", + value: { type: "literal", value: "currency" }, + }, + { + name: "currency", + value: { type: "variable-reference", name: "currency" }, + }, + ], +} +``` + +### Markup + +Markup is represented as pattern parts. This example corresponds to `Click here`. + +```typescript +[ + { type: "text", value: "Click " }, + { type: "markup-start", name: "link" }, + { type: "text", value: "here" }, + { type: "markup-end", name: "link" }, + { type: "markup-standalone", name: "icon" }, +]; +``` + +Markup can include options and attributes: + +```typescript +{ + type: "markup-start", + name: "link", + options: [ + { + name: "href", + value: { type: "literal", value: "/pricing" }, + }, + ], + attributes: [ + { + name: "external", + value: true, + }, + ], +} +``` + +## Selectors And Matches + +Selectors choose which variables a message uses to pick a variant. Matches on each variant must refer to selector names. + +### Literal Match + +```typescript +{ + type: "literal-match", + key: "platform", + value: "ios", +} +``` + +### Catch-All Match + +```typescript +{ + type: "catchall-match", + key: "platform", +} +``` + +## Plural-Style Selector + +Use an input variable for the caller-provided value, a local variable for the derived selector value, and variants that match the local variable. + +```typescript +const messageId = crypto.randomUUID(); +const oneVariantId = crypto.randomUUID(); +const otherVariantId = crypto.randomUUID(); + +{ + id: "items_count", + declarations: [ + { type: "input-variable", name: "count" }, + { + type: "local-variable", + name: "countPlural", + value: { + type: "expression", + arg: { type: "variable-reference", name: "count" }, + annotation: { + type: "function-reference", + name: "plural", + options: [], + }, + }, + }, + ], + messages: [ + { + id: messageId, + bundleId: "items_count", + locale: "en", + selectors: [{ type: "variable-reference", name: "countPlural" }], + variants: [ + { + id: oneVariantId, + messageId, + matches: [{ type: "literal-match", key: "countPlural", value: "one" }], + pattern: [{ type: "text", value: "One item" }], + }, + { + id: otherVariantId, + messageId, + matches: [ + { type: "literal-match", key: "countPlural", value: "other" }, + ], + pattern: [ + { + type: "expression", + arg: { type: "variable-reference", name: "count" }, + }, + { type: "text", value: " items" }, + ], + }, + ], + }, + ], +} +``` + +## Variant Linkage: CRUD Versus Plugin Import + +Use `messageId` when you already have a concrete message row id. This is the normal shape for direct CRUD writes and `insertBundleNested()`. + +```typescript +const messageId = crypto.randomUUID(); + +{ + id: "variant_1", + messageId, + matches: [], + pattern: [{ type: "text", value: "Hello" }], +} +``` + +Use `messageBundleId` and `messageLocale` when returning variants from a plugin's `importFiles()`. In that flow, message ids are often omitted so the SDK can generate or reuse them while importing. + +```typescript +{ + messageBundleId: "greeting", + messageLocale: "en", + matches: [], + pattern: [{ type: "text", value: "Hello" }], +} +``` + +The SDK resolves `messageBundleId` plus `messageLocale` to the matching message and generates ids when needed. + +Rule of thumb: + +- Direct database writes: use `messageId`. +- `insertBundleNested()`: use `messageId`, and reuse the same id from the message object. +- Plugin `importFiles()`: use `messageBundleId` plus `messageLocale`, unless your plugin deliberately manages stable message ids itself. + +## Next Steps + +- [Data Model](/docs/data-model) - Understand bundles, messages, and variants +- [CRUD API](/docs/crud-api) - Insert and query these shapes +- [Writing a Plugin](/docs/write-plugin) - Return these shapes from `importFiles()` diff --git a/docs/plugin-api.md b/docs/plugin-api.md index 8df24a9f26..4744ef7501 100644 --- a/docs/plugin-api.md +++ b/docs/plugin-api.md @@ -1,11 +1,11 @@ # Plugin API -Plugins handle the transformation between external file formats (JSON, i18next, XLIFF) and inlang's internal data model. They only do import/export — they don't touch the database directly. +Plugins handle the transformation between external file formats (JSON, i18next, XLIFF) and inlang's internal data model. `.inlang` is the canonical project format; external translation files are compatibility files. Plugins only do import/export — they don't write the `.inlang` project directly. ``` ┌─────────────────┐ ┌─────────┐ ┌──────────────────┐ │ .inlang file │◄─────►│ Plugins │◄─────►│ Translation files│ -│ (SQLite) │ │ │ │ (JSON, XLIFF) │ +│ │ │ │ │ (JSON, XLIFF) │ └─────────────────┘ └─────────┘ └──────────────────┘ ``` @@ -15,8 +15,8 @@ Plugins handle the transformation between external file formats (JSON, i18next, type InlangPlugin = { key: string; settingsSchema?: TObject; - toBeImportedFiles?: (args) => Promise>; - importFiles?: (args) => Promise<{ bundles, messages, variants }>; + toBeImportedFiles?: (args) => Promise>; + importFiles?: (args) => Promise<{ bundles; messages; variants }>; exportFiles?: (args) => Promise>; meta?: Record>; }; @@ -99,13 +99,15 @@ toBeImportedFiles: async ({ settings }) => { { path: "./messages/en.json", locale: "en" }, { path: "./messages/de.json", locale: "de" }, ]; -} +}; ``` **Parameters:** + - `settings` — Project settings including plugin-specific config **Returns:** Array of file descriptors: + - `path` — Path to the file - `locale` — Locale this file contains - `metadata` — Optional, passed to `importFiles` @@ -125,6 +127,7 @@ importFiles: async ({ files, settings }) => { ``` **Parameters:** + - `files` — Array of files to import: - `locale` — The locale - `content` — Binary file content (`Uint8Array`) @@ -132,6 +135,7 @@ importFiles: async ({ files, settings }) => { - `settings` — Project settings **Returns:** + - `bundles` — Array of `BundleImport` - `messages` — Array of `MessageImport` - `variants` — Array of `VariantImport` @@ -149,16 +153,18 @@ exportFiles: async ({ bundles, messages, variants, settings }) => { content: new TextEncoder().encode(JSON.stringify(data)), }, ]; -} +}; ``` **Parameters:** + - `bundles` — All bundles - `messages` — All messages - `variants` — All variants - `settings` — Project settings **Returns:** Array of files to write: + - `locale` — The locale - `name` — Filename (e.g., `"en.json"`) - `content` — Binary content (`Uint8Array`) @@ -176,9 +182,7 @@ export const PluginSettings = Type.Object({ description: "Path to translation files", examples: ["./messages/{locale}.json"], }), - sort: Type.Optional( - Type.Union([Type.Literal("asc"), Type.Literal("desc")]) - ), + sort: Type.Optional(Type.Union([Type.Literal("asc"), Type.Literal("desc")])), }); ``` @@ -223,7 +227,7 @@ type BundleImport = { ```typescript type MessageImport = { - id?: string; // auto-generated if omitted + id?: string; // auto-generated if omitted bundleId: string; locale: string; selectors: VariableReference[]; @@ -232,7 +236,11 @@ type MessageImport = { ### VariantImport -Variants can reference messages by ID or by bundle/locale: +Variants can reference messages by ID or by bundle/locale. + +Use `messageBundleId` plus `messageLocale` for most `importFiles()` implementations. The SDK can then generate or reuse message ids during import. + +Use `messageId` only when your plugin deliberately manages stable message ids and returns matching message ids in `messages`. ```typescript // By message ID @@ -243,7 +251,7 @@ type VariantImport = { pattern: Pattern; }; -// By bundle/locale (recommended) +// By bundle/locale type VariantImport = { messageBundleId: string; messageLocale: string; @@ -272,4 +280,3 @@ export const plugin: InlangPlugin = { - [Writing a Plugin](/docs/write-plugin) — Step-by-step guide to building a plugin - [Data Model](/docs/data-model) — Understand bundles, messages, and variants - [Architecture](/docs/architecture) — See how plugins fit in the architecture - diff --git a/docs/settings.md b/docs/settings.md index 3a72ff38a8..9ddd8c921e 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -16,7 +16,7 @@ type ProjectSettings = { locales: string[]; modules?: string[]; experimental?: Record; -} +}; ``` ### baseLocale @@ -53,7 +53,7 @@ URIs to plugin modules. Plugins extend inlang with import/export capabilities fo ```json { "modules": [ - "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@3/dist/index.js", + "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@6.1.4/dist/index.js", "./local-plugin.js" ] } @@ -62,6 +62,7 @@ URIs to plugin modules. Plugins extend inlang with import/export capabilities fo - Must be valid URIs (RFC 3986) - Must end with `.js` - Can be absolute (CDN) or relative paths +- Pin plugin versions that support the current SDK import/export API. For i18next, use `@inlang/plugin-i18next@6` or newer. ### experimental @@ -97,7 +98,7 @@ See each plugin's marketplace page for available settings. "baseLocale": "en", "locales": ["en", "de", "fr", "es"], "modules": [ - "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@3/dist/index.js", + "https://cdn.jsdelivr.net/npm/@inlang/plugin-i18next@6.1.4/dist/index.js", "https://cdn.jsdelivr.net/npm/@inlang/plugin-t-function-matcher@3/dist/index.js" ], "plugin.inlang.i18next": { @@ -114,11 +115,11 @@ See each plugin's marketplace page for available settings. These settings are deprecated and will be removed in SDK v3: -| Setting | Replacement | -|---------|-------------| -| `sourceLanguageTag` | Use `baseLocale` | -| `languageTags` | Use `locales` | -| `{languageTag}` in paths | Use `{locale}` | +| Setting | Replacement | +| ------------------------ | ---------------- | +| `sourceLanguageTag` | Use `baseLocale` | +| `languageTags` | Use `locales` | +| `{languageTag}` in paths | Use `{locale}` | ## Next Steps diff --git a/docs/table_of_contents.json b/docs/table_of_contents.json index 88434fe6b5..9b0fd1d041 100644 --- a/docs/table_of_contents.json +++ b/docs/table_of_contents.json @@ -23,6 +23,10 @@ "path": "./data-model.md", "slug": "data-model" }, + { + "path": "./message-shapes.md", + "slug": "message-shapes" + }, { "path": "./unpacked-project.md", "slug": "unpacked-project" diff --git a/docs/unpacked-project.md b/docs/unpacked-project.md index a04eba1c08..0bbabde414 100644 --- a/docs/unpacked-project.md +++ b/docs/unpacked-project.md @@ -2,24 +2,29 @@ ## What is an unpacked project? -An unpacked project is a directory representation of an `.inlang` file. Instead of a single SQLite binary, the project is stored as a folder with human-readable files. +An unpacked project is the Git-friendly representation of an `.inlang` file. The canonical `.inlang` format is a single binary file: a SQLite database with version control via [lix](https://lix.dev). The unpacked directory exists so changes can be reviewed alongside code. + +Messages, variants, and locale data live in the `.inlang` database. In unpacked Git projects, `settings.json` is the only tracked project file by default; translation files such as `messages/en.json` live outside `project.inlang/` and are connected through plugins. + +> **Important:** `saveProjectToDirectory()` does not make translation data magically appear as files. Bundles, messages, and variants are exported only by an import/export plugin from `settings.modules` or `providePlugins`. If no exporter plugin is configured, save the canonical packed file with `project.toBlob()` instead. ``` project.inlang/ ├── settings.json -├── cache/ +├── .gitignore ├── README.md -└── .gitignore +├── .meta.json +└── cache/ ``` ## Packed vs Unpacked -| | Packed (`.inlang` file) | Unpacked (directory) | -|---|---|---| -| **Format** | Single SQLite file | Directory with files | -| **Git-friendly** | No (binary) | Yes (diffable, mergeable) | -| **Portable** | Yes (one file to share) | No | -| **Use case** | Sharing, backups, tools like Fink | Storing in git repos | +| | Packed (`.inlang` file) | Unpacked (directory) | +| ---------------- | --------------------------------- | ------------------------------------- | +| **Format** | Canonical single binary file | Git-friendly directory representation | +| **Git-friendly** | No (binary) | Yes (diffable, mergeable) | +| **Portable** | Yes (one file to share) | No | +| **Use case** | Sharing, backups, tools like Fink | Storing in git repos | ## Why does it exist? @@ -29,13 +34,13 @@ Most codebases use git for version control. Developers want their translations c ### Git doesn't handle binary files well -An `.inlang` file is a SQLite database (binary). Git can store binary files, but you lose: +An `.inlang` file is binary. Git can store binary files, but you lose: - **Readable diffs** — Binary changes show as "file changed", not what changed - **Merge conflict resolution** — Git can't merge binary files - **Code review** — Teammates can't review translation changes in PRs -An unpacked project solves this. Each file is human-readable, diffable, and mergeable. +An unpacked project solves this for the project configuration. The generated `.gitignore` keeps `settings.json` in Git and ignores generated/cache files. Translation files are stored outside `project.inlang/` according to plugin configuration. ## How to use it @@ -53,10 +58,22 @@ const project = await loadProjectFromDirectory({ }); // Query translations -const messages = await project.db - .selectFrom("message") - .selectAll() - .execute(); +const messages = await project.db.selectFrom("message").selectAll().execute(); +``` + +Check plugin and resource-file errors after loading: + +```typescript +const errors = await project.errors.get(); +if (errors.length > 0) { + throw new AggregateError(errors, "Could not load inlang project"); +} +``` + +Close the project in one-off scripts: + +```typescript +await project.close(); ``` ### Saving a project @@ -65,7 +82,7 @@ Use `saveProjectToDirectory()` to save changes back to disk: ```typescript import { saveProjectToDirectory } from "@inlang/sdk"; -import fs from "node:fs/promises"; +import fs from "node:fs"; await saveProjectToDirectory({ fs: fs, @@ -74,6 +91,19 @@ await saveProjectToDirectory({ }); ``` +`loadProjectFromDirectory()` and `saveProjectToDirectory()` both accept `node:fs`. Passing the same `fs` object to both functions is the least surprising Node setup. + +This writes `settings.json`, metadata files, and any resource files produced by configured exporters. Without an exporter plugin, translation data stays in the packed `.inlang` database and cannot be represented by the unpacked directory. + +To save the canonical packed file instead: + +```typescript +import fs from "node:fs/promises"; + +const blob = await project.toBlob(); +await fs.writeFile("project.inlang", new Uint8Array(await blob.arrayBuffer())); +``` + ### File synchronization You can enable automatic syncing between the filesystem and the in-memory database: @@ -90,12 +120,13 @@ When `syncInterval` is set, changes to files on disk are automatically imported, ## Directory structure -| File | Purpose | -|------|---------| -| `settings.json` | Project configuration (locales, plugins, plugin settings) | -| `.gitignore` | Auto-created, ignores everything except `settings.json` (including itself) | -| `README.md` | Auto-created, explains the folder to coding agents (gitignored) | -| `cache/` | Cached plugin modules (gitignored) | +| File | Purpose | +| --------------- | -------------------------------------------------------------------------------------------------------- | +| `settings.json` | Project configuration (locales, plugins, plugin settings). This is the only file kept in Git by default. | +| `.gitignore` | Auto-created, ignores everything except `settings.json` (including itself) | +| `README.md` | Auto-created, explains the folder to coding agents (gitignored) | +| `.meta.json` | Auto-created SDK metadata (gitignored) | +| `cache/` | Cached plugin modules, usually under `cache/plugins/` (gitignored) | Translation files (like `messages/en.json`) are managed by plugins and stored relative to your project based on plugin configuration. diff --git a/docs/write-plugin.md b/docs/write-plugin.md index f9aa78df1b..3009de89fb 100644 --- a/docs/write-plugin.md +++ b/docs/write-plugin.md @@ -16,6 +16,10 @@ Translation files ──► importFiles() ──► Bundles/Messages/Variant Translation files ◄── exportFiles() ◄── Bundles/Messages/Variants ``` +When a project is loaded with `loadProjectFromDirectory()`, the SDK calls `toBeImportedFiles()` and then `importFiles()` for configured import/export plugins. When a project is saved with `saveProjectToDirectory()`, the SDK calls `exportFiles()`. + +Plugin load and resource-file errors are exposed through `await project.errors.get()` on the loaded project. + ## Step 1: Create the plugin file Create a new file for your plugin: @@ -90,7 +94,8 @@ export const plugin: InlangPlugin = { selectors: [], }); - // One variant with the actual text + // One variant with the actual text. Plugins usually link variants by + // bundle + locale because the SDK can generate message ids during import. variants.push({ messageBundleId: key, messageLocale: file.locale, @@ -105,6 +110,8 @@ export const plugin: InlangPlugin = { }; ``` +Use `messageBundleId` plus `messageLocale` for variants returned from `importFiles()`. Use `messageId` only when your plugin also returns stable message ids. Direct CRUD examples use `messageId` because they operate on existing database rows. + ### Understanding the data model - **Bundle** — A translation key (e.g., `"greeting"`). Groups all locale versions. @@ -112,6 +119,7 @@ export const plugin: InlangPlugin = { - **Variant** — The actual text. Most messages have one variant; plurals have multiple. For a simple `{ "greeting": "Hello" }`: + ``` Bundle: id="greeting" └── Message: bundleId="greeting", locale="en" @@ -153,6 +161,10 @@ exportFiles: async ({ bundles, messages, variants }) => { }, ``` +`exportFiles()` must return `{ locale, name, content }` objects. `name` is used as the fallback filename when the plugin settings do not define `pathPattern`. If the project settings include `settings[plugin.key].pathPattern`, `saveProjectToDirectory()` writes to that path instead. + +Relative `pathPattern` values are resolved relative to the directory that contains `project.inlang/`. + ## Step 5: Add settings (optional) Let users configure your plugin with a settings schema: @@ -174,8 +186,8 @@ export const plugin: InlangPlugin<{ settingsSchema: PluginSettings, toBeImportedFiles: async ({ settings }) => { - const pattern = settings["plugin.my.json"]?.pathPattern - ?? "./messages/{locale}.json"; + const pattern = + settings["plugin.my.json"]?.pathPattern ?? "./messages/{locale}.json"; return settings.locales.map((locale) => ({ path: pattern.replace("{locale}", locale), @@ -222,8 +234,8 @@ export const plugin: InlangPlugin<{ settingsSchema: PluginSettings, toBeImportedFiles: async ({ settings }) => { - const pattern = settings["plugin.my.json"]?.pathPattern - ?? "./messages/{locale}.json"; + const pattern = + settings["plugin.my.json"]?.pathPattern ?? "./messages/{locale}.json"; return settings.locales.map((locale) => ({ path: pattern.replace("{locale}", locale), @@ -263,10 +275,11 @@ export const plugin: InlangPlugin<{ for (const message of messages) { const variant = variants.find((v) => v.messageId === message.id); - const text = variant?.pattern - .filter((p) => p.type === "text") - .map((p) => p.value) - .join("") ?? ""; + const text = + variant?.pattern + .filter((p) => p.type === "text") + .map((p) => p.value) + .join("") ?? ""; if (!filesByLocale[message.locale]) { filesByLocale[message.locale] = {}; @@ -287,7 +300,7 @@ export const plugin: InlangPlugin<{ - Handle variables: Parse `{name}` syntax into expression patterns - Handle plurals: Create multiple variants with match conditions +- [Message Shapes](/docs/message-shapes) — Concrete pattern, match, and declaration shapes - [Plugin API](/docs/plugin-api) — Full type reference - [Data Model](/docs/data-model) — Understand bundles, messages, and variants - [Architecture](/docs/architecture) — See how plugins fit in - diff --git a/docs/write-tool.md b/docs/write-tool.md index 189f3b4ede..e7c534c904 100644 --- a/docs/write-tool.md +++ b/docs/write-tool.md @@ -4,7 +4,11 @@ This guide walks through building a tool that flags missing translations. By the ## What tools can do -Tools read and write translations via the CRUD API. Because inlang handles file format conversion through plugins, your tool works with any translation format — JSON, XLIFF, i18next, etc. +Tools read and write translations through the `.inlang` project file format via the CRUD API. Because plugins handle conversion at the boundary, your tool works with any translation format — JSON, XLIFF, i18next, etc. — without parsing each one directly. + +An `.inlang` project is canonically a single binary file. In Git repositories, it is often unpacked into a directory; `loadProjectFromDirectory()` loads that Git-friendly representation. + +If a `project.inlang/` directory already exists, load it with `loadProjectFromDirectory()`. If your tool is generating a new localization project from scratch, start with `newProject()` and save the packed file with `project.toBlob()`; see [Getting Started](/docs/getting-started) for a runnable create-save-reload example. ``` ┌─────────────────┐ @@ -30,9 +34,26 @@ const project = await loadProjectFromDirectory({ }); ``` -That's it. The project is loaded with all translations from your files (via plugins). +That's it. The project is loaded with all translations from your files (via plugins). The external files are compatibility files; the tool works against the shared `.inlang` data model. + +## Step 2: Check load errors + +Plugin import and resource-file errors are exposed through `project.errors.get()`. Check them before trusting query results. + +```typescript +const errors = await project.errors.get(); +if (errors.length > 0) { + throw new AggregateError(errors, "Could not load inlang project"); +} +``` + +In one-off scripts and CLIs, close the project before the process exits: + +```typescript +await project.close(); +``` -## Step 2: Get project settings +## Step 3: Get project settings ```typescript const settings = await project.settings.get(); @@ -43,18 +64,15 @@ console.log("Locales:", settings.locales); // Locales: ["en", "de", "fr"] ``` -## Step 3: Query all bundles +## Step 4: Query all bundles ```typescript -const bundles = await project.db - .selectFrom("bundle") - .selectAll() - .execute(); +const bundles = await project.db.selectFrom("bundle").selectAll().execute(); console.log(`Found ${bundles.length} translation keys`); ``` -## Step 4: Find missing translations +## Step 5: Find missing translations Now let's find bundles that are missing translations for certain locales: @@ -86,7 +104,7 @@ async function findMissingTranslations(project) { } ``` -## Step 5: Put it together +## Step 6: Put it together Here's a complete CLI tool: @@ -95,39 +113,48 @@ import { loadProjectFromDirectory } from "@inlang/sdk"; import fs from "node:fs"; async function main() { - // Load project const project = await loadProjectFromDirectory({ path: "./project.inlang", fs, }); - const settings = await project.settings.get(); - const bundles = await project.db.selectFrom("bundle").selectAll().execute(); - const messages = await project.db.selectFrom("message").selectAll().execute(); + try { + const errors = await project.errors.get(); + if (errors.length > 0) { + throw new AggregateError(errors, "Could not load inlang project"); + } - // Find missing translations - const missing = []; + const settings = await project.settings.get(); + const bundles = await project.db.selectFrom("bundle").selectAll().execute(); + const messages = await project.db + .selectFrom("message") + .selectAll() + .execute(); - for (const bundle of bundles) { - const bundleMessages = messages.filter((m) => m.bundleId === bundle.id); - const localesWithTranslation = bundleMessages.map((m) => m.locale); + const missing = []; - for (const locale of settings.locales) { - if (!localesWithTranslation.includes(locale)) { - missing.push({ bundleId: bundle.id, locale }); + for (const bundle of bundles) { + const bundleMessages = messages.filter((m) => m.bundleId === bundle.id); + const localesWithTranslation = bundleMessages.map((m) => m.locale); + + for (const locale of settings.locales) { + if (!localesWithTranslation.includes(locale)) { + missing.push({ bundleId: bundle.id, locale }); + } } } - } - // Report results - if (missing.length === 0) { - console.log("All translations complete!"); - } else { - console.log(`Found ${missing.length} missing translations:\n`); - for (const { bundleId, locale } of missing) { - console.log(` - "${bundleId}" is missing locale "${locale}"`); + if (missing.length === 0) { + console.log("All translations complete!"); + } else { + console.log(`Found ${missing.length} missing translations:\n`); + for (const { bundleId, locale } of missing) { + console.log(` - "${bundleId}" is missing locale "${locale}"`); + } + process.exitCode = 1; } - process.exit(1); + } finally { + await project.close(); } } @@ -146,33 +173,34 @@ Found 3 missing translations: - "error_404" is missing locale "fr" ``` -## Using SQL for complex queries +## Building report queries -The CRUD API is powered by Kysely. You can write complex queries: +The CRUD API is powered by Kysely. For reports, it is often clearest to query the rows you need and aggregate them in JavaScript: ```typescript // Find bundles missing a specific locale -const missingGerman = await project.db - .selectFrom("bundle") - .where((eb) => - eb.not( - eb.exists( - eb.selectFrom("message") - .where("message.bundleId", "=", eb.ref("bundle.id")) - .where("message.locale", "=", "de") - ) - ) - ) - .selectAll() +const bundles = await project.db.selectFrom("bundle").selectAll().execute(); +const germanMessages = await project.db + .selectFrom("message") + .select("bundleId") + .where("locale", "=", "de") .execute(); +const translatedBundleIds = new Set( + germanMessages.map((message) => message.bundleId), +); +const missingGerman = bundles.filter( + (bundle) => translatedBundleIds.has(bundle.id) === false, +); + // Count translations per locale -const counts = await project.db - .selectFrom("message") - .select("locale") - .select((eb) => eb.fn.count("id").as("count")) - .groupBy("locale") - .execute(); +const messages = await project.db.selectFrom("message").selectAll().execute(); +const counts = Object.entries( + messages.reduce>((result, message) => { + result[message.locale] = (result[message.locale] ?? 0) + 1; + return result; + }, {}), +).map(([locale, count]) => ({ locale, count })); ``` ## Modifying translations @@ -181,7 +209,7 @@ Tools can also create, update, and delete translations: ```typescript // Add a missing translation -await project.db +const message = await project.db .insertInto("message") .values({ id: crypto.randomUUID(), @@ -189,14 +217,15 @@ await project.db locale: "fr", selectors: [], }) - .execute(); + .returning("id") + .executeTakeFirstOrThrow(); // Add the variant with text await project.db .insertInto("variant") .values({ id: crypto.randomUUID(), - messageId: messageId, + messageId: message.id, matches: [], pattern: [{ type: "text", value: "Bonjour!" }], }) @@ -205,22 +234,31 @@ await project.db ## Saving changes -If you're using the unpacked format, changes sync automatically. To explicitly save: +If you're using the unpacked format, changes sync automatically when `syncInterval` is enabled. To explicitly save: ```typescript import { saveProjectToDirectory } from "@inlang/sdk"; +import fs from "node:fs"; await saveProjectToDirectory({ - fs: fs.promises, + fs, project, path: "./project.inlang", }); ``` +`loadProjectFromDirectory()` and `saveProjectToDirectory()` both accept `node:fs`. `saveProjectToDirectory()` writes translation resource files through import/export plugins. If no exporter plugin is configured, save the canonical packed file instead: + +```typescript +import fs from "node:fs/promises"; + +const blob = await project.toBlob(); +await fs.writeFile("project.inlang", new Uint8Array(await blob.arrayBuffer())); +``` + ## Next steps - [CRUD API](/docs/crud-api) — Full reference for query operations - [Data Model](/docs/data-model) — Understand bundles, messages, and variants - [Unpacked Project](/docs/unpacked-project) — Loading projects from git repos - [Architecture](/docs/architecture) — See how tools fit in - diff --git a/packages/sdk/README.md b/packages/sdk/README.md index e06ab23fd4..57a4350dcb 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -17,24 +17,38 @@ ## Introduction -The inlang SDK is the official specification and parser for `.inlang` files. +The inlang SDK is the reference implementation for reading and writing `.inlang` project files. -`.inlang` files are designed to become the open standard for i18n and enable interoperability between i18n solutions. Such solutions involve apps like [Fink](https://inlang.com/m/tdozzpar/app-inlang-finkLocalizationEditor), libraries like [Paraglide JS](https://inlang.com/m/gerre34r/library-inlang-paraglideJs), or plugins that extend inlang. +`.inlang` files are designed to become the open standard for localization data and make i18n tools work together. Build editors, CLIs, runtimes, agents, and plugins on the same shared project format instead of inventing another file structure. + +An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. + +For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. + +`.inlang` is the canonical project format. Plugins import and export formats like JSON, ICU MessageFormat v1, i18next, and XLIFF for compatibility with existing translation files and runtimes. Version control via lix adds file-level history, merging, and change proposals to `.inlang` projects. + +Messages, variants, and locale data live in the `.inlang` database. External translation files such as `messages/en.json` are compatibility files outside `project.inlang/`, connected through plugins. ### Core Features -- 📁 **File-based**: Interoperability without cloud integrations or lock-in. -- 🖊️ **CRUD API**: Query messages with SQL. -- 🧩 **Plugin System**: Extend the capabilities with plugins. -- 📦 **Import/Export**: Import and export messages in different file formats. -- [Lix Icon**Change control**](https://lix.dev/): Collaboration, change proposals, reviews, and automation. +- 📁 **File-based**: A portable project file, no cloud integrations or lock-in. +- 🖊️ **CRUD API**: Read, write, and query messages with SQL. +- 🧩 **Plugin System**: Connect external translation files to the shared message structure. +- 📦 **Import/Export**: Import and export messages in formats like JSON, XLIFF, and i18next. +- [Lix Icon**Version control via lix**](https://lix.dev/): File-level history, merging, change proposals, reviews, and automation. + +### Core data model + +- **Bundle** — one translatable unit across locales +- **Message** — locale-specific translation for a bundle +- **Variant** — text pattern plus selector matches ## Getting Started > [!Note] -> Inlang files can be unpacked and [stored as directories](#unpacked-inlang-files-directories). The long-term goal is to have portable `.inlang` files. Hence, the documentation refers to files instead of directories. +> Inlang files are single binary files. They can be unpacked and [stored as directories](#unpacked-inlang-files-directories) when you want to review changes in Git. The packed file remains the canonical format. ### Installation @@ -55,6 +69,16 @@ const project = await loadProjectInMemory({ project.* ``` +### Loading an unpacked project from Git + +```ts +import { loadProjectFromDirectory } from "@inlang/sdk"; + +const project = await loadProjectFromDirectory({ + path: "./project.inlang", +}); +``` + ### Next steps Go to the [API reference](#api-reference) to learn how to query messages, changes, and save the project. @@ -66,6 +90,19 @@ The inlang SDK supports plugins to extend its functionality. Plugins can be used to import/export messages in different formats, add custom validation rules, and implement specialized workflows. +## For coding agents and tool builders + +If you are building i18n tooling or generating localization code, target the `.inlang` file format instead of inventing your own JSON schema. + +Why target `.inlang` instead of JSON? + +- One file gives every tool the same data model for bundles, messages, variants, locales, metadata, and settings. +- CRUD operations are available through the SDK instead of custom parsing and rewriting. +- Version control via [lix](https://lix.dev) keeps localization changes reviewable and mergeable. +- Plugins handle JSON, ICU, i18next, XLIFF, and other formats for compatibility. + +Do not invent custom translation-file schemas, create a second localization config when `.inlang` already exists, edit generated/cache files in unpacked projects, or bypass `@inlang/sdk`. + ### Available Plugins Find available plugins on https://inlang.com/c/plugins. @@ -155,7 +192,7 @@ console.log(messages); > [!NOTE] > The inlang plugin for lix is work in progress. If you stumble on issues, please open an issue on the [GitHub](https://github.com/opral/inlang). -The inlang file format uses lix for change control. The lix APIs are exposed via `project.lix.*`. Visit the [lix documentation](https://lix.dev/) for more information on how to query changes. +The inlang file format uses version control via lix. The lix APIs are exposed via `project.lix.*`. Visit the [lix documentation](https://lix.dev/) for more information on how to query changes. ```typescript const changes = await project.lix.db @@ -210,11 +247,11 @@ await project.settings.set(settings) ### Unpacked inlang files (directories) > [!NOTE] -> Unpacked inlang files are a workaround to store inlang files in git. +> Unpacked inlang files are the Git-friendly representation of packed `.inlang` files. > -> Git can't handle binary files. **If you don't intend to store the inlang file in git, do not use unpacked inlang files.** +> Git can store binary files, but plain-file review and merge workflows work better with the unpacked directory. **If you don't intend to store the inlang file in git, use the packed binary file.** > -> Unpacked inlang files are not portable. They depent on plugins that and do not persist [lix change control](https://lix.dev/) data. +> Unpacked inlang files are not portable. They depend on plugins and do not persist [version control via lix](https://lix.dev/) data. ```typescript import { diff --git a/packages/sdk/src/lix-plugin/applyChanges.ts b/packages/sdk/src/lix-plugin/applyChanges.ts index 066946d7ce..774d42d544 100644 --- a/packages/sdk/src/lix-plugin/applyChanges.ts +++ b/packages/sdk/src/lix-plugin/applyChanges.ts @@ -116,7 +116,7 @@ async function handleForeignKeyViolation(args: { .selectAll() // heuristic that getting the last bundle value is fine // and using created_at is fine too. if the change is undesired - // , a user can revert it with lix change control + // , a user can revert it with version control via lix .orderBy("created_at", "desc") .where("type", "=", type) .where((eb) => eb.ref("value", "->>").key("id"), "=", id) diff --git a/packages/sdk/src/project/README_CONTENT.ts b/packages/sdk/src/project/README_CONTENT.ts index bbf94ee53b..45866c4b2c 100644 --- a/packages/sdk/src/project/README_CONTENT.ts +++ b/packages/sdk/src/project/README_CONTENT.ts @@ -12,7 +12,9 @@ This is an [unpacked (git-friendly)](https://inlang.com/docs/unpacked-project) i ## At a glance Purpose: -- This folder stores inlang project configuration and plugin cache data. +- This folder is the Git-friendly representation of an \`.inlang\` project. +- The canonical \`.inlang\` format is a single binary file; this directory is the unpacked version for Git. +- This folder stores project configuration and plugin cache data. - Translation files live outside this folder and are referenced from \`settings.json\`. Safe to edit: @@ -23,26 +25,44 @@ Do not edit: - \`.gitignore\` Key files: -- \`settings.json\` — locales, plugins, file patterns (source of truth) +- \`settings.json\` — locales, plugins, file patterns - \`cache/\` — plugin caches (safe to delete) - \`.gitignore\` — generated +- \`README.md\` — generated, explains this folder +- \`.meta.json\` — generated SDK metadata \`\`\` *.inlang/ -├── settings.json # Locales, plugins, and file patterns (source of truth) -├── cache/ # Plugin caches (gitignored) -└── .gitignore # Ignores everything except settings.json +├── settings.json # Locales, plugins, and file patterns; kept in Git +├── .gitignore # Ignores everything except settings.json +├── README.md # Generated, explains this folder +├── .meta.json # Generated SDK metadata +└── cache/ # Plugin caches, usually cache/plugins/ \`\`\` Translation files (like \`messages/en.json\`) live **outside** this folder and are referenced via plugins in \`settings.json\`. ## What is inlang? -[Inlang](https://inlang.com) is an open file format for building custom localization (i18n) tooling. It provides: +[Inlang](https://inlang.com) is an open project file format for localization. An \`.inlang\` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like \`.sqlite\` for relational data, \`.inlang\` packages localization data into one file that tools can share. + +For Git repositories, that binary file can be unpacked into a directory of plain files. The packed file is the canonical format; this directory is the Git-friendly representation. + +Use inlang when multiple tools, teams, automations, or agents need to use the same localization data. The \`@inlang/sdk\` is the reference implementation for reading and writing \`.inlang\` projects. + +\`.inlang\` is the canonical project format. Plugins import and export external translation files for compatibility with existing runtimes and workflows. Messages, variants, and locale data live in the \`.inlang\` database; translation files such as \`messages/en.json\` live outside this folder and are connected through plugins. Version control via lix adds file-level history, merging, and change proposals to \`.inlang\` projects. + +It provides: - **CRUD API** — Read and write translations programmatically via SQL -- **Plugin system** — Import/export any format (JSON, XLIFF, etc.) -- **Version control** — Built-in version control via [lix](https://lix.dev) +- **Plugin system** — Import/export external translation files (JSON, XLIFF, etc.) +- **Version control** — Version control via [lix](https://lix.dev) + +Core data model: + +- **Bundle** — one translatable unit across locales +- **Message** — locale-specific translation for a bundle +- **Variant** — text pattern plus selector matches \`\`\` ┌──────────┐ ┌───────────┐ ┌────────────┐ @@ -65,15 +85,45 @@ npm install @inlang/sdk \`\`\`ts import { loadProjectFromDirectory, saveProjectToDirectory } from "@inlang/sdk"; +import fs from "node:fs"; -const project = await loadProjectFromDirectory({ path: "./project.inlang" }); -// Query messages with SQLite + [Kysely](https://kysely.dev/) under the hood. +const project = await loadProjectFromDirectory({ + path: "./project.inlang", + fs, +}); +// Query messages with the SDK. const messages = await project.db.selectFrom("message").selectAll().execute(); // Use project.db to update messages. -await saveProjectToDirectory({ path: "./project.inlang", project }); +await saveProjectToDirectory({ + path: "./project.inlang", + fs, + project, +}); + +await project.close(); \`\`\` +\`loadProjectFromDirectory()\` and \`saveProjectToDirectory()\` both accept \`node:fs\`. \`saveProjectToDirectory()\` writes translation files through import/export plugins. If no exporter plugin is configured, save the canonical packed file with \`project.toBlob()\` instead. + +## For coding agents and tool builders + +Target \`.inlang\` instead of inventing a custom JSON schema. + +Why target \`.inlang\`? + +- One file gives every tool the same data model for bundles, messages, variants, locales, metadata, and settings. +- CRUD operations are available through the SDK instead of custom parsing and rewriting. +- Version control via [lix](https://lix.dev) keeps localization changes reviewable and mergeable. +- Plugins handle JSON, ICU, i18next, XLIFF, and other formats for compatibility. + +Do not: + +- Invent a custom \`locales/en.json\` schema as the project format. +- Create a new localization config when a \`.inlang\` project already exists. +- Edit generated or cached files in this directory. +- Bypass \`@inlang/sdk\` when reading or writing \`.inlang\`. + ## Ideas for custom tooling - Translation health dashboard (missing/empty/stale messages) diff --git a/packages/sdk/src/project/saveProjectToDirectory.test.ts b/packages/sdk/src/project/saveProjectToDirectory.test.ts index 6dd18e5b12..fcb590b438 100644 --- a/packages/sdk/src/project/saveProjectToDirectory.test.ts +++ b/packages/sdk/src/project/saveProjectToDirectory.test.ts @@ -60,6 +60,76 @@ test("it should overwrite all files to the directory except the db.sqlite file", expect(updatedSettings.locales).toEqual(["en", "fr", "mock"]); }); +test("accepts the node:fs style module with a promises namespace", async () => { + const volume = Volume.fromJSON({}); + + const project = await loadProjectInMemory({ + blob: await newProject({ + settings: { + baseLocale: "en", + locales: ["en"], + }, + }), + }); + + await saveProjectToDirectory({ + fs: volume as any, + project, + path: "/foo/bar.inlang", + }); + + const settings = await volume.promises.readFile( + "/foo/bar.inlang/settings.json", + "utf-8" + ); + expect(JSON.parse(settings as string).locales).toEqual(["en"]); +}); + +test("creates exporter target directories from pathPattern", async () => { + const volume = Volume.fromJSON({}); + const mockPlugin: InlangPlugin = { + key: "mock", + exportFiles: async () => [ + { + locale: "en", + name: "fallback.json", + content: new TextEncoder().encode(JSON.stringify({ greeting: "Hi" })), + }, + ], + }; + + const project = await loadProjectInMemory({ + blob: await newProject({ + settings: { + baseLocale: "en", + locales: ["en"], + modules: [], + mock: { + pathPattern: "./messages/{locale}.json", + }, + }, + }), + providePlugins: [mockPlugin], + }); + + await project.db + .insertInto("bundle") + .values({ id: "greeting", declarations: [] }) + .execute(); + + await saveProjectToDirectory({ + fs: volume as any, + project, + path: "/foo/bar.inlang", + }); + + const exported = await volume.promises.readFile( + "/foo/messages/en.json", + "utf-8" + ); + expect(JSON.parse(exported as string)).toEqual({ greeting: "Hi" }); +}); + // Users were confused by project_id, and without sync a stable id is rarely needed. test("it should not write project_id to disk", async () => { const mockFs = Volume.fromJSON({ @@ -404,6 +474,29 @@ test("emits a .meta.json file with the sdk version", async () => { expect(meta.highestSdkVersion).toBe(ENV_VARIABLES.SDK_VERSION); }); +test("throws when saving translation data to a directory without an exporter plugin", async () => { + const fs = Volume.fromJSON({}); + + const project = await loadProjectInMemory({ + blob: await newProject(), + }); + + await project.db + .insertInto("bundle") + .values({ id: "greeting", declarations: [] }) + .execute(); + + await expect( + saveProjectToDirectory({ + fs: fs.promises as any, + project, + path: "/foo/bar.inlang", + }) + ).rejects.toThrow( + "saveProjectToDirectory cannot write bundles, messages, or variants without an import/export plugin" + ); +}); + test("updates an existing README.md file", async () => { const fs = Volume.fromJSON({ "/foo/bar.inlang/README.md": "custom readme", diff --git a/packages/sdk/src/project/saveProjectToDirectory.ts b/packages/sdk/src/project/saveProjectToDirectory.ts index 69d1e6c15f..081c12ec71 100644 --- a/packages/sdk/src/project/saveProjectToDirectory.ts +++ b/packages/sdk/src/project/saveProjectToDirectory.ts @@ -1,3 +1,4 @@ +import type nodeFs from "node:fs"; import type fs from "node:fs/promises"; import type { InlangProject } from "./api.js"; import path from "node:path"; @@ -18,6 +19,33 @@ async function fileExists(fsModule: typeof fs, filePath: string) { } } +type SaveProjectFs = typeof fs | typeof nodeFs; + +function getPromisesFs(fsModule: SaveProjectFs): typeof fs { + return "promises" in fsModule ? fsModule.promises : fsModule; +} + +async function assertTranslationDataCanBeExported(project: InlangProject) { + const plugins = await project.plugins.get(); + const hasExporter = plugins.some( + (plugin) => plugin.exportFiles || plugin.saveMessages + ); + if (hasExporter) { + return; + } + + const [bundle, message, variant] = await Promise.all([ + project.db.selectFrom("bundle").select("id").limit(1).executeTakeFirst(), + project.db.selectFrom("message").select("id").limit(1).executeTakeFirst(), + project.db.selectFrom("variant").select("id").limit(1).executeTakeFirst(), + ]); + if (bundle || message || variant) { + throw new Error( + "saveProjectToDirectory cannot write bundles, messages, or variants without an import/export plugin. Add a plugin to settings.modules/providePlugins, or save the canonical .inlang file with project.toBlob()." + ); + } +} + /** * Saves a project to a directory. * @@ -26,7 +54,7 @@ async function fileExists(fsModule: typeof fs, filePath: string) { * * @example * await saveProjectToDirectory({ - * fs: await import("node:fs/promises"), + * fs: await import("node:fs"), * project, * path: "./project.inlang", * }); @@ -34,8 +62,10 @@ async function fileExists(fsModule: typeof fs, filePath: string) { export async function saveProjectToDirectory(args: { /** * The file system module to use for writing files. + * + * Accepts either `node:fs` or `node:fs/promises`. */ - fs: typeof fs; + fs: SaveProjectFs; /** * The inlang project to save. */ @@ -55,6 +85,11 @@ export async function saveProjectToDirectory(args: { if (args.path.endsWith(".inlang") === false) { throw new Error("The path must end with .inlang"); } + if (!args.skipExporting) { + await assertTranslationDataCanBeExported(args.project); + } + const fsModule = getPromisesFs(args.fs); + const files = await args.project.lix.db .selectFrom("file") .selectAll() @@ -65,7 +100,7 @@ export async function saveProjectToDirectory(args: { ); const existingMeta = await readProjectMeta({ - fs: args.fs, + fs: fsModule, projectPath: args.path, }); const highestSdkVersion = @@ -83,9 +118,9 @@ export async function saveProjectToDirectory(args: { const readmePath = path.join(args.path, "README.md"); const gitignorePath = path.join(args.path, ".gitignore"); const shouldWriteReadme = - shouldWriteMetadata || !(await fileExists(args.fs, readmePath)); + shouldWriteMetadata || !(await fileExists(fsModule, readmePath)); const shouldWriteGitignore = - shouldWriteMetadata || !(await fileExists(args.fs, gitignorePath)); + shouldWriteMetadata || !(await fileExists(fsModule, gitignorePath)); // write all files to the directory for (const file of files) { @@ -93,17 +128,17 @@ export async function saveProjectToDirectory(args: { continue; } const p = path.join(args.path, file.path); - await args.fs.mkdir(path.dirname(p), { recursive: true }); - await args.fs.writeFile(p, new Uint8Array(file.data)); + await fsModule.mkdir(path.dirname(p), { recursive: true }); + await fsModule.writeFile(p, new Uint8Array(file.data)); } if (shouldWriteGitignore) { - await args.fs.writeFile(gitignorePath, gitignoreContent); + await fsModule.writeFile(gitignorePath, gitignoreContent); } if (shouldWriteReadme) { // Write README.md for coding agents - await args.fs.writeFile( + await fsModule.writeFile( readmePath, new TextEncoder().encode(README_CONTENT) ); @@ -111,7 +146,7 @@ export async function saveProjectToDirectory(args: { if (shouldWriteMetadata) { const metaContent = JSON.stringify({ highestSdkVersion }, null, 2); - await args.fs.writeFile( + await fsModule.writeFile( path.join(args.path, ".meta.json"), new TextEncoder().encode(metaContent) ); @@ -161,15 +196,12 @@ export async function saveProjectToDirectory(args: { pathPattern.replace(/\{(languageTag|locale)\}/g, file.locale) ) : absolutePathFromProject(args.path, file.name); - const dirname = path.dirname(p); - if ((await args.fs.stat(dirname)).isDirectory() === false) { - await args.fs.mkdir(dirname, { recursive: true }); - } + await fsModule.mkdir(path.dirname(p), { recursive: true }); if (p.endsWith(".json")) { try { - const existing = await args.fs.readFile(p, "utf-8"); + const existing = await fsModule.readFile(p, "utf-8"); const stringify = detectJsonFormatting(existing); - await args.fs.writeFile( + await fsModule.writeFile( p, new TextEncoder().encode( stringify(JSON.parse(new TextDecoder().decode(file.content))) @@ -178,10 +210,10 @@ export async function saveProjectToDirectory(args: { } catch { // write the file to disk (json doesn't exist yet) // yeah ugly duplication of write file but it works. - await args.fs.writeFile(p, new Uint8Array(file.content)); + await fsModule.writeFile(p, new Uint8Array(file.content)); } } else { - await args.fs.writeFile(p, new Uint8Array(file.content)); + await fsModule.writeFile(p, new Uint8Array(file.content)); } } } @@ -194,7 +226,7 @@ export async function saveProjectToDirectory(args: { await plugin.saveMessages({ messages: bundlesNested.map((b) => toMessageV1(b)), // @ts-expect-error - legacy - nodeishFs: withAbsolutePaths(args.fs, args.path), + nodeishFs: withAbsolutePaths(fsModule, args.path), settings, }); } diff --git a/packages/website-v2/src/components/Footer.tsx b/packages/website-v2/src/components/Footer.tsx index 21fbd8d179..5d0f121371 100644 --- a/packages/website-v2/src/components/Footer.tsx +++ b/packages/website-v2/src/components/Footer.tsx @@ -89,7 +89,7 @@ export default function Footer() {

- The open file format for localization (i18n). + The open project file format for localization (i18n).

{socialMediaLinks.map((link) => ( diff --git a/packages/website-v2/src/components/Header.tsx b/packages/website-v2/src/components/Header.tsx index c6b05e7e02..89b229ee93 100644 --- a/packages/website-v2/src/components/Header.tsx +++ b/packages/website-v2/src/components/Header.tsx @@ -2,7 +2,7 @@ import { useState } from "react"; import { Link, useLocation } from "@tanstack/react-router"; import { getGithubStars } from "../github-stars-cache"; -const ecosystemLinks = [ +const builtOnInlangLinks = [ { label: "Tools", to: "/c/tools", @@ -242,14 +242,14 @@ export default function Header() {
- {/* Subheader: Ecosystem categories */} + {/* Subheader: format categories */}
- Ecosystem + Built on .inlang