lingui extract-experimental bugs
#1797
Comments
|
Well, i've found a problem why some files not get processed by extractor. Unfirtunately it's not easy to fix, because it's need to change literally everthying in how this is processed now. The simple explanation of the process now:
The problem here is that esbuild while bundling doing a lot of work related to module imports. It renames, deduplicate and etc. Macro is able to understand renamed imports, but macro is also responsible of turning macro input into runtime import. And here where problems appears. Macro is not smart enough to analyze what import already inserted in the file and in what order (the esbuild could generate very complicated setups, because what actually bundlers do) So after macro expannding some imports got deleted and extractor did not recognize these nodes as belonging to the Lingui Esbuild bundle// src/pages/index.page.tsx
import { Plural as Plural2, t as t2, Trans as Trans3 } from "@lingui/macro";
import path from "path";
import Head from "next/head";
// src/components/AboutText.tsx
import { Trans } from "@lingui/macro";
function AboutText() {
return <p>
<Trans>Hello, world</Trans>
<br />
<Trans id="message.next-explanation">Next.js is an open-source React front-end development web framework that enables functionality such as server-side rendering and generating static websites for React based web applications. It is a production-ready framework that allows developers to quickly create static and dynamic JAMstack websites and is used widely by many large companies.</Trans>
</p>;
}
// src/components/Developers.tsx
import { useState } from "react";
import { Trans as Trans2, Plural } from "@lingui/macro";
function Developers() {
const [selected, setSelected] = useState("1");
return <div>
<p><Trans2>Plural Test: How many developers?</Trans2></p>
<div style={{ display: "flex", justifyContent: "space-evenly" }}>
<select
value={selected}
onChange={(evt) => setSelected(evt.target.value)}
>
<option value="1">1</option>
<option value="2">2</option>
</select>
<p><Plural value={selected} one="Developer" other="Developers" /></p>
</div>
</div>;
}
// src/components/Switcher.tsx
import { useRouter } from "next/router";
import { useState as useState2 } from "react";
import { msg } from "@lingui/macro";
import { useLingui } from "@lingui/react";
var languages = {
en: msg`English`,
sr: msg`Serbian`,
es: msg`Spanish`
};
function Switcher() {
const router = useRouter();
const { i18n: i18n2 } = useLingui();
const [locale, setLocale] = useState2(
router.locale.split("-")[0]
);
function handleChange(event) {
const locale2 = event.target.value;
setLocale(locale2);
router.push(router.pathname, router.pathname, { locale: locale2 });
}
return <select value={locale} onChange={handleChange}>{Object.keys(languages).map((locale2) => {
return <option value={locale2} key={locale2}>{i18n2._(languages[locale2])}</option>;
})}</select>;
}
// src/pages/index.page.tsx
import styles from "../styles/Index.module.css";
// src/utils.ts
import { i18n } from "@lingui/core";
import { useRouter as useRouter2 } from "next/router";
import { useEffect, useState as useState3 } from "react";
async function loadCatalog(locale, pathname) {
if (pathname === "_error") {
return {};
}
const catalog = await import(`@lingui/loader!./locales/src/pages/${pathname}.page/${locale}.po`);
return catalog.messages;
}
// src/pages/index.page.tsx
import { useLingui as useLingui2 } from "@lingui/react";
var getStaticProps = async (ctx) => {
const fileName = __filename;
const cwd = process.cwd();
const { locale } = ctx;
const pathname = path.relative(cwd, fileName).replace(".next/server/pages/", "").replace(".js", "");
const translation = await loadCatalog(locale || "en", pathname);
return {
props: {
translation
}
};
};
var Index = () => {
useLingui2();
return <div className={styles.container}>
<Head>
{
/*
The Next Head component is not being rendered in the React
component tree and React Context is not being passed down to the components placed in the <Head>.
That means we cannot use the <Trans> component here and instead have to use `t` macro.
*/
}
<title>{t2`Translation Demo`}</title>
<link rel="icon" href="/favicon.ico" />
</Head>
<main className={styles.main}>
<Switcher />
<h1 className={styles.title}><Trans3>
{"Welcome to "}
<a href="https://nextjs.org">Next.js!</a>
</Trans3></h1>
<h2><Trans3>Plain text</Trans3></h2>
<h2>{t2`Plain text`}</h2>
<h2><Trans3>
<a href="https://nextjs.org">Next.js</a>
{" say hi."}
</Trans3></h2>
<h2><Trans3>
{"Wonderful framework "}
<a href="https://nextjs.org">Next.js</a>
{" say hi."}
</Trans3></h2>
<h2><Trans3>
{"Wonderful framework "}
<a href="https://nextjs.org">Next.js</a>
{" say hi. And "}
<a href="https://nextjs.org">Next.js</a>
{" say hi."}
</Trans3></h2>
<div className={styles.description}><AboutText /></div>
<Developers />
<div>
<Plural2 value={1} one="# Person" other="# Persons" />
<br />
<Plural2 value={2} one="# Person" other="# Persons" />
</div>
</main>
</div>;
};
var index_page_default = Index;
export {
index_page_default as default,
getStaticProps
};Bundle after Macroimport path from "path";
import Head from "next/head";
// src/components/Developers.tsx
import { useState } from "react";
function Developers() {
const [selected, setSelected] = useState("1");
return <div>
<p>
<Trans id={"jryo3h"} message={"Plural Test: How many developers?"} />
</p>
<div style={{
display: "flex",
justifyContent: "space-evenly"
}}>
<select value={selected} onChange={evt => setSelected(evt.target.value)}>
<option value="1">1</option>
<option value="2">2</option>
</select>
<p>
<Trans id={"y0Og8v"} message={"{selected, plural, one {Developer} other {Developers}}"} values={{
selected: selected
}} />
</p>
</div>
</div>;
}
// src/components/Switcher.tsx
import { useRouter } from "next/router";
import { useState as useState2 } from "react";
import { useLingui, Trans } from "@lingui/react";
var languages = {
en:
/*i18n*/
{
id: "lYGfRP",
message: "English"
},
sr:
/*i18n*/
{
id: "9aBtdW",
message: "Serbian"
},
es:
/*i18n*/
{
id: "65A04M",
message: "Spanish"
}
};
function Switcher() {
const router = useRouter();
const {
i18n: i18n2
} = useLingui();
const [locale, setLocale] = useState2(router.locale.split("-")[0]);
function handleChange(event) {
const locale2 = event.target.value;
setLocale(locale2);
router.push(router.pathname, router.pathname, {
locale: locale2
});
}
return <select value={locale} onChange={handleChange}>
{Object.keys(languages).map(locale2 => {
return <option value={locale2} key={locale2}>
{i18n2._(languages[locale2])}
</option>;
})}
</select>;
}
// src/pages/index.page.tsx
import styles from "../styles/Index.module.css";
// src/utils.ts
import { i18n } from "@lingui/core";
import { useRouter as useRouter2 } from "next/router";
import { useEffect, useState as useState3 } from "react";
async function loadCatalog(locale, pathname) {
if (pathname === "_error") {
return {};
}
const catalog = await import(`@lingui/loader!./locales/src/pages/${pathname}.page/${locale}.po`);
return catalog.messages;
}
// src/pages/index.page.tsx
import { useLingui as useLingui2 } from "@lingui/react";
var getStaticProps = async ctx => {
const fileName = __filename;
const cwd = process.cwd();
const {
locale
} = ctx;
const pathname = path.relative(cwd, fileName).replace(".next/server/pages/", "").replace(".js", "");
const translation = await loadCatalog(locale || "en", pathname);
return {
props: {
translation
}
};
};
var Index = () => {
useLingui2();
return <div className={styles.container}>
<Head>
{/*
The Next Head component is not being rendered in the React
component tree and React Context is not being passed down to the components placed in the <Head>.
That means we cannot use the <Trans> component here and instead have to use `t` macro.
*/}
<title>{i18n._(
/*i18n*/
{
id: "HjmF2U",
message: "Translation Demo"
})}</title>
<link rel="icon" href="/favicon.ico" />
</Head>
<main className={styles.main}>
<Switcher />
<h1 className={styles.title}>
<Trans id={"QpzMsa"} message={"Welcome to <0>Next.js!</0>"} components={{
0: <a href="https://nextjs.org" />
}} />
</h1>
<h2>
<Trans id={"pOo4Aa"} message={"Plain text"} />
</h2>
<h2>{i18n._(
/*i18n*/
{
id: "pOo4Aa",
message: "Plain text"
})}</h2>
<h2>
<Trans id={"L7eJFB"} message={"<0>Next.js</0>say hi."} components={{
0: <a href="https://nextjs.org" />
}} />
</h2>
<h2>
<Trans id={"4rdBWT"} message={"Wonderful framework <0>Next.js</0>say hi."} components={{
0: <a href="https://nextjs.org" />
}} />
</h2>
<h2>
<Trans id={"mQ2tsK"} message={"Wonderful framework <0>Next.js</0>say hi. And <1>Next.js</1>say hi."} components={{
0: <a href="https://nextjs.org" />,
1: <a href="https://nextjs.org" />
}} />
</h2>
<div className={styles.description}>
<AboutText />
</div>
<Developers />
<div>
<Trans id={"9C7Ncf"} message={"{0, plural, one {# Person} other {# Persons}}"} values={{
0: 1
}} />
<br />
<Trans id={"9C7Ncf"} message={"{0, plural, one {# Person} other {# Persons}}"} values={{
0: 2
}} />
</div>
</main>
</div>;
};
var index_page_default = Index;
export { index_page_default as default, getStaticProps };Note how code of Solution which i'm actually thinking of is to change order of processing in the following way:
That makes process more safe, because macro would process untouched files and would be more future-safe. Other option could be to implement a separate AST transformation which would hoist all lingui imports to the very top of the file. |
|
continuing developing an idea: SWC plugin unfortunately was not designed to work in extractor pipeline. It always work in a "production" mode means always remove non-essential properties which actually used by extractor. So some additional work would be required here. About hoisting imports - this could fix this exact case and could break in some other, so i would rather skip this option. |
|
It seems so hard to process, but in our big project seems not find out this question. I just find it when I create the reproduction 😂 So use On the other hand, how about the first question: Space is losted when plain text follow JSX element under BTW, Esbuild bundle related code format is broken. |
Yes, it was created as PoC, and actually there not match attention to it, so i could not get enough feedback to make it production ready.
I didn't investigate it, but i think the problem is similar, when code is processed by esbuild it gets reprinted and space might be dropped. |
|
@thekip Is there any plan to refactor the PoC to make it fully usable? |
The issue also happens to |
|
Actually, some spaces are deleted intentionally. You can read unit-tests to understand the cases. The issue would be if the final line would be different between what left by macro in the code and what was extracted to catalog. |
How about make extractor extract id directly regardless of whether module imports? |
|
@thekip I found a another solution, use How about this idea? |
|
I considered that solution in the beginning, even look into that exact library. Solution with esbuild i like more because:
I still think we need to continue with esbuild. The problems mentioned in this issue are fixable, i just don't have a time for that currently. |
|
How far @thekip are we of converting this feature in something stable? Probably if you detail which tasks are pending I can help on my free time :) We noticed that our catalogs are getting large and this splitting by page would be amazing |
|
Hi @semoal i see following tasks:
I think this 3 is essential to at least unblock following adoption of the extractor. From my side, i can help with SWC Rust plugin, and can explain how extractor works now. After basic functionality become stable we can do performance optimizations:
|
|
An update here:
I created a separate discussion, please add your ideas there.
This one is fixed in #1867 separate test case was added |
This reverts commit 797fe4e.
Fixed: #1882 |
|
@thekip Thanks for your great work, I will try |
|
@thekip Unfortunately, with 4.8.0-next.1, const result = curringFoo()()
console.log('curringFoo', result)Throw error: But it works after I try to call currying function step by step like this: const curryingBar = curringFoo()
const result = curryingBar()
console.log('curringFoo', result) |
|
Interesting, thanks for report, that probably related to useLingui and this refactoring I will take a look. |
|
I use .mjs files but in imports I omit the file extension such as |
|
https://esbuild.github.io/api/#resolve-extensions // lingui.config.ts
experimental: {
extractor: {
/// ...
resolveEsbuildOptions: (options: import('esbuild').BuildOptions) => {
options.resolveExtensions = ['.ts', '.js', '.jsx', '.tsx', '.mjs'];
return options;
}
},
}, |
|
It's extracting messages from some of the detected pages but for some reason this one it fails. There is no Event.jsx (just an Event.mjs file). It's the same as the other files which are being extracted. Getting rid of the Event.mjs file removes the error and the other files still extract fine so it must be something within my Event.mjs file (changing the name doesn't do anything). I also would like to take the opportunity to propose an alternate strategy to build-time resolution of dependencies. Currently I use Relay, and I have to compose data requirements from smaller components up until the final page component where it collects all the data requirements. But Relay doesn't auto-detect the data requirements at the page level, but rather has the developer statically define the data dependencies of the sub-components for every level of composed component. For example a UserProfilePage component may compose of UserName, UserEmail, etc, and UserName and UserEmail define the data requirements it needs, and UserProfilePage you define (via GraphQL fragments) the sum of the data requirements of its constituent components. I attempted a similar strategy with Lingui where each component file defines its own loadMessages function which will load the message bundle given the locale, and then call the lingui api to load it and merge the messages, and return this as a promise. So for example UserName and UserEmail indicate that they need to load some ../../locales/en/User[Name/Email].ts file and merge the messages into the Lingui context. The UserProfilePage component then composes the requirements of these two child components, and then when the page route is loaded it executes all its loadMessage functions (provided as an array of Promises). The UserProfilePage render function then uses React Suspense to wait for the message bundles to load first. This actually works fine with current Lingui as is. I just tell the Lingui extractor to split the message bundles into one file per component. The only problem is that there are duplicate translations in the .po files. I don't think it's an issue that there are duplicate entries in the compiled .ts files, but duplicate entries in .po files means it's tough to translate without deduplication. Is there a way to have it generate a merged .po file for a translator to use, and then during compile separate that back into individual per-component .ts files? If not, what do you think of supporting this feature to support component-based bundle loading? |
There is something in the bundle itself, if you can share it (maybe privately) i will fix it. Regarding your strategy - that's sounds interesting, I think that could be an option for some people. Would be nice if you can share it with community. I also use relay on my project, and i understand what you're talking about. However, I'm not sure that this approach would suit all devlopers. Relay, thanks to graphql language and fragment masking (and types if you use TS) make developer experience on very high level, unfortunately achieve the same experience with lingui would be problematic.
Any TMS will help you with this. Deduplicating should happen on the level of TMS, not on the level of individual files. The same challenge would be with experimental extractor as well, messages might be duplicated between pages. |
|
Yes I agree the problem would arise where we forget to call some loadMessages() of a child component and the best case the translation never shows or worst case it doesn't render and permanently Suspends. Actually here is the file. It is generated by the ReScript compiler so it's not hand written JS. |
|
@yunsii unfortunately I realized that the strategy i went with, when bundle everything and then apply macro + extractor will not work properly even with all changes I did past few months. In simple cases that might work, but in more complicated cases like you showed that might be broken. Any bundler might rename / restructure identifier references, and that would cause mismatch between runtime code an extraction. Simple example is: import { t } from "@lingui/macro";
import { name } from "./names";
t`Hello ${name}` // extracts to "Hello {name}"Could be bundled ang changed to: // after esbuild
import { t } from "@lingui/macro";
import _names1 from "./names";
t`Hello ${_names1.name}` // extracts to "Hello {0}"So the only way to go is to process every file by macro before it gets to the bundler. The RsPack now supports swc plugins, and i'm considering to change underlying esbuild to RsPack with lingui SWC plugin, to avoid intercommunication between runtimes which would be the case with esbuild (esbuild <-> js <-> babel / swc) |
|
Wow, it seems really hard. But why |
|
Because it doesn't take any preprocess step (esbuild) and consume sources directly. |
|
RsPack sounds like a good approach, I was thinking along similar lines except using Turbopack. |
|
I have to wait the brand new approach 😂 |
|
@thekip if you need any extra hands on this, let me know. We're currently in the uncomfortable position of only really being able to use the tree-based approach, but buckling under the extremely slow extraction times in our codebase. We may end up trying to build our own extractor as an interim measure anyway. |
|
@AndrewIngram yes i need an extra hands on this. Could you measure what exactly consuming a lot of time? I think the slowest parts would be parsing the bundles with babel and following extracting. Using tree-approach the same files could end up in many bundles and would be parsed many times compared to just file-by-file approach. I see there few options:
|
|
Right now our setup is to have one entry point (we generate a dummy page that imports all the other pages), we do this because ultimately we think one message bundle per page will be too granular, so we're starting from the other end. We've overridden a lot of the esbuild config to:
This got the baseline extraction times down to something reasonable, i.e. when there's only a handful of strings it's fairly quick. But as we've been instrumenting things, it's been getting progressively slow. I've dived into where the bottleneck is, and it's the babel-plugin-macro step, of the 1 minute extract, it consistently takes about 55 seconds. One thing I tried yesterday was to use swc to perform that step, before feeding it into babel for the final string extraction. But there's no way to configure swc to preserve JSX, so it was only able to pick up non-JSX strings -- though it did seem to be faster. If the final extraction plugin was able to operate on the post-JSX version of the AST, I think this would work -- it would also mean an RSpack-based extractor might would be viable, because using a 2nd babel transform for the final step doesn't appear to be a bottleneck. |
How did you measure it? I just want to be sure that this is caused exactly by this plugin and not by the parsing by babel in general. |
|
You also could try with version from |
I wasn't too scientific, I was just shoving logs in at various points in the process and this was the step that took most of the time. But I can try and get something more concrete |
|
I hit the wall I was expecting with Rspack -- Can generate the tree-shaken bundle with the swc plugin used, but as there's no way to get swc to preserve JSX, I can't feed it into babel for the final extraction step. |
|
Yes, Rspack is not the way at least now. The SWC plugin has to be changed in some way to work in extraction mode (it doesn't have a 100% feature parity with babel's version). Also, it could be modified to pick up non-macro Trans components and transform them into Message descriptor so they would be picked up by extractor even if the JSX is not preserved (something like this for reference) Anyway, after you discovered that SWC could not preserve JSX , I think going in the way of RSPack would be quite long path, and i don't have a bandwidth for that now. Let's stick with esbuild and babel. Could you check on your project does all this time caused by babel parsing or by macro plugin itself? You can alter the sources in the |
|
Without the Macro, the babel step takes ~3.5 seconds, with it, it takes ~66 seconds. |
|
Wow, ok. Let's check than, does this change #1867 helps? (you can build a next branch by yourself following these instructions. The problem is, i don't have a big enough project to play with. If you can point me to the one of the opensource projects which may reproduce the performance issue, i will try to debug performance on my own. |
https://github.com/bluesky-social/social-app contains a lingui.config.js, probably is not hard to set-up |
Bluesky isn't using the experimental extractor though, so it's unlikely they have any individually huge files to transform. In my case the bundle file is an absolute beast, approx 25mb, even though the number of instrumented strings is still relatively low (approx 500). |
On 4.7.1 On Next So yeah, you could call that an improvement 😅 |
I'm going to explore this avenue more -- i.e an esbuild plugin that conditionally invokes babel, even if it involves a performance hit at the bundling stage, it feels like the only way to mitigate all the other issues people have raised -- short of rewriting everything for swc. |
|
Okay, in my project i've cobbled together a custom tree shaking extractor, that works as follows:
Despite adding the babel macro plugin to the esbuild step, moving it out of the final extractor resulting in a net perf gain of approximately 5x (on one package, the extraction time goes from 160 seconds to 35 seconds). This also fixes many of the issues people have flagged with the experimental extractor (strings not getting extracted, being extracted incorrectly, or interpolated variables having the wrong names). I expect to see even faster perf if I switch to the babel plugin that's not built on babel-plugin-macros Another idea that might further boost perf is to run the extractor plugin during esbuild too, and then filter the messages to just those whose ID is found in the final generated bundle file (given the pseudorandom ID format, just grepping the file should be sufficient -- no 2nd babel pass needed). |
|
Yep, that is what i exactly wanted to do. Move macro per-file basis. Would you be minded to share your solution in a PR? I than could help you to finalize it to make it merged. You can DM me in discord if you have any questions about contributing.
This also fixed in the
That might not work for non-macro usecases because they have to provide ids by themselves. But the idea could be expanded. For example extractor could annotate extracted symbols with specific mark (say generate an ID) and then do what you said with grepping, |
|
I don't have it implemented as a fork right now, but rather a separate in-house script with a lot of copy-pasted code; so i'll start off by putting together a gist with the key parts. I think the only notable parts are really the esbuild config and the custom extractor. |
|
@AndrewIngram i prepared a PR for the next branch with your ideas #1958 |
This PR merged in new version? |
|
@Shperung it was released in the v5.0.0-next.0. Currently, the |
|
@AndrewIngram do you have any feedback on the deps extractor? Do you use it on your project in production? I'm thinking is it stable enough to be considered non-experimental, and "experimental" is removed from its name. |
|
We're using it in production, and it seems to be working. I think there were some variable alias issues when it came to trying to use the new placeholder annotation functionality, so we've disabled that for now. However, our extract times are gradually creeping up, so I've been thinking of an alternative approach which may perform better on projects with lots of entry points where there's excessive duplicate extraction work with the current tree-based approach. Something like this:
|
Yes, this's what we're going through 😂 |
|
@AndrewIngram thanks for reply.
Could you share an example?
Maybe it would be easier just to enable hard cache for babel in the esbuild, so macro would not call for the same file again and again? Also if you have a lot of entrypoints you can benefit from multithreading on the extraction step. Proposed by you approach might work, but sounds overcomplicated for me. |
Describe the bug
Transchildren has plain text follow JSX element, space is lostedTo Reproduce
https://github.com/yunsii/lingui-examples-nextjs-swc-1797
Expected behavior
v4.5.0@lingui/swc-pluginbabel-macro-pluginThe text was updated successfully, but these errors were encountered: