Repository navigation
Epic: Move away from pages folder #8262
Description
Activity
feel -0.5 to put MDX on
appdir because I don't like to mix routing and content.feel +1 to have a
contentdir that store md/mdx and these are processed by the Next.js app router. And for layout instead of injecting them by frontMatter we can just define it on theapprouter.Reacted by Md Fayaj Nakibfeel -0.5 to put MDX on
appdir because I don't like to mix routing and content.feel +1 to have a
contentdir that store md/mdx and these are processed by the Next.js app router. And for layout instead of injecting them by frontMatter we can just define it on theapprouter.I get the feeling, but right now we have some pretty custom infrastructure that deviates from standard Next.js and App router was made for content too. This differentiation has been the root cause for many problems with our Next.js upgrades + the main piece of effort and issues with our Cloudflare and OpenNext integration. So yeah, I definitely would like to move away from there :)
Reacted by Md Fayaj NakibMy main concern here is that every blog post would need to be its own folder, correct? At least, that's my understanding of https://nextjs.org/docs/app/guides/mdx
As I mentioned above, and as @ovflowd agreed on Slack, the "one page per folder" structure is a major blocker for this change. It's infeasible for us to make a new folder for every new blog post and learn article we have.
Reacted by Efe Karasakal and Tobias NießenReacted by Md Fayaj Nakibis the reasoning beyond busy work? we can script/automate that away
is the reasoning beyond busy work? we can script/automate that away
The reasoning is DX. The
pagesfolder, IMO, is very neat. Making a new folder for every page is a lot less... 'neat'.Regarding a script, however, I wrote:
import fs from 'node:fs/promises'; import path from 'node:path'; const base = new URL(import.meta.resolve('./apps/site/')); const pagesBase = new URL('./pages/', base); const pages = await Array.fromAsync( fs.glob('**/*.{md,mdx}', { cwd: pagesBase }) ); for (const page of pages) { const src = new URL(page, pagesBase); const ext = path.extname(page); const parsed = path.parse(page); // If the file is named "index", don't create a subdirectory const destPath = parsed.name === 'index' ? path.join('app', parsed.dir, `page${ext}`) : path.join('app', parsed.dir, parsed.name, `page${ext}`); await fs.cp(src, new URL(destPath, base)); }After doing a bunch of testing, the only viable way for use to use
@next/mdxis via the app router without dynamic imports. This means, as Claudio said in the issue description, a 1:1appfolder to page routes. (This is because we have too many pages for Turbopack to compile into a bundle in a reasonable time.1)However, this structure brings it's own set of hoops.
First, with layouts. Currently, we specify the layout via MDX frontmatter. At the same time, we also attach
headingsandreadingTimeto the VFile containing our Markdown. Unfortunately, when using the/app/(path/to/my/route)/page.mdxway of writing pages, this data cannot be passed to the layouts. In order to pass data to the layouts, we'd need Next.js to resolve vercel/next.js#87990, OR write a custom Remark plugin to have each page export it's data to the layouts.Second, with internationalization. Currently, our current i18n setup relies on a dynamic
/[locale]/route segment. If we move all of our pages into theappfolder, locales will be explicitly defined (like in ourpagesfolder, meaning/[locale]/can no longer remain fully dynamic. This change raises a question: How do we have internationalization work without a[locale], but still honor the locale prefixes in our URLs.Footnotes
-
https://github.com/hashicorp/next-mdx-remote#background--theory
Webpack is a JavaScript bundler, forcing it to load hundreds/thousands of pages of text content will blow out your memory requirements. Webpack stores each page as a distinct object with a large amount of metadata. One of our implementations with a couple hundred pages hit more than 8GB of memory required to compile the site. Builds took more than 25 minutes.
This is behavior that I was able to reproduce. Now, I'm wondering if Turbopack compiles all routes beforehand. If so, then I doubt we can use
@next/mdxat all, given that compiling all of our routes takes several minutes. ↩
Reacted by Claudio Wunder-
Also a note with
next-mdx-remoteis that it would load the MDX asgetStaticPropswhich means, all that raw MDX would also be attached to the page's HTML, as Next.js "imprints/stamps" the static data on the HTML within a<script>tag. -- There are many reasons why we went away from that library, unironically our custom router right now is as performant as it gets.I also foresee that if Vercel addresses vercel/next.js#87990, we can simplify a few things on our custom router. Ideally, @avivkeller we can even make this whole custom MDX router a package and distribute to npm as an alternative to next-mdx-remote. We validated it as being something good. Then we just need to figure out which pieces we outsource and how to make it an easily importable thing that isn't stuck to just our way of structuring pages. That can be a good contribution to the community as right now there isn't really a good alternative to
next-mdx-remote... I've seen thisnext-mdx-remote-clientpackage but no idea how good it is, and if it covers our needs...Note that the package I mentioned above feels supercomplex and probably not the way we want to do things...
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Back in 2023/2024 when we were releasing the redesigned Node.js Website, we had to create a custom dynamic router around our infrastructure (Next.js) because
pagesfolder structure is what @nodejs/collaborators were used to work with. This was intentionally done to not make the move to Next.js feel as big as an atomic move as it could be. This was also done to keep DX for existing collaborators simple.Fast forwarding to now, release blog post generation is getting automated and releasers and other collaborators that have worked with us are used and more familiar with our Next.js environment.
The time has probably come. It's time to move away from our custom
next.dynamic.mjsrouter (including all the bells-and-whistles) and adopt Next.js's built-in MDX, which usesmdx-rs, a Rust-based MDX parser/compiler (https://github.com/web-infra-dev/mdx-rs) and that at the moment also supports custom JS-based Rehype and Remark Plugins.This work effectively means that:
/pageswould be moved under the respective folder under /app/ routerSome of the benefits:
The cons?
cc @nodejs/web-infra @nodejs/nodejs-website