Provide entire docs in a single text file for LLMs

[fhammerschmidt]

There is a standard for providing LLMs with better information about the contents of a website: https://llmstxt.org/

This can make a huge difference with the suggestions LLMs make about ReScript code.
The goal is to generate the following files from a script:

pages/docs/llms
├── llms-full.txt // full documentation (all markdown files in pages/docs/manual + pages/docs/react with examples)
├── llms-small.txt // full documentation (a compressed version of the former, without examples)
└── llms.txt // to tell the LLM which files are available

Here are the Svelte docs as an example: https://svelte.dev/docs/llms

Thanks to forum user for johnny.b.good for mentioning that topic.

[lucasguellis]

Hey Florian! I noticed this issue and would love to work on it! This would be my first contribution to an open-source project. Can you assign it to me?
I just have a doubt on how to run this script, should I create an npm run, to run it during the build process, like the update-index that already exists? And did you thought about the versions? Should we generate the llm files for all versions or just the last one?

[fhammerschmidt]

Welcome!

Right, it's probably better to do pages/docs/manual/v11.0.0/llms and pages/docs/manual/v12.0.0/llms. Earlier versions should not have it though, since that was before uncurried mode and some other features that make ReScript easier to write today.

I think yes, it can just be appended to the update-index scripts.

Thank you so much for having a stab at this!

[lucasguellis]

Hey @fhammerschmidt! Sorry for the delay. I finished the script to generate the llm files, but i'm struggling into creating the route to access the files, do I have to do this into switch url on the App.res file?
I'm thinking about creating a route like /docs/manual/v11.0.0/llms/llms-small.txt for each file, that returns just the content of the file, just like the svelte docs does, but adding it to the App.res file it only ruturns me a 404 page. I'm not used to this project yet haha, could you help me out with this please?

For now, the page looks like this:

[fhammerschmidt]

I think we should host all the txt files in the static public directory, maybe just under public/llms/<versionNumber>.

1. Save the file from the screenshot as pages/docs/manual/v11.0.0/llms.mdx
2. Add a preamble like this to llms.mdx

---
title: "LLMs"
description: "Documentation for LLMs"
canonical: "/docs/manual/v11.0.0/llms"
---

3. Adapt the links to the public folder:

- [llms.txt](/llms/v11.0.0/llms.txt)
- [llms-full.txt](/llms/v11.0.0/llms-full.txt)
- [llms-small.txt](/llms/v11.0.0/llms-small.txt)

4. I guess we also want this in the sidebar, so let's add it to sidebar_manual_v1100.json under "Extra":

  ...
  "Extra": [
    "newcomer-examples",
    "project-structure",
    "faq",
+  "llms"
  ]

4. run npm run update-index
5. refresh/restart your dev server and it should hopefully appear and the links should work

[fhammerschmidt]

Or rather public/llms/manual since we also will have public/llms/react