Skip to content

Commit 423df0a

Browse files
alothaloth
authored
Merge pull request #53 from Waynting/docs/architecture
Architecture note from #49: docs/ARCHITECTURE.md plus comments in client.ts explaining the browser-session model and the fallbacks that exist because Overleaf changed its markup. Author: @Waynting. The merge commit on the branch resolves a README conflict with #52, which landed a minute earlier: both branches appended a pointer to the Contributing section, one to CONTRIBUTING.md and one to docs/ARCHITECTURE.md. Both lines stay. Verified on the merged state: npm ci, lint and build clean, 58 tests passing, CI green on Node 20.18.1 and 24. Also corrects the client.ts file header, which claimed to provide programmatic access to Overleaf's REST APIs - the exact misconception the document exists to correct.
2 parents 1043c44 + 1e04e44 commit 423df0a

3 files changed

Lines changed: 228 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -426,6 +426,9 @@ Contributions are welcome! Please open an issue or submit a pull request.
426426
See [CONTRIBUTING.md](CONTRIBUTING.md) for local setup, which tests need a real
427427
Overleaf account, and what to expect from CI on a pull request.
428428

429+
[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) explains how the client works —
430+
there is no public Overleaf API, so it authenticates as a browser session.
431+
429432
## License
430433

431434
MIT © [Alexander Loth](https://alexloth.com)

‎docs/ARCHITECTURE.md‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# Architecture
2+
3+
Orientation for anyone about to open `src/client.ts` and wonder why it looks
4+
like that. For setup and pull request mechanics, see
5+
[CONTRIBUTING.md](../CONTRIBUTING.md).
6+
7+
## The premise: there is no Overleaf API
8+
9+
Overleaf publishes no REST API for the free tier. olcli is not an API client —
10+
it **authenticates as a logged-in browser session and calls the same endpoints
11+
the web editor's own JavaScript calls.**
12+
13+
Almost every design decision follows from that one fact:
14+
15+
- Authentication is a session cookie, because that is what a browser holds.
16+
- Reading project data means parsing Overleaf's HTML, because that is where a
17+
server-rendered page puts it.
18+
- Write endpoints are whatever the web editor posts to.
19+
- Nothing is versioned or documented, so **anything here can break when
20+
Overleaf ships a redesign.** The layered fallbacks scattered through
21+
`client.ts` are not defensive habit; each one is a redesign that already
22+
happened.
23+
24+
Requests carry a `User-Agent` of `olcli/<version>`. olcli does not pretend to
25+
be Chrome — the session is a real one belonging to the user running it.
26+
27+
## Authentication
28+
29+
Two ways in, both ending at the same place:
30+
31+
| Entry point | What it does |
32+
|---|---|
33+
| `OverleafClient.fromSessionCookie()` | Takes a cookie the user copied from their browser |
34+
| `OverleafClient.fromPasswordLogin()` | Submits the login form (self-hosted instances without reCAPTCHA) |
35+
36+
Both then need a second credential. Overleaf requires a **CSRF token** on every
37+
state-changing request, which is what stops another site using your cookie
38+
against it. The token is not secret — the web editor needs it in the page to
39+
make its own requests — so olcli fetches a page with the cookie and reads the
40+
token out of the HTML (`extractCsrfToken`). From then on every request carries
41+
both:
42+
43+
```
44+
Cookie: overleaf_session2=...
45+
X-Csrf-Token: ...
46+
```
47+
48+
`applySetCookieHeaders()` folds any `Set-Cookie` from each response back into
49+
the in-memory jar, so a session that rotates mid-run keeps working — the same
50+
bookkeeping a browser does.
51+
52+
Credential *storage* lives in `src/config.ts`, deliberately apart from the
53+
client: env var, then `.olauth` in the current directory, then the global
54+
config file. The client itself never reads any of them.
55+
56+
## Reading: HTML scraping, then Socket.IO
57+
58+
Project data is server-rendered into `<meta name="ol-*">` tags, so
59+
`listProjects()` and `getProjectInfo()` parse the page with `cheerio`. Each has
60+
several fallbacks tried in order, because the tag names and shapes have changed
61+
more than once.
62+
63+
The file tree is the awkward one. It used to live in `ol-project`; it no longer
64+
does. `getProjectFromSocket()` recovers it by **speaking Socket.IO 0.9 by
65+
hand** — handshake for a session id, `xhr-polling` for packets, decode the
66+
frames, answer the `2::` heartbeats, and pull the tree out of the
67+
`joinProjectResponse` event.
68+
69+
This is the most fragile surface in the repository, and the least
70+
self-evident. It is also unavoidable: that payload is where the tree is now.
71+
Results are cached per project in `folderTreeCache` so a multi-file upload
72+
does not repeat the whole dance for every file.
73+
74+
## Writing: upload replaces, it does not edit
75+
76+
| Operation | Request |
77+
|---|---|
78+
| Read all files | `GET /project/<id>/download/zip` |
79+
| Write a file | `POST /project/<id>/upload?folder_id=<id>` (multipart, field `qqfile`) |
80+
| Delete | `DELETE /project/<id>/{doc,file,folder}/<entityId>` |
81+
| Rename an entity | `POST /project/<id>/<type>/<entityId>/rename` |
82+
| Rename the project | `POST /project/<id>/rename` |
83+
| Compile | `POST /project/<id>/compile` |
84+
85+
**The most important thing to understand about writes:** typing in the Overleaf
86+
editor sends character-level operations over the collaboration socket — an
87+
operational transform stream that merges concurrent edits. `uploadFile()` does
88+
not do that. It posts a whole file to the upload endpoint, exactly as if you
89+
had dragged a same-named file into the web UI.
90+
91+
So a `push` **overwrites**. It does not merge, and it cannot: there is no
92+
three-way merge to perform, only a file replacing a file. That is why
93+
`olcli diff` exists — previewing what a push will overwrite is the only
94+
protection against a collaborator's edit being replaced — and why `diff`
95+
fetches the remote fresh rather than comparing against the last pull.
96+
97+
Reading the whole project is one request, not one per file: `downloadProject()`
98+
returns the entire project as a zip. `pull`, `sync` and `diff` all use it.
99+
100+
## The transport
101+
102+
Everything goes through one private method, `httpRequest()`, built on
103+
`node:http`/`node:https` rather than `fetch`. That is not preference: `fetch`
104+
validates response headers as Latin-1 and throws on a `Content-Disposition`
105+
carrying a non-ASCII project name, which made downloads fail for anyone with an
106+
accented title ([#2](https://github.com/aloth/olcli/issues/2)). It also handles
107+
redirects, timeouts, and serialising `FormData` into a multipart body.
108+
109+
`--verbose` makes it log every request and response to stderr, which is the
110+
first thing to reach for when Overleaf changes something.
111+
112+
## Module map
113+
114+
Which files need an Overleaf account to exercise, and which do not. This is the
115+
main thing to know before adding a feature, because it decides where the logic
116+
should go.
117+
118+
**Pure — data in, data out. No network, no filesystem, unit-tested:**
119+
120+
| Module | Responsibility |
121+
|---|---|
122+
| `diff.ts` | Compare two file trees; render unified diffs |
123+
| `ignore.ts` | The three ignore layers and the `.pdf`-next-to-`.tex` rule |
124+
| `paths.ts` | Remote path normalisation; zip-slip containment |
125+
| `rename-plan.ts` | Plan bulk project renames before applying any |
126+
| `prompt.ts` | Keystroke handling for the password prompt |
127+
| `scan.ts` | Walk a local directory, applying ignore rules |
128+
129+
**Talks to Overleaf:**
130+
131+
| Module | Responsibility |
132+
|---|---|
133+
| `client.ts` | Every request. The browser-session model lives here |
134+
| `config.ts` | Credential resolution and storage |
135+
136+
**Entry points, all thin over the two above:**
137+
138+
| Module | Binary |
139+
|---|---|
140+
| `cli.ts` | `olcli` — argument parsing and terminal output |
141+
| `mcp.ts` | `olcli-mcp` — the same operations as MCP tools |
142+
| `remote-helper.ts` | `git-remote-overleaf` — `gitremote-helpers(7)` protocol |
143+
| `index.ts` | The programmatic API re-exported from the package root |
144+
145+
New logic belongs in the pure column wherever it can go. That is why `scan.ts`
146+
exists at all: `push` and `sync` each carried their own copy of the same walk
147+
loop and had already drifted apart, and `diff` would have made a third. The
148+
same reasoning produced `rename-plan.ts` and `diff.ts`.
149+
150+
`client.ts` request *construction* can also be tested without an account, by
151+
pointing the client at a local HTTP server that captures the outgoing request —
152+
see `test/client.test.ts`.
153+
154+
## When Overleaf breaks it
155+
156+
The usual failure is a redesign moving data somewhere else. Reliable order:
157+
158+
1. `olcli --verbose <command>` — see the actual request and response.
159+
2. If a page parse returns nothing, fetch the page in a browser with devtools
160+
and look for the `ol-*` meta tag. Add a fallback; keep the existing ones,
161+
since self-hosted instances run older versions.
162+
3. If the file tree is what broke, suspect `getProjectFromSocket()` first.
163+
4. `olcli check` reports which credential source is in play, without printing
164+
any secret.

‎src/client.ts‎

Lines changed: 61 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,18 @@
11
/**
2-
* Overleaf API Client
2+
* Overleaf client
33
*
4-
* Provides programmatic access to Overleaf's REST APIs for project
5-
* management, file operations, and LaTeX compilation.
4+
* Project management, file operations and LaTeX compilation against an
5+
* Overleaf instance.
6+
*
7+
* These are not Overleaf's public APIs - there are none for the free tier.
8+
* This client authenticates as a logged-in browser session and calls the same
9+
* endpoints the web editor's own JavaScript calls: a session cookie plus a
10+
* CSRF token scraped from the page, project data parsed out of `ol-*` meta
11+
* tags, and the file tree recovered over the collaboration socket. Nothing
12+
* here is versioned or documented by Overleaf, so the layered fallbacks below
13+
* are not defensive habit - each one is a redesign that already happened.
14+
*
15+
* Read docs/ARCHITECTURE.md before changing anything in this file.
616
*/
717

818
import * as cheerio from 'cheerio';
@@ -391,6 +401,22 @@ export class OverleafClient {
391401
return new OverleafClient({ cookies: bootstrapClient.cookies, csrf: projectCsrf, baseUrl });
392402
}
393403

404+
/**
405+
* Pull the CSRF token out of a rendered Overleaf page.
406+
*
407+
* Overleaf requires this on every state-changing request, which is what
408+
* stops another site from using your session cookie against it. It is not a
409+
* secret - the web editor needs it in the page to make its own requests - so
410+
* reading it back out of the HTML is the intended way for a session to
411+
* obtain one. See docs/ARCHITECTURE.md.
412+
*
413+
* The three lookups are not belt-and-braces. Each is where the token lived
414+
* at some point: the `ol-csrfToken` meta tag is current, the hidden form
415+
* input is what older releases shipped, and the inline-script scrape catches
416+
* self-hosted instances older still. Removing the later ones breaks
417+
* self-hosted users without breaking anything on overleaf.com, so the
418+
* failure would not show up here.
419+
*/
394420
private static extractCsrfToken($: cheerio.CheerioAPI): string | undefined {
395421
let csrf = $('meta[name="ol-csrfToken"]').attr('content');
396422
if (!csrf) {
@@ -571,7 +597,11 @@ export class OverleafClient {
571597
const html = response.body as string;
572598
const $ = cheerio.load(html);
573599

574-
// Try new Overleaf structure first (PR #82)
600+
// There is no projects API; the list is server-rendered into a meta tag,
601+
// so this parses Overleaf's own HTML. The three methods below are three
602+
// successive shapes that tag has had - newest first, oldest last. A
603+
// self-hosted instance can be running any of them, which is why the older
604+
// ones stay. See docs/ARCHITECTURE.md.
575605
let projectsData: any[] = [];
576606

577607
// Method 1: ol-prefetchedProjectsBlob (newest Overleaf)
@@ -757,6 +787,17 @@ export class OverleafClient {
757787
* Fetch the full project object via the collaboration socket.
758788
* Returns the `project` field of the joinProjectResponse, which contains
759789
* the rootFolder tree and other metadata that used to live in ol-project.
790+
*
791+
* This is a hand-written Socket.IO 0.9 client: handshake for a session id,
792+
* `xhr-polling` for packets, decode the frames, answer the `2::` heartbeats,
793+
* disconnect with `0::`. No library - the protocol is old enough that
794+
* depending on one to speak it would cost more than the forty lines below.
795+
*
796+
* It is the most fragile surface in the repository and the least obvious,
797+
* because it reimplements an undocumented internal protocol rather than
798+
* calling an endpoint. It exists because the file tree left the meta tags
799+
* and this payload is where it went; there is no HTTP route that returns it.
800+
* When the tree is what broke, suspect this method first.
760801
*/
761802
private async getProjectFromSocket(projectId: string): Promise<any | null> {
762803
let sid: string | null = null;
@@ -1631,6 +1672,22 @@ export class OverleafClient {
16311672
* If folderTree is provided and fileName contains a path (e.g. 'figures/img.png'),
16321673
* the file will be uploaded into the correct subfolder, creating it if needed.
16331674
*/
1675+
/**
1676+
* Upload a file, replacing any file of the same name.
1677+
*
1678+
* This **overwrites**; it does not edit. Typing in the Overleaf editor sends
1679+
* character-level operations over the collaboration socket, and those merge
1680+
* with concurrent edits. This posts a whole file to the upload endpoint -
1681+
* the same thing as dragging a same-named file into the web UI - so whatever
1682+
* was there is gone.
1683+
*
1684+
* That is why `push` has no merge semantics and cannot grow any: there is no
1685+
* three-way merge available, only a file replacing a file. It is also why
1686+
* `olcli diff` exists, and why it fetches the remote fresh rather than
1687+
* comparing against the last pull - previewing what a push will overwrite is
1688+
* the only thing standing between a collaborator's edit and its replacement.
1689+
* See docs/ARCHITECTURE.md.
1690+
*/
16341691
async uploadFile(
16351692
projectId: string,
16361693
folderId: string | null,

0 commit comments

Comments
 (0)