Read build files once and publish outputs from memory in OCaml rewatch (#9170)

* Read each build file once per OCaml rewatch build

Why:
    Every in-process compile read the interfaces it uses from disk again,
    above all the runtime's, and unmarshalled them again, and finding one
    took a failed `open` in each load-path directory before the one that
    has it. On macOS an `open` costs about 15 µs and gets slower when
    several threads call it at once, so a clean build of a 2000-module
    project spent a third of its CPU time in `open`.

The compiler now reads and writes build files through `Build_file_store`.
Rewatch enables it for the duration of each build: a file is read from
disk at most once, a file written through the store is kept for later
reads, and each domain keeps the interfaces it has unmarshalled until the
file is written again. `bsc` leaves the store disabled and reads the disk
as before. Writers render their output in memory first, so the bytes
written are unchanged: every artifact of the fixtures below, `.cmt` files
included, is byte-identical to master's.

IO modified (file system calls of a clean build, 2000 modules, 8 domains):
Before (text):
    open .cmi       70856
    stat .cmj       30856
    open .cmj       14856
    open .res       10000
    open .ast        8000
    all calls      211523
After (text):
    open .cmi       5454
    stat .cmj       1731
    open .cmj       5722
    open .res       8000
    open .ast       6000
    all calls       101462

Build modified (wall time, median of 5, Apple M4 Pro):
Before (text):
    synthetic 2000 modules, clean    2.42 s
    tests/tests, clean               0.71 s
After (text):
    synthetic 2000 modules, clean    2.11 s
    tests/tests, clean               0.67 s

Review: Can a request read a stale file from the store? Within a build a
file is read only after the request that writes it has finished, because
rewatch orders requests by their dependencies, and writes go through the
store. Requests spell paths relative to their working directory, such as
`lib/bs/../ocaml/A.cmi`, so keys remove `.` and `..` lexically. The store
starts every build empty, so it sees changes made between builds.

Review: Is it safe to reuse an unmarshalled interface in later requests?
Only on the domain that read it: copying a signature marks its types in
place until the copy ends, which is why domains don't share them. A request
that fails drops the domain's interfaces, in case it stopped with a type
still marked, and every request resets the abbreviations it memoized,
including those in cached interfaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Read AST headers once, in parallel, in OCaml rewatch

Why:
    Before anything compiles, a build reads the header of every AST twice
    on the main domain: the published copy in lib/ocaml, to know its
    source and dependencies, and again the one in lib/bs, to build the
    module graph. For a 2000-module project that was two thirds of a build
    after a one-file edit.

The module graph now takes the dependencies of a module that wasn't parsed
in this build from its published AST, which is a copy of the one in
lib/bs, and those of a module parsed in this build from memory. The
published ASTs are inventoried on all worker domains, and a header is read
with one small read instead of through a channel that buffers 64 KiB of
the AST.

Build modified (wall time after a one-file edit, median of 5, Apple M4 Pro):
Before (text): synthetic 2000-module project
    no-op build             0.139 s
    edit a leaf module      0.155 s
After (text): synthetic 2000-module project
    no-op build             0.089 s
    edit a leaf module      0.104 s

Review: When can the published AST's dependencies differ from the one in
lib/bs? The published AST is a byte copy made when the module is parsed.
A module whose source is newer than its published AST, or whose published
AST is missing, is parsed again, and its dependencies are then read from
the new AST in memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Publish outputs from memory, on the worker that made them, in OCaml rewatch

Why:
    Publishing copies each output of the compiler into lib/ocaml, and each
    source into lib/bs and lib/ocaml, by reading back the files just
    written. After parsing, the main domain alone also copied every AST
    and source, one after another: in a clean build of a 2000-module
    project all workers sat idle for half a second.

Copies now take their contents from `Build_file_store`, where the compiler
left what it wrote, and keep the published interfaces and `.cmj` files
for dependents to read; ASTs, typed trees and JavaScript leave the store
once published. Each parse publishes its AST and source on the worker
that ran it. The source copy that compiling publishes again is written
only if it differs from the one the parse published. Every rewatch-side
write or removal of a file updates the store.

IO modified (file system calls of a clean build, 2000 modules, 8 domains):
Before (text):
    all calls       99063
After (text):
    all calls       69125

Build modified (wall time, median of 5, Apple M4 Pro):
Before (text):
    synthetic 2000 modules, clean    2.07 s
After (text):
    synthetic 2000 modules, clean    1.80 s

Review: Is a file a hook or PPX reads still on disk when it runs? Yes:
writes through the store reach the disk before `write` returns; the store
only spares reading them back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Test the build file store's failure paths

Codecov flagged the store's failure handling as untested. The new cases
check that an interface cut short reads as corrupted from memory as it
does from disk, that a failed write keeps nothing for its path, and that
a failed request makes its domain read its interfaces again while a
successful one keeps them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Correct the clean-build figure in the build file store's changelog entry

The entry said clean builds became about a quarter faster, as measured
with the disk nearly full. Measured again with free space, a clean build
of the synthetic 2000-module project takes 2.21 s on master and 1.47 s
with this branch, about a third less.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Say in the OCaml rewatch guide that workers publish their outputs

The guide still said the main domain publishes every artifact, which
stopped being true when workers began publishing what they produce from
memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Read an AST's header with one parser, in the compiler and OCaml rewatch

Why:
    The header of a binary AST was parsed in three places: in
    `Binary_ast.read_ast_exn`, and twice in OCaml rewatch's `Ast_header`,
    once to find where the header ends in the first block read and once
    to read it. `read_ast_exn` also raised `Invalid_argument`, not
    `End_of_file`, for a header with a negative or impossible dependency
    length.

`Binary_ast.read_header` finds where the dependency block and the source
path end in the first bytes of an AST, or says they don't hold the whole
header. `read_ast_exn` raises `End_of_file` for a file without one, and
rewatch reads one block of a published AST, and all of it only when the
header may be longer.

Review: Does rewatch read any header differently? An AST with no newline
after its source path now reads as damaged, so its module parses again;
before, rewatch took the path to run to the end of the file. `write_ast`
always writes the newline, so only a file cut short ends this way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Publish each parse from its task's result hook, in OCaml rewatch

Why:
    Parses were published through an `on_success` hook added to
    `Compiler_process.task` and `run_requests`, which looked each parse up
    by its index, though `Process.map_result` already runs a function on
    the worker when a task finishes. Compile assets also regrouped the
    inventory by directory through a hashtable, though `Parallel.map`
    returns it in the order of the directories.

`run_parses` maps each parse task's result to publish it, and `task` and
`run_requests` are as they were before the build file store.
`Compile_assets.create` takes each directory's entries in turn. The store
says why its `enabled` and `generation` fields are read without its lock.

Review: Is a parse still published on its worker, and does a failed
publication still fail the parse? `Process.run_concurrent_task` applies
the result hook on the worker, inside the handler that turns an exception
into a failed task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Take a parsed module's dependencies from its new AST, in OCaml rewatch

Why:
    The module graph reads a module's dependencies from the header of its
    published AST, which the build read before parsing. It told a module
    parsed in this build by its AST being held in the build file store, so
    a store that no longer held the AST, or no store, would have given the
    module the dependencies of its previous source, and the build would
    have skipped recompiles.

`Module_graph.initialize` takes the paths that this build parsed
successfully, which `Build_preparation` already knows, and reads their
dependencies from their new AST in `lib/bs`. The store's copy of that AST
only saves the disk read.

Review: Does any other caller rely on the store to tell a fresh parse?
The incremental path in `Build` reads every changed module's AST from
`lib/bs` without the published headers, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Bound each domain's interface cache, in the compiler

Why:
    Every compiler worker domain kept each interface it decoded for the
    rest of the build. A decoded interface takes a few times the size of
    its file, so a project with large interfaces could hold several copies
    of all of them, one per domain.

A domain caches interfaces whose contents add up to at most 16 MB, and
starts over when the next one would exceed that. The runtime's interfaces
take 2 MB. An interface rewritten during the build replaces its entry and
its size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Raise End_of_file for a truncated AST or .cmj, in the compiler

Why:
    Reading from the build file store, an AST or `.cmj` cut short inside
    its marshalled value raised `Invalid_argument` from
    `Marshal.from_string`, though the AST reader is documented to raise
    `End_of_file` for a file a killed build cut short. A channel's
    `input_value` raised `Failure` there.

`Ext_io.unmarshal` checks that the marshalled value ends within the
contents, and raises `End_of_file` when it doesn't. The AST and `.cmj`
readers use it, and read their file with `Ext_io.load_file` instead of
their own copy of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Keep the build file store in step with removed files, in the compiler and OCaml rewatch

Why:
    `Misc.remove_file`, which removes stale source maps and the `.cmi` of
    a failed save, deleted files without telling the build file store, so
    a later read in the same build would have returned the removed
    contents. Nothing reads those paths again today. An implementation
    compiled with `-bs-read-cmi` also read its interface's `.cmi` in
    `lib/bs` back into the store, which then kept it for the rest of the
    build. And when `Parallel.map` failed to start a domain, it raised
    without joining the domains it had started.

`Misc.remove_file` forgets the path it removes, and publishing an
implementation that has an interface evicts the `.cmi` that its compile
read. `Parallel.map` shares the items among the domains that started. The
store's interface no longer suggests `keep:false` for `.cmt` files, which
rewatch copies out of the store.

New tests cover a removed file, a file that exists but can't be read,
several domains using the store at once, more spellings of one path, and
AST headers that end at the edge of the first read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Test which files publishing keeps in the build file store, in OCaml rewatch

Why:
    Publishing a module keeps its interface and `.cmj` in the store for
    its dependents, and evicts the rest so that a clean build's memory
    stays bounded. No test checked which files it kept, so an eviction
    that was dropped would only have shown as higher peak memory.

The new test publishes a module and checks that the published `.cmi` and
`.cmj` stay in the store, while the typed tree, the AST, the sources, the
JavaScript output and every copy leave it. It also publishes an
implementation that has an interface, whose compile reads its `.cmi` in
`lib/bs` again, and checks that this `.cmi` leaves the store too. Each
check fails without the eviction it covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Publish parses that print warnings after their cleanup, in OCaml rewatch

Why:
    A worker published each successful parse into `lib/ocaml` as soon as
    it finished, including a parse that printed warnings. The cleanup
    that removes such an AST, so that the next build parses the source
    again and repeats its warnings, is only registered once all the
    package's parses are done. A build interrupted between the two left
    the AST published and newer than its source, so the next build
    skipped the parse and never reported warnings such as the deprecated
    `(. ...)` syntax.

The worker now publishes only parses that printed nothing.
`Package_parse` publishes the others itself, after it registers their
cleanup, as it did for every parse before the worker took over
publication.

A new test parses one clean source and one that warns, and checks that
only the clean one is published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

* Merge the two OCaml rewatch build I/O changelog entries into one

The branch added one entry for the build file store and publication and
another for the AST header scan. Each said it made builds about a third
faster, without saying which builds, so they read as two similar claims.
One entry now describes the three steps together, with the figures the
pull request measured on the synthetic 2000-module project: a clean
build takes 2.21 s on master and 1.47 s with the branch, a no-op build
0.154 s and 0.089 s, and a build after editing a leaf module 0.174 s and
0.100 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>

---------

Signed-off-by: Paul Tsnobiladzé <2479216+tsnobip@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Author: tsnobip

changes/ocaml-rewatch-build-file-store.polish.md

@@ -0,0 +1 @@
+- OCaml rewatch: read each interface, `.cmj`, AST and source at most once per build, read the headers of existing ASTs once and in parallel, and publish outputs from memory instead of reading them back. On a large project, this cuts the time of a clean build by about a third, and of a no-op build or one that recompiles a single module by about 40%.

compiler/bsc/rescript_compiler_driver.ml

@@ -569,12 +569,28 @@ let start_request () =
   Predef.reset_for_request ();
   Env.reset_initial_for_request ()
 
+(* Interfaces that this domain keeps for later requests must not keep this
+   request's memoized abbreviations, so they are reset while the request's
+   memo list is still current. A failed request may also have stopped with
+   some of their types marked, so the domain drops them. *)
+let leave_cached_interfaces_clean action =
+  match action () with
+  | result ->
+    Btype.cleanup_abbrev ();
+    if result.exit_code <> 0 then Bs_cmi_load.discard_cached_cmis ();
+    result
+  | exception exn ->
+    Btype.cleanup_abbrev ();
+    Bs_cmi_load.discard_cached_cmis ();
+    raise exn
+
 let with_fresh_request_states ~cwd action =
   Predef.with_fresh (fun () ->
       Env.with_fresh_initial (fun () ->
           Ident.with_fresh (fun () ->
               Request_local.with_fresh (fun () ->
-                  Compiler_request_state.with_fresh ~cwd action))))
+                  Compiler_request_state.with_fresh ~cwd (fun () ->
+                      leave_cached_interfaces_clean action)))))
 
 let run_argv ?run_external ~capture ~cwd argv =
   with_fresh_request_states ~cwd (fun () ->

compiler/core/bs_cmi_load.ml

@@ -22,22 +22,83 @@
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. *)
 
+(* Reads build files missing from [Build_file_store]. Open each file once and
+   read that descriptor: on case-insensitive filesystems a lowercase alias can
+   remain visible to [Sys.file_exists] briefly after its CMI is removed, and a
+   vanished file must count as missing. *)
+let read_file path =
+  match
+    Unix.openfile path [Unix.O_RDONLY; Unix.O_CLOEXEC; Unix.O_SHARE_DELETE] 0
+  with
+  | descriptor ->
+    let channel = Unix.in_channel_of_descr descriptor in
+    Fun.protect
+      ~finally:(fun () -> close_in_noerr channel)
+      (fun () -> Some (really_input_string channel (in_channel_length channel)))
+  | exception Unix.Unix_error ((Unix.ENOENT | Unix.ENOTDIR), _, _) -> None
+  | exception Unix.Unix_error (error, _, _) ->
+    raise (Sys_error (path ^ ": " ^ Unix.error_message error))
+
+(* Interfaces read by earlier requests on this domain, by path, with the
+   stored contents they were read from. Reading the same interfaces, above all
+   the runtime's, is a sizable part of compiling a small module. The values
+   can't be shared between domains: copying a signature marks its types in
+   place for the duration of the copy. *)
+type cached_cmi = {contents: string; cmi: Cmi_format.cmi_infos}
+
+type cmi_cache = {
+  mutable generation: int;
+  mutable size: int;  (** The length of the cached contents. *)
+  cmis: (string, cached_cmi) Hashtbl.t;
+}
+
+(* A decoded interface takes a few times the size of its contents, and every
+   domain keeps its own, so a domain that would cache more contents than this
+   starts over. The runtime's interfaces take 2 MB. *)
+let cmi_cache_budget = 16 * 1024 * 1024
+
+let cmi_cache_key =
+  Domain.DLS.new_key (fun () ->
+      {generation = -1; size = 0; cmis = Hashtbl.create 64})
+
+let empty_cmi_cache cache ~generation =
+  Hashtbl.reset cache.cmis;
+  cache.size <- 0;
+  cache.generation <- generation
+
+let read_cached_cmi filename path contents =
+  let cache = Domain.DLS.get cmi_cache_key in
+  if cache.generation <> Build_file_store.generation () then
+    empty_cmi_cache cache ~generation:(Build_file_store.generation ());
+  match Hashtbl.find_opt cache.cmis path with
+  | Some cached when cached.contents == contents -> cached.cmi
+  | previous ->
+    let cmi = Cmi_format.read_cmi_contents filename contents in
+    Option.iter
+      (fun previous ->
+        cache.size <- cache.size - String.length previous.contents)
+      previous;
+    if cache.size + String.length contents > cmi_cache_budget then
+      empty_cmi_cache cache ~generation:cache.generation;
+    Hashtbl.replace cache.cmis path {contents; cmi};
+    cache.size <- cache.size + String.length contents;
+    cmi
+
+(* A request that fails can stop with a type of a cached interface still
+   marked, so the domain reads its interfaces again. *)
+let discard_cached_cmis () =
+  empty_cmi_cache (Domain.DLS.get cmi_cache_key) ~generation:(-1)
+
 let load_cmi ~unit_name : Env.Persistent_signature.t option =
-  (* On case-insensitive filesystems a lowercase alias can remain visible to
-     [Sys.file_exists] briefly after its CMI is removed. Open each candidate
-     once and parse that descriptor, so a vanished CMI is a missing module. *)
   Misc.find_map_in_path_uncap (Config.get_load_path ()) (unit_name ^ ".cmi")
     (fun filename ->
       let path = Compiler_request_state.resolve_path filename in
-      match Unix.openfile path [Unix.O_RDONLY; Unix.O_CLOEXEC] 0 with
-      | descriptor ->
-        let channel = Unix.in_channel_of_descr descriptor in
+      match Build_file_store.read path with
+      | None -> None
+      | Some contents ->
         let cmi =
-          Fun.protect
-            (fun () -> Cmi_format.read_cmi_channel filename channel)
-            ~finally:(fun () -> close_in_noerr channel)
+          if Build_file_store.enabled () then
+            read_cached_cmi filename path contents
+          else Cmi_format.read_cmi_contents filename contents
         in
-        Some Env.Persistent_signature.{filename; cmi}
-      | exception Unix.Unix_error ((Unix.ENOENT | Unix.ENOTDIR), _, _) -> None
-      | exception Unix.Unix_error (error, _, _) ->
-        raise (Sys_error (path ^ ": " ^ Unix.error_message error)))
+        Some Env.Persistent_signature.{filename; cmi})

compiler/core/bs_conditional_initial.ml

@@ -24,6 +24,7 @@
 
 let setup_env () =
   Env.Persistent_signature.load := Bs_cmi_load.load_cmi;
+  Build_file_store.set_disk_reader Bs_cmi_load.read_file;
   Matching.make_test_sequence_variant_constant :=
     Polyvar_pattern_match.make_test_sequence_variant_constant;
   Matching.call_switcher_variant_constant :=

compiler/core/js_cmj_format.ml

@@ -69,26 +69,15 @@ let make ~(values : cmj_value Map_string.t) ~effect_ ~package_spec ~case : t =
   }
 
 (* Serialization .. *)
-(* Channels are closed also when decoding fails: the compiler can run inside a
-   long-lived build process, which would otherwise keep the file open. *)
 let from_file name : t =
-  let ic = open_in_bin (Compiler_request_state.resolve_path name) in
-  Fun.protect
-    ~finally:(fun () -> close_in_noerr ic)
-    (fun () ->
-      let _digest = Digest.input ic in
-      (input_value ic : t))
+  Ext_io.unmarshal (Ext_io.load_file name) Ext_digest.length
 
 let for_sure_not_changed (name : string) (header : string) =
-  if Sys.file_exists (Compiler_request_state.resolve_path name) then
-    let ic = open_in_bin (Compiler_request_state.resolve_path name) in
-    Fun.protect
-      ~finally:(fun () -> close_in_noerr ic)
-      (fun () ->
-        match really_input_string ic Ext_digest.length with
-        | holder -> holder = header
-        | exception End_of_file -> false)
-  else false
+  match Build_file_store.read (Compiler_request_state.resolve_path name) with
+  | Some contents ->
+    String.length contents >= Ext_digest.length
+    && String.sub contents 0 Ext_digest.length = header
+  | None -> false
 
 (* This may cause some build system always rebuild
    maybe should not be turned on by default
@@ -98,13 +87,9 @@ let to_file name ~check_exists (v : t) =
   let cur_digest = Digest.string s in
   let header = cur_digest in
   if not (check_exists && for_sure_not_changed name header) then
-    let oc = open_out_bin (Compiler_request_state.resolve_path name) in
-    Fun.protect
-      ~finally:(fun () -> close_out_noerr oc)
-      (fun () ->
-        output_string oc header;
-        output_string oc s;
-        close_out oc)
+    Build_file_store.write
+      (Compiler_request_state.resolve_path name)
+      (header ^ s)
 
 let key_comp a b = Ext_string.compare a b.name
 

compiler/core/lam_compile_main.ml

@@ -453,26 +453,42 @@ let remove_stale_source_map ?(remove_stale_map = true) target_file =
   if remove_stale_map && not !((Clflags.current ()).dont_write_files) then
     Misc.remove_file (target_file ^ ".map")
 
+(* Where a program is printed: the request's stdout, or a buffer that becomes
+   the output file. *)
+type sink = {printer: Ext_pp.t; add_string: string -> unit}
+
+let channel_sink ~track_positions channel =
+  {
+    printer = Ext_pp.from_channel ~track_positions channel;
+    add_string = output_string channel;
+  }
+
+let buffer_sink ~track_positions buffer =
+  {
+    printer = Ext_pp.from_buffer ~track_positions buffer;
+    add_string = Buffer.add_string buffer;
+  }
+
 let dump_deps_program_with_source_map ?(remove_stale_map = true) ~target_file
-    ~output_prefix module_system lambda_output chan =
+    ~output_prefix module_system lambda_output sink =
   let builder =
     Js_source_map.make ~generated_file:target_file
       ~source_root:!((Js_config.current ()).source_map_root)
       ~sources_content:!((Js_config.current ()).source_map_sources_content)
   in
   Js_source_map.with_builder builder (fun () ->
       Js_dump_program.pp_deps_program ~output_prefix module_system lambda_output
-        (Ext_pp.from_channel ~track_positions:true chan));
+        sink.printer);
   match !((Js_config.current ()).source_map) with
   | Linked ->
     let json = Js_source_map.json builder in
-    output_string chan
+    sink.add_string
       (Js_source_map.linked_comment ~map_file:(target_file ^ ".map"));
     Ext_io.write_file (target_file ^ ".map") json
   | Hidden ->
     Ext_io.write_file (target_file ^ ".map") (Js_source_map.json builder)
   | Inline ->
-    output_string chan
+    sink.add_string
       (Js_source_map.inline_comment ~json:(Js_source_map.json builder));
     remove_stale_source_map ~remove_stale_map target_file
   | No_source_map -> ()
@@ -490,7 +506,8 @@ let lambda_as_module (lambda_output : J.deps_program) (output_prefix : string) :
       in
       dump_deps_program_with_source_map ~remove_stale_map:false ~target_file
         ~output_prefix Commonjs lambda_output
-        (Compiler_request_output.stdout_channel ())
+        (channel_sink ~track_positions:true
+           (Compiler_request_output.stdout_channel ()))
     | _ ->
       Js_dump_program.dump_deps_program ~output_prefix Commonjs lambda_output
         (Compiler_request_output.stdout_channel ())
@@ -505,18 +522,19 @@ let lambda_as_module (lambda_output : J.deps_program) (output_prefix : string) :
           Ext_path.package_dir () // path
           // basename (* #913 only generate little-case js file *)
         in
-        let output_chan chan =
-          match !((Js_config.current ()).source_map) with
+        if not !((Clflags.current ()).dont_write_files) then (
+          let buffer = Buffer.create 4096 in
+          (match !((Js_config.current ()).source_map) with
           | No_source_map ->
-            Js_dump_program.dump_deps_program ~output_prefix module_system
-              lambda_output chan;
+            Js_dump_program.pp_deps_program ~output_prefix module_system
+              lambda_output
+              (Ext_pp.from_buffer buffer);
             remove_stale_source_map target_file
           | Linked | Inline | Hidden ->
             dump_deps_program_with_source_map ~target_file ~output_prefix
-              module_system lambda_output chan
-        in
-        if not !((Clflags.current ()).dont_write_files) then
-          Ext_pervasives.with_file_as_chan target_file output_chan)
+              module_system lambda_output
+              (buffer_sink ~track_positions:true buffer));
+          Build_file_store.write target_file (Buffer.contents buffer)))
 
 (* We can use {!Env.current_unit = "Pervasives"} to tell if it is some specific module,
     We need handle some definitions in standard libraries in a special way, most are io specific,

compiler/depends/binary_ast.ml

@@ -28,19 +28,42 @@ type 'a kind = 'a Ml_binary.kind =
   | Ml : Parsetree.structure kind
   | Mli : Parsetree.signature kind
 
-let read_ast_exn (type t) ~fname (_ : t kind) : t =
-  let ic = open_in_bin (Compiler_request_state.resolve_path fname) in
-  Fun.protect
-    ~finally:(fun () -> close_in_noerr ic)
-    (fun () ->
-      let dep_size = input_binary_int ic in
-      seek_in ic (pos_in ic + dep_size);
-      let sourcefile = input_line ic in
-      Location.set_input_name sourcefile;
-      input_value ic)
-
 let magic_sep_char = '\n'
 
+type header = {source_start: int; source_end: int}
+
+let read_header ?length contents =
+  let length = Option.value length ~default:(String.length contents) in
+  if length < 4 then None
+  else
+    let source_start = 4 + Int32.to_int (String.get_int32_be contents 0) in
+    if source_start < 4 || source_start > length then None
+    else
+      let rec source_end index =
+        if index >= length then None
+        else if contents.[index] = magic_sep_char then
+          Some {source_start; source_end = index}
+        else source_end (index + 1)
+      in
+      source_end source_start
+
+let header_dependencies contents header =
+  String.sub contents 4 (header.source_start - 4)
+  |> String.split_on_char magic_sep_char
+  |> List.filter (fun name -> name <> "")
+
+let header_source contents header =
+  String.sub contents header.source_start
+    (header.source_end - header.source_start)
+
+let read_ast_exn (type t) ~fname (_ : t kind) : t =
+  let contents = Ext_io.load_file fname in
+  match read_header contents with
+  | Some header ->
+    Location.set_input_name (header_source contents header);
+    Ext_io.unmarshal contents (header.source_end + 1)
+  | None -> raise End_of_file
+
 (*
    Reasons that we don't [output_value] the set:
    1. for performance , easy skipping and calcuate the length 
@@ -57,13 +80,15 @@ let write_ast (type t) ~(sourcefile : string) ~output (kind : t kind) (pt : t) :
         (* filter *predef* *)
         Ext_buffer.add_string_char buf s magic_sep_char)
     output_set;
-  let oc = open_out_bin (Compiler_request_state.resolve_path output) in
-  Fun.protect
-    ~finally:(fun () -> close_out_noerr oc)
-    (fun () ->
-      output_binary_int oc (Ext_buffer.length buf);
-      Ext_buffer.output_buffer oc buf;
-      output_string oc sourcefile;
-      output_char oc '\n';
-      output_value oc pt;
-      close_out oc)
+  let length = Bytes.create 4 in
+  Bytes.set_int32_be length 0 (Int32.of_int (Ext_buffer.length buf));
+  Build_file_store.write
+    (Compiler_request_state.resolve_path output)
+    (String.concat ""
+       [
+         Bytes.unsafe_to_string length;
+         Ext_buffer.contents buf;
+         sourcefile;
+         "\n";
+         Marshal.to_string pt [];
+       ])

compiler/depends/binary_ast.mli

@@ -25,6 +25,8 @@
 type _ kind = Ml : Parsetree.structure kind | Mli : Parsetree.signature kind
 
 val read_ast_exn : fname:string -> 'a kind -> 'a
+(** Raises [End_of_file] when the file ends before its header or its AST does,
+    as when a killed build cut it short. *)
 
 val write_ast : sourcefile:string -> output:string -> 'a kind -> 'a -> unit
 (** [write_ast ~sourcefile ~output kind ast] writes to [output]:
@@ -33,5 +35,22 @@ val write_ast : sourcefile:string -> output:string -> 'a kind -> 'a -> unit
       (predefined names excluded), each followed by a newline;
     - [sourcefile] and a newline;
     - the marshalled [ast].
-    [read_ast_exn] skips the dependency block; rewatch reads it in
-    [get_dep_modules] (rewatch/src/build/deps.rs). *)
+    [read_ast_exn] skips the dependency block; OCaml rewatch reads it with
+    {!read_header}, and Rust rewatch in [get_dep_modules]
+    (rewatch/src/build/deps.rs). *)
+
+type header = {source_start: int; source_end: int}
+(** Where the parts of an AST's header end: the dependency block takes the
+    bytes from 4 to [source_start], the source path those up to
+    [source_end], a newline, and the marshalled AST follows. *)
+
+val read_header : ?length:int -> string -> header option
+(** [read_header ?length contents] reads the header of an AST whose first
+    [length] bytes, all of them by default, are those of [contents]. It is
+    [None] when they end before the header does, or give an impossible length
+    for the dependency block. *)
+
+val header_dependencies : string -> header -> string list
+(** The module names in the dependency block. *)
+
+val header_source : string -> header -> string