Add API for incremental compilation followed by watching changes and updating buildinfo

[perbergland]

Search Terms

incremental watch compiler api

Suggestion

Support incremental + watch compilation in the compiler API

Use Cases

When using the compiler API to build a typescript compiler plugin for a build system that supports "developer mode" (a mode where changes are made incrementally and expected to quickly turn around with transpiled files for changes) it becomes necessary to support both incremental compilation (loading buildinfo from disk at startup and writing changes back) and watch mode (to not rescan the entire source tree for every change since this takes tens of seconds or even minutes).

It does not seem to be possible using the existing compiler API when I study it, and a request to show an example of this has gone unanswered (microsoft/TypeScript-wiki#260).

Examples

This would be used by me in my typescript compiler for meteorjs to keep a watching build instance around for a directory for subsequent compilations:

https://github.com/Meteor-Community-Packages/meteor-typescript-compiler/blob/master/meteor-typescript-compiler.ts

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
• This feature would agree with the rest of TypeScript's Design Goals.

[perbergland]

My attempt to implement this by providing a createProgram function to createWatchCompilerHost and have that createProgram invoke createIncrementalProgram is not working:

prepareIncrementalProgram(
    program: BuilderProgramType,
    buildInfoFile: string
  ) {
    const diagnostics = [
      ...program.getConfigFileParsingDiagnostics(),
      ...program.getSyntacticDiagnostics(),
      ...program.getOptionsDiagnostics(),
      ...program.getGlobalDiagnostics(),
      ...program.getSemanticDiagnostics(), // Get the diagnostics before emit to cache them in the buildInfo file.
    ];

    const writeIfBuildInfo = (
      fileName: string,
      data: string,
      writeByteOrderMark: boolean | undefined
    ): boolean => {
      if (fileName === buildInfoFile) {
        info(`Writing ${getRelativeFileName(buildInfoFile)}`);
        ts.sys.writeFile(fileName, data, writeByteOrderMark);
        return true;
      }
      return false;
    };

    /**
     * "emit" without a sourcefile will process all changed files, including the buildinfo file
     * so we need to write it out if it changed.
     * Then we can also tell which files were recompiled and put the data into the cache.
     */
    const emitResult = program.emit(
      undefined,
      (fileName, data, writeByteOrderMark, onError, sourceFiles) => {
        if (!writeIfBuildInfo(fileName, data, writeByteOrderMark)) {
          if (sourceFiles && sourceFiles.length > 0) {
            const relativeSourceFilePath = getRelativeFileName(
              sourceFiles[0].fileName
            );
            if (fileName.match(/\.js$/)) {
              info(`Compiling ${relativeSourceFilePath}`);
              this.numCompiledFiles++;
              this.addJavascriptToCache(relativeSourceFilePath, {
                fileName,
                source: data,
              });
            }
            if (fileName.match(/\.map$/)) {
              this.cache?.addSourceMap(relativeSourceFilePath, data);
            }
          }
        }
      }
    );

    this.writeDiagnostics(diagnostics);
  }

createWatcher(directory: string): WatcherInstance {
    info(`Creating new Typescript watcher for ${directory}`);

    const configPath = ts.findConfigFile(
      /*searchPath*/ "./",
      ts.sys.fileExists,
      "tsconfig.json"
    );
    if (!configPath) {
      throw new Error("Could not find a valid 'tsconfig.json'.");
    }

    const buildInfoFile = ts.sys.resolvePath(
      `${this.cacheRoot}/buildfile.tsbuildinfo`
    );
    if (!process.env.METEOR_TYPESCRIPT_CACHE_DISABLED) {
      this.cache = new CompilerCache(
        ts.sys.resolvePath(`${this.cacheRoot}/v1cache`)
      );
    }
    const optionsToExtend: ts.CompilerOptions = {
      incremental: true,
      tsBuildInfoFile: buildInfoFile,
      noEmit: false,
      sourceMap: true,
    };

const createProgram: ts.CreateProgram<BuilderProgramType> = (
      rootNames = [],
      options = {},
      providedHost,
      oldProgram,
      configFileParsingDiagnostics,
      projectReferences
    ) => {
      const innerCreateProgram: ts.CreateProgram<BuilderProgramType> = (
        ...args
      ) => {
        return ts.createEmitAndSemanticDiagnosticsBuilderProgram(...args);
      };

      const standardHost = ts.createIncrementalCompilerHost(options);
      const host: ts.CompilerHost = {
        ...standardHost,
        readFile: (fileName) => {         
          return standardHost.readFile(fileName);
        },
      };
      const program = ts.createIncrementalProgram({
        rootNames,
        options,
        configFileParsingDiagnostics,
        projectReferences,
        host,
        createProgram: innerCreateProgram,
      });

      return program;
    };

const watchHost = ts.createWatchCompilerHost(
      configPath,
      optionsToExtend,
      ts.sys,
      createProgram,
      (diagnostic) => this.writeDiagnostics([diagnostic]),
      (...args) => this.reportWatchStatus(...args),
      watchOptionsToExtend
    );

    watchHost.afterProgramCreate = (program) => {
      // The default implementation is to emit files to disk, which we absolutely do not want
      this.prepareIncrementalProgram(program, buildInfoFile); // this method calls program.emit
    };

    const watch = ts.createWatchProgram(watchHost);

While the compilation does work, the first compilation does not take advantage of the buildinfo file but recompiles all files, and subsequent compilations (using a cached "watch") generate a new program every time watch.getProgram() is invoked even if no source file has been changed, so it does not seem that watch mode is working.

[perbergland]

I’m closing this now as I believe the issues I see are bugs, not missing features. I will create a standalone minimal example that shows that when part of a watch build, incremental builds don’t work properly (maybe just under some circumstances that I have yet to figure out).

[andersekdahl]

Any update here? Noticed that you created a repro in a repo, but can't see that you've created an issue with it?

[perbergland]

I filed a separate issue for the bug that made me think this was not possible in the current API #41690