Adding script to generate compile commands database

[guimafelipe]

When using a text editor or IDE to work with C++, usually it is common to see a big amount of errors in the code even though everything is building correctly locally. This can be annoying, distracting and also slow down productivity.

Clangd is a Language Server Protocol for C/C++ that can help solve these problems. It removes the annoying errors (and enable real ones), enables tools like "Go to definition/Go to declaration", enables code completion, and also helps AI to do a better job in the codebase. It works on different code editors like Visual Studio, VS Code, (neo)vim and so on.

For clangd to work, it needs a compile_commands.json file to be generated on build, which is a compile commands database. Clangd automatically finds it in the project directory tree and links the files on it. The compile commands database is a JSON file with the commands used by Cl.exe for compiling each of the files in the exact way it was done in the successful build process.

Unfortunately, MSBuild does no generate a compile_commands.json file natively at the current moment, which makes it necessary to use an external tool. Our build process uses the BuildAll.ps1 script, which does various operations before actually calling MSBuild on the solution, which also makes it harder for these tools to infer the compile compile commands database based only on the project files.

A microsoft engineer, facing this same problem, developed the tool ms2cc. This tool receives a msbuild.log file and the project root and generates the database.

Unfortunately, for this tool to work it needs to infer for each Cl.exe command, the correct absolute path to the files so it can generate the correct output. And .vcxproj files in our project uses relative paths, making it impossible to infer the full path from it.

So, I added a preparation script to use the context of the CL task in the binlogs to set the full paths of all the files before passing the list to ms2cc, making it work and generating the correct command compile database.

After that, install clangd on your favorite editor and enjoy all the tools and productivity.

A microsoft employee must use /azp run to validate using the pipelines below.

WARNING:
Comments made by azure-pipelines bot maybe inaccurate.
Please see pipeline link to verify that the build is being ran.

For status checks on the main branch, please use TransportPackage-Foundation-PR
(https://microsoft.visualstudio.com/ProjectReunion/_build?definitionId=81063&_a=summary)
and run the build against your PR branch with the default parameters.

[DrusTheAxe]

Why the $(MSBuildThisFileDirectory) prefix on every header and source?

VS doesn't do this when you add a file in Solution explorer (new, or point at existing).

Aren't these files relative to the project (and typically in same dir)?

[guimafelipe]

For developers that work on WinAppSDK, it is common that regardless of the code editor used, there is always a lot of errors that does not help with productivity even if the project builds correctly locally. There is also no consistent Language Server Protocol support (things like "Go to Definition", "See references", etc.).

This PR is from some research and work I put to try to make this all available for WinAppSDK developers.

For this, we can use CLang Driver (https://clangd.llvm.org/), available in most of the editors, to have this support. For clangd to work, it needs a "compile commands database", which is basically a JSON file with all the commands executed by the compiler for building each of the files in the project, with the include folders needed for each of them, so clangd can do all the correct linking between the files.

Unfortunately, MSBuild does not produce a compile commands database naturally, so we need to use an external tool for that. There are some Visual Studio extensions for that, but they not work because our build system uses powershell scripts moving lots of stuff around before even calling msbuild directly.

In my research, I found the https://github.com/freddiehaddad/ms2cc tool, from another microsoft software engineer, that was attempting to solve the same problem in their project. This tool works getting a msbuild log and converting it into a compile commands database, which is what the powershell script I added is doing.

The problem is: if we don't pass $(MSBuildThisFileDirectory) to CLCompile includes on our project files, the compile command will be something like:
cl.exe ... [includess] /Fo="C:\WindowsAppSKD\obj\..." pch.cpp

This would not be a problem is there were only one file named pch.cpp in the project. We could search for it in our file tree and know what is the full path. But there are various pch.cpp files in WinAppSDK, and there is no way to know only from the compilation path. We could use the object folder for that (which was also being deleted, losing track of some of the include files referenced by the tests), but there is no pattern to follow either.

By including it with $(MSBuildThisFileDirectory), it will have the full path for every .h or .cpp file compiled in the msbuild logs, which will make the linking possible as there will be no ambiguity of which file is being referenced.

This way, the text editors now have full access to the LSP beneficies from clangd and there will be no more weird errors everywhere even though everything builds correctly.

WinAppSDK was already using $(MSBuildThisFileDirectory) in part of the includes, and broadcasted it to the rest of the project and it worked really well for generating the database. I am currently investigating if there is a way to set VS to add it automatically.

[DrusTheAxe]

My concern is VS is our primary build tool, and adding files to projects is commonly done via VS' Solution Explorer.

I didn't say primary editor. Some use VS, I (like many) rarely use VS' editor (I'm a longstanding TSE fan, others use VScode and other tools). But for build and project + solution management VS is the primary tool.

While command line support is nice, I'm not keen on the friction of requiring $(MSBuildThisFileDirectory) prefixed on all files in *proj when VS doesn't do that. I don't see us banning VS, and most devs aren't going to know better (or human error will occasionally forget) to edit *.*proj files after making any edits in VS. This is just going to lead to a constant tax where command line builds regularly break as most devs expand their projects using VS, and not this additional manual editing step.

(Which, BTW, the VS team says "Do not directly edit *.*proj files." Even though many do, or need to do because the GUI lacks some GUI means to make certain edits.

This will create pains and friction for the team. That's the wrong vector - we want LESS friction for the dev team, not more.

TL;DR I'm sympathetic to letting devs use their preferred tools, but not in this way. Not benefiting some developers by imposing a permanent and recurring tax on all future development. The ROI isn't justified.

Have you talked to the VS team about this? That seems the better answer -- for VS to create/manage a compiler command database and any related changes needed (e.g. $(MSBuildThisFile)foo.cpp). Then more tools are supported but w/o the added dev tax to do non-VS-standard behavior.

[guimafelipe]

I also use Visual Studio regularly, which is also a editor. Clangd is also naturally supported in Visual Studio, which is also currently having this same problems as the other editors (showing errors where there is none, not having complete support for C/C++ LSP and so on).

I agree with you that having $(MSBuildThisFileDirectory) is a big friction and I am still trying to find alternatives to that. For now was the only thing that make it possible for us to correctly index all the files.

Have you talked to the VS team about this? That seems the better answer -- for VS to create/manage a compiler command database and any related changes needed (e.g. $(MSBuildThisFile)foo.cpp). Then more tools are supported but w/o the added dev tax to do non-VS-standard behavior.

I didn't talk to them yet. I was expecting that generating a command compile database would be something that already existed by default, but it most of the tools I found related to that are external and didn't work well for our case either. I like the idea of starting a discussion about it with them.

[Scottj1s]

@guimafelipe I'm also concerned with having to modify every ClCompile and ClInclude. Note that ClInclude is only an IDE amenity - it has nothing to do with the build. If you're processing the binlog to find the $task cl, you should be able to get the enclosing vcxproj location and infer the source location from that.

For completeness, another approach: cvdump on the pdb to collect the cl.exe command line and all parameters. But my preference would be groveling the binlog for the vcxproj location.

[guimafelipe]

@DrusTheAxe Thank you for the feedback, I worked on an alternative via powershell scripts for that and all is working fine now.
@Scottj1s Thank you for the tip on looking for the rest of the logs for context I could use to regenerate the full paths!

[DrusTheAxe]

What is a "compile commands database"?

[Scottj1s]

or just this, and have pipelines supply $CleanIntermediateFiles = $true.

Suggested change

	if ($WindowsAppSDKBuildPipeline -eq 1)
	{
	$CleanIntermediateFiles = $true
	}
	$cleanIntermediateFilesArg = if ($CleanIntermediateFiles) { "/p:WindowsAppSDKCleanIntermediateFiles=true" } else { "" }
	if ($CleanIntermediateFiles -eq $true)
	{
	$cleanIntermediateFilesArg = "/p:WindowsAppSDKCleanIntermediateFiles=true"
	}

[guimafelipe]

I implemented this change in all the calls of BuildAll.ps1in the pipeline. Let me know what you think about it. Maybe the standard can be for the variable to be true so we don't need to change all these calls.

[Scottj1s]

A general comment - most developers don't invoke special scripts like BuildAll.ps1. My own preference is to use the standard build tools: VS, MSBuild, dotnet - especially for inner loop. You might consider refactoring this into a custom MSBuild target with AfterTargets="ClCompile" to hook into every C++ compilation. Then you'd have direct access to the cl.exe args and project location and could write/append to a compile_commands.json directly. The advantage is that you'd always have an up to date compile commands DB, regardless of your IDE or build tool. You can implement a custom target with any arbitrary commands, including PS1 scripts, although I prefer to use custom inline C# tasks to enable debugging.