Brewfile.lock.json's filename is misleading

[boardfish]

#552 seems to suggest that the Brewfile.lock.json file generated by brew bundle install is for debugging purposes. However, the name suggests to me that it is responsible for ensuring that brew bundle install has the same results for different users in the same way that, for example, Gemfile.lock does against Ruby's bundler gem. With a Gemfile.lock in place, bundle install will attempt to install the same versions of the specified gems across two different machines. Likewise, Brewfile.lock.json's name (alongside this extension's intentional similarity to bundler) suggests to me that this file is responsible for the same thing. (There's a bit of oversimplification there for the sake of argument, but you get my gist.)

It came up in code review just now that perhaps we should add the filename to our .gitignore, and I had to seek out that PR to confirm that for myself. However, this could be avoided for folks in future if the generated file has a different name. The real responsibility of the file is for debug logging, so maybe brew-bundle-debug-log-YYMMDDHHMM.json would be a good idea.

Whatever we go with here, it might also be good to get it merged into GitHub's own macOS gitignore, along with Brewfile.lock.json.

[MikeMcQuaid]

We've had this feedback a few times and are still considering how best to address it without breaking existing usage.

[jacobbednarz]

i'm personally in favour of Brewfile.state.json which denotes its intended position. as for the rollout, i would expect that Brewfile.lock.json would be read but if detected, renamed to Brewfile.state.json during that run (or deleted once used for the new file to be written out). this approach also gets us a gradual upgrade as opposed to forcing others to do work to continue using it.

[MikeMcQuaid]

Yes, that approach would make sense to me @jacobbednarz.

[boardfish]

The migration path sounds good. Should we also offer the user the chance to add it to the .gitignore in the directory from which brew bundle was run? One concern there might be that they're not running it from an interactive session – so folks use this in their macOS GitHub Actions workflows, for example?

As for the naming of the file, I would argue that state feels like a loaded term still – to me, there's a vague implication of that state being coupled to brew bundle in either direction. It's correct that brew bundle would potentially update this "state", but it leaves another question open: would this "state" also dictate the behaviour of brew bundle? That brings us back to the cause of the issue, so I'm not sure it's the optimal solution.

I'd mentally frame it as a log or report of the previous run, hence my suggestion – it captures what happened when we ran brew bundle on this system at this time, and it installed these dependencies. That feels separate from the implication of state controlling brew bundle – the run is finished, the log is written, and nothing more gets done with it. That naming helps to capture the idea that this file is purely an output of brew bundle. I'm open to discuss further if it doesn't sound optimal.

[MikeMcQuaid]

Should we also offer the user the chance to add it to the .gitignore in the directory from which brew bundle was run?

Not sure I understand the question @boardfish, can you elaborate?

would this "state" also dictate the behaviour of brew bundle?

I think this is definitely less obvious than lock but I'm open to other naming. Brewfile.last_run_state.json, Brewfile.last_run.json, etc.?

Also perhaps worth noting that the implementation that never really got done here but could/should be as part of this work: on a failure, diff what would be written to a new Brewfile.lock.json with the existing one and output the differences. That's the bit that needs done by hand right now, is pretty unintuitive, and should actually reveal what changes may have broken things (e.g. system changes vs. Homebrew changes vs. version changes etc.).

I'd mentally frame it as a log or report of the previous run

Agreed.

---

Once we flesh this out: if you're game to create a PR @boardfish, we'd love that! I know you're a talented Ruby developer and I'm happy to help get any draft/WIP PR along to completion.

Thanks for your thoughtful responses and your own OSS work ❤️ !

[boardfish]

Should we also offer the user the chance to add it to the .gitignore in the directory from which brew bundle was run?

I'll break down what I'm guessing this might look like in practice, since the experience from start to finish sort of plays into where and when we do it.

It might not be apparent to the user that the Brewfile.lock.json shouldn't generally be committed due to its name – as we migrate towards this new default, it'd be good to communicate the change, which is where:

Brewfile.lock.json would be read but if detected, renamed to <the new filename> during that run (or deleted once used for the new file to be written out).

We might log something here to the effect of:

A report of this run has been written to <the new filename>. If you previously
had a Brewfile.lock.json here,it has been renamed to <the new filename>.old.

Since folks probably don't want to commit these files, we might then prompt:

Would you like to add <pattern for the new filename> to the `.gitignore` in <pwd>? [y/N]

...if one exists. Some things we might be concerned with that might make this tougher or less viable are:

• We need to check that the .gitignore exists
• We need to check that it has a pattern that would match the log files
• We need to prompt the user inside brew bundle, which might be an issue if they're running in CI
• We need to make sure we only prompt them for this once somehow – how might we track this state?
• We're making a (safe?) assumption they're using Git for version control

It's a small thing that might help folks with the migration and make them more aware of it, but those concerns and how they might differ across environments might make it a heavier lift.

[MikeMcQuaid]

We might log something here to the effect of:

👍🏻 this makes sense (but ideally without the "if" because we can print a different message depending).

Since folks probably don't want to commit these files, we might then prompt:

👎🏻 I think we should leave this up to the docs to inform.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[colindean]

I'd like to see this.

I like Brewfile.last_run.json as the new file name.

I like writing to that file if nothing exists or writing to Brewfile.lock.json if it already exists.

I don't like altering files for users unless the program completely "owns" that file, so suggesting they take a manual action will have to suffice.

I like more actively advising the user that the file can be ignored, perhaps with a nudge to add to .gitignore or to suppress it entirely with --no-lock or the envvar (or just link to the docs or a new command that outputs that doc section).

I also like setting deprecation and abiding by it, so the lock -> last_run code could be removed in like a year, and it goes back to just writing Brewfile.last_run.json.

No matter what, I want the new file to have a version number, e.g., what I did in #1175, but perhaps without the other things I did that made that PR unsuitable.

---

Other ideas while we're considering a rename

One idea I'd had at some point was to switch from JSON to JSONL and just append the new run's state. Then the user can ingest the file through something that looks at the differences per object instead of having to spelunk through git history. There might be some storage-related pros and cons of whole file replacement vs. appending a line, but the impact is so small that I'm not interested in finding out how many bytes difference it'd be. My problem with JSONL is versioning it, something I want to do.

Another idea I had was not to use JSON but rather a simpler key-value output that would probably just look like gron'd JSON. That'd be easier for a human to act and consume with ye olde diff.

[MikeMcQuaid]

I don't like altering files for users unless the program completely "owns" that file

brew bundle does completely own this file.

so suggesting they take a manual action will have to suffice.

I disagree: we should diff with the previous version and present that output to users.

I like more actively advising the user that the file can be ignored, perhaps with a nudge to add to .gitignore or to suppress it

This is already documented?

No matter what, I want the new file to have a version number, e.g., what I did in #1175, but perhaps without the other things I did that made that PR unsuitable.

Really don't see a need for this. Homebrew Bundle is rolling release and I've seen no evidence of this file having external consumers.

One idea I'd had at some point was to switch from JSON to JSONL and just append the new run's state. Then the user can ingest the file through something that looks at the differences per object instead of having to spelunk through git history.

This feels like overkill, sorry.

Another idea I had was not to use JSON but rather a simpler key-value output that would probably just look like gron'd JSON. That'd be easier for a human to act and consume with ye olde diff.

Given the data is hierarchical: I don't think optimising for diff usage is gonna be a good outcome and will make things worse.

[colindean]

I don't like altering files for users unless the program completely "owns" that file

brew bundle does completely own this file.

so suggesting they take a manual action will have to suffice.

I disagree: we should diff with the previous version and present that output to users.

Sorry, my remark was ambiguous. I don't want brew bundle to alter the .gitignore. The user would need to do that manually.

I like more actively advising the user that the file can be ignored, perhaps with a nudge to add to .gitignore or to suppress it

This is already documented?

It is, but the traffic asking about indicates that we could do better informing the user more actively about how brew bundle is operating. I'd like to see something like this in the output of brew bundle:

…
Using unixodbc
Using pkg-config
Using [email protected]
Homebrew Bundle complete! 9 Brewfile dependencies now installed.
A record of this installation is in Brewfile.last_run.json; for more information about this record, see https://github.com/Homebrew/homebrew-bundle/somedocsurl.

The verbosity of what I'd really like to see might be too much:

…
Using unixodbc
Using pkg-config
Using [email protected]
Homebrew Bundle complete! 9 Brewfile dependencies now installed.
A record of this installation is in Brewfile.last_run.json.
Pass --no-lock to disable writing this record. 
For more information, see https://github.com/Homebrew/homebrew-bundle/somedocsurl.

No matter what, I want the new file to have a version number, e.g., what I did in #1175, but perhaps without the other things I did that made that PR unsuitable.

Really don't see a need for this. Homebrew Bundle is rolling release and I've seen no evidence of this file having external consumers.

I've never regretted having a version number in a data dump, but I have regretted not putting one in until it would be useful. We're talking about migrating from one filename convention (effectively v1) to another (effectively v2) for a filename that is not guaranteed.

One idea I'd had at some point was to switch from JSON to JSONL and just append the new run's state. Then the user can ingest the file through something that looks at the differences per object instead of having to spelunk through git history.

This feels like overkill, sorry.

Probably, given how there's

no evidence of this file having external consumers

so I'll drop that idea.

Another idea I had was not to use JSON but rather a simpler key-value output that would probably just look like gron'd JSON. That'd be easier for a human to act and consume with ye olde diff.

Given the data is hierarchical: I don't think optimising for diff usage is gonna be a good outcome and will make things worse.

Yeah, it's probably unnecessarily complex for this use case.

---

Another idea: What if this file was written to XDG_CACHE_HOME or XDG_STATE_HOME? Maybe a brew bundle run from /home/colin/Source/homebrew/homebrew-bundle goes into

~/.cache/homebrew/bundle/records/home__colin__Source__homebrew__homebrew-bundle__Brewfile.last_run.json

The eliminates checking it in, though, for people who want to do that.

[boardfish]

I like the idea of putting the logfile in a less conspicuous location that Homebrew is responsible for, that's a great shout!

[MikeMcQuaid]

Sorry, my remark was ambiguous. I don't want brew bundle to alter the .gitignore. The user would need to do that manually.

👍🏻

It is, but the traffic asking about indicates that we could do better informing the user more actively about how brew bundle is operating. I'd like to see something like this in the output of brew bundle:

I don't think the traffic warrants outputting this message for every user every time. If we did it once on e.g. first creation: 👍🏻

I've never regretted having a version number in a data dump, but I have regretted not putting one in until it would be useful.

I've definitely regretted merging PRs with such things only for them to go unused and cause unneeded bloat and confusion on reading the code later.

so I'll drop that idea.

❤️

Another idea: What if this file was written to XDG_CACHE_HOME or XDG_STATE_HOME? Maybe a brew bundle run from /home/colin/Source/homebrew/homebrew-bundle goes into
I like the idea of putting the logfile in a less conspicuous location that Homebrew is responsible for, that's a great shout!

Please no 😁

That's making it even less intuitive/expected than it is currently.

Let's not blow the scope up on this one. We've got enough decided here for someone to open a PR if desired. If we want to iterate beyond that: let's do so when the initial issue is resolved.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[MikeMcQuaid]

Here's an interesting use of these lock files: https://github.com/advanced-security/brew-dependency-submission-action