flakes: reuse cached results for `self.packages`

[KiaraGrouwstra]

Is your feature request related to a problem?

flakes currently cache results at the level of the attribute built from the cli, if i understand correctly, whereas such attributes may be accessed from code as potentially intermediate values as well, e.g. through the self fix-point.

Proposed solution

while perhaps not a high priority, it would be nice to see cached results for packages used when accessing these at the code level through self.packages as well.

Alternative solutions

• incremental builds seem prior art

Additional context

i recently took it from @roberth that this feature was not yet implemented.
/cc @edolstra, who is probably most well-versed in performance optimization for flake-based builds.

Checklist

• checked latest Nix manual (source)
• checked open feature issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[roberth]

A previous attempt was (effectively) rejected because of a significant slowdown in attrset performance.

• Cache the result of getFlake #4511

I think another attempt could be made, either
a: using a primop thunk for any "unevaluated" but cached values, therefore requiring no changes to the performance critical attrset representation, or
b: adding a second internal attrset type backed by a C++ virtual interface

(a) is conceptually simpler and guaranteed to be fast. Note that the number of attrs locally in flake outputs is tiny vs the attrs in e.g. the Nixpkgs packages used.
(b) could enable some interesting other functionality like splitting retrieval and listing, improving performance and helping with ${system}

Note that this is not a localized thing. This puts all of call-flake.nix in a big fixpoint that must be used for all output accesses. It probably involves computing which parts of the flake inputs graph can influence each other due to follows, etc. Might be easier to sidestep that by not treating dependencies as individual cache keys and doing everything in the context of the root flake; having the root flake be part of all cache keys.