Idea - Chaining spec generation using projectReferences

[liamnichols]

👋 I've been spending some time recently working out a plan for modularising our project and have been looking into recent discussions in the issues about how XcodeGen might be able to further help with this.

It mostly relates to the RFC/thread for workspace generation (#168) back in 2017, however since the original discussion, XcodeGen has grown quite a bit and now has support for things like subprojects using projectReferences that were introduced in #701.

From my (somewhat limited) understanding of the project, the projectReferences seem like a potentially good candidate for expansion to allow the automatic generation of a number of Xcode Projects so I tried putting some feelers out there in #776 but didn't get much of a response. Since I had some time this weekend, I wanted to give a POC a go to better understand things.

In this draft PR, I've expanded SpecLoader to introduce a new method:

SpecLoader.loadProjects(path:projectRoot:variables:)

This method combined with the new optional spec property on a ProjectReference allows the loader to resolve all dependant specs in order to compute an array of Project objects that can then be passed into the generator to generate dependant projects in the correct order.

---

As an example, lets say that I have the following specs in my workspace:

./Global/project.yml

name: Global
projectReferences:
  Frameworks:
    path: ../Frameworks/Frameworks.xcodeproj
    spec: ../Frameworks/project.yml
  GlobalCore:
    path: ../GlobalCore/GlobalCore.xcodeproj
    spec: ../GlobalCore/project.yml
  Settings:
    path: ../GlobalFeatures/Settings/Settings.xcodeproj
    spec: ../GlobalFeatures/Settings/project.yml
# ...

./Frameworks/project.yml

name: Frameworks
# ...

./GlobalCore/project.yml*

name: GlobalCore
projectReferences:
  Frameworks:
    path: ../Frameworks/Frameworks.xcodeproj
    spec: ../Frameworks/project.yml
# ... 

./GlobalFeatures/Settings/project.yml

name: Settings
projectReferences:
  Frameworks:
    path: ../Frameworks/Frameworks.xcodeproj
    spec: ../Frameworks/project.yml
  GlobalCore:
    path: ../GlobalCore/GlobalCore.xcodeproj
    spec: ../GlobalCore/project.yml
# ... 

I can now run the generator command on my entry project, I'll get the following output:

$ xcodegen generate --project ~./Global --spec ~./Global/project.yml 
Resolved projects in order:
1. ./Frameworks/Frameworks.xcodeproj
2. ./GlobalCore/GlobalCore.xcodeproj
3. ./GlobalFeatures/Settings/Settings.xcodeproj
4. ./Global/Global.xcodeproj
⚙️  Generating plists...
⚙️  Generating project...
⚙️  Writing project...
Created project at ./Frameworks/Frameworks.xcodeproj
⚙️  Generating plists...
⚙️  Generating project...
⚙️  Writing project...
Created project at ./GlobalCore/GlobalCore.xcodeproj
⚙️  Generating plists...
⚙️  Generating project...
⚙️  Writing project...
Created project at ./GlobalFeatures/Settings/Settings.xcodeproj
⚙️  Generating plists...
⚙️  Generating project...
⚙️  Writing project...
Created project at ./Global/Global.xcodeproj

As a result, I then end up with all four projects generated as I expect! This helps me to avoid having to maintain a wrapping script for XcodeGen that currently consists of me manually invoking the generator a given number of times since all of the specs are detected based on my projectReferences that I already have to define anyway 🚀

---

So, while all of this works, it's far from being correct. I've identified a number of things that would probably need addressing before hand, but before we look into that, I want to get some thoughts on the concept in general 🙏

Here are a few things to discuss:

1. xcodegen the CLI is very tailored to a specific project/spec since both GenerateCommand and DumpCommand inherit from ProjectCommand. Would this still make sense? Should this feature be implemented into a new command entirely, or should the existing CLI commands be rearchitected?
2. This approach works by running the generator a number multiple times to keep changes minimal, but if loaded all projects and generated in one go then we could benefit dump by being able to visualise the workspace/repo's dependencies cross-projects
3. Because projectReferences is keyed by the name that is used for reference in the spec, this has the potential to be different to the name of the actual project and this could get confusing but this can't be solved unless we load the referenced specs before resolving the project itself.
4. SpecLoader might need rewriting, I didn't get time to look at why it stores project but maybe that breaks things since there are now multiple. Might be better to investigate loading multiple SpecFile objects first and resolving them first?
5. Having to match up path and spec in ProjectReference could lead to misconfigurations where the same spec points to different project paths, or different project paths use the same spec. We can validate against that but maybe there would also be a way to avoid this?
6. Is building on top of projectReferences the right approach to this? By doing so, it means that you can use them both with existing xcodeproj references and also specs to be generated. I think that's a good thing but I might be overlooking something?

I'm really keen to help out on a feature like this, but I think the idea still needs fleshing out a bit so any input on this would be great! Thanks!

[pavel-trafimuk]

Anew news for this request? cause we have the same issue
Thanks

[ghost]

Hi,
Are there any updates?

[karim-alweheshy]

in big projects, this approach makes Xcode much faster as it have to edit a small xcproj file in contrast to a 200MB main project file. it also shows only folder related targets when adding a new file to a fileGroup.

Let's give this priority please!