[Bug]: build pipelines not working for third party dependencies (PTE nor appsource deps)

[SteveKrisjanovsD365]

AL-Go version

6.0

Describe the issue

we are currently standing up several AL Go repositories (both AL Go PTE and AL Go Appsource).

• The appsource AL GO repos depend on external (i.e. not owned by us) appsource dependencies
• The PTE AL GO repos depend on those same external appsource dependencies AND external PTE dependencies (the PTE deps are also not owned by us, rather they privately share their PTE apps with us via secure channels)

according to the AL Go workshop documentation, the build pipelines should work out of the box for the following use cases

• when depending on an external AppSource dependency, AL Go should fetch the dependency via the default whitelisted public nuget and use that app for the build pipeline
• when depending on an external PTE dependency, AL Go should fetch the dependency from the "installApps" JSONArray property (in our case we're momentarily using google drive with authentication turned off - one URL per app in the "installApps" setting)

as far as i understand, the build pipelines work both when docker is used ("useCompilerFolder": false, "doNotPublishApps": false) and when docker IS NOT used ("useCompilerFolder": true, "doNotPublishApps": true). My observations suggest otherwise.

• when our AL GO repo has NO PTE dependencies NOR any Appsrouce dependencies, the build SUCCEEDS
• when our AL GO repo has an external appsrource dependency (via default nuget) and docker is used, the build FAILS
• when our AL GO repo has an external appsrource dependency (via default nuget) and docker is NOT used, the build SUCCEEDS
• when our AL GO repo has an external PTE dependency (via default nuget) and docker is used, the build FAILS
• when our AL GO repo has an external PTE dependency (via default nuget) and docker is NOT used, the build FAILS

the external dependencies (both appsource and PTE) are not written by my company. As such, the external PTE dependencies are hosted as full apps on google drive (for now) and we expose them to AL go via the InstallApps property setting. For external AppSource dependencies, default nuget functionality is used where ALGO fetches the dependency from the whitelisted public nuget repository.

Expected behavior

all scenarios should pass the CICD build pipeline. no build failures.

Steps to reproduce

request access to https://github.com/SteveKrisjanovsD365/PTEDemo/ where I have the following branches to reproduce (you can run the CICD in the non-main branches as we have no delivery steps added - this is a throwaway repo)

• main (NO external appsource NOR external PTE dependencies, use docker) << SUCCEEDS
• 20241108-appsourcedep-usedocker-no (one external appsource dependency - do NOT use docker) << SUCCEEDS
• 20241108-appsourcedep-usedocker-yes (one external appsource dependency - use docker) << FAILS
• 20241108-ptedep-usedocker-no (one external PTE dependency - do NOT use docker) << FAILS
• 20241108-ptedep-usedocker-yes (one external PTE dependency - use docker) << FAILS

Alternately you can reproduce it from scratch if you prefer. "installApps" setting is used fore the external PTE dependencies (one full app per URL, not a runtime pkg), and "useCompilerFolder" + "doNotPublishApps" to utilize or skip docker (true, true to skip docker, false, false to use docker)

the branches that FAIL have the following build errors:

20241108-appsourcedep-usedocker-yes (compiles fine in VS Code):

Error: Unexpected error when running action. Error Message: Extension compilation failed src/pageextensions/Activities.PageExt.al(2,47): error AL0247: The target Page 'MAW Activities' for the extension object is not found, StackTrace: at Invoke-ScriptInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 113 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 379 <- at Publish-BcContainerApp, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 154 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 941 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2359 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2328 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2313 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1057 <- at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1019 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 391 <- at <ScriptBlock>, D:\a\_temp\365735ec-5cdc-4b81-abce-b5b8cf04d53d.ps1: line 3 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\Invoke-AlGoAction.ps1: line 17 <- at <ScriptBlock>, D:\a\_temp\365735ec-5cdc-4b81-abce-b5b8cf04d53d.ps1: line 2 <- at <ScriptBlock>, <No file>: line 1
Error: Process completed with exit code 1.

20241108-ptedep-usedocker-no (compiles fine in VS Code):

Compilation started for project 'PTEDemo' containing '2' files at '15:38:03.546'.

Error: AL1022 A package with publisher 'Steve Krisjanovs', name 'ALProjectPTE', and a version compatible with '1.0.0.0' could not be found in the package cache folders: D:\a\PTEDemo\PTEDemo\.project1\.packages

Compilation ended at '15:38:09.427'.

Files in build artifacts folder:
Run-AlPipeline Telemetry Correlation Id: 6c49ec69-16e1-4503-8a99-f7be89e8b83b
Applying settings from D:\a\PTEDemo\PTEDemo\.github\AL-Go-Settings.json
No settings found in D:\a\PTEDemo\PTEDemo\.AL-Go\settings.json
No settings found in D:\a\PTEDemo\PTEDemo\.github\CICD.settings.json
No settings found in D:\a\PTEDemo\PTEDemo\.AL-Go\CICD.settings.json
No settings found in D:\a\PTEDemo\PTEDemo\.AL-Go\SteveKrisjanovsD365.settings.json
Enabling Microsoft telemetry...
Error: Unexpected error when running action. Error Message: App generation failed, StackTrace: at Compile-AppWithBcCompilerFolder, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\CompilerFolderHandling\Compile-AppWithBcCompilerFolder.ps1: line 511 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 920 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2076 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1608 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1601 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1057 <- at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1019 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 391 <- at <ScriptBlock>, D:\a\_temp\a39e368c-6256-42e5-87d3-8a9f3da3eadf.ps1: line 3 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\Invoke-AlGoAction.ps1: line 17 <- at <ScriptBlock>, D:\a\_temp\a39e368c-6256-42e5-87d3-8a9f3da3eadf.ps1: line 2 <- at <ScriptBlock>, <No file>: line 1
Error: Process completed with exit code 1.

20241108-ptedep-usedocker-yes (compiles fine in VS Code):

Error: Unexpected error when running action. Error Message: No apps to publish, StackTrace: at Publish-BcNuGetPackageToContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\NuGet\Publish-BcNuGetPackageToContainer.ps1: line 93 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 338 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 321 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1434 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1416 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1057 <- at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1019 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 391 <- at <ScriptBlock>, D:\a\_temp\61046ac8-bb0b-46dd-9f25-dafb492e3366.ps1: line 3 <- at <ScriptBlock>, D:\a\_actions\microsoft\AL-Go-Actions\v6.0\Invoke-AlGoAction.ps1: line 17 <- at <ScriptBlock>, D:\a\_temp\61046ac8-bb0b-46dd-9f25-dafb492e3366.ps1: line 2 <- at <ScriptBlock>, <No file>: line 1
Error: Process completed with exit code 1.

Additional context (logs, screenshots, etc.)

see attached logs for each of the relevant branches (see "steps to reproduce" notes)

the only one that succeeds is branches [main] (no dependencies) and [20241108-appsourcedep-usedocker-no] (one external appsource dep, docker NOT used). All the ohers fail.

logs_30668419473-20241108-appsourcedep-usedocker-yes.zip
logs_30668621671-20241108-ptedep-usedocker-no.zip
logs_30669787918-20241108-ptedep-usedocker-yes.zip
logs_30668189200-20241108-appsourcedep-usedocker-no.zip

[freddydk]

It looks like the URL provided in installApps isn't a direct download url.
Investigating that, I get:

Which is HTML and not an .app file, meaning that when AL-Go needs to use this app, it will not work.

[SteveKrisjanovsD365]

@freddydk you're right. I tested the installApps URLs in postman and indeed they return an HTML preview of the file. not the file itself. Let me see if I can correct this and re-run the CI/CD pipelines

[SteveKrisjanovsD365]

@freddydk thanks for the suggestion! That was was the issue (google drive turns out is a poor CDN endpoint). All pipelines are passing now that I moved the dependency app files from Google drive to a dedicated GitHub repo and changed the URLs in installApps accordingly

Next question...

That dependencies repo is currently public and we want to make it private-only via GitHub personal access tokens

The raw download of such a file according to my research is as follows:

https://[email protected]////

I assume I need to use classic tokens and not fine-grained tokens? I've been trying to use a fine grained token, granting the content:read-only permission to the repository where the dependencies are and setting our org as the owner. Despite this I'm getting a 404 error from postman. GPT suggests that fine-grained tokens require request bearer headers but I'm unsure if BCContainerHelper supports that (my guess is no), hence me exploring the classic token route.

[freddydk]

I would use NuGet instead.

And then provide the people who need access to the NuGet feed access, so they themselves can create their secrets,

[SteveKrisjanovsD365]

I would use NuGet instead.

And then provide the people who need access to the NuGet feed access, so they themselves can create their secrets,

duly noted! I'm going to reach out our dependency ISVs if they have private nuget feeds I can consume