Chocolatey/ProGet Assets/Azure DevOps Pipelines: Unauthorized asset downloads

[norbertstoll]

Checklist

• I have verified this is the correct repository for opening this issue.
• I have verified no other issues exist related to my problem.
• I have verified this is not an issue for a specific package.
• I have verified this issue is not security related.

What You Are Seeing?

We are using packages with external dependencies like this one: https://community.chocolatey.org/packages/GoogleChrome
Since we're not allowed to use external sources on client systems, we are downloading the installers from official sources then uploading it to ProGets assets feed. The next step is to re-create the package by editing the download url of the installer in chocolateyInstall.ps1. Everything else remains unchanged.

ProGet and IIS have windows integrated authentication enabled. Permissions are set correctly.

Using chocolatey within an authenticated user session resp. PowerShell-session everything is working fine with those packages:
Chocolatey first downloads the nupkg-file, then the installer itself from ProGets asset feeds. Installations are finished successfully.

We also use Azure DevOps pipelines to get our stuff automated and that's where it comes to issues though we're using windows integrated authentication, too:

The nupkg gets downloaded but the download of the installer from the asset feed fails with HTTP 401 - Unauthorized.

---

The edited url64bit value in chocolateyInstall.ps1:

$packageArgs = @{
  packageName            = 'googlechrome'
  fileType               = 'MSI'
  url                    = 'https://dl.google.com/tag/s/dl/chrome/install/googlechromestandaloneenterprise.msi'
  url64bit               = 'https://my.internalrepo.net/endpoints/Chrome/content/114.0.5735.199/googlechromestandaloneenterprise64.msi'
  checksum               = '455a4ad2f39f8e3817d3e5831bf8e0be81bf3c7ad16025226fc3ef320bdcf1b0'
  checksum64             = '6cf6084f8504f6879266e906048babaf84c0575c56f58db8e71fb33da92acbda'
  checksumType           = 'sha256'
  checksumType64         = 'sha256'
  silentArgs             = "/quiet /norestart /l*v `"$($env:TEMP)\$($env:chocolateyPackageName).$($env:chocolateyPackageVersion).MsiInstall.log`""
  validExitCodes         = @(0)
}

Google Chrome's package in ProGet's asset feed:

What in the end leads to this:

PS C:\> choco list
Chocolatey v2.1.0
chocolatey 2.1.0
chocolatey-compatibility.extension 1.0.0
chocolatey-core.extension 1.4.0
GoogleChrome 114.0.5735.199
4 packages installed.
PS C:\>

The relevant part of code in ADO pipelines:

  - task: PowerShellOnTargetMachines@3
    displayName: Install required packages
    inputs:
      Machines: "${{ parameters.Servername }}"
      UserName: '$(UserName)'
      UserPassword: '$(PASSWORD)'
      InlineScript: |
        
        choco install $InstallPackage -f -u "$(UserName)" -p "$(PASSWORD)"

      CommunicationProtocol: 'Http'

What is Expected?

Download and installation of nupkg-files with dependencies on ProGet asset feeds, having windows authentication enabled, are working fine in ADO pipelines, too.

How Did You Get This To Happen?

Environment:
We're using ProGet in combination with Microsoft IIS:

• Version 2023.9 (Build 9)
• Basic license
• SSL enabled with cert issued by internal CA
• Local file repository resp. feeds (chocolatey and asset feeds)
• ProGet: Windows Integrated Authentication enabled
• IIS: Windows Authentication enabled (anything else disabled within IIS's Authentication)
• MSSQL database on dedicated server

System Details

• Operating System: Windows Server 2019
• Windows PowerShell version: 5.1.17763.3770
• Chocolatey CLI Version: 2.1.0
• Chocolatey Licensed Extension version: -
• Chocolatey License type: free
• Terminal/Emulator: Azure DevOps Pipelines / PowerShell

Installed Packages

PS C:\> choco list
Chocolatey v2.1.0
chocolatey 2.1.0
chocolatey-compatibility.extension 1.0.0
chocolatey-core.extension 1.4.0
3 packages installed.
PS C:\>

Output Log

2023-07-11T11:52:12.4895430Z ##[section]Starting: Install required packages
2023-07-11T11:52:12.5329003Z ==============================================================================
2023-07-11T11:52:12.5329414Z Task         : PowerShell on target machines
2023-07-11T11:52:12.5329559Z Description  : Execute PowerShell scripts on remote machines using PSSession and Invoke-Command for remoting
2023-07-11T11:52:12.5329803Z Version      : 3.200.0
2023-07-11T11:52:12.5329973Z Author       : Microsoft Corporation
2023-07-11T11:52:12.5330108Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/powershell-on-target-machines
2023-07-11T11:52:12.5330327Z ==============================================================================
2023-07-11T11:52:13.9848909Z PSSession created for Machines:'MYSERVER'
2023-07-11T11:52:14.2274374Z ================================================ MYSERVER ================================================
2023-07-11T11:52:19.2346606Z ================================================ MYSERVER ================================================
2023-07-11T11:52:19.2789067Z ##[command]& 'C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe' -NoLogo -NoProfile -NonInteractive 
2023-07-11T11:52:19.2790061Z -ExecutionPolicy Unrestricted -Command ". 
2023-07-11T11:52:19.2799805Z 'C:\Users\MYADOSERVICEACCOUNT\AppData\Local\Temp\b80d6b4f-061a-452a-ae6a-80b535fbd5ca.ps1'"
2023-07-11T11:52:19.2800067Z 
2023-07-11T11:52:19.2964928Z Chocolatey v2.1.0
2023-07-11T11:52:19.2965151Z 
2023-07-11T11:52:19.3005578Z Installing the following packages:
2023-07-11T11:52:19.3012098Z 
2023-07-11T11:52:19.3031150Z GoogleChrome
2023-07-11T11:52:19.3031281Z 
2023-07-11T11:52:19.3097259Z By installing, you accept licenses for the packages.
2023-07-11T11:52:19.3097712Z 
2023-07-11T11:52:19.3194165Z 
2023-07-11T11:52:19.3201381Z 
2023-07-11T11:52:19.3234245Z Progress: Downloading GoogleChrome 114.0.5735.199... 61%
2023-07-11T11:52:19.3261690Z 
2023-07-11T11:52:19.3596913Z Progress: Downloading GoogleChrome 114.0.5735.199... 100%
2023-07-11T11:52:19.3622394Z 
2023-07-11T11:52:19.3622648Z 
2023-07-11T11:52:19.3623423Z 
2023-07-11T11:52:19.3664728Z GoogleChrome v114.0.5735.199 (forced)
2023-07-11T11:52:19.3664987Z 
2023-07-11T11:52:19.3701258Z GoogleChrome package files install completed. Performing other installation steps.
2023-07-11T11:52:19.3701503Z 
2023-07-11T11:52:19.3715412Z WARNING: No registry key found based on  'Google Chrome'
2023-07-11T11:52:19.3718329Z 
2023-07-11T11:52:19.3735506Z WARNING: Unable to find the architecture of the installed Google Chrome application
2023-07-11T11:52:19.3736026Z 
2023-07-11T11:52:24.3705237Z ================================================ MYSERVER ================================================
2023-07-11T11:52:24.3717189Z Attempt to get headers for https://my.internalrepo.net/endpoints/Chrome/content/114.0.5735.199/googlechromestandaloneenterprise64.msi failed.
2023-07-11T11:52:24.3717787Z 
2023-07-11T11:52:24.3737233Z   The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://my.internalrepo.net/endpoints/Chrome/content/114.0.5735.199/googlechromestandaloneenterprise64.msi'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (401) Unauthorized."
2023-07-11T11:52:24.3737870Z 
2023-07-11T11:52:24.3760239Z Downloading googlechrome 64 bit
2023-07-11T11:52:24.3760802Z 
2023-07-11T11:52:24.3782558Z   from 'https://my.internalrepo.net/endpoints/Chrome/content/114.0.5735.199/googlechromestandaloneenterprise64.msi'
2023-07-11T11:52:24.3782923Z 
2023-07-11T11:52:24.3805171Z ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://my.internalrepo.net/endpoints/Chrome/content/114.0.5735.199/googlechromestandaloneenterprise64.msi'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (401) Unauthorized."
2023-07-11T11:52:24.3805752Z 
2023-07-11T11:52:24.3824783Z The install of GoogleChrome was NOT successful.
2023-07-11T11:52:24.3825258Z 
2023-07-11T11:52:24.3842615Z Error while running 'C:\ProgramData\chocolatey\lib\GoogleChrome\tools\chocolateyInstall.ps1'.
2023-07-11T11:52:24.3842976Z 
2023-07-11T11:52:24.3858666Z  See log for details.
2023-07-11T11:52:24.3859388Z 
2023-07-11T11:52:29.3862842Z ================================================ MYSERVER ================================================
2023-07-11T11:52:29.3873878Z 
2023-07-11T11:52:29.3875032Z 
2023-07-11T11:52:29.3892792Z Chocolatey installed 0/1 packages. 1 packages failed.
2023-07-11T11:52:29.3893284Z 
2023-07-11T11:52:29.3909898Z  See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
2023-07-11T11:52:29.3910840Z 
2023-07-11T11:52:29.3926190Z 
2023-07-11T11:52:29.3927323Z 
2023-07-11T11:52:29.3942206Z Failures
2023-07-11T11:52:29.3943268Z 
2023-07-11T11:52:29.3959192Z  - GoogleChrome (exited 404) - Error while running 'C:\ProgramData\chocolatey\lib\GoogleChrome\tools\chocolateyInstall.ps1'.
2023-07-11T11:52:29.3959709Z 
2023-07-11T11:52:29.3974903Z  See log for details.
2023-07-11T11:52:29.3975551Z 
2023-07-11T11:52:29.3991617Z 
2023-07-11T11:52:29.5665098Z ##[error]Non-Zero exit code: '1' for ComputerName: 'MYSERVER'
2023-07-11T11:52:29.7191321Z ##[error]Atleast one remote job failed. Consult logs for more details. ErrorCodes(s): 'RemoteDeployer_NonZeroExitCode'
2023-07-11T11:52:29.7517019Z ##[section]Finishing: Install required packages

Additional Context

Succesful installation of Google Chrome as a logged on user:

PS C:\> choco install GoogleChrome
Chocolatey v2.1.0
Installing the following packages:
GoogleChrome
By installing, you accept licenses for the packages.
Please provide credentials for: https://my.internalrepo.net/nuget/choco_test/
User name:
Password:
No password specified, this will probably error.
Progress: Downloading GoogleChrome 114.0.5735.199... 100%

GoogleChrome v114.0.5735.199
GoogleChrome package files install completed. Performing other installation steps.
File appears to be downloaded already. Verifying with package checksum to determine if it needs to be redownloaded.
Hashes match.
Hashes match.
Installing googlechrome...
googlechrome has been installed.
  GoogleChrome may be able to be automatically uninstalled.
 The install of GoogleChrome was successful.
  Software installed as 'MSI', install location is likely default.

Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
PS C:\> choco list
Chocolatey v2.1.0
chocolatey 2.1.0
chocolatey-compatibility.extension 1.0.0
chocolatey-core.extension 1.4.0
GoogleChrome 114.0.5735.199
4 packages installed.
PS C:\>

[corbob]

@norbertstoll I'm trying to replicate this setup, but without an AZDO pipeline, I'm not sure if I fully can. From what I'm reading in the issue here, I think you might actually be encountering a double-hop issue. If I'm reading the AZDO pipeline configuration and documentation properly, it sounds like it uses WinRM to connect to the server as the username/password you provide. It then runs the choco install command passing in the username and password provided. This allows Chocolatey to connect to the feed, but not to connect to the URL inside.

In my testing, I have the following setup:

System	Role
dc	Domain Controller
web1	ProGet Server
web2	Chocolatey Client

To test this theory, first I setup a test user on web2 that was not a domain user, and didn't match a domain user at all. I then attempted to perform the steps you mentioned. This worked as I expected, and I got to the point of downloading the msi and realized there would be no way for me to get it to work as it's not a domain user. So I started looking at how the AZDO worked, and I realized it was doing remoting. So from web1 I did Enter-PSSession web2.domain.local. From here I attempted with my internal build to install and search from web1. This didn't work at all, didn't even prompt me for username and password (just showed it as though it tried, but didn't prompt... remoting is fun for interactive things 😅 ). I downgraded Chocolatey to 1.4.0 and tried it to. This is where I found that it also didn't work. But it did work directly on web2. If I ran choco install googlechrome -u vagrant -p vagrant -s http://web1.domain.local:8624/nuget/my-choco/, I got the same thing you did: it downloads the nupkg, but then can't get the msi:

(Note: the screenshot shows the IP address, I did try updating it to use the FQDN afterwards and still get the same behaviour)

Based on this testing, I believe this is a feature request (and there may in fact already be some existing for adding authentication abilities to the chocolatey web handling cmdlets). So I will label this as such.

[TheCakeIsNaOH]

Is this potentially the same thing as #1021 ?

[norbertstoll]

Hi @corbob,

as this is my second request within a short time thanks again for a quick response and your analysis!

We've been thinking of a double hop issue, too and also I tried to get around it by using CredSSP (though it has its security downside) in ADO. Sadly this isn't an option and not allowed because of internal guidelines.

@TheCakeIsNaOH I've been looking at this function, too, thinking that might be the right place to implement credential support.

Regards,
Norbert

[gep13]

@norbertstoll are you able to confirm whether this is something that was working for you in 1.4.0 of Chocolatey and something which has stopped working now that you have moved to 2.x, or whether this is actually a new feature/improvement that you are looking for.

[norbertstoll]

@gep13 This hasn't been working in any version starting from 1.3.1.

I'm not quite sure if this a new feature but some kind of circumstance when using ProGet assets combined with Chocolatey.

[gep13]

@norbertstoll said...
This hasn't been working in any version starting from 1.3.1.

Just to drill down a little further on this, to make sure we are talking about the same thing...

Are you saying that this used to work in a version of Chocolatey CLI before 1.3.1? Or are you saying that since you started using Chocolatey CLI, which was with 1.3.1, this hasn't worked?

[norbertstoll]

@norbertstoll said...
This hasn't been working in any version starting from 1.3.1.

Just to drill down a little further on this, to make sure we are talking about the same thing...

Are you saying that this used to work in a version of Chocolatey CLI before 1.3.1? Or are you saying that since you started using Chocolatey CLI, which was with 1.3.1, this hasn't worked?

Apologies...

We started using Chocolatey with version 1.3.1. So I can't say anything about versions earlier than that one.

[gep13]

@norbertstoll said...
Apologies...

There is no need to apologise here. I just wanted to make sure that we have all the facts, in order to know how to move forward with this issue.

[corbob]

I've tested this scenario with Chocolatey CLI versions 1.0.0, 1.1.0, 1.2.0, and 1.3.0. None of them are able to download the msi from a windows authenticated web server, they also do not download the nupkg without being provided the username/password along with the source. At this point, I'm going to close this issue as a duplicate of #1021 as TheCakeIsNaOH suggested. If we determine a scenario where this isn't a duplicate, and isn't related to the double hop issue, we can always reopen this issue.

[corbob]

Duplicate of #1021