VssUnauthorizedException: VS30063: You are not authorized to access using Windows Impersonation

[cvbc2010]

I am trying to connect to Azure DevOps Server 2019 using Windows Identity with Imperosnation.

Currently we have Azure devops server 2019 installed on Server A with https://abcd.domain.com
IIS on Server B
App running under AzureDevopsHelper website with No Managed Code as App Pool CLR an running under service account.

Application built under .NET 5 using windows authentication

we have an Extension method which returns Impersonated Clients to perform external calls

public static async Task<T> GetClientWithImpersonation<T>(this ControllerBase controllerBase, string connectionURL) where T : VssHttpClientBase
        {
            var user = (WindowsIdentity)controllerBase.Request.HttpContext.User.Identity;
            var client = await WindowsIdentity.RunImpersonatedAsync(user.AccessToken, async () =>
             {
                 VssClientCredentials creds = new VssClientCredentials(new Microsoft.VisualStudio.Services.Common.WindowsCredential(true),
                                              new VssFederatedCredential(true));
                                                  //CredentialPromptType.PromptIfNeeded);
                 var connection = new VssConnection(new Uri(connectionURL), creds);
                 return await connection.GetClientAsync<T>();
             });
            return client;
        }

and this is resulting in
Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access https://abcd.domain.com

On visual Studio I am able to access and fetch or create work items.
After deploying to IIS I am getting this error on remote machines.

On local accessing to localhost url is working from the same machine.
but while accessing the url from remote machine is failing with unauthorized issue.

Can anyone looking this issue.