EchoBot throws exceptions during the initialization and terminates the call on joining.

[genemgh]

We try to run the EchoBot example (GitHub: microsoftgraph/microsoft-graph-comms-samples/Samples/PublicSamples/EchoBot/).
We have a Microsoft 365 Developer sandbox as a part of Visual Studio subscription, and a tenant in the sandbox.
We use Postman to hit the bot.

Versioning:
- We use Windows 11, Visual Studio 2022 Professional, C#, and .NET version 6.0.
- README in the EchoBot example recommends .NET 4.7.1, but it looks like it's just wrong.
(The project file for the EchoBot example contains net6.0).
- We downloaded .NET 4.7.1 SDK from the NET web site, installed it, switched to .NET Framework 4.7.1,
did the NuGet clean and restore. Does not compile.

Ngrok:
- We have Ngrok (free version) running and providing two endpoints locally, one for HTTPS, and one for TCP.

Certificates:
- We created a self-signed certificate for the TCP domain and installed it in the local certificate store for
both User / Personal and Local Machine / Personal.
- The certificate is being picked up by the bot (without the certificate installed, the bot does not start at all).

Graph Application and Azure Bot registrations:
- The application is registered in MS Entra for our sandbox tenant, all the permissions are granted and consented.
- The Azure Bot is also registered in the Azure portal.
- We also added the Microsoft Teams Channel to the bot channels in the bot registration, enabled calling in the Calling tab,
and added the correct domain name to Webhook (for calling) field: https://.ngrok-free.app/api/calling

Local environment:
- The application / client ID and the application / client secret are added to the .env file in the bot project.
- The Ngrok endpoints and ports are added to the .env file.
- The certificate thumbprint is added to the .env file.

When we run the bot from Visual Studio in the debug mode, during the bot initialization, the following 12 different types of exceptions are logged:
1. onecore\net\netprofiles\service\src\nsp\dll\namespaceserviceprovider.cpp(613)\nlansp_c.dll!00007FFF45F1F6BD:
(caller: 00007FFF619FACF6) LogHr(1) tid(59e8) 8007277C No such service is known. The service cannot be found in the specified name space. (3)
2. Microsoft C++ exception: std::system_error at memory location 0x00000050203BA850.
3. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: WinRT originate error - 0x80040155 :
'Failed to find proxy registration for IID: {79EAC9E4-BAF9-11CE-8C82-00AA004BA90B}.'.
4. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x80040155: Interface not registered. (2)
5. MediaPerf is not registered: no key found at SYSTEM\CurrentControlSet\Services\MediaPerf\Performance
6. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x000006D9: There are no more endpoints available from the endpoint mapper.
7. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x000006BA: The RPC server is unavailable. (3)
8. 'System.InvalidOperationException' in System.Diagnostics.PerformanceCounter.dll: Category does not exist.
9. 'System.IO.IOException' in System.Net.Sockets.dll, System.Net.Security.dll, System.Private.CoreLib.dll
10. 'System.InvalidOperationException' in Unity.Container.dll.
No public constructor is available for type Microsoft.Extensions.Options.IPostConfigureOptions`1[Microsoft.Extensions.Logging.LoggerFilterOptions]
(and for other interfaces, about 60 of them).
Inner Exception: InvalidRegistrationException: Exception of type 'Unity.Exceptions.InvalidRegistrationException' was thrown.
11. System.IO.IOException in System.Net.Security.dll, System.Private.CoreLib.dll. Received an unexpected EOF or 0 bytes from the transport stream. (5)
12. System.IO.IOException: 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..'

But despite of all the exceptions, the bot starts:
Microsoft.Hosting.Lifetime: Information: Now listening on: http://[::]:9442
Microsoft.Extensions.Hosting.Internal.Host: Debug: Hosting started.
And the GET health check request from Postman hits Ngrok, hits the bot, and returns 200 OK to the Postman.
During the bot initialization, the BotMediaStream is created, the audio socket is initialized,
the bot sends status active for media, and the media player is created.

Testing;
- We start the bot in Visual Studio (as administrator) locally.
- We start the Teams client locally, in our sandbox tenant.
- We start a private scheduled meeting in Teams (Calendar -> Schedule -> Join Meeting).
- We use the meeting JoinURL for the bot to join the meeting call from Postman with a JSON POST request
{ "joinURL": <JOIN_URL> }.
- It hits Ngrok, hits the bot, the bot returns 200 OK and reports that it joined the call:
CallController: Joined the call, call ID: 1e005980-8e2c-4831..., URL: https://teams.microsoft.com/l/meetup-join/19%3ameeting_NjZh...
- The bot also receives two notifications via api/calling/notification endpoint after the join call request and replies with 202 Accepted:
PlatformCallController: Notification result: StatusCode: 202, ReasonPhrase: 'Accepted', Version: 1.1, Content: System.Net.Http.EmptyContent.
- We monitor the wire in WireShark for the media TCP ports.

The problem:
It tries to establish the call:
17:01:28:954 CallHandler: Call status updated to Establishing.
And then immediately terminates it:
17:01:29:954 CallHandler: Call status updated to Terminated - Server Internal Error. DiagCode: 500#1203002.@.
And then throws the following exceptions:
Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Private.CoreLib.dll
A task was canceled.
Exception thrown: 'System.IO.IOException' in System.Net.Sockets.dll
Unable to read data from the transport connection:
The I/O operation has been aborted because of either a thread exit or an application request.
THE BOT DOES NOT APPEAR IN THE MEETING, and NO MEDIA IS SENT TO THE BOT.

Questions:
1. All the exceptions are triggered deeply in the native libraries. Can it mean that there are some versioning issues?
2. What can cause the exceptions in the bot during the initialization? Are those exceptions significant?
3. Can those exceptions cause that the bot is not joining the call and also the media stream issues?
4. Is it OK to use a self-signed certificate?
5. What would you recommend to further debug the issue?

Thank you!

[1fabi0]

I'll start with question 4 it is not ok to use a self signed certificate. Regarding the other questions probably they resolve themself if you use a certificate with a certificate to the root chain. Maybe as tip with ngrok try to get a connection to 0.tcp.-endpoint of ngrok by restarting on the other endpoints there sometimes seem to be problems with the firewall settings from ngrok.

[genemgh]

Thank you for the advice! We'll try a root chain certificate.

Our Ngrok is configured to generate the following two endpoints:
TCP: tcp://N.tcp.ngrok.io:PORT -> localhost:8445, N = 0,2,4.6,8, PORT is a random int usually between 10000 and 30000.
HTTP: https://DOMAIN.ngrok-free.app -> http://localhost:9442, DOMAIN is a random string like d7b9-63-209-137-19

We generate the certificates for domain N.tcp.ngrok.io, the TCP connection domain, no wildcard *. Is that correct?

=========================================================

Another possible reason of the issue:
Our application registration and the Azure bot registration technically belong to two different tenants.

We have a sandbox available as a part of our Microsoft 365 Developer Subscription for Visual Studio Professional.
So, the application is registered in the sandbox tenant (and it's multi-tenant).
The Azure bot registration is a part of our organization tenant
(but it's based on the application registration, and it's also multi-tenant).
That's because the real Azure subscription is under our organization tenant as a part of 365 Subscription.

We run the Teams client from our sandbox tenant.
During the call joining process, the bot does not ask for any admin consent.
Can it (the registration in two different tenants) be the reason of the call termination?

=========================================================

And would anybody please confirm that it's supposed to work for .NET 6.0 :-)

Thank you!

[1fabi0]

I think your tenant situation is not a problem as long as you gave admin permission for the teams operating tenant.
Regarding the certificate for the tcp ngrok address, as it might be rather complicated to get a valid certificate for the ngrok domain you can use a cname on a custom domain that then points to e.g. 0.tcp.ngrok.io and get the certificate for your custom domain(this implies using the custom domain instead of the ngrok domain in the bot).

Yes .Net 6 works, but hosting with IIS doesn't

[genemgh]

Thank you for your help! We'll try it.

A deployment question. Does it have to be deployed in Azure VMSS? Is it possible to deploy it on-premises on a Windows Server in Kubernetes?

[1fabi0]

Yes, it is possible to deploy it on-premises in a Kubernetes cluster, but keep in mind it is not officially supported by Microsoft. It is recommended to run the bot as close as possible to the Azure datacenter location where the teams meeting is hosted to reduce packet loss and roundtrip times.

However regarding Kubernetes, you can have a look at the Azure Kubernetes Service Sample that I also worked on. The k8s charts and deployment steps should be compatible/portable.

[genemgh]

In our case, we'll have to redirect the media stream to our on-premises platform for processing and analysis anyway, so the distance is not very important. Anyway, thank you for the advice, we'll consider it.

[genemgh]

BTW, is it possible to use HTTPS for the media stream instead of TCP? Web Sockets, for example?
Thank you!

[1fabi0]

No, it seems like the media platform is somewhere based on WCF nowadays core WCF so you can not replace the TCP endpoint with websockets, hopefully Microsoft will refactor the media platform one day and pay some technical debt and remove the dependency to WCF.

[genemgh]

Finally, we have our root chain certificate issued and the DNS cname records added.
It works now (in the non-debug mode).
Thank you very much for your help!

In the debug mode it triggers the same exceptions.
Is it supposed to work in the debug mode?
It would provide a lot of useful information...

Does the audio stream provide the participant diarization?

Thank you!

[genemgh]

Our real-time media bot throws the following exception. Could anybody help us to explain what we do wrong?
The MS servers sometimes forcibly closes connection. (The bot is able to join a Teams meeting and it receives the audio.)
Thank you!

[24-12-04 17:02:56.5681] fail: PTBotService.Bot.MediaLogger[0]
[AvMP][AppId:63b1acf6-88cb-4b73-a92a-e9eb26872450] TL_ERROR(TF_COMPONENT)
30684.122::12/04/2024-17:02:56.567.00000009 (MPAZAPPHOST,LogError:AzAppMPHostLogger.cs(39))

[EventManager] Sending of notification message failed to Uri :
https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback
for HttpMethod: POST

with exception : System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.)
(The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.) --->
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. --->
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.ContextForwardingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationTokenoriginalCancellationToken)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.MtlsHttpRequestSender.InternalSendAsync[T](T message, Uri requestUri, HttpMethod httpMethod, Uri implicitCallbackLink, String contentType, String authToken)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.MtlsHttpRequestSender.SendAsync[T](T message, Uri requestUri, String authToken, HttpMethod httpMethod)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.EventManager.SendNotificationAsync[T](T notificationMessage, Uri requestUri, HttpMethod httpMethod, Boolean sendNotificationUsingCompositeTransport, Uri implicitCallbackLink, String authToken) --->
(Inner Exception #1) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. --->
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)<--- --->
(Inner Exception #2) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. --->
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)<---

[adityaramgopal]

@genemgh a few questions:

1. What nuget version of Microsoft.Skype.Bots.Media are you running?
2. What target framework (is it net framework or net6.0?)
3. What version of TLS / SSL are you configured with on your machine?

[genemgh]

Hi @adityaramgopal !
Answering you questions:

1. Microsoft.Skype.Bots.Media version: 1.27.0.2.alpha
2. It’s Net 6.0.
3. TLS 1.2 and TLS 1.3 are enabled on my machine (Windows 11)
   Thank you!

[adityaramgopal]

@genemgh do you have any firewall rules on outbound traffic?

[genemgh]

@adityaramgopal
In fact, I do. Our IT team manages them.
Which port should be open to communicate with the MS media server?
The media we need from a Teams meeting comes through to our local bot, so it looks like it does not happen every time...
Firewall would block all the media. Is that correct?

[adityaramgopal]

@genemgh can you temporarily disable all outbound firewall rules and see if the exception is thrown? Just to clarify we're asking about disabling outbound firewall rules (not inbound)

[genemgh]

I'm not supposed to do it. And it's a security risk.

[adityaramgopal]

without eliminating that as the reason it's going to be hard to figure out the root cause.

[genemgh]

It's an HTTPS request which goes through port 443, which is always open. So, I don't think it's firewall.
And usually there are better ways to debug than disabling your firewall :-) WireShark?
Thank you anyway.

[adityaramgopal]

Are you saying that you allow all outbound traffic to port 443 and don't constrain them to specific IP addresses?

[adityaramgopal]

The media we need from a Teams meeting comes through to our local bot, so it looks like it does not happen every time...

Also, can you clarify what you mean by this comment. This issue reproduces sometimes but not always? Or does it reproduce in a specific machine (say local) but not in another machine?

[genemgh]

What I mean is that I'm not sure how often that http request fails (on one single machine). And I'm not sure how this request is related to the media stream. But we are getting the media stream, so it means that the request is either not critical or does not fail every time. Do you know what that request does and how it's related to the media stream?

[genemgh]

To my understanding, if an outbound firewall rule blocks an outgoing request, the request won't be sent.
In our case, it looks like it's reaching the remote http server, and the remote host closes the connection.
So, it does not look like a firewall issue.

[adityaramgopal]

@genemgh can you share a sample call id of a successful call vs a failing call? Might help us debug further. Maybe a wireshark trace of a failing call might also help

[genemgh]

In order to reproduce it, we tried to hit the same URL directly from Curl in the verbose mode (both from Windows and WLS):
Windows> curl -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

• Host 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701 was resolved.
• IPv6: (none)
• IPv4: 172.171.152.231
• Trying 172.171.152.231:10701...
• Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (172.171.152.231) port 10701
• schannel: disabled automatic use of client certificate
• ALPN: curl offers http/1.1
• Recv failure: Connection was reset
• schannel: failed to receive handshake, SSL/TLS connection failed
• closing connection #0
  curl: (35) Recv failure: Connection was reset

WSL Ubuntu:~$ curl -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

• Trying 172.171.152.231:10701...
• Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (172.171.152.231) port 10701 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701
• Closing connection 0
• TLSv1.0 (OUT), TLS header, Unknown (21):
• TLSv1.3 (OUT), TLS alert, decode error (562):
  curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701

WSL Ubuntu:~$ curl -k -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

• Trying 172.171.152.231:10701...
• Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (172.171.152.231) port 10701 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701
• Closing connection 0
• TLSv1.0 (OUT), TLS header, Unknown (21):
• TLSv1.3 (OUT), TLS alert, decode error (562):
  curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701

WSL Ubuntu:~$ telnet 172.171.152.231 10701

• Trying 172.171.152.231...
• Connected to 172.171.152.231.
• Escape character is '^]'.
• Connection closed by foreign host.