Can’t handle more than 1Gbps traffic on Edge

[germanbuchmuller]

Hi, we’ve got an Origin-Edge architecture using OriginMapStore. Our edge server has a dedicated 10Gbps network, however, when streaming @3500kbps, no more than 300 viewers can watch our stream.
Executing nload shows that we are at 1,1 Gbps network usage. We have implemented Linux Kernel settings as explained in OMEs troubleshooting page. We have also tested the server’s network speed connection with speedtest, and obtained +3Gbps upload speeds.
We are using WebRTC for streaming. Here is our Edge Conf:
`

BigSalaEdgeMediaEngine edge * false

<!-- 
To get the public IP address(mapped address of stun) of the local server. 
This is useful when OME cannot obtain a public IP from an interface, such as AWS or docker environment. 
If this is successful, you can use ${PublicIP} in your settings.
-->
<StunServer>stun.ovenmediaengine.com:13478</StunServer>

<Modules>
    <!-- 
	Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.			
	-->
    <HTTP2>
        <Enable>true</Enable>
    </HTTP2>

    <LLHLS>
        <Enable>false</Enable>
    </LLHLS>

    <!-- P2P works only in WebRTC and is experiment feature -->
    <P2P>
        <!-- disabled by default -->
        <Enable>false</Enable>
        <MaxClientPeersPerHostPeer>1</MaxClientPeersPerHostPeer>
    </P2P>
</Modules>

<!-- Settings for the ports to bind -->
<Bind>
    <!-- Enable this configuration if you want to use API Server -->

    <Managers>
        <API>
            <Port>${env:OME_API_PORT:8085}</Port>
            <TLSPort>${env:OME_API_TLS_PORT:8086}</TLSPort>
            <WorkerCount>1</WorkerCount>
        </API>
    </Managers>



    <Providers>
        <!-- Pull providers -->
        <OVT>
            <WorkerCount>2</WorkerCount>
        </OVT>
    </Providers>

    <Publishers>

        <!--
        <LLHLS>
            <Port>${env:OME_LLHLS_STREAM_PORT:3335}</Port>
            <TLSPort>${env:OME_LLHLS_STREAM_TLS_PORT:3336}</TLSPort>
            <WorkerCount>1</WorkerCount>
        </LLHLS>

        -->

        <WebRTC>
            <Signalling>
                <Port>${env:OME_SIGNALLING_PORT:3333}</Port>
                <TLSPort>${env:OME_SIGNALLING_TLS_PORT:3334}</TLSPort>
                <WorkerCount>2</WorkerCount>
            </Signalling>
            <IceCandidates>
                <IceCandidate>${env:OME_HOST_IP:*}:${env:OME_WEBRTC_CANDIDATE_PORT:10005-10009/udp}</IceCandidate>
                <TcpRelay>${env:OME_HOST_IP:*}:${env:OME_WEBRTC_TCP_RELAY_PORT:3478}</TcpRelay>
                <TcpForce>true</TcpForce>
                <TcpRelayWorkerCount>2</TcpRelayWorkerCount>
            </IceCandidates>
        </WebRTC>

    </Publishers>
</Bind>

<!-- P2P works only in WebRTC -->
<!--
<P2P>
    <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
</P2P>
-->

<Managers>
    <Host>
        <Names>
            <Name>*</Name>
        </Names>
        <!--
        <TLS>
            <CertPath>/certs/signed.crt</CertPath>
            <KeyPath>/certs/domain.key</KeyPath>
        </TLS>
         -->
    </Host>
    <API>
        <AccessToken>root:PASS</AccessToken>
        <CrossDomains>
            <Url>*</Url>
        </CrossDomains>
    </API>
</Managers>

<VirtualHosts>
    <!-- You can use wildcard like this to include multiple XMLs -->
    <VirtualHost include="VHost*.xml" />
    <VirtualHost>
        <Name>default</Name>
        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value is output to the events log, so you can use it to aggregate statistics. -->
        <Distribution>ovenmediaengine.com</Distribution>

        <!-- Settings for multi ip/domain and TLS -->
        <Host>
            <Names>
                <!-- Host names
					<Name>stream1.airensoft.com</Name>
					<Name>stream2.airensoft.com</Name>
					<Name>*.sub.airensoft.com</Name>
					<Name>192.168.0.1</Name>
				-->
                <Name>*</Name>
            </Names>
            <TLS>
                <CertPath>/certs/signed.crt</CertPath>
                <KeyPath>/certs/domain.key</KeyPath>
            </TLS>
        </Host>

        <!--
        <SignedPolicy>
            <PolicyQueryKeyName>policy</PolicyQueryKeyName>
            <SignatureQueryKeyName>signature</SignatureQueryKeyName>
            <SecretKey>OURPASS</SecretKey>

            <Enables>
                <Publishers>llhls</Publishers>
            </Enables>
        </SignedPolicy>
        -->

        <AdmissionWebhooks>
            <ControlServerUrl>https://OURPAGE.com/stream/admission</ControlServerUrl>
            <SecretKey>SERVERPASS$</SecretKey>
            <Timeout>20000</Timeout>
            <Enables>
                <Publishers>webrtc</Publishers>
            </Enables>
        </AdmissionWebhooks>


        <!-- Settings for ProxyPass (It can specify origin for each path) -->

        <Origins>
            <Properties>
                <NoInputFailoverTimeout>10000</NoInputFailoverTimeout>
                <UnusedStreamDeletionTimeout>120000</UnusedStreamDeletionTimeout>
            </Properties>
        </Origins>


        <!--
		When Edge tries to pull a stream, it looks first in <Origins> in local, and in OriginMapStore. 
		Therefore, it is recommended to disable Origins when using OriginMapStore.
        -->
        <OriginMapStore>
            <RedisServer>
                <Host>REDISSERVER:6379</Host>
                <Auth>REDISPASS$</Auth>
            </RedisServer>
        </OriginMapStore>


        <!-- Default CORS Settings -->
        <CrossDomains>
            <Url>https://OURPAGE.com</Url>
            <Url>https://OURPAGE.com</Url>
            <Url>https://OURPAGE.com</Url>
            <Url>https://OURPAGE.com</Url>
        </CrossDomains>

        <Applications>
            <Application>
                <Name>livestream</Name>
                <Type>live</Type>
                <OutputProfiles>
                    <OutputProfile>
                        <Name>passthrough</Name>
                        <OutputStreamName>${OriginStreamName}</OutputStreamName>
                        <Encodes>
                            <Video>
                                <Bypass>true</Bypass>
                            </Video>
                            <Audio>
                                <Bypass>true</Bypass>
                            </Audio>
                        </Encodes>
                    </OutputProfile>
                </OutputProfiles>
                <Providers>
                    <OVT />
                </Providers>
                <Publishers>
                    <AppWorkerCount>{env:OME_APPWORKERCOUNT:2}</AppWorkerCount>
                    <StreamWorkerCount>{env:OME_STREAMWORKERCOUNT:4}</StreamWorkerCount>

                    <WebRTC>
                        <Timeout>30000</Timeout>
                        <Rtx>false</Rtx>
                        <Ulpfec>false</Ulpfec>
                        <JitterBuffer>false</JitterBuffer>
                    </WebRTC>

                    <!--
                    <LLHLS>
                        <OriginMode>false</OriginMode>
                        <ChunkDuration>3</ChunkDuration>
                        <SegmentDuration>6</SegmentDuration>
                        <SegmentCount>5</SegmentCount>
                        <DVR>
                            <Enable>false</Enable>
                        </DVR>
                        <DRM>
                            <Enable>false</Enable>
                        </DRM>
                        <CrossDomains>
                            <Url>https://OURPAGE.com</Url>
                            <Url>https://OURPAGE.com</Url>
                            <Url>https://OURPAGE.com</Url>
                            <Url>https://OURPAGE.com</Url>
                        </CrossDomains>
                    </LLHLS>
                    -->

                </Publishers>
            </Application>
        </Applications>
    </VirtualHost>
</VirtualHosts>

`

Thanks for any advice, and please let us know if more information is required.

[bchah]

Hi, I've recently done some high-load testing with OME. There are some easy things to check here:

1. StreamWorkerCount - when scaling out a stream to many viewers, the StreamWorker process will consume a lot of CPU. If you have this limited to 4 threads you should increase that number. The higher your overall throughput, the more CPU will be consumed. Use 'top' to get the PID of OvenMediaEngine, then run 'top -H -p {OME PID}' to see the individual thread usage. The StreamWorker threads should never sit above 75% max in my experience, to accommodate spikes in traffic. Given the bitrate you indicated, my gut feeling is that this is the root cause of your issue.

2. Check your network, for real. A speed test is a simple use case, but if you are serving hundreds or thousands of connections through a physical layer, it needs to be very well-tuned. I discovered some performance gaps in my physical network infrastructure by using OvenRtcTester. Running it on a docker network between two containers I was able to get much better performance than through the physical switch.

3. Try some more networking tweaks on the host:

• Some users have success when they disable TCP segmentation offload: "ethtool -K {interface} tx off sg off tso off"
• Try changing the congestion control method to BBR: "net.ipv4.tcp_congestion_control=bbr"

Good luck and keep us posted!

[germanbuchmuller]

Hi, thanks for your reply! Our edge server has an Intel Xeon E-2386G CPU @ 3.50Ghz, with 6 physical, 12 logical cores, and 64GB RAM. What threads configuration do you recommend for having 30+ live streams, but only a few of them with high traffic?

According to your suggestion, we set them like this:
In the Section, OVT WorkerCount is set to 1.
In Publishing, Signalling WorkerCount is set to 1 and IceCandidates WorkerCount is also set to 1.
Then, AppWorkerCount is set to 2, and StreamWorkerCount is set to 6.

Are this threads settings kind of correct?

If we stream 720p @3000kbps, and the Edge's network speed is 10Gpbs, then we should expect at least 3000 concurrent viewers right? Or is the calculation not so linear? Configuration should be the same if we expect all traffic from one live stream, than if traffic is evenly distributed accross all streams? (Eg: 3000 viewers in one stream vs 10 viewers in each of the 30 livestreams)

We've also changed congestion control method to BBR, and disabled TCP segmentation as you mentioned. We're going to be testing with real traffic in a couple of hours. We will let you know how it goes!

Thanks again for your fast reply!

[bchah]

I think your CPU is the bottleneck. 6C/12T is not much to work with at that scale. To serve a similar throughput, I was pretty much lighting up a Threadripper 7965WX with 24C/48T.

I don't work a lot with Origin/Edge but I believe the AppWorkerCount is more relevant to your Origin server in this case, since it is the one actually receiving the 30 incoming streams. On the Edge server, do that 'top -H -p {OME PID}' check to see that OVTWorker isn't overwhelming the single thread or increase that number if so. Also for StreamWorker which I still believe could be the root issue. It will also reveal if any other process are maxing out threads. You need to keep the processes balanced since they will begin to split CPU time if there are not enough threads available.

As far as your bandwidth calculation goes, I usually reserve a 30% overhead, i.e. treat your 10Gb connection as if it were a 7Gb connection. This accounts for spikes in bitrate, traffic, and the extra bits that repackaging the RTP stream adds (you'll notice your incoming and outgoing bitrate are slightly different). But I don't think this is your immediate bottleneck.

Related to this, I am organizing a test on a ML research system with a 192C/384T CPU 🤓 Will report back with the results once I can get the time reserved on the server.

[germanbuchmuller]

Related to this, I am organizing a test on a ML research system with a 192C/384T CPU 🤓 Will report back with the results once I can get the time reserved on the server.

It sounds great! Sure we want to see those reports as soon as they are published.

Regarding our server, we are on a tight budget right now. Maybe in a month or so we can upgrade it to a more powerful one. Does LLHLS require less CPU power for streaming at same bitrate, with same number of viewers, compared to WebRTC?

[bchah]

Yes, LLHLS uses less CPU versus WebRTC for playback since it is just focusing on serving out the chunk files and managing the sessions. I haven't done tests at scale with LLHLS but anecdotally others have observed it uses noticeably less resources.

[naanlizard]

You'll really want to put OME's llhls endpoint behind an nginx cache or something, cpu usage directly connecting to OME is insanely high.

Latency will just be a bit higher on llhls than webrtc, ˜0s to ˜2-4s

[bchah]

Trust the wisdom of the lizard! They have done a lot more production testing with LLHLS than I have 👍

[germanbuchmuller]

We are running OME through Docker. Is any other configuration needed to enable multithreading?
This is our docker-compose.yml:
edge:
container_name: edge
image: IMAGE
ports:
- "3333:3333"
- "3334:3334"
- "443:3334"
- "3478:3478"
- "10000-10005:10000-10005/udp"
- "3336:3336"
- "8085:8085" #API
environment:
- TYPE=edge
- OME_HOST_IP=HOST
- OME_WEBRTC_SIGNALLING_PORT=3333
- OME_WEBRTC_SIGNALLING_TLS_PORT=3334
- OME_WEBRTC_TCP_RELAY_PORT=3478
- OME_WEBRTC_CANDIDATE_PORT=10000-10005/udp
- OME_LLHLS_STREAM_TLS_PORT=3336
- OME_API_PORT=8085
- OME_APPWORKERCOUNT=1
- OME_STREAMWORKERCOUNT=8
volumes:
- /data/ssl_certs/DOMAIN/production:/certs
restart: always

Also, how can we get OME's PID from Docker's container see each threads usage?

[bchah]

More anecdotal advice here but I believe there is a network performance impact when you use Docker networking (forwarding particular ports) rather than using passthrough host networking ("--network host"). There are some security considerations to this but it's worth a test.

Also, how can we get OME's PID from Docker's container see each threads usage?

You need to enter the Docker container first:

docker exec -it OMECONTAINERNAME /bin/bash

And here's a one liner to get the PID then check it (which assumes OME is running under a 'www-data' user):

p=$(ps -u www-data | grep OvenMediaEngine | awk '{print $1}'); top -Hp $p

[germanbuchmuller]

@bchah thanks for all this help. I replaced Docker networking with Hosts networking, and executed top-H to see threads usage. When there is one total viewer, with Edge Conf as I showed above, this is threads usage:

I will be posting thread usage with real traffic in a couple of hours when we test it.
Thanks again for your help!

[bchah]

Cool, that's the readout! If you are using LLHLS you may need to adjust the LLHLS WorkerCount instead. But this readout will tell you which processes are bottlenecking.

[germanbuchmuller]

After adding some few viewers, I only see AW-WebRTC0 thread usage increasing. Even though StreamWorkerCount is set to 6, there is no StreamWorker thread appearing as https://airensoft.gitbook.io/ovenmediaengine/performance-tuning#monitoring-the-usage-of-threads shows.

[bchah]

Here's my readout for the same command:

AW-WebRTC is the AppWorker for WebRTC, i.e. the incoming stream(s). How many viewers do you have connected?

EDIT: To clarify, Dech264 is the thumbnail encoder.

[germanbuchmuller]

I found out some weird behaviour when AppWorkerCount, StreamWorkerCount, and WebRTC's Publisher Signalling and TCPRelay WorkerCounts are set through environment variables. StreamWorker threads would not be created, as you can see in my post above.
I commented out those two lines, and only set AppWorkerCount and StreamWorkerCount.

Now we have these threads usage for 20 WebRTC streams @4000kbps:

[bchah]

Interesting! I use another means of filling in the Server XML on system boot so haven't hit that issue before.

Good luck on the big test 👍

[germanbuchmuller]

Hello again! After testing with real traffic, we found these limits with a single Edge Server as I mentioned above:

• @4000kbps: 1400 viewers (no delay or packet loss), 1800 viewers (with 3-6 sec delay due to cpu usage)
• @2000kbps: 2200 viewers (no delay or packet loss), 2800 viewers (with 3-6 sec delay due to cpu usage)

With 1 Publishers AppWorkerCount, 12 Publishers StreamWorkerCount, 6 WebRTC Signalling WorkerCount, and 6 WebRTC TcpRelayWorkerCount, we found the most appropiate configuration. We noticed that bitrate has a great impact not only in network usage, but in CPU usage too.
When TcpRelayWorkerCount is too low (1-4), new connections are rejected even though CPU usage is very low.

We will be scaling our edge servers as our budget increases. Thanks @bchah for all your help!

[naanlizard]

Do you control the encoding of the streams? As in the original source?

4000kbps is quite a lot depending on what you're doing. If you need help with encoder settings I might be able to optimize it some

[bchah]

Those are great numbers for the amount of data you're pulling. Good tuning, especially for a 6-core CPU!

My use case is similar where users generally stream at a minimum of 4Mbps. They use 10-bit 4:2:2, HDR, all the other fun stuff too. I'm orchestrating a load test soon on a 192-core machine, that'll be fun 😵‍💫

[germanbuchmuller]

Do you control the encoding of the streams? As in the original source?

4000kbps is quite a lot depending on what you're doing. If you need help with encoder settings I might be able to optimize it some

Our Origin Server is only doing Audio Re-Encoding (due to WebRTC Opus). Video is bypassed. We use OBS to encode.
We are streaming 1080p @4000kbps. We found out that the perfect resolution-bitrate relation really depends on video content.
When there is a lot of movement, such as in Sports Games, we need to set bitrate quite high to achieve some good quality. However, with other kind of content such as Cartoons, we noticed that a low bitrate (such as 1080p@2000kbs) is enough.

Here are our OBS settings:

[germanbuchmuller]

Those are great numbers for the amount of data you're pulling. Good tuning, especially for a 6-core CPU!

My use case is similar where users generally stream at a minimum of 4Mbps. They use 10-bit 4:2:2, HDR, all the other fun stuff too. I'm orchestrating a load test soon on a 192-core machine, that'll be fun 😵‍💫

We figured out 4Mbps is quite the minimum for streaming @1080p, unless the video has many static segments such as Cartoons where bitrate can be lowered without compromising quality. Are you doing transcoding in your server?

[bchah]

Are you doing transcoding in your server?

Generally not. It works OK if you have basic inputs but gets very complex to handle things like different frame rates, colour spaces, etc... If you are implementing transcodes you'll almost certainly want to use the TranscodeWebook feature to decide how and when things are transcoded - otherwise it sends all streams to all defined encoders.

On the topic of video quality, try a hardware encoder! OBS can utilize the encoders found in Apple computers and NVIDIA GPUs, or you can go for something like a Magewell Ultra Encode. These can give you better images at the same bitrate versus a software encoder.

[naanlizard]

On the topic of video quality, try a hardware encoder! OBS can utilize the encoders found in Apple computers and NVIDIA GPUs, or you can go for something like a Magewell Ultra Encode. These can give you better images at the same bitrate versus a software encoder.

This is the exact opposite of what we've found. Nvenc and amd's encoder give worse quality at the same bitrate. Saves CPU though of course

I'll look at your OBS settings later I'm out now.

[bchah]

This is the exact opposite of what we've found.

This is computers, everything is exact and nothing is exact 😝 GPU and software encoders both perform different under different conditions. I've found that GPUs handle motion better, and encoders with a dedicated FPGA like Magewell, Osprey, Haivision etc. work best. Here's a completely unscientific look at x264 Software vs Apple H264 Hardware both at 2Mbps:

I think NVENC has improved over time and the 6th-gen cards onward look pretty good. My clients these to run encodes to HEVC 10-bit @ 12Mb and they look superb. Although you're certainly not wrong with the AMD encoders, those have always lagged behind. It's funny that they also make the XILINX platform but can't ship a decent encoder on their own SoCs.

[naanlizard]

Ahh, yeah I will admit most of my experience is with relatively low bitrate, 1080 at about 2.5mbps or less typically, lots of "slow" content for sure in comparison.

We also have all sorts of input encoders, from brand new to super old GPUs so maybe that's coloring my experience too

[naanlizard]

Notably with those settings, your keyframe interval is super short. I like 4 as a nice middle ground, but I think it makes webrtc playback start more slowly. But bandwidth requirements (or, quality at the same bandwidth) go way up with lower keyframe intervals. Try backing off on that and bitrate to see how that looks.

[basisbit]

Can confirm x264 with ffmpeg and preset medium or better looks a lot better than AMD's hardware encoder and a bit better (or on par) with NVENC, at typical 1080p bitrates and 0 b-frames. AMD's encoder looks especially garbage when having fast bright-dark transitions for example when live streaming a DJ rave party.

Apple's H264 hardware implementation seems to be good on some devices, and garbage on other devices - I assume it adjusts depending on temperature / TDP availability.

[getroot]

I happened to reread this article. I think I was on vacation last year when this discussion was going on. This is a really good article that could help a lot of people. I think I should pin this.

[getroot]

@bchah Do you have permission to use a PIN on this discussion board? If not, please let me know. And I think it would be great if you could PIN these great discussions.

[bchah]

Looks like you already pinned it - I am able to add a "pinned" label, but I don't know if there's another action somewhere for pinning.

[getroot]

Oh, what I meant was, if there is another good discussion like this, please PIN it. Thanks!