Split the monolithic services file into one file per service

[dodgepong]

I wanted to test the waters a bit on this concept before submitting an RFC, so I figured I'd give the new Github discussions feature a whirl just to see how it goes. There was an internal discussion about this idea a few days ago, and I figured it was worth sharing more publicly.

Since OBS started, we've been maintaining a monolithic services file that lists every RTMP service that we support, along with some encoding restrictions for each service. This file populates the services drop-down in the Stream settings.

This works, but it's a bit cumbersome to use for a few reasons:

• When new services want to add themselves to OBS, they have to submit a PR to add themselves to the services list. Oftentimes they either don't know how to do that, or they get the PR wrong and forget things like updating the version number in the package.json file.
• If users want to add a custom service themselves, if they update the services.json file, their changes will get clobbered by the next update. The only way for users to stream to custom services is via the custom RTMP server output.
• We have an open RFC for the implementation of a URI handler that would allow users to click buttons on websites and have those websites communicate with OBS. One major feature this would enable is the addition of new services to OBS without having to submit a PR, especially if that service doesn't want to have themselves listed as a default service for whatever reason. However, we currently don't have a way to durably store custom-added services that would be added this way. If we stored them in the services.json file, they would be clobbered as well.

A more flexible solution to these problems would be to split out the services each into their own files rather than have them live in a single monolithic file. Services would live in a folder, rather than a file, and the services list would be populated by iterating through each JSON file in the given folder.

Furthermore, we could register these JSON files with a custom extension, such as .obssvc, which means services could distribute these files for download, and users could simply double-click them to add them to OBS without the need to manually place them in a folder. Services could still be installed into OBS via the URI handler (whenever it arrives) with a one-click button on the website, but registering a file handle would also be a good step toward making the files easier to use.

Benefits:

• Allows the addition of new services by adding new files rather than editing one existing file
• Users could create files in this folder at-will for their own use. Since these are user-added files, they won't get overwritten by updates.
• Services can add themselves to OBS without first requiring an PR to the OBS repository by either distributing a service file, or, in the future, through the URI handler.

Addressing a few questions:

• How will service files that were automatically managed by OBS be deleted if they are deleted from the OBS-managed list?
  • Service files could contain a flag in them to indicate that they are automatically managed by OBS, and if a file has that flag and is not in the list of files that the OBS service updater manages, then it is deleted automatically.
• How will custom services receive automatic updates from the services they were gotten from?
  • Service files could include a URL that they can go to in order to fetch new updates from the service itself at the same time that OBS updates its own service files. These files would be cached in the services folder so that if the update from that URL fails, it keeps the last-known good service file. It may be necessary for OBS to do some sanity checking on the formatting and content of these files to be sure that the server isn't sending down nonsense before it caches the result of an update request.
• If services can update their own services file hosted on their server through that URL, and users can fetch updates directly from those services through that URL, why is OBS hosting a services file at all?
  • This is actually a great question, and maybe one day we can move to a world where services do indeed host their own services file. Until then, we will need to host our own in case services don't want to host it themselves, though that means they have to go through the PR process when they update their ingest list or encoding requirements. OBS will still need to maintain, at the very least, a list of services that we want to distribute with the program by default, whether that's in the form of JSON files we distribute ourselves, and/or a list of URLs that OBS should know it can fetch from on first launch.
• Why do all this work if we're going to revamp the way we do services at some time in the future?
  • While the concept of services absolutely needs an overhaul, especially in the context of services that want integration via OAuth, I think this is a reasonable stop-gap in the meantime, since such an overhaul would be a much larger undertaking that nobody has started designing, and could probably still use several of the ideas found here.

I'm interested to hear thoughts on this idea, or if someone has a better idea. Note that @RytoEX has already implemented a portion of this feature here: https://github.com/RytoEX/obs-studio/tree/add-user-defined-rtmp-services

[RytoEX]

A possible benefit:
Isolating service data like this could more easily allow people to package or modify an OBS setup so that it has a curated set of services available to users.

[coral]

I wanted to add some comments here as I wrote the RFC linked above and clarify some of why I haven’t really updated the RFC, because I do think Dodgepong sort of gets to the bottom of why that RFC probably wasn’t the right approach.

To start with, I work at the Video Platform at Twitch, also building Amazon IVS (Interactive Video Service). The RFC’s intent stemmed from a desire to help customers correctly configure OBS for use cases that are outside of the traditional Twitch stream. For example, a customer streaming to IVS today using OBS gets the default 4 second keyframe interval together with x264’s standard scenecut parameter enabled (inserting keyframes at scene changes). This has a direct impact on the latency that Twitch & IVS is able to offer customers due to how we chunk video and leaves customers confused why OBS works great when using the “Twitch” profile but not the “Custom RTMP”.

As OBS is a tool intended for many platforms, I thought it would be silly to try to change the default behaviour to fit Twitch & IVS. Instead I tried to come up with a solution where the service could take on the burden on specifying the configuration for the user so new IVS customers didn’t have to add hundreds of pull requests to the services file.

Another use case of this that I didn’t touch on in the RFC is the desire to provide deeper recommendations from the platform on what parameters to encode with. Of course we’re not asking to “mandate” the users configuration but in many cases we see users stream based on recommendations that are generic that for example aren’t working for their specific ISP. Being able to give the user a link of “hey this configuration might work better for you” would enable a better streaming experience for the user from the platform point of view.
I think the .obssvc approach is a much cleaner design that addresses a lot of the problems in the URI approach.

• It solves the problem of defining what actions the URI should be able to do. Part of the problem was that I initially drafted the RFC with a specific goal in mind but as people on the issue identified, there is a lot of ambiguity in what that means for other services and for other use cases.
• Allows the service to be more specific on what settings should be configured. The RFC proposal was before I realized that the scenecut=0 parameter isn’t set by default for “Custom RTMP” which is something customers of IVS would benefit from. If the .obssvc file is broken out similar to how services are defined in the general JSON array right now, the service can easily add these parameters.
• Overcomes the URI payload limit. As was brought up in the RFC, the URI has a payload limitation which means that if you want to add for example tons of custom RTMP endpoints to the service list, you are likely to run out of the URI payload limit.

I agree that this approach would work great as well as a bridge until an URI handler exists (if it ever does). Having the user download a file and automatically opening in OBS is easier than having to guide the user to copy JSON files into specific directories or provide 5 screenshots of how to configure OBS correctly.

I would add some feedback to this approach. If I skim the services.json file I can sse that it supports specifying blocks like these:

"recommended": 
{
	"keyint": 2,
	"profile": "main",
	"supported resolutions": [
	    "1280x720",
	    "852x480",
	    "480x360"
	],
	"max fps": 30,
	"max video bitrate": 6000,
	"max audio bitrate": 128
}

Just make sure that these are also resolutions that can be vertical, as broadcasting for mobile aspect ratios (vertical video) is becoming more popular. Being able for a service to specify 4 resolutions that are vertical would be great.

// coral - [email protected]

[notr1ch]

I wrote up a lengthy reply to why we should continue hosting service files on obsproject.com for updates, but lost it before posting in my clumsiness. TLDR: For privacy (exposing users to 3rd party sites) and security (potentially malicious / DoS JSON payloads) I'd suggest we keep updates on our site where possible. Bonus: At some point we can ditch the whole PR-a-JSON-file process and have a user-friendly service submission web UI.

[coral]

There are a lot of solution to this but I think this is almost a separate issue. Being able for third parties to ship a downloadable file is a feature that really doesn't need long term auto update, even if the auto update would be nice.

Addressing the auto update however, the @derrod approach is interesting as OBS could technically provide validation that the configs adhere to specific accepted JSON which should remove most of the attack vector + being able to filter to the services who uses OBS.

To be clear, from Twitch's point of view, if the feature ends up not having auto update then that's fine. Our goal is to just enable easier way for users to configure OBS for other purposes than the normal "16:9 normal resolution stream to established service".

[maxstoller]

Hi all, I work with coral. I like the .obssvc approach. It would enable one-click configuration of OBS for us and any other service. That would certainly make OBS easier to use.

Is the main question now how do we enable automatic configuration updates for .obsvc users? I can see how that significantly complicates this proposal, but I’m not sure it’s needed? How often do services need to update their configurations? In our case, the answer is not often.

I think the ability to distribute static config files is a big win itself. Maybe automatic updates could be considered as part of a v2 if needed?

[dodgepong]

Thanks for the feedback @coral and @maxstoller, it's possible it could be left for later implementation.

Though it occurs to me that there is another security concern that should probably at least be considered, and that is the authoritativeness of a service file, i.e. are we sure that a service file wasn't maliciously altered by a third party. For example, if we are doing manual updates, then double-clicking on an .obssvc file would open a prompt in OBS saying "You already have a service named X, do you want to update it?" But these .obssvc files could come from anywhere, so someone who doesn't know what they are doing could overwrite their settings for one service with unknowingly-malicious settings that someone else gave them. By having a central repository, people can generally trust that submissions have been vetted by a human that has users' interests in mind.

At the very least, we will need to make it impossible for someone to overwrite a service file that OBS ostensibly controls from our central repository. That said, this might be an issue regardless of what we settle on regarding auto-updates. I'm not convinced it's a problem we need to solve, or that it's a blocker, but I wanted to at least state that it is at least somewhat of a concern.

[coral]

@dodgepong

I think there's a rather easy solution to that actually. You mention this approach (I think) and I think this is the best path personally. As in never allowing third-party .obssvc files to overwrite the services provided by OBS, instead just add to a "custom services" list or similar. That way OBS maintains the control over the services that are important to the regular "inside the box" user and new services can easily send .obssvc files to OBS users and have them get a configuration.

It's possible that you could have a "Twitch" and "Custom Services -> Twitch Special Settings" at the same time but I think that as long as you indicate that the configuration is not "blessed" by OBS that shouldn't cause an issue.

[dodgepong]

Yes, that's the approach I meant. My concern was only related to services that were not controlled by OBS, since without auto-updates, the only way to get updates for such services would be procure new files to overwrite existing ones.

[Fenrirthviti]

I'd like to reopen this discussion, as it's becoming increasingly clear that some level of custom service handling will be useful for OBS, specifically for plugins that might want to roll their own lists and have full control over them.

The solution proposed and implemented by RytoEX seems to be a fair compromise between all the raised points.

Are there any strong objections to this approach?