Letterbox rewrite

[defagos]

Some Letterbox issues are deeply related to how it was supposed to work at the time it was written. We discovered a few issues since then, some of them more annoying than others.

If we wanted to fix them all we could rethink SRG Media Player and SRG Letterbox, that means listing everything that is supported first, then thinking about an architecture that could help us solve the following issues (which ones are relevant should be agreed on) while preserving our feature set, performance and battery life:

• Continuous playback does not work when the application is in the background (requires AVPlayerQueue).
• The behavior could be improved when playing the same media (Improve same media playback completion handler #226 and Improve same media playback behavior #227).
• The UI could provide some customization options, or might be entirely customizable from smaller bricks. At least it should be possible to implement a custom player bar. A good approach might be to deliver a Letterbox view model with a default associated view, letting applications create custom user experiences without the need for small bricks (or at least not too many).
• Playback notifications are sometimes missed, especially at the end of playback.
• The design team would like to completely update the segment / chapter selection experience.
• Google Cast could be integrated directly (control from a Letterbox view, integration behaving well with AirPlay and PiP, maybe support for playlists but might be a bit touchy since the playlist is managed at the receiver level). It should be investigated whether this would be better integrated at the Media Player or at the Letterbox level (so that AirPlay or Picture in Picture can be mutually exclusive with Google Cast in a more reliable way).
• Lightweight players should be possible (e.g. for stories or display in video feeds). Preload should be possible so that playback starts fast.
• Improved behavior when sports streams stop (the user should be able to continue watching the content, even if timeshift was used).
• Support for multiple camera angles (sports events).
• Perfect responsiveness, even down to very small / zero sizes.
• Better, more consistent atomic changes and associated API. Currently we update internal state (e.g. media and media composition) in several steps, and all these properties can be observed with KVO, with inconsistent results (the recommended way is therefore currently to listen to a notification sent at the very end of the updates). We should probably rather encapsulate all data as a single struct and update the whole thing at once. This would make it possible to publish changes reactively.
• Fast deallocation when the player is released (and no more issue with playback sometimes still continuing in background for some reason).
• Media service reset resilience.
• Akamai QoS SDK integration Will be discontinued soon
• Possibility to have several Letterbox views displaying a video played by a single controller (feasibility must be validated with simple AVPlayer and AVPlayerLayers first).
• (To be discussed) Integration of any kind of analytics solutions next to our own official solutions.
• Improve testability / mockability with protocols.
• Low-data mode support.
• CarPlay integration helpers.
• Other data sources than the IL: We could provide an abstraction layer to be able to use any kind of data source instead of relying on an IL-compliant endpoint. Not sure if worth it or too complex.
• If possible provide a more universal way to hook into UI-triggered events, e.g. toggle play / pause from the UI, or change subtitles for example.

This is of course no small tasks with no guarantee it will ever happen, but this issue can help us track important ideas which could help us think about Letterbox future.

If the idea of rewriting Letterbox is validated so that all these new features and issues can be fixed, we should:

• List all features we currently support.
• List all additional features we want to support.
• Gather feedback from other teams if we missed some ideas.
• Have a look at all Letterbox, Media Player and Play issues and write down all behaviors that should be fixed in the new version.
• Discuss with all Letterbox team members about the scope of Letterbox, what belongs to the view, what is the responsibility of Letterbox, etc. to define a new scope for what Letterbox is.
• Design the main architecture principles together to address the new requirements.
• Proceed with the implementation. PoC should probably be made to verify:
  • Queue player integration and background behaviour.
  • Playback states and reactive API model.
  • UI concepts.
  • Gesture recognizers. Apparently it is still possible to define relationships and exclusions between gesture recognizers.
  • Finding performance limits in scroll views. Starting with a simple AVQueuePlayer and a single AVPlayerLayer, we can namely investigate if scrolling hiccups can be eliminated and what cannot be eliminated (work on the main thread required by media services, e.g.).
  • etc.
• Create a kind of Letterbox manifesto which clearly states what Letterbox is as a product, what it supports (e.g. devices like iPhones, iPads, old and recent Apple TVs for AirPlay, Chromecast devices, requirements for thumbnail seeking on all platforms, having subtitles and audio tracks identical between HLS and DASH, etc.), and precisely documents how streams and metadata must be produced for compatibility with it.
• Ask for stable test streams matching Letterbox manifesto.

[defagos]

I made a small PoC with a slider and several gesture recognizers:

• Tap.
• Double tap.
• Pinch.
• Drag.

Gesture recognizers are easy to implement and can be easily prioritized (e.g. .gesture(doubleTap.exclusively(before: tap))). The slider behaves well.

I am confident we can implement a player UI in SwiftUI.

[defagos]

We had a pretty interesting discussion within the team, mostly about playlist and continuous playback support. Since AVQueuePlayer need to be at the center, we discussed general design ideas about how we could better implement playlists if we were to rethink Letterbox:

• Have a more stateful approach, i.e. have the player store the items to be played, which is currently not the case on iOS. A playlist orchestrator (the controller or outside it) would let the playlist be mutated, as AVQueuePlayer allows to do (insertion, deletion, relocations). Items in the playlist would probably be defined by a URN, start time, settings (e.g. standalone parameter), etc.
• The playlist orchestrator would be given the tasks to do (mutable) and would be responsible of preparing ahead of time the data.
• Maybe we could use resource loaders to prepare the whole playback context, i.e. play URLs with a custom scheme so that the resource loader is self-sufficient, retrieving media and media composition when playback must be prepared. This would make AVQueuePlayer the orchestrator itself. We would be able to adopt a player API resembling the one of AVQueuePlayer, both at the Media Player and Letterbox level, so that playlist support is guaranteed to work well. Of course, with all the other behaviors we currently have on top AVPlayer playback (metadata, segments, etc.), we likely have to retrieve the media / media composition by the controller directly, but maybe not for playback purposes (if this can be achieved by a custom resource loader). Of course this concept, inspired by @StaehliJ who investigated a same idea with ExoPlayer, should be verified with a PoC first.

An important question when using custom resource loaders is whether we can have good error reporting, as we know there are potential issues. This should be verified as well.

We should also get inspiration by reading through ExoPlayer documentation, as some ideas might prove to be helpful.

[defagos]

If I were to recommend a perimeter for a PoC based on the remarks above, a good scope would probably be:

• Be able to play items at the Media Player level. Design a basic API supporting playback of single items and playlists.
• Check that a consolidated publisher-based API is possible to have common mechanisms for notifications, KVO and playback time observers. Verify that reliable playback state management can be implemented.
• Implement a basic Letterbox API supporting single items and playlists. Implement a resource loader taking custom URLs as input so that media and media composition are retrieved independently by AVQueuePlayer for playback purposes.
• Implement basic media composition based behaviors in Letterbox, like the kill switch. Maybe such features can also be implemented at the resource loader level.

[defagos]

An interesting idea to PoC for playlist support: Playing urn:<urn> in the player, and letting the resource loader manage metadata retrieval. This way we could prepare content without preparing metadata ahead. Does not mean that other metadata updates would occur through this channel, we could have a separate data retrieval process for other purposes. Also we need to check whether error handling can be handled properly and if this works correctly over AirPlay.

Other important question: When does resource pre-loading happen? Can we guarantee that an Akamai token is retrieved at the very last time? We should study when resource pre-loading might happen in an AVQueuePlayer and possibly use resource renewal to load the tokenized URL at the latest moment (for token-protected content only, other content can be pre-loaded at any time).

[defagos]

I think we should also discuss going the monorepo way with Letterbox. We could then iterate faster on our player code base while still maintaining a clear separation between components by having each of them (core player, data provider, analytics, diagnostics, LB player) be a separate product within a single SPM package.

Moreover we should drop all the dependencies that are superfluous, like FXReachability, libextobjc, Mantle or SRG Logger. for example. If we later decide to separate the packages (e.g. the core player) we can easily do it. We would also only require a single demo project and we could share some test helpers which are now duplicate between projects.

[defagos]

I think we would likely need to support the legacy Letterbox for a while, which is maybe why we could use a codename for a new Letterbox, e.g. Pillarbox, until the final release. This would make it easier to identifiy about both products while they coexist .

[defagos]

Google Cast integration would also be necessary in a PoC. Not at the core player level probably (since the player is on the receiver, while AVPlayer is local), but at the UI level.

[defagos]

For controller configuration we should introduce as much immutable configuration as possible (e.g. update intervals, skip intervals if customizable, etc.).

For that we should have a configuration builder, something like:

let controller = SRGLetterboxController()
    .configure { builder in
        builder
            .setUpdateInterval(30)
            .setSkipInterval(10)
    }
}

The builder pattern could also be used for construction of other immutable instances, e.g. analytics labels.

[defagos]

Some use case mentioned by SwissTXT which a better playlist could help implementing: Catch up with live, which means play all missed highlights in sequence and then switch to the livestream automatically.

This would likely be something easy to implement with a better playlist management and player items responsible of getting the media composition, though the feature would likely be implemented in an app rather than in Letterbox itself (more business logic involved).

[defagos]

Front-end teams to involve in discussions for a new Letterbox:

• News / sports product development teams (SRF, RTS, RSI, RTR, SWI).
• Play product development team.
• Play Suisse development team.
• SwissTXT / Sports stream team.
• Meteo app team.
• Other?

[defagos]

Given all the issues we recently had with stream packaging and segments / standalone playback, and given the cost-efficiency desired by some teams, we could discuss dropping segments entirely if this makes sense from a user experience point of view. Segments namely introduce a lot of complexity which might not be needed anymore.

For apps needing standalone content each individual clip would then be published independently, thus limiting the impact to only a few clips instead of all segments.

[defagos]

Snippet support would likely be interesting in the future.

[defagos]

We must definitely use the new async API for querying asset properties, see https://developer.apple.com/wwdc21/10146 and https://developer.apple.com/wwdc22/110379. iOS 15 for most of them, otherwise iOS 16+.

[defagos]

✅ I made a first PoC for a queue player with a list of items, each of them responsible of retrieving the media composition entirely via a resource loader. The result is very promising and matches all my current expectations:

• The queue player immediately retrieves the main entry point of the stream to be played and, for HLS, only loads chunks when the item actually starts playing.
• There is no problem with tokens: As the master playlist has been successfully retrieved, the stream can play successfully any time later.
• If resource loading fails for some resource playback automatically continues with the next item in the list. This is both the case if processRequest immediately returns false or if finishLoading reports an error to the resource loader after asynchronous resource retrieval.
• Background continuous playback works fine.
• Chaining items is very efficient and smooth.
• Livestreams with or without DVR can be added to a playlist and play fine. If the playback speed is larger than 1 they will reach their end and playback will continue with the next item. If playback speed is smaller than 1 they will continue playing as expected. Moving to the next item must therefore be triggered explicitly, otherwise only if the stream ends will playback automatically continue. Note that our playback speed implementation near the live edge should take care that, no matter the playback speed, playback never skips livestream items.
• Continuous playback over AirPlay works well, for audio and video.

We can therefore imagine a core player implementing playlist with AVQueuePlayer and, working with AVPlayerItem and self-contained resource loading, would make it possible to play any kind of stream at the Letterbox level, provided we define a common media playback context format which can be bridged with IL or Play CH metadata, making it possible to play and mix any kind of content.

[defagos]

Other PoCs to be made afterwards:

• ✅ Error reporting from resource loader. Are errors now correctly forwarded to the parent AVPlayer?
• ✅ Reliable KVObservation via publishers.
• ✅ Reactive view updates with a SwiftUI view having basic controls (and controller switch on-the-fly).
• ✅ Gesture management for usual player interactions (UI toggle, double tap, pinch). Pinch-to-zoom should also be available only when the video is full screen, so it would be interesting to see how this condition can be implemented in SwiftUI.
• Continuous playback overlay / delay, on iOS and tvOS. We likely must pause the player during the transition if there is no dedicated API to pause item dequeuing.
• ✅ Consolidation of all reactive change sources with Combine (notifications, KVO, periodic time observation). The controller should be an ObservableObject with @Published state information. States will be enums with associated values, e.g. a seek will provide start and destination in its associated values directly. We can also likely implement a periodic time observer published property backed by a traditional periodic time observer (no custom publisher is likely needed to achieve this result).
• ☑️ SwiftUI previews for Swift packages (having binary dependencies). Warning: Test local package as well as remote package.
• ✅ Media metadata and player items. In particular how could a generic player (metadata contract and player items) API look like and how IL / Play CH modules implementing this contract would look like.
• ✅ Letterbox view and safe area management.
• ✅ Integration of a Letterbox view:
  • Full screen mode
  • Status bar management
  • Home indicator management
• ✅ Async / await / actor PoC to better understand how these can help, especially with the new AVPlayer async APIs.
• ✅ Lightweight player for display in feeds.
• Google Cast integration concept. Likely not at the core player level but rather at the Letterbox level.
• ✅ Kill switch implementation with resource loader renewal requests (also see AVAssetResourceLoadingContentInformationRequest renewalDate property).
• ✅ Documentation with DocC, from code documentation to guides, articles and tutorials (also with packages as dependencies). Also interesting to investigate behavior with subpackages as well as how we could publish the documentation per version on GitHub pages.
• ℹ️ Playgrounds for packages with binary dependencies. DocC tutorials and articles are better
• Interstitials: With interstitials available on iOS 16 and a controller to manage them we can maybe entirely manage blocked regions using them. See WWDC 2022 main session and related sessions.
• ☑️ Project scaffolding: We must check that we can create and iOS + tvOS project, with a demo, UTs, working SwiftUI previews, documentations and all binary dependencies (Google Cast SDK, analytics, Akamai QoS, Mapp SDK). This project should have the following products:
  • Media player
  • Appearance
  • Diagnostics
  • Analytics
  • Letterbox
• ✅ Workflow (branching model, rebase, PRs and comments, issue tracking, etc.).
• ✅ Test modern background video playback API.
• tvOS continuous playback native support with item playlists.
• ✅ Test modern control center integration.
• ✅ API to play the previous item with AVQueuePlayer.
• ℹ️ Use of AVPlayerItemPlaybackStalledNotification for stall detection. Seems that _ isPlaybackLikelyToKeepUp suffices_
• ✅ Kill switch concept with resource renewal.
• ✅ Handling loading states (like buffering or preparing in the current SRG Media Player). Maybe such distinction is not needed anymore if we use items and observe when they are loading. Then no matter whether we are preparing (i.e. fetching data in the item internally) or stuck buffering, the state can be the same. _Implemented without a PoC. isPlaybackLikelyToKeepUp works as expected.
• ✅ Possible to attach control center metadata to items directly so that control center information is bound to items?
• Use of boundary time observers for segments (instead of periodic updates):
  • Reliabie?
  • Can we identify segment transtions?
  • What about segment updates for highlights?
  • Can we use boundary time observers to implement blocked regions?
• Player pool API: Maybe having a player pool API (to manage a list of assets, ensuring a player is ready to play the content) would be valuable.
• Use of AVPlayerItemMediaDataCollector for metadata retrieval. Seems that data collection can also be associated with time ranges.
• Chapter native support.

[defagos]

✅ I made a second PoC of a video feed. Though the implementation is fairly basic this shows that performance is quite good when scrolling. This is likely the best we can achieve (except if things can be further optimized with async asset loading somehow), but at least it is a lot better than what we have with the current Letterbox version.

[defagos]

✅ I made a third PoC for error propagation from a resource loader. Loading the media composition requires us to better propagate errors, which was historically an issue. This was preventing us from performing DRM error reporting at the Letterbox level, but things have improved, and I discovered by chance that the type of the error and its value (non-zero) are especially important so that propagation succeeds. This also was clearly improved in iOS 15, which is probably a hint this is the version we should target.

[defagos]

I just discovered that AVPlayer supports various policies for background video playback since iOS 12, see audiovisualBackgroundPlaybackPolicy. We will likely be able to implement the same behavior as before in a much simpler way.

Before starting any serious work on the player we should really look at all AVFoundation and AVKit headers to discover new APIs we might have missed.

[defagos]

The AVFoundation-Combine library introduces a few publishers for AVPlayer, in particular one for periodic time observation. The publisher is directly implemented and might be worth a look.

I also stumbled on an article which applies what I think should be sufficient, namely a current value subject or a passthrough subject.

Anyway, a good solution should likely register the time observer upon subscription and release it automatically when the subscription is cancelled. Maybe this is possible with a simple publisher and handleEvents, otherwise a custom publisher might be better.

[defagos]

✅ I made a fourth PoC showing how a reactive layer can be built on top of AVPlayer. Works nicely and should be a pretty solid inspiration for our implementation. I used AVFoundation-Combine implementation above to implement periodic time publishers and checked that removal and associated player / subscription deallocations do not leak.

[defagos]

✅ I made a fifth PoC showing how a reactive player with business logic can be built on top of a lower-level reactive player. This PoC also shows that we can easily attach a player to an existing view, but also to several views at the same time, while keeping the UI in sync.

[defagos]

✅ I made a sixth PoC for documentation with DocC.

Learnings

• Symbols appearing in the documentation are checked so broken links will be detected when compiling the documentation.
• Code appearing in the documentation is not compiled or checked (was to be expected).
• We can use a plugin to have documentation generated via the SPM command line.
• Each product in a package made of different libraries has its own documentation. No links between documentations or no top-level documentation is possible. This means we cannot e.g. link from the Analytics documentation to the Media Player documentation directly and with the compiler checking symbols.
• We can structure the documentation with a landing page, sections grouped by topic. Articles can be written for further topic exploration.
• Long documentation in code can be moved in extensions bearing the same name (totally or partially) in a documentation catalog.
• Tutorials are a bit painful to write IMHO. Images are required and the format is a bit verbose. We should focus on landing pages and articles first.
• Disambiguation is sometimes required and some conventions have to be followed (e.g. title of the landing page). Everything is well described in the official documentation.
• A short GitHub readme for the SPM integration is probably still a good idea, though the rest of the documentation could be provided with a documentation catalog.
• When generating documentation for a project linking against SPM packages, documentation for the packages is generated as well. So there is no real need to distribute the documentation separately, the documentation bundled with the code will always be aligned with the code itself.

Useful links:

• Official documentation.
• Short article.

Remark:

I also found that with SPM 5.7 we can no longer provide a package name to refer it in target dependencies. This most likely means we should match the repository name with the package name (though aliases can now be defined).

Conclusion

I think that using DocC is very valuable. We should use Xcode as much as possible to adopt common documentation style (e.g. /// in general) and use separate markdown files only sparingly, e.g. for a repository welcome page with basic integration instructions.

[defagos]

☑️ I made a seventh PoC for project structure and SPM packaging.

I found two annoying issues:

1. Even if specifying that a binary dependency is iOS-only (with .when(platforms:)), dependencies without tvOS binaries like Google Cast SDK will make compilation fails (also see associated linked thread). The issue likely affects source packages, except if the sources can be compiled for all platforms (which is what I think happens with Mapp SDK). To be done:
   • Create small sample iOS + tvOS package using iOS-only binary dependency.
   • Create small sample iOS + tvOS package using iOS-only source dependency.
   • Check the results.
   • Investigate if this can be fixed in an SPM PR.
2. SwiftUI previews do not work when binary dependency packages are involved as of Xcode 14 (beta 3). Was working in Xcode 13 Fixed in Xcode 14 beta 4:
   • Create small sample iOS package with a binary dependency.
   • Investigate if this can be fixed in an SPM PR. I think there is a slight chance we can do it at the SPM level, as apps using this kind of binary package (like Play, e.g.) are able to display previews. Maybe it suffices to copy binaries somewhere when generating previews for a Swift package to fix the issue.
   • If a solution was found, check it works with other kinds of packags (iOS + tvOS package with iOS + tvOS binary dependency / iOS + tvOS package with iOS-only binary dependency).

Workarounds:

1. Package dummy tvOS binary (unused) in Google Cast binary package. I tweaked the XCFramework locally and this works fine. We can use a small repackaging script which builds two dummy tvOS binaries, get the iOS ones from the official XCFramework and packages a universal XCFramework. The only downside is that the dummy tvOS framework is then embedded into the tvOS app (but never used and small, so this is not really a problem).
2. Report the issue to Apple (if not already) and cross fingers it is fixed in Xcode 14 later pre-releases. There is likely no workaround. Issue fixed in Xcode 14 beta 4

Remark

The workaround is not necessary for the Mapp iOS SDK because it is built from source. But the WebKit dependency makes compilation fail for tvOS SwiftUI previews. If we need to integrate this SDK we can either work on a fix (because its packaging is currently a bit messy) or simply integrate the common SDK .product(name: "MappIntelligenceSDK", package: "MappIntelligence-iOS-v5") since the iOS and tvOS SDKs have very few additions we will likely never need anyway.

[defagos]

I made a PoC for better workflow using GitHub and it is promising.

• We can make PRs and rebase branches while preserving code discussions and annotations. Rebased commits mentioned in messages will be dangling but the corresponding code is still accessible so discussions can still be understood.
• GitHub Projects is simple but could be used to manage our backlogs. This still remains to be discussed within the team, though.

[defagos]

An idea about states (enums with with associated labels where appropriate):

• Core player level: States like idle, preparing, playing, paused, seeking, etc.
• At the Letterbox level: States like idle, preparing, waiting (for content not available yet), playing, paused, seeking, etc.

Letterbox states are a bit richer than core player ones so they should be a superset if the core player states. With precisely defined states writing a (possibly custom) UI should be a breeze.

[defagos]

I tried integration Akamai QoS SDK but without success (missing core dependency), and it seems to be packaged with its own player AmpPlayer. Likely a bad idea.

[defagos]

✅ I updated the reactive player PoC to support background video playback with the audiovisualBackgroundPlaybackPolicy API. It works really as expected.

One limitation, though: We still have to detach the layer if we want to support background playback with the screen locked. This might be easy to implement but, if not, I would likely recommend dropping this feature, as it relies on non-official APIs we should rather avoid in general.

[defagos]

Pillarbox is born, built on many ideas outlined in this issue. I guess we can now close it.