Feat: Add ios support for accessing WebVTT Subtitle Content

[coofzilla]

📝 Summary

This PR adds the capability to access and expose the content of subtitles within RN; but, currently only implements the ios portion. The core of this functionality leverages AVFoundation framework's AVPlayerItemLegibleOutput, enabling the app to intercept subtitle data as attributed strings during video playback. This data is subsequently passed to do the RN layer through the newly introduced event, onSubtitleTracks; thus, allowing components to reactively update based on this subtitle content.

Motivation

Our motivation for integrating the onSubtitleTracks event handler with react-native-video is to enhance user engagement with HLS VODs of sports content. Utilizing WebVTT tracks, we aim to dynamically display game information like scores as overlays on the video, based on the playback position. This feature ensures viewers have access to contextual sports data as they navigate through the video, significantly improving their viewing experience by making it more informative and interactive. This implementation aligns with the capabilities of the WebVTT API, optimizing our app's use of text tracks for an enriched sports viewing experience.

The primary purpose of WebVTT files is to add text overlays to a <video>

source

Changes

📲 iOS Implementation:

• RCTPlayerObserver class implements AVPlayerItemLegibleOutputPushDelegate to observe and handle subtitle tracks.
• Configured legibleOutput on the player item and set the current class as the delegate to manage legible output.
• Implemented the legibleOutput delegate method to capture subtitle data, processing it via the handleLegibleOutput method to extract and utilize subtitle strings.

🌉 RN Bridge:

• Added an @objc property onSubtitleTracks to facilitate event emission, enabling the RN layer to receive subtitle data.
• Implemented method handleLegibleOutput to handle attributed strings from subtitles, triggering the onSubtitleTracks event with the subtitle text for consumption within RN components.

🥳 RN Layer

• Exposed the onSubtitleTracks property.

Example:

const App = () => {
  const handleSubtitleTracks = useCallback((e: OnSubtitleTracksData) => {
    console.log('Subtitles:', e.subtitleTracks);
  }, []);

  return (
    <Video
      source={{ uri: 'video-url.mp4' }} // Replace 'video-url.mp4' with your video source
      onSubtitleTracks={handleSubtitleTracks} // new event handler
      // Other props for the Video component
    />
  );
};

📄 Docs

• updated events.md to include a description aligned with current standards.

Test plan

✅ Tested in example app with uri included in this PR

Set the tracks you want to use, for my demo I used the below; however, SelectedTrackType.LANGUAGE also works. For my actual use-case, we'll be using TITLE so I wanted to make sure that worked.

• Apply following changes to test in example app:

// VideoPlayer.tsx

    // selectedTextTrack={this.state.selectedTextTrack}
    selectedTextTrack={{
      type: SelectedTrackType.TITLE,
      value: 'English',
    }}

Comment out this line; due to textTracks being undefined, I believe it has to do with how data.textTracks is being set; but, I didn't want to address that on this PR as its a separate issue.

// VideoPlayer.tsx
  {/* {this.state.textTracks.map(track => (
    <Picker.Item
      label={track.language}
      value={track.language}
      key={track.language}
    />
  ))} */}

Comment out this line for the app to build( I believe this is a known issue):

// metro.config.js

// /(.*\/react-native-video\/node_modules\/.*)$/,

Demo of how it works:

demo.mp4

[freeboub]

few comments, but looks good !
on android I think the code shall be in onCue function if you want to implement it.

[coofzilla]

few comments, but looks good ! on android I think the code shall be in onCue function if you want to implement it.

Thank you for the feedback 🙏🏽! I'll take a look at the android implementation once I get this merged 😄 I'm going to apply the requested changes here shortly

[coofzilla]

All feedback has been addressed :)

I've also tested it again in the sample app.

[KrzysztofMoch]

LGTM, Thank you very much 👍
Let's wait for @freeboub review (this may take some time, for what I know he's busy lately)

[coofzilla]

LGTM, Thank you very much 👍 Let's wait for @freeboub review (this may take some time, for what I know he's busy lately)

Awesome, happy to help! Excited to get this merged in 😊

[coofzilla]

@KrzysztofMoch any other blockers I need to take care of to get this merged? 😅

[KrzysztofMoch]

No 😄