1.3.9. Adaptive Streaming

Overview

Adaptive streaming is feature of the SDK that allows a feed’s quality to be adjusted according to different performance criteria - e.g. network performance. When a feed supports adaptive streaming, it allows the application to specify certain quality criteria (e.g. frame rate or bit rate) and how they may be modified.

Concepts

When creating an outgoing feed, the application can specify one or more quality “rules” that are associated with a that feed. Each rule refers to a specific attribute of the feed (e.g. frame rate, bit rate, etc.) and how it may be adjusted. Once the feed is created with its quality rules, the application can request the feed to adjust to a given quality level. Quality levels are represented with AdaptiveQuality values, which are floating-point values ranged from 0.0 (lowest quality) to 1.0 (highest quality).

Allowing a feed to respond to a quality request is one aspect of the adaptive streaming feature. Indeed, the application can directly use per-feed quality levels to implement custom adaptive streaming logic. However, the SDK also provides a quality monitor feature, which automatically collects performance metrics and makes requests of any feeds assigned to it.

Configuring adaptive streaming

Creating a feed object

Currently, only outgoing video feeds (i.e. MediaServerOutgoingVideoFeed) support adaptive streaming. This is because key performance attributes such a frame rate, resolution, and bit rate are set at source. The process of creating feeds is described in the section Managing feeds.

Setting up quality rules

Once the feed has been created, we can set up quality rules. Because a rule applies to an attribute of the feed, like frame rate, it is applied to the component that owns/uses that attribute. For example, the video source component allows rules to adjust its frame rate or resolution, while the video’s encoder component would be used to adjust the encoding bit rate.

Here we define a rule using a AdaptiveMinMaxRule<T> object templated for the FrameRate type.

    // Create an adaptive rule to interpolate frame rate between 10 and 60
    auto frameRateRule = PxMedia::AdaptiveMinMaxRule<PxMedia::FrameRate>::create(10, 60);

Once a rule is created, it can be used as part of the properties of the component used to make the feed.

    PxMedia::VideoInputFeedV4Linux2 video;
    video.deviceProperties().device("/dev/video0");
    video.videoCapabilities().frameRate(30).adaptiveFrameRate(frameRateRule);

It’s important to note that multiple rules could be applied to the components for the feed.

    video.videoCapabilities()
        .frameRate(30)
        .adaptiveFrameRate(frameRateRule)
        .adaptiveResolution(resolutionRule);

As before, the component is then passed to the feed creation using the factory MediaServerOutgoingVideoFeed::create method.

Applying quality levels

Manual quality adjustment

Once a feed object has been created with quality rules, it will apply them when requested. At the simplest level, this can be performed directly on the feed object:

    // Request that the feed adjusts its quality directly
    feed->applyAdaptiveQuality(0.5);

Of course, the application needs to determine when to adjust the quality level and to what value. For this purpose, the SDK provides a quality monitor.

Quality monitors

The SDK already has the capability to report on the performance of a feed, namely collecting WebRTC feed statistics (see Feed statistics for details).

The FeedQualityMonitor class can be used by the application to monitor feeds by collecting their WebRTC stats, and making quality requests based on performance metrics. The application chooses which feeds to assign to a given quality monitor, allowing full control over which feeds are assessed, and how.

To allow for customisation of monitoring, (e.g. to allow for using other performance metrics such as CPU or memory usage), the feed quality monitor accepts a “performance assessor” object that examines performance and decides what quality level to request of the feeds that it is monitoring. This is analogous to the quality rules that are set on the feed, object but in this case the assessor looks at the feed’s performance and decides a quality level (recall, in the range 0.0 to 1.0) to request.

The following snippet shows how a feed quality monitor is created with an assessor:

    // Create a quality monitor and use a packet loss assessor to adapt with
    PxMedia::FeedQualityMonitor monitor(std::make_unique<PxMedia::FeedPacketLossAssessor>());

The SDK provides an out-of-the-box assessor FeedPacketLossAssessor that looks at packet loss for a feed. This has some default settings for behaviour of packet loss over time, but can be customised by the application. See the FeedPacketLossAssessor documentation for more details.

Once the monitor is created, you can assign feeds to it, and set the monitoring interval:

    using namespace std::chrono_literals;  // Allows us to use ms notation

    // Add a feed to the monitor
    monitor.registerFeed(feed);

    // Start monitoring the feed every 2.5 seconds
    monitor.start(2500ms);

The application can dynamically add and remove feeds from the monitor, as well as change the internal or even stop the monitor altogether.