2.2.11. Adaptive Streaming (Experimental)

Note

Adaptive streaming described here is an experimental feature that previews a more flexible approach using FeedRequest objects. We foresee this replacing the existing approach, which is described in the current Adaptive Streaming section in a future release of the SDK.

2.2.11.1. Overview

Adaptive streaming is a feature of the SDK that allows a feed’s quality to be adjusted according to different performance criteria - e.g. network performance. When a feed supports adaptive streaming, it allows the application to specify certain quality criteria (e.g. frame rate or bit rate) and how they may be modified.

2.2.11.2. Concepts

Quality levels

The AdaptiveQuality class represents a quality level, which is a floating-point value in the range from 0.0 (lowest quality) to 1.0 (highest quality). Feeds that support adaptive streaming can respond to a RequestAdaptiveQuality feed request to set a “quality level” for the feed.

Quality mapping

When creating a feed, the application can provide additional configuration to specify how the feed should adjust its quality. Depending on the feed and components it uses, various aspects may be adjusted - e.g. frame rate, resolution, encoding, etc. This is achieved using a quality mapping which simply specifies a quality range and corresponding attribute to use when the quality requested matches.

Note that mappings are set on feeds individually for these different aspects, allowing different feeds to respond differently to a given quality - e.g. one feed may be deemed low priority and drop its output quality whilst another high priority feed only compromises on quality at a much lower level.

Setting quality

The mechanism for setting a feed’s quality is by sending a RequestAdaptiveQuality request. Whilst the application can directly use per-feed requests to implement custom adaptive streaming logic, the SDK also provides a quality monitor feature, which automatically collects performance metrics for certain feed types, and makes quality requests of any feeds assigned to it.

2.2.11.3. Configuring adaptive streaming

Feed quality mappings are set on the components used to build the feed. As discussed in Feed components, components are the building blocks allowing the application to build feeds with different capabilities and configurations.

Note

Currently the experimental adaptive streaming feature only applies to outgoing media server video feeds (i.e. MediaServerOutgoingVideoFeed). Future updates will add further feed types.

Quality bands

All quality components take a set of “quality bands”, which express how the quality range (0.0 to 1.0) maps to the attribute being adjusted. Naturally, different types of attribute will use different target values - e.g. frame rate using FrameRate or video resolution using VideoResolution.

Example:

Quality band

Frame rate

From

To

0.0

0.25

5fps

0.25

0.5

20fps

0.5

0.75

30fps

0.75

1.0

60fps

To avoid mistakes with disjoint, overlapping or incorrectly sized bands, the SDK APIs take a vector of size/value pairs, e.g.:

// Equal sized bands (the same proportional size of 1.0), with target frame rate values
auto frameRateQuality =
    PxMedia::experimental::AdaptiveQualityMapping<PxMedia::FrameRate>::create(
        {{1.0, PxMedia::FrameRate(5)},
         {1.0, PxMedia::FrameRate(20)},
         {1.0, PxMedia::FrameRate(30)},
         {1.0, PxMedia::FrameRate(60)}});
if (!frameRateQuality) {
    // The provided bands were invalid - e.g. empty vector, non-positive band sizes, etc.
    // Handle error...
}

Note the creation of the quality mapping object using a factory create() function. This returns an outcome result with valid quality mapping object when successful, or an error outcome if the quality bands are invalid (e.g. empty, non-positive band sizes, etc.). You should check the outcome as usual before accessing the quality mapping value.

Example: Frame rate quality

The process of creating feeds is described in the section Managing feeds. To set up adaptive streaming capabilities, we additionally create the relevant component with quality bands, and pass it to the feed creation.

using VideoAdaptiveFrameRateComponent = PxMedia::experimental::VideoAdaptiveFrameRateComponent;

// Video input/output components as normal
PxMedia::VideoInputFeedV4Linux2 videoInput;
videoInput.deviceProperties().device("/dev/video0");
PxMedia::VideoOutputFeedAuto localOutput;

// An adaptive frame rate component and its bands
VideoAdaptiveFrameRateComponent adaptiveFrameRate(frameRateQuality.value());

// Create the feed using the builder
auto created = PxMedia::MediaServerOutgoingVideoFeed::Builder(session, props, videoInput)
                   .adaptiveFrameRate(adaptiveFrameRate)
                   .localOutput(localOutput)
                   .create();
if (!created) {
    // Handle error...
}

Notes:

  • The feed takes components for video input and output as usual

  • We additionally create a VideoAdaptiveFrameRateComponent component and set the quality bands (as created in the previous example)

  • Then, we use the MediaServerOutgoingVideoFeed::Builder helper class to create the feed, passing the adaptive frame rate component to it.

  • After creation, the application will go on to start the feed as normal.

Of course you can apply multiple quality components to a feed using the builder pattern:

using VideoAdaptiveResolutionComponent =
    PxMedia::experimental::VideoAdaptiveResolutionComponent;

// An adaptive resolution component and its bands
auto adaptiveResolution = VideoAdaptiveResolutionComponent::create({
    {1.0, {320, 240}},  // Lowest quality resolution
    {1.0, {640, 480}},  // Medium resolution
    {2.0, {1280, 720}}  // Best resolution (2x the proportional size of the other bands)
});
if (!adaptiveResolution) {
    // Handle error...
}

// Create with both adaptive components
auto created =
    PxMedia::MediaServerOutgoingVideoFeed::Builder(session, props, videoInput)
        .adaptiveFrameRate(adaptiveFrameRate)            // Pass component directly
        .adaptiveResolution(adaptiveResolution.value())  // Component value from the outcome
        .localOutput(localOutput)
        .create();

In this example, we used an alternative method to create the component. Instead of creating a validated mapping and passing to the component constructor, we can use a factory create() method to create the component directly with the quality bands. Because the bands still need to be verified, this returns an outcome result, which needs to be checked before using the value to create the feed.

Note

In the above example for resolution, the last band has a proportional size of 2.0 which means that band is double-sized compared to the other bands which use 1.0.

Quality band

Resolution

Size

Proportion

0.0

0.25

320 x 240

1.0

〕 25%

0.25

0.5

640 x 480

1.0

〕 25%

0.5

1.0

1280 x 720

2.0

⎫ 50%

⎭

There is no requirement to “match” bands between different adapting components in terms of sizes or number of bands, they can be arranged to change for different quality levels. e.g. a frame rate could drop at quality 0.75 whilst the resolution remains high until quality 0.5.

Example: Bit rate quality

The previous examples demonstrated components whose purpose was to define adaptive behaviour. Where a component is already mandatory to create the feed (e.g. the encoding component for a video feed), quality mappings are applied directly to the component itself.

For encoding, the application can choose which type of encoding to use (e.g. VP8 or H264), and these different encoders adapt in different ways. Hence, the quality mapping needs to be applied directly to the chosen encoding component. As described in Per-feed encoders, you can set the encoder to use when creating a feed.

We need quality bands typed to the correct attribute for the encoder, in this case bit rate:

using namespace PxMedia::Literals;

// Bit rate quality mapping
auto bitRateQuality = PxMedia::experimental::AdaptiveQualityMapping<PxMedia::BitRate>::create({
    {1.0, 250_kbps},   // Lowest quality
    {1.0, 1000_kbps},  // Medium quality
    {1.0, 4000_kbps},  // High quality
});
if (!bitRateQuality) {
    // The provided bands were invalid - e.g. empty vector, non-positive band sizes, etc.
    // Handle error...
}

Then we can use this to create the encoder for use in the feed:

// A VP8 encoder component with the adaptive quality mapping and any other settings
PxMedia::VideoWebRtcEncodeFeedVp8 vp8adaptive;
vp8adaptive
    .endUsage(PxMedia::VideoWebRtcEncodeFeedVp8::EndUsageType::CBR)  // e.g. set CBR
    .rtpMtu(1300)                                                    // e.g. set RTP MTU
    .adaptiveQuality(bitRateQuality.value());                        // Adaptive bit rate
auto created = PxMedia::MediaServerOutgoingVideoFeed::Builder(session, props, videoInput)
                   .localOutput(localOutput)
                   .encoder(vp8adaptive)
                   .create();

Just for demonstrative purposes, the example above sets other properties like CBR.

Note: VP8 encoding is currently fixed to software encoding, but H264 can use different encoding methods (e.g. software x264 or hardware VAAPI); the principle is the same:

// Use a x264 as the underlying encoder (could just as easily be VAAPI, etc.)
PxMedia::X264EncoderFeedComponent x264adaptive;
x264adaptive.adaptiveQuality(bitRateQuality.value());

// Set the x264 encoder on the H264 WebRTC encoder
PxMedia::VideoWebRtcEncodeFeedH264 h264adaptive(x264adaptive);

// Create the feed with the adaptive H264 encoder
auto created = PxMedia::MediaServerOutgoingVideoFeed::Builder(session, props, videoInput)
                   .localOutput(localOutput)
                   .encoder(h264adaptive)
                   .create();

For brevity we only set the adaptive bit rate in the above example, but could equally set other (non-adapting) encoder-specific properties as needed, as demonstrated in the VP8 example.

Hardware (VAAPI) encoder: automatic bit-rate clamping

The software encoders (x264 and VP8) and the VAAPI hardware encoder (H264VaapiEncoderComponent) all support runtime adaptation across all levers (bit rate, frame rate and resolution) and recover quality when a higher quality level is subsequently requested.

The VAAPI encoder has one hardware-specific consideration that the SDK handles for you: it sizes its coded (output) buffer from the encode resolution alone, while CBR rate control budgets bit rate ÷ (8 × frame rate) bytes per frame. If a low resolution is combined with a low frame rate at a high bit rate, the per-frame budget can exceed the coded buffer and wedge the encoder. To keep simultaneous resolution + frame-rate adaptation safe, the H264VaapiEncoderComponent automatically clamps the effective bit rate to what the coded buffer can carry at the current resolution and frame rate.

This clamp is transparent for typical quality bands; it only reduces the bit rate at very low resolutions and/or frame rates — which is the desired behaviour for a low-quality band anyway. No configuration is required, and you can freely adapt all three levers with this encoder.

2.2.11.4. Applying quality levels

Setting quality levels directly

Once a feed has been created, it can accept requests to adjust its quality level. The mechanism for this is simply to send a RequestAdaptiveQuality request to the feed, with the desired quality level.

// Send a request to the feed to adjust to medium quality (0.5)
auto requestResult = feed->feedRequest(PxMedia::experimental::RequestAdaptiveQuality{0.5});
if (requestResult) {
    // Feed accepted the request and will adjust quality accordingly...
} else {
    if (requestResult.error() == PxMedia::PxMediaErrorCode::FeedRequestUnhandled) {
        // The feed does not support adaptive quality requests...
    } else {
        // Some other error occurred...
    }
}

If the feed has not been created with adaptive capabilities, it will respond to the quality request with a PxMediaErrorCode::FeedRequestUnhandled error.

Of course, the application needs to determine when to adjust the quality level and to what value. For this purpose, the SDK provides a quality monitor.

Quality monitor for WebRTC feeds

The SDK already has the capability to report on the performance of some feeds, e.g. collecting WebRTC feed statistics (see Media session feed statistics for details).

The FeedQualityMonitor class can be used by the application to monitor feeds by collecting their WebRTC stats, and making quality requests based on performance metrics. The application chooses which feeds to assign to a given quality monitor, allowing full control over which feeds are assessed, and how.

To allow for customisation of monitoring, (e.g. to allow for using other performance metrics such as CPU or memory usage), the feed quality monitor accepts a “performance assessor” object that examines performance and decides what quality level to request of the feeds that it is monitoring. The assessor looks at the feed’s performance and decides a quality level to request.

The following snippet shows how a feed quality monitor is created with an assessor:

// Create a quality monitor and use a packet loss assessor to adapt with
PxMedia::FeedQualityMonitor monitor(std::make_unique<PxMedia::FeedPacketLossAssessor>());

The SDK provides an out-of-the-box assessor FeedPacketLossAssessor that looks at packet loss for a feed. This has some default settings for behaviour of packet loss over time, but can be customised by the application. See the FeedPacketLossAssessor documentation for more details.

Once the monitor is created, you can assign feeds to it, and set the monitoring interval:

using namespace std::chrono_literals;  // Allows us to use ms notation

// Add a feed to the monitor
monitor.registerFeed(feed);

// Start monitoring the feed every 2.5 seconds
monitor.start(2500ms);

The application can dynamically add and remove feeds from the monitor, as well as change the polling interval, or stop the monitor altogether.