Publishing & Subscribing with the IVS Android Broadcast SDK | Real-Time Streaming

This document takes you through the steps involved in publishing and subscribing to a stage using the IVS real-time streaming Android broadcast SDK.

Concepts

Three core concepts underlie real-time functionality: stage, strategy, and renderer. The design goal is minimizing the amount of client-side logic necessary to build a working product.

Stage

The Stage class is the main point of interaction between the host application and the SDK. It represents the stage itself and is used to join and leave the stage. Creating and joining a stage requires a valid, unexpired token string from the control plane (represented as token). Joining and leaving a stage are simple.

Stage stage = new Stage(context, token, strategy);

try {
	stage.join();
} catch (BroadcastException exception) {
	// handle join exception
}

stage.leave();

The Stage class is also where the StageRenderer can be attached:

stage.addRenderer(renderer); // multiple renderers can be added

Strategy

The Stage.Strategy interface provides a way for the host application to communicate the desired state of the stage to the SDK. Three functions need to be implemented: shouldSubscribeToParticipant, shouldPublishFromParticipant, and stageStreamsToPublishForParticipant. All are discussed below.

Subscribing to Participants

Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

When a remote participant joins the stage, the SDK queries the host application about the desired subscription state for that participant. The options are NONE, AUDIO_ONLY, and AUDIO_VIDEO. When returning a value for this function, the host application does not need to worry about the publish state, current subscription state, or stage connection state. If AUDIO_VIDEO is returned, the SDK waits until the remote participant is publishing before subscribing, and it updates the host application through the renderer throughout the process.

Here is a sample implementation:

@Override
Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return Stage.SubscribeType.AUDIO_VIDEO;
}

This is the complete implementation of this function for a host application that always wants all participants to see each other; e.g., a video chat application.

More advanced implementations also are possible. Use the userInfo property on ParticipantInfo to selectively subscribe to participants based on server-provided attributes:

@Override
Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	switch(participantInfo.userInfo.get(“role”)) {
		case “moderator”:
			return Stage.SubscribeType.NONE;
		case “guest”:
			return Stage.SubscribeType.AUDIO_VIDEO;
		default:
			return Stage.SubscribeType.NONE;
	}
}

This can be used to create a stage where moderators can monitor all guests without being seen or heard themselves. The host application could use additional business logic to let moderates see each other but remain invisible to guests.

Configuration for Subscribing to Participants

SubscribeConfiguration subscribeConfigurationForParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

If a remote participant is being subscribed to (see Subscribing to Participants), the SDK queries the host application about a custom subscribe configuration for that participant. This configuration is optional and allows the host application to control certain aspects of subscriber behavior. For information on what can be configured, see SubscribeConfiguration in the SDK reference documentation.

Here is a sample implementation:

@Override
public SubscribeConfiguration subscribeConfigrationForParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
    SubscribeConfiguration config = new SubscribeConfiguration();

    config.jitterBuffer.setMinDelay(JitterBufferConfiguration.JitterBufferDelay.MEDIUM());

    return config;
}

This implementation updates the jitter-buffer minimum delay for all subscribed participants to a preset of MEDIUM.

As with shouldSubscribeToParticipant, more advanced implementations are possible. The given ParticipantInfo can be used to selectively update the subscribe configuration for specific participants.

We recommend using the default behaviors. Specify custom configuration only if there is a particular behavior you want to change.

Publishing

boolean shouldPublishFromParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

Once connected to the stage, the SDK queries the host application to see if a particular participant should publish. This is invoked only on local participants that have permission to publish based on the provided token.

Here is a sample implementation:

@Override
boolean shouldPublishFromParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return true;
}

This is for a standard video chat application where users always want to publish. They can mute and unmute their audio and video, to instantly be hidden or seen/heard. (They also can use publish/unpublish, but that is much slower. Mute/unmute is preferable for use cases where changing visibility often is desirable.)

Choosing Streams to Publish

@Override
List<LocalStageStream> stageStreamsToPublishForParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);
}

When publishing, this is used to determine what audio and video streams should be published. This is covered in more detail later in Publish a Media Stream.

Updating the Strategy

The strategy is intended to be dynamic: the values returned from any of the above functions can be changed at any time. For example, if the host application does not want to publish until the end user taps a button, you could return a variable from shouldPublishFromParticipant (something like hasUserTappedPublishButton). When that variable changes based on an interaction by the end user, call stage.refreshStrategy() to signal to the SDK that it should query the strategy for the latest values, applying only things that have changed. If the SDK observes that the shouldPublishFromParticipant value has changed, it will start the publish process. If the SDK queries and all functions return the same value as before, the refreshStrategy call will not perform any modifications to the stage.

If the return value of shouldSubscribeToParticipant changes from AUDIO_VIDEO to AUDIO_ONLY, the video stream will be removed for all participants with changed returned values, if a video stream existed previously.

Generally, the stage uses the strategy to most efficiently apply the difference between the previous and current strategies, without the host application needing to worry about all the state required to manage it properly. Because of this, think of calling stage.refreshStrategy() as a cheap operation, because it does nothing unless the strategy changes.

Renderer

The StageRenderer interface communicates the state of the stage to the host application. Updates to the host application’s UI usually can be powered entirely by the events provided by the renderer. The renderer provides the following functions:

void onParticipantJoined(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

void onParticipantLeft(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo);

void onParticipantPublishStateChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull Stage.PublishState publishState);

void onParticipantSubscribeStateChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull Stage.SubscribeState subscribeState);

void onStreamsAdded(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams);

void onStreamsRemoved(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams);

void onStreamsMutedChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams);

void onError(@NonNull BroadcastException exception);

void onConnectionStateChanged(@NonNull Stage stage, @NonNull Stage.ConnectionState state, @Nullable BroadcastException exception);

For most of these methods, the corresponding Stage and ParticipantInfo are provided.

It is not expected that the information provided by the renderer impacts the return values of the strategy. For example, the return value of shouldSubscribeToParticipant is not expected to change when onParticipantPublishStateChanged is called. If the host application wants to subscribe to a particular participant, it should return the desired subscription type regardless of that participant’s publish state. The SDK is responsible for ensuring that the desired state of the strategy is acted on at the correct time based on the state of the stage.

The StageRenderer can be attached to the stage class:

stage.addRenderer(renderer); // multiple renderers can be added

Note that only publishing participants trigger onParticipantJoined, and whenever a participant stops publishing or leaves the stage session, onParticipantLeft is triggered.

Publish a Media Stream

Local devices such as built-in microphones and cameras are discovered via DeviceDiscovery. Here is an example of selecting the front-facing camera and default microphone, then return them as LocalStageStreams to be published by the SDK:

DeviceDiscovery deviceDiscovery = new DeviceDiscovery(context);

List<Device> devices = deviceDiscovery.listLocalDevices();
List<LocalStageStream> publishStreams = new ArrayList<LocalStageStream>();

Device frontCamera = null;
Device microphone = null;

// Create streams using the front camera, first microphone
for (Device device : devices) {
	Device.Descriptor descriptor = device.getDescriptor();
	if (!frontCamera && descriptor.type == Device.Descriptor.DeviceType.Camera && descriptor.position = Device.Descriptor.Position.FRONT) {
		front Camera = device;
	}
	if (!microphone && descriptor.type == Device.Descriptor.DeviceType.Microphone) {
		microphone = device;
	}
}

ImageLocalStageStream cameraStream = new ImageLocalStageStream(frontCamera);
AudioLocalStageStream microphoneStream = new AudioLocalStageStream(microphoneDevice);

publishStreams.add(cameraStream);
publishStreams.add(microphoneStream);

// Provide the streams in Stage.Strategy
@Override
@NonNull List<LocalStageStream> stageStreamsToPublishForParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return publishStreams;
}

Display and Remove Participants

After subscribing is completed, you will receive an array of StageStream objects through the renderer’s onStreamsAdded function. You can retrieve the preview from an ImageStageStream:

ImagePreviewView preview = ((ImageStageStream)stream).getPreview();

// Add the view to your view hierarchy
LinearLayout previewHolder = findViewById(R.id.previewHolder);
preview.setLayoutParams(new LinearLayout.LayoutParams(
		LinearLayout.LayoutParams.MATCH_PARENT,
		LinearLayout.LayoutParams.MATCH_PARENT));
previewHolder.addView(preview);

You can retrieve the audio-level stats from an AudioStageStream:

((AudioStageStream)stream).setStatsCallback((peak, rms) -> {
	// handle statistics
});

When a participant stops publishing or is unsubscribed from, the onStreamsRemoved function is called with the streams that were removed. Host applications should use this as a signal to remove the participant’s video stream from the view hierarchy.

onStreamsRemoved is invoked for all scenarios in which a stream might be removed, including:

Because onStreamsRemoved is invoked for all scenarios, no custom business logic is required around removing participants from the UI during remote or local leave operations.

Mute and Unmute Media Streams

LocalStageStream objects have a setMuted function that controls whether the stream is muted. This function can be called on the stream before or after it is returned from the streamsToPublishForParticipant strategy function.

Important: If a new LocalStageStream object instance is returned by streamsToPublishForParticipant after a call to refreshStrategy, the mute state of the new stream object is applied to the stage. Be careful when creating new LocalStageStream instances to make sure the expected mute state is maintained.

Monitor Remote Participant Media Mute State

When a participant changes the mute state of their video or audio stream, the renderer onStreamMutedChanged function is invoked with a list of streams that have changed. Use the getMuted method on StageStream to update your UI accordingly.

@Override
void onStreamsMutedChanged(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo, @NonNull List<StageStream> streams) {
	for (StageStream stream : streams) {
		boolean muted = stream.getMuted();
		// handle UI changes
	}
}

Get WebRTC Statistics

To get the latest WebRTC statistics for a publishing stream or a subscribing stream, use requestRTCStats on StageStream. When a collection is completed, you will receive statistics through the StageStream.Listener which can be set on StageStream.

stream.requestRTCStats();

@Override
void onRTCStats(Map<String, Map<String, String>> statsMap) {
	for (Map.Entry<String, Map<String, string>> stat : statsMap.entrySet()) {
		for(Map.Entry<String, String> member : stat.getValue().entrySet()) {
			Log.i(TAG, stat.getKey() + “ has member “ + member.getKey() + “ with value “ + member.getValue());
		}
	}
}

Get Participant Attributes

If you specify attributes in the CreateParticipantToken endpoint request, you can see the attributes in ParticipantInfo properties:

@Override
void onParticipantJoined(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	for (Map.Entry<String, String> entry : participantInfo.userInfo.entrySet()) {
		Log.i(TAG, “attribute: “ + entry.getKey() + “ = “ + entry.getValue());
	}
}

Continue Session in the Background

When the app enters the background, you may want to stop publishing or subscribe only to other remote participants’ audio. To accomplish this, update your Strategy implementation to stop publishing, and subscribe to AUDIO_ONLY (or NONE, if applicable).

// Local variables before going into the background
boolean shouldPublish = true;
Stage.SubscribeType subscribeType = Stage.SubscribeType.AUDIO_VIDEO;

// Stage.Strategy implementation
@Override
boolean shouldPublishFromParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return shouldPublish;
}

@Override
Stage.SubscribeType shouldSubscribeToParticipant(@NonNull Stage stage, @NonNull ParticipantInfo participantInfo) {
	return subscribeType;
}

// In our Activity, modify desired publish/subscribe when we go to background, then call refreshStrategy to update the stage
@Override
void onStop() {
	super.onStop();
	shouldPublish = false;
	subscribeTpye = Stage.SubscribeType.AUDIO_ONLY;
	stage.refreshStrategy();
}

Enable/Disable Layered Encoding with Simulcast

When publishing a media stream, the SDK transmits high-quality and low-quality video streams, so remote participants can subscribe to the stream even if they have limited downlink bandwidth. Layered encoding with simulcast is on by default. You can disable it by using the StageVideoConfiguration.Simulcast class:

// Disable Simulcast
StageVideoConfiguration config = new StageVideoConfiguration();
config.simulcast.setEnabled(false);

ImageLocalStageStream cameraStream = new ImageLocalStageStream(frontCamera, config);

// Other Stage implementation code

Video-Configuration Limitations

The SDK does not support forcing portrait mode or landscape mode using StageVideoConfiguration.setSize(BroadcastConfiguration.Vec2 size). In portrait orientation, the smaller dimension is used as the width; in landscape orientation, the height. This means that the following two calls to setSize have the same effect on the video configuration:

StageVideo Configuration config = new StageVideo Configuration();

config.setSize(BroadcastConfiguration.Vec2(720f, 1280f);
config.setSize(BroadcastConfiguration.Vec2(1280f, 720f);

Handling Network Issues

When the local device’s network connection is lost, the SDK internally tries to reconnect without any user action. In some cases, the SDK is not successful and user action is needed. There are two main errors related to losing the network connection:

If the first error is received but the second is not, the SDK is still connected to the stage and will try to reestablish its connections automatically. As a safeguard, you can call refreshStrategy without any changes to the strategy method’s return values, to trigger a manual reconnect attempt.

If the second error is received, the SDK’s reconnect attempts have failed and the local device is no longer connected to the stage. In this case, try to rejoin the stage by calling join after your network connection has been reestablished.

In general, encountering errors after joining a stage successfully indicates that the SDK was unsuccessful in reestablishing a connection. Create a new Stage object and try to join when network conditions improve.

Using Bluetooth Microphones

To publish using Bluetooth microphone devices, you must start a Bluetooth SCO connection:

Bluetooth.startBluetoothSco(context);
// Now bluetooth microphones can be used
…
// Must also stop bluetooth SCO
Bluetooth.stopBluetoothSco(context);