Integrating Video Content: VideoPlayer Applet
The VideoPlayer Applet is a separate Embedded Wizard unit included in each VideoPlayer Applet Add-On Package. It provides the class VideoPlayerApplet, derived from Views::Applet, to integrate video content into a GUI application via a platform-managed surface that is composited independently of the Embedded Wizard GUI layer. Unlike ExternVideo, which delivers decoded frames as bitmaps into the GUI pipeline, the VideoPlayer Applet renders video directly into a separate video layer - for example a Wayland SubSurface or a dedicated hardware video plane. The GUI application controls the position and size of the video area and drives the player via the VideoPlayerApplet object — starting, pausing, and stopping playback and setting the video source. In return, the platform reports the player state back to the GUI application along with playback data such as frame rate, total number of frames, and current frame number. The video content itself never passes through the Embedded Wizard bitmap pipeline.
The VideoPlayerApplet class is derived from Views::Applet, inheriting all its properties and behavior. This is relevant primarily for the Chora side of the integration — how the Applet is embedded in a GUI component, how its position and size are bound, and how playback is controlled from Chora. This is described in the article using-video-player-applet (available soon). The present article focuses on the platform-specific C interface that the Applet relies on.
Architecture of the VideoPlayer Applet approach: the Applet controls position and size of the video area while the platform renders decoded frames directly into a separate surface managed by the display compositor. No video data passes through the Embedded Wizard bitmap pipeline.
The VideoPlayerAppletNative interface
The interface is declared in VideoPlayerAppletNative.h and must be implemented for each target platform. All interface functions are called by the generated Embedded Wizard application code — the GUI application only needs to control playback and position via the VideoPlayerApplet object, and the actual rendering is handled entirely by the platform.
The interface is based on an opaque instance handle XVideoPlayerAppletNativeInst that your implementation allocates in VideoPlayerAppletNativeCreate() and frees in VideoPlayerAppletNativeDestroy(). All other functions receive this handle as their first parameter.
The following table gives an overview of all interface functions:
Function |
Description |
|---|---|
VideoPlayerAppletNativeCreate() |
Allocates the instance and prepares the platform video surface at the given position and size in GUI coordinates. Returns the instance handle. |
VideoPlayerAppletNativeDestroy() |
Stops playback, releases all resources, and frees the instance. |
VideoPlayerAppletNativeSetSource() |
Sets the video source URI and prepares the decoding backend. Must only be called in state STOPPED. Returns non-zero on success. |
VideoPlayerAppletNativeStart() |
Starts or resumes video playback. |
VideoPlayerAppletNativePause() |
Pauses video playback. |
VideoPlayerAppletNativeStop() |
Stops video playback and resets the decoding backend. |
VideoPlayerAppletNativeProcess() |
Called periodically from the EW main loop. Processes pending events from the decoding backend. Returns non-zero if further immediate processing is requested. |
VideoPlayerAppletNativeGetPlayerState() |
Returns the current player state. |
VideoPlayerAppletNativeGetFps() |
Returns the current playback frame rate. |
VideoPlayerAppletNativeGetTotalNbOfFrames() |
Returns the total number of frames of the current media. |
VideoPlayerAppletNativeGetCurrentFrameNumber() |
Returns the current frame number. |
VideoPlayerAppletIsSeparateLayer() |
Returns non-zero if the implementation renders into a separate display layer rather than into the EW GUI surface. |
VideoPlayerAppletNativeSetName() |
Assigns a debug name to the instance. Useful when multiple instances are active simultaneously. |
VideoPlayerAppletNativeGetBitmap() |
Returns the current video frame as XBitmap*, or NULL if the implementation uses a separate layer. |
VideoPlayerAppletNativeNeedUpdate() |
Returns non-zero if the GUI layer needs to be redrawn, e.g. because a new bitmap frame is available. |
VideoPlayerAppletNativeGetUpdateArea() |
Returns the region of the GUI layer that needs to be redrawn. |
Player states
The player state machine uses the following states:
State |
Description |
|---|---|
UNDEFINED |
Initial state before any initialization. |
STOPPED |
No active playback. SetSource may be called. |
BUFFERING |
The backend is buffering data, e.g. from a network stream. |
PLAYING |
Video is actively playing. |
PAUSED |
Playback is paused. Also the state after a successful SetSource call. |
END_OF_STREAM_REACHED |
Playback reached the end of the media. |
ERROR |
An unrecoverable error occurred. |
Separate layer vs. bitmap delivery
The interface supports two fundamentally different rendering modes, which can coexist in different implementations of the same interface:
•If VideoPlayerAppletIsSeparateLayer() returns non-zero, video is rendered into a platform-managed surface outside the GUI bitmap pipeline. GetBitmap() returns NULL and NeedUpdate() returns 0 in this case. This is the intended use case for the VideoPlayer Applet.
•If VideoPlayerAppletIsSeparateLayer() returns zero, the implementation delivers frames as XBitmap* via GetBitmap() into the GUI layer — inherited from the Views::Applet base class.
If your use case requires delivering video frames as bitmaps into the GUI layer, the ExternVideo Interface is the better choice. It provides the same bitmap-based frame delivery with additional capabilities such as zoom, rotation, warping, and perspective transformation that are not available via the VideoPlayer Applet. The bitmap delivery mode of the VideoPlayer Applet is primarily useful for prototyping intrinsic modules as described in section 3.
Example: GStreamer with Wayland SubSurface
The following example implements the VideoPlayerAppletNative.h interface using GStreamer as the decoding backend and a Wayland SubSurface as the render target. GStreamer's waylandsink element renders decoded frames directly into the SubSurface without any involvement of the GUI bitmap pipeline.
This example covers video playback only. Audio decoding is not supported.
Instance structure
typedef struct XVideoPlayerAppletNativeInstance { char Name[ VIDEO_PLAYER_APPLET_NAME_MAX_LEN + 1 ]; XVideoPlayerAppletNativePlayerState PlayerState; uint32_t FrameRate; uint32_t NbOfFrames; uint32_t Speed; void * SubSurface; void * VideoSurface; GstElement * GstPipeline; XPoint Position; XPoint Size; } XVideoPlayerAppletNativeInstance;
SubSurface is the platform-level Wayland SubSurface object returned by GfxSystemCreateSubSurface(). VideoSurface is the underlying wl_surface used as the render target for the waylandsink.
Creating the instance and SubSurface
XVideoPlayerAppletNativeInst VideoPlayerAppletNativeCreate( XPoint aPosition, XPoint aSize ) { XVideoPlayerAppletNativeInst instance; void * subSurface; subSurface = GfxSystemCreateSubSurface( aSize.X, aSize.Y ); if ( !subSurface ) return NULL; instance = ( XVideoPlayerAppletNativeInst ) EwAlloc( sizeof( XVideoPlayerAppletNativeInstance )); if ( !instance ) return NULL; memset( instance, 0, sizeof( XVideoPlayerAppletNativeInstance )); if ( !gst_is_initialized()) gst_init( 0, 0 ); instance->SubSurface = subSurface; instance->VideoSurface = GfxSystemSubSurfaceGetSurface( subSurface ); instance->Position = aPosition; instance->Size = aSize; instance->PlayerState = VIDEO_PLAYER_APPLET_PLAYER_STATE_STOPPED; instance->FrameRate = 50; return instance; }
The Wayland SubSurface is created via GfxSystemCreateSubSurface(), a platform BSP function that allocates a wl_subsurface and positions it beneath the main EW GUI surface in the compositor stacking order. GfxSystemSubSurfaceGetSurface() returns the associated wl_surface pointer that is later passed to waylandsink as its render target.
GStreamer is initialized lazily on the first call to VideoPlayerAppletNativeCreate() if it has not been initialized already. If your application initializes GStreamer elsewhere at startup, this call is a no-op.
Setting the source and building the pipeline
int VideoPlayerAppletNativeSetSource( XVideoPlayerAppletNativeInst aInstance, XString aSource ) { char source[ MAX_SOURCE_LENGTH + 1 ]; char pipeline[ MAX_GST_PIPELINE_LENGTH + 1 ]; XGfxSystemInfo gfxSystemInfo; GError * err = NULL; GstElement * videosink; float scaleX, scaleY; int x, y, w, h; EwStringToAnsi( aSource, source, MAX_SOURCE_LENGTH, '_' ); snprintf( pipeline, MAX_GST_PIPELINE_LENGTH, "uridecodebin uri=%s ! videoconvert ! waylandsink name=sink", source ); aInstance->GstPipeline = gst_parse_launch( pipeline, &err ); if ( err ) goto onError; /* pass the Wayland display handle to the pipeline */ GfxSystemGetInfo( &gfxSystemInfo ); GstContext * context = gst_context_new( "GstWaylandDisplayHandleContextType", TRUE ); gst_structure_set( gst_context_writable_structure( context ), "handle", G_TYPE_POINTER, gfxSystemInfo.NativeDisplay, NULL ); gst_element_set_context( aInstance->GstPipeline, context ); /* assign the SubSurface as render target and set the render rectangle */ videosink = gst_bin_get_by_name( (GstBin *)aInstance->GstPipeline, "sink" ); gst_video_overlay_set_window_handle( GST_VIDEO_OVERLAY( videosink ), (guintptr) aInstance->VideoSurface ); scaleX = ( float )gfxSystemInfo.Width / EwScreenSize.X; scaleY = ( float )gfxSystemInfo.Height / EwScreenSize.Y; x = aInstance->Position.X * scaleX; y = aInstance->Position.Y * scaleY; w = aInstance->Size.X * scaleX; h = aInstance->Size.Y * scaleY; gst_video_overlay_set_render_rectangle( GST_VIDEO_OVERLAY( videosink ), x, y, w, h ); wl_surface_commit( aInstance->VideoSurface ); VideoPlayerAppletNativeNotifyPlayerState( aInstance, VIDEO_PLAYER_APPLET_PLAYER_STATE_PAUSED ); return 1; onError: VideoPlayerAppletNativeNotifyPlayerState( aInstance, VIDEO_PLAYER_APPLET_PLAYER_STATE_ERROR ); return 0; }
The GStreamer pipeline uridecodebin uri=... ! videoconvert ! waylandsink name=sink handles container demuxing, codec detection, and decoding automatically. videoconvert ensures the decoded frames are in a format accepted by waylandsink. The sink renders directly into the Wayland SubSurface — no pixels pass through the CPU or the GUI bitmap pipeline.
The render rectangle is specified in physical display coordinates. Since the GUI may be scaled relative to the physical display resolution, the position and size passed in GUI coordinates are multiplied by scaleX / scaleY before being passed to gst_video_overlay_set_render_rectangle().
The videoconvert element performs a color space conversion if the decoder output format is not directly supported by waylandsink. On platforms where the display controller supports YUV compositing natively, this element can be omitted or replaced by a hardware-accelerated converter, keeping the video data in YUV format all the way to the display.
Processing GStreamer messages
int VideoPlayerAppletNativeProcess( XVideoPlayerAppletNativeInst aInstance ) { GstBus * pBus; GstMessage * pMsg; if ( !aInstance || !aInstance->GstPipeline ) return 0; pBus = gst_element_get_bus( aInstance->GstPipeline ); while (( pMsg = gst_bus_pop_filtered( pBus, GST_MESSAGE_ANY ))) VideoPlayerAppletGstMsgHandler( aInstance, pMsg ); g_object_unref( pBus ); return 0; }
VideoPlayerAppletNativeProcess() is called from the EW main loop on every update cycle. It drains the GStreamer message bus, handling end-of-stream events, state change notifications, and errors. State changes detected here are reported back to the Applet via VideoPlayerAppletNativeNotifyPlayerState(), which updates aInstance->PlayerState and allows the Chora application to react accordingly.
VideoPlayer Applet and Prototyping
Since the VideoPlayer Applet interface is platform-specific and renders video into a platform-managed surface outside the GUI pipeline, it is not available in the Embedded Wizard Prototyper or Composer window by default. When a VideoPlayerApplet object attempts to start playback during prototyping, a runtime warning is reported.
★Option 1: Ignore the warning. The video area remains empty during prototyping. The GUI layout and control flow can still be tested without any video output. When the generated code is integrated on the target system and the VideoPlayerAppletNative interface is implemented, video playback will work as expected. If the interface functions are missing at link time, the linker will report unresolved external symbol ... errors.
★Option 2: Provide an intrinsic module that renders video into a separate window on the Windows desktop, simulating the behavior of a hardware video layer or Wayland SubSurface. VideoPlayerAppletIsSeparateLayer() returns non-zero, and video playback is visible in a separate window positioned to match the video area in the GUI. This closely reflects the actual target behavior and is the most realistic prototyping option.
★Option 3: Provide an intrinsic module that decodes video frames and delivers them as bitmaps via VideoPlayerAppletNativeGetBitmap() into the GUI layer. VideoPlayerAppletIsSeparateLayer() returns zero in this case. This is simpler to implement and sufficient for testing GUI layout and playback control flow, but does not simulate the separate layer behavior of the target platform.
How intrinsic modules are implemented is described in the chapter Implementing Prototyper intrinsics.
Available add-on packages
Ready-to-use implementations of the VideoPlayer Applet interface are available as add-on packages for specific build environments. Each package contains the complete platform-specific implementation, example applications, and a ReadMe with build and integration instructions.
VideoPlayer GStreamer SubSurface Add-Ons: The STM32MP1-OpenGL-Wayland-AddOn-VideoPlayer-GStreamer-SubSurface and RasPi-CM4-Wayland-AddOn-VideoPlayer-GStreamer-SubSurface packages implement the VideoPlayer Applet interface using GStreamer with Wayland SubSurface on their respective target platforms. Video is rendered directly into a Wayland SubSurface beneath the GUI surface, bypassing the GUI bitmap pipeline entirely.
Further add-on packages for additional target platforms may be available. If you are interested in a VideoPlayer Applet add-on package for your platform, please contact our support team at support@embedded-wizard.de with a brief description of your requirements and the target platform you are using.
IMPORTANT
The availability and hardware-specific details of add-on packages may change between releases. Always refer to the ReadMe file included in each package for build and integration instructions specific to your build environment version.