The Mosaic framework provides the class Resources::ExternVideo to integrate video content into a GUI application at runtime. Video frames are delivered as Embedded Wizard bitmaps, which the GUI engine can display, scale, rotate, or apply perspective transformations to - just like any other bitmap resource.

This functionality depends on a platform-specific interface you have to implement in your target system. The interface bridges between the Embedded Wizard runtime and your chosen video decoder - whether a software library like libjpeg, a hardware-accelerated JPEG decoder, or a full multimedia framework like GStreamer. You are free to use any decoder available on your platform. The interface simply expects you to deliver each decoded frame as an XBitmap*.

This article describes how to implement this interface. For a general overview of video integration approaches and guidance on whether ExternVideo is the right choice for your use case, see Integrating video content. How Resources::ExternVideo is used within a GUI application is described in the article using-extern-video (available soon).

Overview of the components involved in video playback and how they interact: the GUI application layer with Resources::ExternVideo and Image View, the C glue layer implementing the ExternVideo interface, and the platform layer containing the video decoder, data source and graphics system.

Before implementing the ExternVideo interface, one key decision needs to be made: how to hand the decoded frame over to Embedded Wizard as an XBitmap*:

Aspect	Copy approach (EwLockBitmap)	Zero-Copy approach (EwCreateBitmapWrapped)
Works on any graphics system	Yes	No - requires graphics system support (e.g. OpenGL/EGL)
Pixel transfer by CPU	Yes	No
Color conversion required in decoder	Yes - frame must be in native bitmap format before writing into bitmap memory	Depends - can be performed by the graphics system if supported
EW_SURFACE_ROTATION must be considered	Yes - pixels must be written in rotated order	No - handled by the graphics system

With the copy approach, the decoded frame is written directly into the memory of an Embedded Wizard bitmap using EwLockBitmap() and EwUnlockBitmap(). This works on any platform and any graphics system, but requires the decoder to output pixels in the native bitmap format, and the pixel data is transferred through CPU memory.

With the zero-copy approach, the decoder delivers its output in an externally managed buffer - for example an OpenGL texture - which is wrapped into an Embedded Wizard bitmap using EwCreateBitmapWrapped(). No pixel data is copied through the CPU. Color conversion from YUV to RGB, if needed, can be performed by the graphics system during texture upload - provided the platform supports this. On platforms where the graphics system performs this conversion efficiently in hardware (e.g. NXP i.MX8MP), this approach significantly reduces CPU load compared to a software color conversion in the copy approach.

The ExternVideo interface consists of a fixed set of C functions that are called by the generated Embedded Wizard application code during the lifetime of a video playback. All functions operate on an opaque XHandle representing the decoder instance. You allocate and initialize this handle in EwExternVideoCreate() and release it in EwExternVideoDestroy().

You must implement all of the following functions and ensure they are compiled and linked with your application. The interface is declared in the header file EwExternVideo.h:

Function	Short description
XHandle EwExternVideoCreate( XString aSource )	Creates and initializes a new decoder instance. The instance is paused by default. The first frame should already be decoded and available.
void EwExternVideoDestroy( XHandle aInstance )	Stops playback and releases all resources.
void EwExternVideoResume( XHandle aInstance )	Starts or resumes video playback.
void EwExternVideoPause( XHandle aInstance )	Pauses playback. The decoder stops at the latest decoded frame.
XBitmap* EwExternVideoGetNewBitmap( XHandle aInstance )	Returns the next decoded frame as a bitmap, or NULL if no new frame is available yet.
int EwExternVideoGetState( XHandle aInstance )	Returns the current playback state. See EW_EXTERN_VIDEO_STATE_XXX defines.
XRect EwExternVideoGetBounds( XHandle aInstance )	Returns the video frame dimensions as an XRect.
int EwExternVideoGetNoOfFrames( XHandle aInstance )	Returns the total number of frames in the video.
int EwExternVideoGetCurrentFrame( XHandle aInstance )	Returns the currently displayed frame number (first frame = 0).
int EwExternVideoSetCurrentFrame( XHandle aInstance, int aCurrentFrame )	Seeks to the specified frame. When paused, the decoder should decode and provide the new frame immediately.
float EwExternVideoGetFrameRate( XHandle aInstance )	Returns the frame rate in frames per second.
int EwExternVideoGetSpeed( XHandle aInstance )	Returns the current playback speed as a percentage value (100 = normal speed, -400 = 4x rewind).
int EwExternVideoSetSpeed( XHandle aInstance, int aSpeed )	Sets the playback speed as a percentage value.
int EwExternVideoGetError( XHandle aInstance )	Returns the current error state. See EW_EXTERN_VIDEO_ERROR_XXX defines.

An ExternVideo instance moves through a defined set of states during its lifetime. The function EwExternVideoGetState() returns one of the following values:

State constant	Meaning
EW_EXTERN_VIDEO_STATE_UNKNOWN	State not yet determined.
EW_EXTERN_VIDEO_STATE_BUFFERING	Decoder is initializing or buffering data (e.g. network stream).
EW_EXTERN_VIDEO_STATE_READY	Decoder is initialized, first frame is available, playback not yet started.
EW_EXTERN_VIDEO_STATE_RESUMING	Playback transitioning to playing (transient state).
EW_EXTERN_VIDEO_STATE_PLAYING	Playback is active and frames are being delivered.
EW_EXTERN_VIDEO_STATE_PAUSED	Playback is paused.
EW_EXTERN_VIDEO_STATE_END_OF_STREAM	The end of the media has been reached.
EW_EXTERN_VIDEO_STATE_ERROR	A fatal error occurred. Use EwExternVideoGetError() to obtain details.

After EwExternVideoCreate() the instance should be in READY state (or BUFFERING if initialization requires time), with the first frame already decoded and available via EwExternVideoGetNewBitmap(). EwExternVideoResume() transitions to PLAYING, EwExternVideoPause() to PAUSED. When the last frame has been delivered the implementation should report END_OF_STREAM.

When the state EW_EXTERN_VIDEO_STATE_ERROR is reported, the function EwExternVideoGetError() returns one of the following error codes:

Error constant	Meaning
EW_EXTERN_VIDEO_ERROR_NONE	No error. Normal state during successful playback.
EW_EXTERN_VIDEO_ERROR_NAME_NOT_FOUND	The video source URI could not be found or resolved.
EW_EXTERN_VIDEO_ERROR_ACCESS_ERROR	The video source exists but could not be opened or accessed (e.g. permission denied, I/O error).
EW_EXTERN_VIDEO_ERROR_CONTENT_ERROR	The video source was opened but its content is invalid or unsupported (e.g. corrupt file, unsupported codec or container).
EW_EXTERN_VIDEO_ERROR_OTHER_ERROR	An unspecified error occurred that does not fit the categories above.

Embedded Wizard calls EwExternVideoGetNewBitmap() periodically from within its main processing loop. Your implementation checks whether a new decoded frame is available and returns it as an XBitmap*, or returns NULL if no new frame is ready yet - in which case the previously returned bitmap remains valid and continues to be displayed.

When using the copy approach (EwLockBitmap), the physical layout of the Embedded Wizard framebuffer must be taken into account. If EW_SURFACE_ROTATION is set to 90 or 270, the framebuffer is stored physically rotated. Embedded Wizard bitmaps follow this same physical layout. A video decoder, however, always delivers unrotated frames.

This has two consequences for your implementation:

Bitmap dimensions must be swapped. When allocating the bitmap with EwCreateBitmap(), width and height must be exchanged for 90 and 270 degree rotation:

Pixels must be written in rotated order. When writing the decoded frame into the locked bitmap memory, the pixel data must be transposed accordingly - rows and columns are exchanged compared to the natural unrotated frame layout. This operation must be performed for every frame and can be CPU-intensive.

One way to reduce this per-frame cost is to pre-rotate the video content during encoding. If the video material is already stored in the rotated orientation, the decoder output matches the expected bitmap layout directly and a simple row-by-row copy suffices. This optimization is specific to workflows where the encoding step can be controlled.

When using the zero-copy approach (EwCreateBitmapWrapped with e.g. an OpenGL texture), EW_SURFACE_ROTATION does not need to be considered in the decoder implementation. The graphics system handles the rotation transparently.

The following example demonstrates a complete implementation of the ExternVideo interface for playing MJPEG video stored in AVI container files, using the copy approach. It is suitable for microcontroller targets with or without an RTOS as well as for Linux systems where no OpenGL-based zero-copy path is available.

The example uses three helper components that are part of the ExternVideo add-on packages:

EwInputStream abstracts the data source. It can be replaced by a custom implementation to read from SD card, SPI Flash, network, or any other source.

EwAviParser parses the AVI container and extracts individual MJPEG frames with their presentation timestamps.

EwJpegDec abstracts the JPEG decoder. The same API supports both software decoding via libjpeg and hardware-accelerated decoding (e.g. the hardware JPEG unit of the STM32H747), making it straightforward to switch between the two without changing the surrounding integration code.

The following code shows the essential structure of EwExternVideoCreate(). Error handling and further details are omitted for clarity — the complete implementation is included in the add-on packages.

The function EwExternVideoGetNewBitmap() locks one of the pre-allocated bitmaps, calls the JPEG decoder to write the next frame directly into the bitmap memory at lock->Pixel1 with the row stride lock->Pitch1Y, then unlocks and returns it:

The decoder writes directly into the locked bitmap memory, avoiding any intermediate copy. The EwJpegDec abstraction ensures that EwExternVideoDecodeNextFrame() works identically regardless of whether a software or hardware JPEG decoder is used underneath.

The MJPEG implementation supports two playback timing strategies.

In time-based mode (the default during active playback), the decoder tracks the presentation timestamp of each frame. If decoding is fast enough, it waits until the correct wall-clock time before returning the frame. If the system is under load and decoding falls behind, frames can be dropped to keep the playback advancing at the correct real-time speed. This behavior is controlled by the compile-time switch:

When frame dropping is disabled, every frame is presented in sequence regardless of timing, resulting in slow-motion playback under load. When enabled, the video advances at the correct speed at the cost of visible skips if the system is consistently overloaded.

In frame-by-frame mode, EwExternVideoSetCurrentFrame() can be used to seek to a specific frame. The decoder seeks to the requested frame, decodes exactly that one frame, and delivers it via the next call to EwExternVideoGetNewBitmap(). EwExternVideoSetCurrentFrame() can be called regardless of the current playback state — during active playback as well as while paused.

The MJPEG example described above is designed to be portable. Real deployments often benefit from platform-specific enhancements.

Hardware JPEG decoding: The EwJpegDec abstraction layer allows replacing the libjpeg software decoder with a BSP wrapper around a hardware JPEG peripheral. On targets such as the STM32H747, the hardware JPEG unit can decode frames directly into a memory-mapped output buffer. When DMA is configured to write into the locked Embedded Wizard bitmap memory, the entire decode-to-display path requires no CPU involvement for the pixel data itself.

Hardware-accelerated MCU reordering, chroma upsampling, and color space conversion: After entropy decoding, JPEG processing involves MCU (Minimum Coded Unit) reordering, chroma upsampling (for 4:2:0 and 4:2:2 streams), and color space conversion from YCbCr to RGB. On some SoCs, dedicated hardware accelerates one or more of these stages - for example, the DMA2D peripheral on STM32 can perform pixel format conversion and blending. Your EwJpegDec implementation can leverage such hardware to reduce CPU load and improve throughput.

Multi-threaded decoding with double buffering: When the target supports RTOS or POSIX threads, decoding and display can be decoupled. The MJPEG implementation supports this via the compile-time switch EW_JPEG_DEC_MULTI_THREADING. When enabled, two bitmap buffers are used: a dedicated decoding thread continuously decodes frames into the back buffer while the Embedded Wizard main thread retrieves the front buffer via EwExternVideoGetNewBitmap(). Access is synchronized via semaphores so that both threads can operate without stalling each other.

On Linux targets with OpenGL/EGL support, GStreamer provides a flexible video decoding pipeline. The EwExternVideoGStreamer.c implementation demonstrates the zero-copy approach: decoded frames are delivered as OpenGL textures and wrapped directly into Embedded Wizard bitmaps via EwCreateBitmapWrapped() - no pixel data passes through the CPU.

The GStreamer implementation runs a pipeline in a dedicated GLib main-loop thread. The following pipeline is used as a starting point:

uridecodebin handles source demuxing and decoding for a wide range of formats. glupload uploads the decoded frame to GPU memory as an OpenGL texture. If the graphics system supports it, color space conversion from YUV to RGB is performed during this upload step in hardware - avoiding a CPU-based conversion entirely. A fakesink with a handoff callback captures the resulting buffer for Embedded Wizard to consume.

For the glupload element to share textures with Embedded Wizard, both must use the same OpenGL context. During EwExternVideoCreate(), the EGL display and context are retrieved from the Embedded Wizard graphics subsystem via GfxSystemGetInfo() and wrapped as a GstGLContext:

When a new buffer arrives at the fakesink, its OpenGL texture handle is extracted and wrapped into an Embedded Wizard bitmap using EwCreateBitmapWrapped(). A DeleteBitmapProc callback ensures that the GStreamer buffer reference is released when Embedded Wizard no longer needs the bitmap:

Because Embedded Wizard treats the wrapped OpenGL texture as a regular bitmap, all GUI compositing operations remain available without any additional effort: the video can be scaled, rotated, warped, or overlaid with GUI elements.

On Linux systems where OpenGL-based color conversion is not sufficiently performant, an alternative approach using a separate video layer is available via the VideoPlayer Applet. See VideoPlayer Applet for details.

Since the ExternVideo interface is platform-specific and tightly coupled to the target hardware, it is not available in the Embedded Wizard Prototyper or Composer window by default. When a Resources::ExternVideo object attempts to start playback during prototyping, a runtime warning is reported.

Ready-to-use implementations of the ExternVideo interface are provided as add-on packages for specific build environments. Each package contains the complete glue code, example applications, and a ReadMe with build and integration instructions.

STM32H747-Discovery ExternVideo MJPEG Add-On: Implements the ExternVideo interface using the hardware JPEG decoder of the STM32H747. Video streams are stored as built-in data in external QSPI Flash memory. The data source can be replaced by providing a custom EwInputStream implementation (e.g. for SD card access).

i.MX8M Mini ExternVideo GStreamer Add-On: Implements the ExternVideo interface using GStreamer with OpenGL texture transport on the NXP i.MX8M Mini. Supports FullHD video playback via hardware-accelerated decoding with full GUI integration.

Further add-on packages for additional target platforms may be available. If you are interested in an ExternVideo add-on package for your platform, please contact our support team at support@embedded-wizard.de with a brief description of your requirements and the target platform you are using.