Platform Integration Aspects: Blob files

Before version 14 constants and resources were stored inside the generated code. After compiling the generated code, the corresponding data became a fixed part of the resulting binary (the application). This is a simple and convenient approach to deal with constants and resources. A disadvantage of this approach is that it was inevitable to rebuild the entire binary when constants or resources changed. Simply exchanging the data retrospectively without modifying the binary was not possible. Following use cases demonstrate when it can be useful to exchange constants and/or resources without rebuilding the binary:

Customer needs to replace logos and texts (e.g. copyright notice) in a device created by an OEM.

ROM (flash) is not large enough to store all language variants needed in the device. In such case, the set of languages is divided in language packs each containing one or few relevant languages. Before the device is used in a region, the region specific language pack is installed on it.

Entire set of bitmap resources and color constants need to be replaced to support themes without maintaining all theme variants in the binary and occupy up ROM flash memory unnecessarily.

To avoid laborious test repetitions after localization of an application, the binary is not allowed to be changed. The localized texts and bitmaps may not belong to the binary.

In order to meet these or similar requirements, the storage of constants and resources has been redesigned and enhanced by so-called Blob files. From technical point of view, a Blob file stores the binary representation of the affected constants and resources as if they were compiled and linked by a C compiler. Embedded Wizard created application can thus process the content of a Blob file similarly as it does with data which is fixed compiled inside the application. Additionally, the Blob file contains a directory -> a list of constants, resources and languages existing inside this Blob file.

The data, thus, is not part of the binary. The code generated for the application limits to refer the data members existing in the Blob file only. At the runtime when accessing a constant or resource from the Blob file, the directory inside the Blob file is searched for the referenced member and the found member is used. By exchanging the Blob file, the contents of constants and resources can be modified easily. The application itself (the binary) can remain untouched in such case.

Control the storage location

Unless you have configured it explicitly, all constants and resources are stored inside generated code just as it was the case in previous versions. By configuring the attribute Storage of a constant or resource with the value Blob the storage location for that member changes and the member can be stored inside the Blob file. You can configure this attribute for each constant and resource individually:

In case, the storage location should be configured generally for all constants and/or resources, the profile member provides two attributes StorageOfConstants and StorageOfResources. For example, configuring StorageOfConstants with the value BlobIfStrings will have the effect of all constants containing a string to be located inside the Blob file instead of inside the generated code:

Find and prepare the Blob file

Once the code generation is finished, two files blob.bin and blob_names.txt are stored in the output directory together with other generated files:

The first file blob.bin contains the actual data of all affected constants and resources. Assuming the Blob file matches the version of the binary (the binary was compiled from C files generated along with the Blob file), the Blob file is ready to use. You can write this file to flash memory in your device, copy it to a SD card, memory stick, etc..

If the versions of the Blob file and the binary differ, you must perform an additional step to re-link the Blob file to the binary before it can be used in the device. This is the case if the binary was built from C files that do not belong to the version to which the Blob file belongs. Mixing versions of Blob files and binaries without the re-link process will lead to a runtime error.

In order to find a member inside the Blob file, the included members are identified by numbers. Unfortunately, the numbers are not reliable across different projects and project versions. There is no guarantee that number e.g. 5 referring originally some constant e.g. Application::CopyrightText does it also in a newer version after the project has been modified. The numbers may change after each project modification.

To resolve this inconsistency, the numbers used in the Blob file must be updated to match the numbers present in the binary. For this purpose exists the second file blob_names.txt. It contains the names of all members existing in the Blob file together with their corresponding numbers. With the help of this file the numbers found in the Blob file can be converted in the original names. Knowing the names, a reverse operation can be performed to find the matching numbers valid in the version of the binary.

This re-link process is done by the bloblinker.exe tool found in the installation directory of Embedded Wizard Studio. Start this tool from the Windows command line and provide it with three parameters: (1) the blob.bin file to update its numbers, (2) the blob_names.txt file corresponding to the blob.bin file, (3) the blob_names.txt file from the code generated to build the binary. For example:

The updated Blob file is now ready to be used with the binary. You can write this file to flash memory in your device, copy it to a SD card, memory stick, etc..

Use the Blob file

Writing the Blob file to Flash memory of your device or copying it to its file system is just the first step to make the file available. In the next step you have to connect the file to Embedded Wizard's Runtime Environment so that its contents are accessible. You invoke for this purpose the function EwSetBlobData(). You do that at the startup time of the application at the very beginning of the initialization sequence. The function EwSetBlobData() expects two pointers addressing the start and the end of a memory area where contents of the Blob file will be mapped.

Depending on your use case and your device's architecture the memory area will exist in flash ROM, in RAM or it will be a virtual, non physically existing memory address space. In any case the size of this area has to be equal to the size of the Blob file. The following sub sections provide a detailed overview of the typical use cases and the corresponding integration approaches.

Use case 1: Blob file resides in CPU accessible ROM

This is the simplest and most convenient approach. In this approach the Blob file was written to ROM (e.g. Flash memory) and the ROM is fully accessible from the CPU. All you have to do in such case is to provide in the invocation of EwSetBlobData() the start and end addresses of the ROM area containing the Blob file.

Use case 2: Blob file resides in not-accessible ROM

In this use case the Blob file was written to ROM which unfortunately is not accessible by the CPU. Regardless of that you still invoke EwSetBlobData() with the start and the end addresses of the memory area in the non directly accessible ROM. Additionally you have to implement a so-called Flash Area Reader. It is an interface to take care of mapping all accesses to not accessible ROM to RAM memory. See the function EwRegisterFlashAreaReader() for more details.

Important: Register the flash reader before invoking EwSetBlobData()!

Use case 3: Blob file resides on a file system and you have enough RAM

This approach is very straightforward. The Blob file resides on a file system. At the startup time, the application reads the complete file and copies its content into RAM. This use case assumes there is enough RAM in your device to accommodate the entire Blob file. The memory itself could be allocated on your memory heap or you could reserve the memory area explicitly by appropriately configuring linker sections. In any case, you provide the start and the end addresses of the RAM area containing the Blob file's copy to the function EwSetBlobData().

Use case 4: Blob file resides on a file system and RAM is scarce

This approach is probably the most complex one. The Blob file resides on a file system. The RAM, however, is scarce so that loading the complete Blob file into the RAM in advance is not possible. As first step estimate which CPU address space in your device is free and large enough to enclose the complete Blob file. You can check the linker file or the resulting MAP file at the end of the build process for this purpose. The start and the end addresses of this virtual address space should be provided then in the invocation of EwSetBlobData().

In the second step implement a so-called Flash Area Reader. Usually it is an interface to take care of mapping all accesses to not accessible ROM to RAM memory. However, it is also well suitable to map accesses to files, etc. too. It's finally up to you what the reader does. For more details concerning the reader, see the function EwRegisterFlashAreaReader().

In this use case, you will register the reader to take care of all accesses to the above mentioned virtual address space reserved for the Blob file. The reader calculates from the distance between the requested address and the start address of the memory area the corresponding offset within the file. It loads the addressed fragment from the file in a RAM area and returns a pointer to this area.

Important: Register the flash reader before invoking EwSetBlobData()

Example 1: Individualization of existing devices

Let's assume there is a finished device, e.g. developed by an OEM. The application for this device is already implemented and the resulting binary is stored in the device. Constant and resources (at least the relevant part of them) are stored in a Blob file. Now let's assume there is a customer of OEM and the customer desired to individualize the device, it means to exchange some welcome strings, copyright notice, logos, etc. without modifying the binary in the device:

Step 1: The customer opens the original OEM's project in Embedded Wizard Studio. Note: this step assumes that the OEM has provided the Embedded Wizard Project to the customer.

Step 2: All relevant constants (e.g. containing the copyright notice) and resources (e.g. logos) are modified.

Step 3: From this modified version the customer generates code.

Step 4: In the generated code the customer looks for the files blob.bin and blob_names.txt.

Step 5: By using the the bloblinker.exe tool, the customer adapts the new Blob file to match the version of the binary. Note: this step assumes that the OEM has provided the blob_names.txt file matching the version of the binary.

Step 6: Finally, the modified Blob file can copied to the device or written to its Flash memory, etc.

Example 2: Managing language packs

Let's assume, the customer's application is localized for many languages and text fragments in all language variants are too large to be stored in the device. The device can store only one or few language variants at the same time. To manage this situation it is necessary to divide the languages in language packs and use only the language pack which is relevant for the region where the device is used:

Step 1: First the customer has to complete the development and the localization of the application. Constants containing localized strings and language dependent resources should be configured to be stored in the Blob file.

Step 2: From this project version the customer generates code.

Step 3: From the generated code the customer builds the binary. The binary can be copied to the device.

Step 4: Also from the generated code the customer saves the file blob_names.txt. Let's name the file blob_names_binary.txt. It means the file corresponds to the version of the binary.

Step 5: Now, the customer disables the code generation for all language members existing in the project except the language or languages which should belong to the first language pack.

Step 6: The customer generates again the code.

Step 7: In the generated code the customer looks for the files blob.bin and blob_names.txt.

Step 8: By using the the bloblinker.exe tool, the customer adapts the new Blob file to match the version of the binary. Here the above saved blob_names_binary.txt file is used.

Step 9: Finally, the modified Blob file is ready to be used in the device.

Step 10: To create more language packs, the customer repeats the steps 5-9.

Please note: when using language packs, the language support at the runtime of the application is limited to languages existing in the Blob file only. This may confuse the user if your application implements a menu system providing the entire list of languages. Selecting a language not available in the Blob file will then has no effect and the default language is used automatically. Therefore it is recommendable to enhance the language selection menu by functionality to detect which languages are available.

A simplest approach to achieve that could be the usage of constants of type bool within the project. Each constant corresponds to one language. When the language is available in the language pack, the constants should contain the value true. Languages which are excluded from the language pack should be signed with the corresponding constant containing the value false. At the initialization time of the menu (when the menu is composed of menu items representing the languages) limit to add only those languages which have the constant initialized with true. This approach requires the constants to be modified manually before generating code for a language pack.

Example 3: Add new languages to existing application

Let's assume, the application should be localized for different languages. The languages, however, are unknown at the design time. The customer wants the languages to be added later after the application's development is finished. This approach is similar to the above described Example 2: Managing language packs. However, it has a peculiarity that there are no concrete language names predefined in the application. The languages are unknown. Instead of using concrete language names (e.g. German, English, Spanish, ...) the application will use place holders for the future languages:

Step 1: First complete the development of the application. Strings which are intended to be localized later (to depend on the selected language) have to be represented by constant members. All language dependent constants and resources should be configured to be stored in the Blob file.

Step 2: Then decide in advance how many languages the application will maximally support. Accordingly, the application should be enhanced by the corresponding number of language members. These will act as place holders for the further languages. Name the language members, for example, Language_1, Language_2, ... Language_N.

Step 3: Additionally, add one constant member for each planed language. That means, if the application is intended to manage up to 10 languages, add 10 constants to your project. Configure the constants to store strings. Name the constants Language_1, Language_2, ... Language_N according to how the languages are named. These constants are intended to store the real names of the corresponding languages.

Step 4: Wherever the name of a language is displayed in your application, use the above mentioned constants Language_1, Language_2, ... Language_N.

Step 5: To switch the language, just assign the corresponding language name Language_1, Language_2, ... Language_N to the global variable language.

Step 6: If the application displays a list of available languages, implement the list to be composed dynamically of the available languages only. For this purpose you could implement a loop to test the contents of the above mentioned Language_1, Language_2, ... Language_N constants one after another. If a constants is empty, then this language is not assigned and should be ignored. In other case use the string stored in the constant as name to display in the language list.

Step 7: From this project version generate code.

Step 8: From the generated code build the binary. The binary can be copied to the device.

Step 9: Also from the generated code save the file blob_names.txt. Let's name the file blob_names_binary.txt. It means the file corresponds to the version of the binary.

Step 10: Now, the customer can localize the application. For example Language_1 could be localized for the language German. In such case, the customer adapts in all affected constants the language variant Language_1 to contain German strings. Similarly, all language specific resources can be modified and if desired font ranges enhanced to include glyphs necessary in the just added language. Also, the above mentioned constant Language_1 should be initialized with the name of the language, e.g. Deutsch (means German).

Step 11: The customer generates again the code.

Step 12: In the generated code the customer looks for the files blob.bin and blob_names.txt.

Step 13: By using the the bloblinker.exe tool, the customer adapts the new Blob file to match the version of the binary. Here the above saved blob_names_binary.txt file is used.

Step 14: Finally, the modified Blob file is ready to be used in the device. The file contains the just added language, e.g. German.

Step 15: To add further languages, repeat the steps 10-14.