Pushing PNG Images to a Display

Sometimes a display is just a display. You don't need an interactive user interface, you just need to show an image. Using the Moddable SDK, this is easy to do by building a small application to draw the image. There are examples to display a PNG, an animated GIF, and to stream JPEG images from the web. All of these approaches are great if you are a programmer. If not, you are out of luck.

Or not. The pngdisplay example was created for use by user interface designers, not developers. Instead of changing the code to change the picture, you use the curl command line tool to push PNG images to the display. This is a fast and easy way to review design renderings on the target device display. Of course, you need to install pngdisplay once on your device, so you still need a friend to install (at least for now - we're working on simplifying this too).

The source code of pngdisplay is included in the Moddable SDK. This article walks through how to use and build the example, as well as notes on how it is implemented.

Using pngdisplay

This section explains how to create PNG files that are compatible with pngdisplay and how to use the curl command line tool to push them to the device running pngdisplay.

Preparing the PNG

In this article, we are using a QVGA (240 x 320) display. The images you upload should be no bigger than this size, to avoid unnecessary processing and possible out of memory failures.

The pngdisplay example supports a subset of the options of the PNG format. Most common PNG images are compatible, but you should be aware of the limitations:

  • images must be either 3 or 4 channels
  • each channel must be 8-bits
  • interlaced PNG images are not supported

Applications, including the venerable Adobe Photoshop, export PNG images with extra information (metadata) that increases the size of the file but is not part of actual image. This may just be a small nuisance, causing a longer upload time, or a problem, because the metadata requires more memory causing the upload to fail. As a rule, the PNG image uploaded to pngdisplay shouldn't be larger than 50 KB (and usually quite a bit smaller).

Displaying a PNG

To push a PNG image to the display, you need to know the IP address of the display. When pngdisplay connects to Wi-Fi it outputs its IP address to the xsbug debugger's console. You can copy the IP address out of the console to use with curl. 

curl -T $MODDABLE/examples/commodetto/pngdisplay/test.png http://10.0.1.4/upload

This assumes you are running pngdisplay connected to a debugger. That's probably not the case. Fortunately, you can also access the display by name thanks to mDNS. The mDNS protocol lets individual devices claim a name on your Wi-Fi network. The pngdisplay example claims the name pngdisplay which makes it available at pngdisplay.local. Here's the curl command using the mDNS name:

curl -T $MODDABLE/examples/commodetto/pngdisplay/test.png http://pngdisplay.local/upload

A thin status bar is displayed while the image is being uploaded. The image is displayed once it is completely received. If an error occurs, the entire screen turns red. While the device is receiving a new image, the entire screen turns green.

After uploading a PNG, the device is ready to receive the next PNG. There's no need to restart the device.

Building pngdisplay

You build pngdisplay following the same steps as other examples in the Moddable SDK. If you have attached your display using the Moddable Zero style ESP32 wiring, the following command line builds and deploys pngdisplay:

cd $MODDABLE/examples/experimental/pngdisplay
mcconfig -d -m -p esp32/moddable_zero ssid="your wifi" password="wifi password"

You'll need to fill in the correct values for your Wi-Fi access point.

The default orientation of the display is portrait mode. You can build pngdisplay in other orientations by changing the rotation to 90, 180, or 270. The following command line rotates the orientation ninety degrees to landscape mode.

mcconfig -d -m -r 90 -p esp32/moddable_zero ssid="your wifi" password="wifi password"

How pngdisplay Works

The pngdisplay examples brings together several different technologies:

  • mDNS to claim a the local name pngdisplay.local
  • An HTTP server to receive the uploaded PNG image
  • A PNG image decoder to decode the image line by line
  • The Commodetto graphics library for pixel format conversion
  • The Poco renderer to draw to the screen

It is an example of how JavaScript helps create solutions by concisely weaving together different software modules.

mDNS Name

The example claims the name pngdisplay.local on the local network to allow the device to be accessed by name, not just IP address. The claiming process takes a couple seconds. A callback function notifies the application when a name has been successfully claimed. The claiming process begins when the MDNS class is instantiated. When the example begins the claiming process it fills the screen with dark gray:

fill(poco.makeColor(64, 64, 64));

If the claiming process succeeds, it paints the screen white; if it fails, it paints it red. 

import MDNS from "mdns";

const HOSTNAME = "pngdisplay";
const mdns = new MDNS({hostName: HOSTNAME}, function(message, value) {
    switch (message) {
        case 1:
            fill(poco.makeColor(255, 255, 255));
            trace(`MDNS - claimed hostname is "${value}"\n`);
            break;

        default:
            if (message < 0) {
                trace("MDNS - failed to claim, give up\n");
                fill(poco.makeColor(255, 0, 0));
            }
            break;
      }
});

HTTP Server PUT

The pngdisplay example includes an HTTP server to receive the PNG image uploaded by the curl command line tool. The server implementation follows the pattern in the httpserverputfile example. However, instead of writing the received data to a file, it creates a SharedArrayBuffer large enough to store the compressed PNG data and writes the data to the buffer. The size of the buffer is determined by the Content-Length HTTP header:

if ("content-length" === value) {
    this.byteLength = parseInt(etc);    
    try {
        this.png = new Uint8Array(new SharedArrayBuffer(this.byteLength));
    }
    ...
    this.png.position = 0;

As data is received, it is read from the HTTP server and written into the buffer:

const data = new Uint8Array(this.read(ArrayBuffer, value));
this.png.set(data, this.png.position);
this.png.position += data.byteLength;

Note that the pngdisplay performs a single allocation for the buffer for the entire compressed PNG image. This approach uses less overall peak memory that receiving the data into separate buffers that are later combined.

PNG Decoding

Commodetto includes a PNG image decoder. The decoder is usually only used during the build process to convert PNG images to a format that is more easily handled by embedded devices. The pngdisplay example uses the PNG decoder on embedded devices. The decoder decompresses one line of the PNG image at a time, from top to bottom.

The PNG decoder is instantiated by passing in the buffer containing the compressed PNG data:

import PNG from "commodetto/PNG";

const png = new PNG(data);

The png instance has properties that describe the image, including width, height, and number of channels. The PNG decoder's read call returns a single scan line of data in a Uint8Array. The process of reading the pixels is straightforward.

for (let y = 0; y < png.height; y++) {
    let scanLine = png.read();
    ...render
}

PNG Display

The PNG decoder provides pixels in the same format they are stored in the PNG, typically RGB or RGBA with 8 bits per channel. The display, on the other hand, is usually 16 bit RGB (565), although it can be 256 gray or RGB 332. The PNG pixels must be converted to the screen's pixel format for display. Commodetto contains a pixel format converter, which takes a buffer of pixels in one format and returns it in another. The first step in using the converter is to get the format of the display:

const pixelFormat = Bitmap[config.format];

The next step is to instantiate a pixel format converter from the PNG format, which depends on the number of channels in the PNG file:

import Convert from "commodetto/Convert";

const convert = new Convert((3 == png.channels) ? Bitmap.RGB24 : Bitmap.RGBA32, pixelFormat);

The last step in setting up the conversion is allocating an output buffer for the converted pixels.

const scanOut = new ArrayBuffer((png.width * Bitmap.depth(pixelFormat)) >> 3);

Converting the pixels is straightforward.

for (let y = 0; y < png.height; y++) {
    let scanLine = png.read();
    convert(scanLine.buffer, scanOut);
    // scanOut is now in the same format as the screen
    ...render
}

The Poco renderer draws the PNG pixels to the screen. It requires that the pixels are contained in a bitmap. After allocating scanOut, the bitmap is allocated around it. The bitmap holds a single scan line of the PNG file, so it has a height of 1 and a width equal to the width of the PNG file.

import Bitmap from "commodetto/Bitmap";

let bits = new Bitmap(png.width, 1, pixelFormat, scanOut, 0);

Inside the rendering loop, the bitmap is drawn for each scan line:

for (let y = 0; y < png.height; y++) {
    convert(png.read().buffer, scanOut);

    poco.begin(0, y, png.width, 1);
    poco.drawBitmap(bits, 0, y);
    poco.end();
}

Rotation

The Poco renderer used by pngdisplay to draw to the screen implements rotation by a combination of build-time and run-time software. At build-time it rotates all images to match the screen orientation so that at run-time only coordinate transformations are needed. This simplifies and accelerates the runtime code on the device, which is important given the limited capabilities of most microcontrollers. Unfortunately, when displaying a PNG that has been uploaded, the build-time tools haven't been run on the image, so the image is unrotated, which means it will be displayed incorrectly on a rotated display.

The pngdisplay code rotates the decompressed PNG image as it is displayed without the use of an additional memory buffer. This results in a somewhat more complicated rendering loop than the one shown above. Rotation by multiples of 90 degrees isn't mathematically difficult, but it is an interesting challenge to implement it in a performance and memory efficient way. The details of how this is done are left as an exercise for the curious reader.

Memory

The pngdisplay example requires a relatively large amount of memory, primarily due to the zlib algorithm used to decompress the PNG image. The complete decompressed PNG image is never held entirely in memory. Instead it is decoded one line at a time and sent to the display. Still, the zlib implementation requires enough memory that this example runs only on the ESP32, not the ESP8266.

Challenge: It should be possible to decode some PNG images on the ESP8266. This would require a different zlib decoder, one that allocates memory as needed rather than up-front as a single block as the miniz library does. If you are knowledgable about zlib decoder implementations, please share your ideas. There's a Moddable T-shirt waiting for you if you help us get PNG image decoding on an ESP8266 (with no external RAM!).

Conclusion

The pngdisplay example is a practical tool for designers to quickly review their design ideas on the screen of the target embedded device. The code here can also be used as a jumping off point for your own projects, for example downloading PNG images for display in your application's user interface. At about two pages of source code, pngdisplay is a reminder of how the power of JavaScript can be harnessed to create real-world solutions with a minimum of work, even for the inexpensive hardware used in many IoT products.