Raspberry Pi 5 release, info-beamer information and development diary

The release is now ready for testing. Get in contact if you’re interested.

Updates since last post:

  • Optimized memory usage on Pi1 (those have 128MB usable memory :slight_smile: )
  • Improved snapshotting while info-beamer isn’t running
  • Added fan support for Pi5

The last remaining larger topic is now sound support on the Pi4/Pi5. Compared to previous versions this will now use Alsa and as a result should now support outputting audio to two displays at the same time. Still not entirely sure how to best approach this topic and it will take a bit before it’s being completed.

Another interesting topic is HDR support on the Pi4 and Pi5. Given that info-beamer now uses DRM/KMS for placing output on the screen, this should now be possible. Still not entirely sure how that works and if OpenGL content, for example, might also make use of that. All very interesting…

1 Like

:warning: Known issues for the preview info-beamer hosted OS release running on the Pi4 and Pi5:


  • No audio support yet
  • Flickering when using transparent videos across multiple displays on the Pi5: This is most likely a Linux kernel bug in the VC4 drm implementation. I’ve opened an issue here.
  • Putting displays to sleep doesn’t really work yet as CEC support isn’t implemented yet.
1 Like

A first beta1 of the info-beamer pi tool is now available too. If you have a Pi5 with Bookworm 32bit running, you can download the version here. A 64bit version is also possible, but there wasn’t enough time for testing yet and I don’t really expect any notable performance difference.

Would love to try the RPi5 version Florian - sounds amazing :boom:

More Pi5 stuff:

  • Noticed that the 1366x768 mode can’t be snapshotted and info-beamer currently aborts when doing so. This mode is notorious for being problematic as its width isn’t divisible by 8. I guess simply using a 1366x768 frame buffer as writeback target doesn’t work. I also tried using 1368 as width instead but that also doesn’t work. Not sure how to handle this. The current workaround is to forcefully replace that mode with 1280x720 :-\

  • Added overscan support. I you configured your display wrong, info-beamer can now slightly shrink the output so the content isn’t cut off. Of course this ends up double scaling (first info-beamer shrinking down the output, then the TV scaling it up again) which looks like garbage. Especially as it’s a digital signal. There’s really no reason to not use pixel perfect output settings on a display, but if for some reason that’s not possible, info-beamer can help now.

Updates to the preview OS release:

  • It should now play HDR content on HDR capable displays. At least in theory: All the test screens around here are SDR only :}
  • CEC abilities are restored. Attached displays can now be turned off/on and the “Keep attached HDMI screen on” feature should now work across two displays as well.

Updates:

Next up: Audio. Everything is now going to use Alsa on Pi4/Pi5 and the already existing audio engine needs a bit of a rework to also work with videos. That’s going to take a moment…

Reworked the audio engine. Previously only the audio playback used ALSA. Now there’s a central mixer that allows audio as well a video players to allocate a channel which then get mixed and sent to potentially both HDMI outputs.

Right now there’s still no proper synchronization between audio and video playback, but for short videos that’s probably not noticeable. Still not entirely sure how to handle sync: The libao I’m using doesn’t provide any good insight into how it’s processing the audio submitted via ao_play and calls to it return after different durations, so it’s not possible to infer the exact playback position of when audio is sent to the speakers. I’ll probably use monotonic system time as a reference and then see if audio playback desyncs from that by taking a rough moving average of PTS from submitted audio packets.

Updates:

  • Fixed an issue for 90/270 degree rotated videos. They should now render correctly. Previously some configurations resulted in videos not ending up on the screen :-}

    Update an hour later: And now it has a resource leak instead when playing videos. I’ll fix that later.

  • Experimenting with not restarting the info-beamer process on Pi4/Pi5. This prevents the kernel from briefly taking back control over DRM during the restart. The result is that video output is now fully controlled by info-beamer and there’s no video mode switch when assigning different setups. This will prevent overlays from popping up that some screens show in that case. In anyone has an idea on how to prevent th

  • Optimized memory usage on Pi1 (yes the Pi1 from 2012/13!). This release will now be a bit faster to work with on that type of device.

Updates:

  • Optimized the logical layer-to-drm-plane placement algorithm. Fully opaque layers now allow skipping of fully covered layers beneath them as they would not be visible anyway. This reduces the load on the Pi HVS hardware in some cases. I still have more ideas on how to optimize even further, but those optimizations get more complicated each time :-}

  • Optimized the handling for drm framebuffers. They are now all reference counted which greatly simplified some parts of the code. It also allows complete gapless and glitchless transitions between two video files. Previously the last frame of a video might be removed internally while it’s still on the screen. The kernel side then removes the framebuffer but that results in visual glitches: basically the frame then stretches diagonally across the screen. That’s gone now.

  • More CMA memory for Pi4/Pi5, so two 4K videos should be loadable at all times.

  • Provide the HDR status of attached displays to the backend.

Wrote a compatibility shim that allows using /sys/class/gpio with the newest 6.6 kernel series. The /sys/class/gpio method of accessing GPIO pins is being deprecated with its use already slightly changing in 6.6 and probably completely going away in a later major kernel release. But there’s still a ton of devices out there using that interface. So I need a way to make everything backwards compatible as expecting all devices and custom code to get updated is unrealistic. So I wrote a FUSE based filesystem tool that overlays /sys/class/gpio and emulates the old behavior. Existing package services should not notice any difference. Behold: https://github.com/info-beamer/sysfs-gpio-shim.

1 Like

Unsuccessfully tried to modify the kernel to minimize used memory while rendering to HDMI. As I understand it, the problem is that scaling a video to a non-native resolution (say a FullHD to 1920x1000 to leave 80 pixels free for a scroller) requires a small patch of memory used by the specialized hardware generating the HDMI signal. These memory chunks needs to be managed. Right now two copies of them might be allocated at the same time: One for the next frame and one for the current frame. When rendering a lot of content (for example GL and two videos across two HDMI outputs), this exceeds the memory available and info-beamer crashes[1]. I tried to modify the VC4 driver code to avoid keeping the state twice and (I think!?) it works somewhat, but there’s green flimmering from time to time. Clearly there’s something wrong. I don’t really understand how the VC4 kernel driver works, so I’m not sure how to proceed. If anyone is interested in all of that: Here’s the Pi forum thread.

[1] In theory it would be possible to handle this case, but there isn’t really any solution that would look good: The only way to get out of that situation is to remove, for example, a video that should be on the screen. In my opinion, hiding errors like that is worse than crashing.

Updates:

  • Fixed non-functional WiFi on Pi5. I’m pretty sure that worked at some point but I guess the switch to kernel 6.6 broke it again. Oops. The brcmfmac module depends on brcmfmac-wcc, but insists on loading that module itself. A manual load of brcmfmac-wcc is too late!? Instead it requires the modprobe mechanism configured via /proc/sys/kernel/modprobe.

Updates:

  • Fixed crashes on Pi1/Pi 0 due to “illegal instructions”.

  • Restored support for Edimax v2 WiFi USB stick.

  • Further improved plane clipping optimizations. Still doesn’t solve running out of memory when showing 4K videos across two displays with additional overlays. But the algorithm is pretty solid now:

    Animated plane clipping visualization

  • The ‘bleeding’ OS version will be promoted to ‘testing’ this week.

1 Like
  • info-beamer now probes the selected mode and falls back to 1920x1080 and then 640x480 in case the mode is not supported by the attached display. That’s better than not starting at all if, for example, 4K is configured but only a FullHD display is attached.
  • Optimized PNG and JPEG loading. PNG loading should now be around 20% faster.

Added the following options to transparently flip display output:

  • INFOBEAMER_FLIP=1 flip both displays, horizontally and vertically, effectively rotating the output by 180 degree
  • INFOBEAMER_PRIMARY_FLIP=1, same but only first display
  • INFOBEAMER_SECONDARY_FLIP=1, same but only second display
  • INFOBEAMER_FLIP_H=1 flip both displays horizontally. Useful for rear projection
  • INFOBEAMER_PRIMARY_FLIP_H=1, same, but only first display
  • INFOBEAMER_SECONDARY_FLIP_H=1, same, but only second display
  • INFOBEAMER_FLIP_V=1 flip both displays vertically. Useful for rear projection
  • INFOBEAMER_PRIMARY_FLIP_V=1, same, but only first display
  • INFOBEAMER_SECONDARY_FLIP_V=1, same, but only second display

When taking screenshots, the flipping is reversed, so the screenshots don’t change orientation. It’s basically an option that allows you to rotate your screen without that having any effect on the content or other configuration at all[1]. All options use hardware acceleration, so there is no loss of playback performance.

[1] Assuming content that’s played back using the info-beamer software. The fullscreen browser unfortunately can’t handle these settings :-\

Revived an old idea: It would be nice to augment the normal info-beamer content (videos/images) with real plugins, implemented as external processes. Now that I’ve worked quite a bit with KMS/DRM, this suddenly seems rather easy to implement. So I did a first prototype. The idea is that info-beamer generates a texture, starts and external program and then allows an that program to modify the shared texture. If that external process is a browser, you suddenly have browser content right in info-beamer itself. The main problem was getting CEF working in a way that it renders into a buffer that happens to be shared with the main info-beamer process. ChatGPT was useful navigating that minefield. Behold:

From Lua this is rather easy to embed:

local b = resource.create_content_process(
    "browser", 800, 800, "https://info-beamer.com"
) 

"browser" is the name of the binary using CEF to render web pages. 800x800 is the texture size and the url is passed directly to the browser as command line argument. The returned value b is an info-beamer object that behaves basically like an image. So you can rotate or otherwise modify its position and draw it together with other content:

b:draw(0, 0, WIDTH, HEIGHT, 0.8)

Yes: it can also be transparent. The embedding of browser content is zero-copy, so there’s no overhead bringing such external data into the main info-beamer process. The rendering within CEF is currently not hardware accelerated though.

The plugin interface is incredibly simple: Create a program that parses WIDTH, HEIGHT from the process environment, mmap width x height x 4 (RGBA BGRA) bytes of file descriptor 3, provided to you by info-beamer, and then directly modify the mmapped data (preferably using sync ioctl to avoid tearing).

This all won’t make it into the next OS release, but I’m certainly looking at how this can be implemented fully production ready in the future. Actually it was easier that expected. So it will be included :slight_smile:

1 Like

Woah. It was actually not too difficult to get that working: I’ve now bundled Chromium Embedded Framework with required dependencies into an info-beamer package. It’s only about 170MB in size total. It can either run as a standalone, like the existing full screen browser, or be added as a plugin to the HD player version 2 package that will be out soon. The latter then allows you to freely mix browser content with images and videos, all running fully inside info-beamer.

The HD player V2 also has video wall and synced playback support that that also works with the browser plugin too. Assuming a configured website produces the same results when fetched from different devices at roughly the same time, the content can be spread across multiple displays.

About the content process API: It now works as follows:

local p = resource.create_content_process{
    file = "browser.exe",
    width = 800,
    height = 800,
    args = {"https://info-beamer.com"}
}

By convention, info-beamer OS will now mark content with the .exe extension as executable. The above code will fork a background process, do a few sandboxing steps (the plugins run as their own user ID) and optionally overlay mount an package-provided overlay.squashfs file system that can be shared with a package service. It then spawns the process with file descriptor 3 being a DMA handle, fd 2 being debug output, \n separated lines written to fd 1 ending up in p:status() and fd 0 being fed by what’s provided via p:command("foo"). Everything in args is provided as normal command line arguments in argv.

p:state() works similarlay to how images or videos work. It’s first in the "loading" state and once the first new line separated string is send to fd 1, the state switches to "loaded". This allows the background process to signal when its ready and at best already filled a first frame with content. Exiting with exit status 0 results in the state switching to "finished" while any other exit status result in "error".

Explicitly calling p:dispose() or a garbage collection sends a SIGKILL to the process. There is no nice way to end content processes. They are expected to handle this. The content process currently has no way to write any persistent files: Everything is fully disposed once the content process object is garbage collected. The advantage is that there is never any leaked state across multiple invocations of the same executable.

Here’s an example C program that produces fully red output:

#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <linux/dma-buf.h>

#define DMA_FD 3

static void die(const char *msg) {
    fprintf(stderr, "fail: %s\n", msg);
    exit(1);
}

int main(int argc, const char **argv) {
    int width = atoi(getenv("WIDTH"));
    int height = atoi(getenv("HEIGHT"));

    char *mem = mmap(NULL, width * height * 4, PROT_WRITE, MAP_SHARED, DMA_FD, 0);
    if (mem == MAP_FAILED)
        die("cannot map DMA-buf");

    fcntl(0, F_SETFL, fcntl(0, F_GETFL, 0) | O_NONBLOCK);

    while (1) {
        struct dma_buf_sync sync;

        char buf[200];
        int ret = read(0, buf, 200);
        if (ret >= 0)
            fprintf(stderr, "COMMAND: '%.*s'\n", ret, buf);

        sync.flags = DMA_BUF_SYNC_START | DMA_BUF_SYNC_WRITE;
        ret = ioctl(DMA_FD, DMA_BUF_IOCTL_SYNC, &sync);
        if (ret)
            die("sync failed");

        char bgra[4] = {0, 0, 255, 255}; // Currently BGRA
        for (int i = 0; i < width * height * 4; i+=4) {
            memcpy(mem+i, bgra, 4);
        }

        sync.flags = DMA_BUF_SYNC_END | DMA_BUF_SYNC_WRITE;
        ret = ioctl(DMA_FD, DMA_BUF_IOCTL_SYNC, &sync);
        if (ret)
            die("sync failed");
        if (i == 0)
            write(1, "ready\n", 6); // signal being ready: switches to "loaded"
        usleep(16600);
    }
    return 0;
}

Similar to a browser, it actually shouldn’t be too difficult to use something like mupdf and its C API to render PDF files the same way.

While that’s all pretty exciting, one downside is that this will only work on the Pi4 and Pi5. Theoretically the Pi3 should be able to run the required KMS/DRM based version of info-beamer as well, but in my experiments, that’s quite slow compared to the broadcom APIs.

1 Like

The new browser package has been updated to showcase the new possibilities of having browser content native within info-beamer: 4 different transition effects between web pages: The most impressive one is probably a cube rotation. Here’s a screenshot:

image

and here’s a quickly captured video showing the rotation in action:

You can import the experimental browser package from here to run it on the latest testing (240427-cb91f4) release on a Pi4 or Pi5:

Import

Ok. To my great surprise it looks like it’s even possible to backport this to at least the Pi3 using the proprietary Broadcom APIs instead of DRM: Here’s a snapshot of a first version:

There are a few minor issues that need fixing:

  • The orientation is off. Seems like the memory layout is different. I’ll have to look into that.
  • The texture size has to be a power-of-two, unlike on the Pi4 or Pi5. So a 1920x1080 browser snapshot would have to reserve a 2048x2048 texture and then crop the result. This also means that the API towards the rendering process needs not only the WIDTH, but also a STRIDE parameter, so it can properly write into the buffer.
  • There’s a memory leak somewhere:
    usrvcsm: [vcsm_malloc_cache]: [3356] [EGL_IMAGE_BRCM_VCSM]: ioctl mem-alloc FAILED [-1] (hdl: 0)
    eglCreateImageKHR:  failed to create image for buffer 0x7e99fe64 target 161034435 error 0x300c
    CRITICAL ERROR: Failed to create vcsm texture: Cannot allocate memory
    
  • It’s realllllllllly slow. The browser takes 8 seconds or so to load. Stupid software bloat.

Not sure if this is really useful in the end.