Can't use shaders on videos

[theturai]

I’d like to implement chroma keying on videos, but none of the shaders seem to apply properly. There are no compile errors and videos are loaded with raw = false. All shaders work properly with images and rects but not videos.
What I’ve tried so far:

• passing the video as a texture => has no texid() method => throws an error
• using my other shaders, none of them work
• copy-pasted the shader, shader:use and load_video parts from the scheduled player package, still nothing, however, when I change the sensitivity to 1.0 I get a black texture (works properly on framegrabs from the same video)

I don’t know what I’m doing wrong.
My video files are encoded with h264, resolution doesn’t matter.

[infobeamer-fw]

I guess you’re using the Pi4? If so, I just realized that the video player object, regardless of what raw is set to, doesn’t have a texid() method attached. In fact it’s not even implemented for the Pi4. So right now chroma keying won’t work there unfortunately. I’ll see that this get fixed. Sorry about the confusion.

[theturai]

Yes, I’m using RPi4s. Thanks.

[infobeamer-fw]

Are you using info-beamer pi or info-beamer hosted?

[theturai]

The latest pi version. I can’t use the hosted version for offline playback.

[infobeamer-fw]

You can try out the prerelease of 1.0.2 on https://info-beamer.com/download/player#prerelease

In my tests it now works, although due to the some internal changes how GL videos work on the Pi4, you need to update the following in your shader code:

#ifdef INFOBEAMER_FKMS
// Allows access to the GL video texture on the Pi4
uniform samplerExternalOES Texture;
#else
uniform sampler2D Texture;
#endif

Using this code should work with both pre-Pi4 and the Pi4 model. The texture is then accessible texture2D.

Oh, and regarding offline: The hosted version will have offline too. In fact if you purchase an license for info-beamer pi, which is required for most use cases anyway, you can run your info-beamer hosted device offline as well. So there’s even less reason to use the raw info-beamer pi software now.

[theturai]

Awesome, it works
It crashes when I call video:stop(), here’s the output:

[mmal_video_render.c] video render shutdown started
[mmal_video_render.c] video render forwarding thread stopped
mmal: mmal_port_pool_destroy: port 0xa8401030, pool 0xa8403830 destroyed while port enabled
[mmal_video_render.c] video render shutdown completed
[mmalvideo.c] [0x10493c8/playground/teszt.mov] no audio stream found
[mmalvideo.c] [0x10493c8/playground/teszt.mov] setting initial pts: 0
[mmalvideo.c] [0x10493c8/playground/teszt.mov] setting initial dts: 0
[mmalvideo.c] [0x10493c8/playground/teszt.mov] starting paused
CRITICAL ERROR: Cannot alloc texture: invalid operation

[infobeamer-fw]

Damn. My guess is that this happens once you load an image in the meantime? Because that’s the on place where that error is generated. I’ll have a look at this again. The difference in how image/video textures are accessed (see here) makes all this shader logic more annoying that it should me.

[theturai]

Indeed, I have an image, however it’s loaded before the video and has a different shader applied. I’m trying to use the video as an overlay on top of this image.
If that matters regarding this crash, my stop command removes elements from bottom to top, the image gets disposed first.

[infobeamer-fw]

I can’t seem to be able to reproduce the issue in my attempts. The error message can only be triggered while an image texture is allocated as a result of a resource.load_image call. At that point, glGetError() is called with the intention of detecting allocation errors. As the glGetError is a state that lingers around and cannot be attributed to a single GL call, my guess would be that the error flag is set earlier as the result of setting up a shader. But I’m not able to trigger this condition, regardless of how I mix image loading/shader usage and video playback.

Do you use the the texture-as-shader-argument feature? So shader:use{ texture = my_video } or similar? Or do you activate the shader and then draw the video/images like this:

shader:use()
my_video:draw(...)
shader:deactivate()

Is the issue reproducible? And obivously, do you actually have enough GPU memory available (gpu_mem in /boot/config.txt)?

[theturai]

Removing the texture = image line fixed this. I don’t understand why… this code is weeks old and never crashed before.

[infobeamer-fw]

Because prerelease didn’t handle all cases correctly. I’ve released a new version that should fix the issue. Can you download the latest version from https://info-beamer.com/download/player#prerelease and check if everything works now?

[theturai]

Looks like you’ve fixed it, it stops successfully with the reverted lua code.

[infobeamer-fw]

Nice. Thanks for testing. Overall the feature is probably still a bit weird and probably needs an API rework in the future to prevent unintended side effects when used outside of the happy path. But for now it’s probably good enough.