I guess that depends™. I would suggest you give it a try. The issue with that on the Pi3 was that at some point especially when using two FullHD 60FPS videos, the Pi was too slow to generate an HDMI signal. The result was occasional connection loss to the attached display, depending on how tolerant it is towards that.
The Pi4 should be a bit faster, but I wouldn’t expect too much. To reduce the load, I would suggest you reduce the framerate of your HDMI displays from 60 to 50Hz. You can change that in the device configuration editor on the device details page. This along frees up a bit of computation for the output generator (see also this old blog post). If you limit yourself to videos with 25 or at most 30 frames per second, it will most likely work.
Additionally you should also set the performance from “powersave” to “performance” (also in the configuration editor).