x server – Why does xserver require a physical screen?

I was searching for a solution to run a graphical application without a physical screen. The applikation is showing multiple hardware accelerated video streams with gstramer-vaapi.

I dont want to render anything on a physical screen, but still be able to make screen recordings of the app. e. g. via gstreamer ximagesrc

Tried: Xvfb Xdummy Xpra Xrdp

Nothing really worked out so far.

I am wondering why a normal xserver needs a physical screen at all? Just a software issue or is it really a hardware requirement?