Almost fresh copy of Ubuntu 20.04 is laggy any time only monitors are connected through displaylink dock

If 1 or 2 monitors are connected only through the displaylink dock, the mouse moves slowly, but any action lags around a second. If I connect one monitor through the computers video card slots and one in the dock, everything behaves as expected. I’m trying to use this computer at my work workstation utilizing a USB switch to switch between my work computer running ubuntu and work laptop running windows.

On the problem computer
Kernel: 5.8.0
X.Org X Server: 1.20.9
Displaylink: 5.4.0-55.153

Seems like it might be related to glxgears problem referenced at https://github.com/DisplayLink/evdi/issues/181 , for which there is a patch
to X.Org X Server 1.20.8. Should I just downgrade that and try to apply the
patch and keep Ubuntu from ever updating that?