Another thing I have noticed is that the GUI is way more laggy with Xorg. Do you have some ideas that I can fix Xwayland?
Only good fix might be to totally eliminate xwayland and go wayland. Not sure if that is practical.
Might try getting an identical board and see if you still have the problems. Start out with absolutely nothing installed, just the factory image and see what happens. When you start adding and removing packages is when the problems can start. If it is working fine start adding more programs and keep an exact list of what is installed.