Confirm with you again:
1. Your VIM works fine with USB keyboard, right?
2. You've try with many different USB mouse, no one works fine, right?
Please reply above one by one, so we can locate the issues you met now.
Following things would be also help us:
1. Tell us your usb mouse model
2. take a photo with the usb mouse together VIM.