Not exactly sure how one would go about that.
A dirty workaround would be to enable Developer Options* in settings. In Developer Options there is a setting, “Disable USB audio routing”, it is off by default. Setting that option will ignore the TB, but will require a restart. Of course, to go back to the TB after unsetting the option, would also require a restart. As I said, a dirty workaround, emphasis on dirty.
Audio settings in CoreELEC or similar may have more flexibility in choosing the audio device, but that is speculation, as I don’t have much experience with those OSes.
It may also be possible to switch off power to the TB, but that would require some physical mods and still would not be automatic.
You could unmount the TB from the 3L and use the TB as a standalone USB DAC using a USB cable from the VIM3L to the TB, plug and unplug as needed, again, not automatic.
As far as doing it automatically based on which file or source is used, I have no suggestions.
*To enable Developer Options. Go to Settings/More settings/Device Preferences/About, scroll to Build and click on it 7 times. That will enable Developer Options. Go back to Settings/More settings/Device Preferences and you will now see Developer options, look in there for the Disable USB audio routing.
Caution… Avoid making changes in developer options, if you are not familiar with a particular setting or what it does, it is best to leave it be.