I love USB devices, they are Sooooo much nicer than messing with old serial devices trying to get the com parameters set up right between the device and PC. One problem with USB though is it can be susceptible to electrical noise.
Part of the problem stems from its use of a quasi differential signal (that is still ground referenced!) and the fact that you can see quite a difference in ground potential at each end of the cable give the right combination of PC and environmental conditions. Some PCs implementation of USB also leaves a lot to be desired.
Many times using an externally powered USB hub, or powering the USB device separately can be a big help. Using a good quality cable is also a good idea.
You could also have noise getting back into your FEZ through the IO lines you have driving things, this in turn can put enough noise on the ground to through off USB. If you are driving AC loads with the relays put arc snubbers across the contacts, this will reduce RFI a lot. Make sure you don’t have low voltage lines (I/O from FEX, USB cable, etc,) run in parallel with high voltage (AC) lines.
Hope that is of some help