I would imagine that the device plugged into the Xantrex inverter isn't happy with the modified sine wave output. I've used them in the past and 98% of everthing I ran with it ran fine, however, the other 2% wasn't happy.
I would bet your 300 watt inverter that was working in this setup was a pure sine wave inverter.