As a nation, Americans are terrible at taking vacation. The last statistic I saw, 52% of paid time off goes unused. Vacations, breaks and rest are great!
Working hard is great. This includes time in the gym, not just at your profession. Having these days off will help you recharge and come back better, physically and mentally refreshed.
When I am ready to go on a trip, I overreach on my training to the point I need some extra recovery time. This not only gives my body the time to recover but also can spur new growth and progress.
On trips I do like to workout but equipment and schedule can often make that unpredictable. Next time you go on that trip, relax, you've earned it.