idler's remorse
noun phrase
Regret for not having made the most of limited free time.
Some weekends, Tarou was too tired to do anything but stay home and relax. He knew he needed this recovery time, but that didn’t prevent the idler’s remorse from settling in on Sunday evening when he had done nothing of value and knew he’d soon be returning to work.