Everytime I do the time advance thing, I always tap the "fountain of happiness" or whatever it's called, to boost their happiness (I've got two of them, plus one "tree of joy"). That keeps their happiness up. Failing that, do a bit of beautifying to give 'em a boost.

I demolished the Astari village entirely, which really bummed them out for a while, but they're naturally cheerful souls and were happy again in no time.
And, by the way, I checked if all my previous beautifying was still in effect and it does appear to be so. I don't see any fading anywhere. I wonder if it's just an "Antheas world" thing?