Simple: All you need to do is go to open Game Center, tap on Games, then tap on Hay Day and then tap on Players on top and you will see the recent activities, there you can tap on each player that you think they are inactive and unfriend them.
As for followers. You would have to do this ingame. Tap on the friends button on the bottom RH Corner and tap on Followers. Go to their farm and follow and then unfollow them. Keep in mind that it only hides them from your followers tab. They still have the access to visit your farm.
