There are a couple of issues here.
(1) The mute control - be it the switch or multi-task bar control - only mutes system sounds - i.e. new email etc - it does not mute other audio - say music or videos
(2) The volume control works separately for headphones and speaker. So, if you turn the volume all the way down for the headphones and then unplug them, the iPad 'remembers' the previously set speaker volume and uses that. If you want to turn the speakers down, you have to do that as a separate operation.
Tim