It depends on your settings in Settings > Sounds.
If you have Change with Buttons turned on, then turning down the volume will affect all sounds on the iPad. If it is turned off, then alert and system sounds will be at the volume shown by the slider, no matter what the master volume of the iPad is set for. In that case you need to turn Mute on to silence thees sounds.
Mute is meant to 'just' silence system sounds and alert, while allowing most other apps (including music apps) to continue playing. It's handy for people in meetings, giving presentations, or other situation where they may want some sounds from apps to play, but not be interrupted by alerts, notification, or other system sounds.