A couple of possibilities....
One - the volume control on the iPad is independent for the speaker and headphones. So, it's possible to have the speakers on maximum volume and then plug in a headset and hear nothing, because the volume setting for the headset was previously set to zero when they were last plugged in.
Another possibility is that the socket/plug for the headset is a tight fit and it needs to be pushed quite firmly home for it to connect properly.
Tim