The iPad has to be on, and it has to have an internet connection, either through 3G or Wifi (sleep mode is fine). The WiFi iPad will be less accurate than the 3G, because it does not have a GPS, instead it estimates it's position by comparing what hotspots it can see against a database.
If someone takes the iPad and restores it, it will not be trackable even when it is connected. At least not by the Find My iPhone app.