AliBaba is generally correct in his reply, however, there are a few additional considerations.
1) If you are in the same place as you were when you turned it off, it will figure out where it is pretty quickly, as long as it has not been turned off longer than about a month. If you have changed location considerably (more than 200 km), it pretty much needs to start from scratch. Not scratch like the first time it was ever turned on, but the satellites won't be where it expects them to be, because you moved the GPSR without telling it. So, expect it to be slow to figure out its position if you have moved it more than 200 km's.
2) Almost all of them do. Some GPSR's that don't have a soft power down feature won't write their last position to non-vol memory before they are turned off. But they will have already written the last satellite almanac to memory, which accomplishes 90% of what writing the last position does, again, presuming you don't move it a great (100's of km's) distance between shutdown and startup.
3) Even if the internal battery fails, the GPSR will get the correct time from the first satellite it picks up. But, in the case of older GPS units (typically pre 2001) that did not write the waypoints, tracks, etc. to non-vol but relied on the battery to keep this information alive, you will lose any user settings, waypoints, tracks, etc. if your battery goes dead.
The battery issue is kind of moot - sort of like batteries in PC's - by the time the internal battery goes dead, the hardware is so out of date you probably will have upgraded anyway.