George pretty much hit the nail on the head there. Many travellers to 'Oz' start off on the main routes here and then discover there's better to be had off the beaten track as it were.
A big single is the way to go as suggested if you have the money, otherwise 2 - 2.5K will get you a fairly decent 250 out here, and there's plenty of folks circling the continent on 'em. In a country where Honda's WR400 is the biggest selling bike you'll have no problem picking up a good used dirt bike.
Some of the more major dirt highways such as the Oodnadatta Track are easily negotiated on a road bike (I took a GTR1000 up there), however anything more then that will require at least a 'trail' bike (dual sport), some areas only negotiable on a full out dirt squirter.
$2500 will buy you more dirt bike then road bike in Oz, dirt bikes de-valuing at a much quicker pace...so you'll probably get a more recent bike for your money. My advice would be to go for a 4-stroke single..250cc or bigger and travel light.
Hope that helps,