The coastal belt between Walvis Bay & Swakopmund is open for dune riding, whereas further to the east it's national parks area, but in that NP there are some tougher tracks/pistes you can do. Just make sure to acquire a permit before you go off into the NP's.
Another dusty bike riding area is going down the west rim of the Fish River canyon. There you can make a loop via Rosh Pinah and up towards Aus near Luederitz. Along the canyon west rim you'll pass a lodge situated right on the rim, about 90 kms south of the tarmac Keetmanshoop - Luederitz. Absolutely stunning and hardly any tourists!
Only when we pause to wonder
do we go beyond the limits of our little lives.