Obviously a number of factors have to be taken into account - the rider, the bike, the road surface, the distances, the amount of gear, the design and construction of the backpack....
I carried all my gear in a backpack (Gearsack) on my rides across Australia and wore it on my back.
The first time I had too much stuff and the straps cut into my shoulders, even though the shoulders don't support all the weight.
After learning my lesson I took only the essentials on subsequent rides and had no problems at all.
If you're out in the middle of nowhere on a highway with no bend in sight there's no problem loosening the straps and letting the seat behind you take some of the load. Handling really is not an issue in that situation.
Two advantages of wearing my backpack:
1. It helps to keep me warm on a cold day and stops the wind whistling up my back.
2. The reflective writing and triangles on it make me more visible at night.