I've got the Latitude as well, and will be using it as my only jacket for my Americas journey next year. I used it for the first time on a 4-day trip in Ecuador, that had nearly every climate and weather possible. With the right underlayers, it can handle anything (well, up to a point, but at that point no one is riding).

I normally will just wear my t-shirt and the jacket under regular temps. But for added warmth I use a merino wool long sleeve baselayer. Smartwool or Icebreakers. Expensive ($80) but good. The wool wicks away sweat, so actually is fine in moderately warm temps, too. Use the 150 or 200 weight, depending on whether you naturally run hot or cold; 250 is waayyy to warm.

Over that, if it gets colder or I'm at altitude, I actually have used a down jacket. This instead of a sweatshirt or fleece pullover because down packs much smaller. The light/ultralight down jackets on the market today are excellent, and designed for high altitude mountaineering where lightweight is as important as warmth. The Mountain Hardware Ghost Whisperer ($250) is the cream of the crop (7 oz!), but there are plenty of alternatives that are nearly as good and cheaper (I've used s Uniqlo ultralight jacket ($80) under my Latitude just fine).

People don't think of wearing a down mid-layer under their moto-jacket, but it can add good warmth, is not too bulky or obstructive, and the best part is it takes up little room in your panniers. On that note, a mountaineering trick is to NOT use the stuff sack they provide. Instead, mush the jack in the crevasses and empty space---it takes up less room than the neat little stuff sack ball. Plus, you have a nice warm jacket for cool evenings off the bike that doesn't make you look like Mad Max or some spaceman.