"... a letter attesting that we had permission to use the car"
I travelled with two friends who had a letter written on UK Police stationery and signed by the police chief of their local town, authorising them to use the car.
Their big error was failing to get an official stamp on it.
The Moroccans took one look and rejected it - and they were put on the next boat back home.
The next two friends, seasoned travellers both over 70 years old, were also thrown out at the dockside a year later. Their V5 (Carte Gris) didn't have the correct name on it.
Morocco is a busy smuggling route for stolen European cars, and the Moroccans are strict about car imports.
An official stamp, as CS says above, is required for everything.