A bid to use a rural barn near Wisbech to sell caravans and camper vans have been refused over road safety fears.