Thanks! That looks like a really close match for such few polygons!
Yes, starting from topology that originates from the road seems to be the way to go to save some time.
Ended up using the loft still, but with less splines. Just the left and right side of the road, then 2 simple freehand drawn splines in the distance.
Doesn't give a square, but good enough for the cause.
After subdividing the loft to somewhat square polys it wasn't too difficult using the smooth tool getting rid of creases. Using stitch & sew for the parts that disconnected by accident.