I agree with your comments on vehicle size, and the current 3 series is pushing on the limit of what I would want in terms of a larger vehicle for daily driving.
The M2 is an *outstanding * daily driver , perhaps slightly noisy a little of a rough ride but nothing compared to the Z4 chassis which can have a choppy ride at the rear. This same lively rear end and the mechanical steering will have you rejoicing in the Z4m as a weekender.
Then back to the M2 and it's very soft, nearly antiseptic controls for the daily grind.
I would expect that you would be tempted to track the M2 on occasion, and it will be Way faster than your modified Z4M, but likely not more fun an overall experience. Good luck in your wagon decision, if you don't need the utility of a wagon the M2 would be perfect