20-30 would be my preference, 20 early game when the number of troops in each city is more important, so you keep your cities smaller in order to fit in more troops, mid to late game it doesn't make a difference anymore so you might just as well build to the max level to speed up the recruitment time.
To be exact 10 levels of harbor is 44 population. By itself that is not significant, you're right - but if players can gold it up and aren't concerned about building times of barracks, senate, harbor, etc then saving population wherever you can means much deadlier attacks.
It makes a difference man. 7k city is gonna whoop butt compared to a 13k.
And harbor itself adds up to it. But you do have valid points as well.