I dunno about most. But lots do I think. Disabling it has no benefit, mostly an old wives tale so to speak. But I still think it's worthwhile to set it.

I've had to argue against external pen-testing providers contracted by customers because for them finding a Server header in a response is a "critical issue" (absurdly once for a service that is behind a proxy anyway).

I found your reasoning against removing the header some time ago in some discussion. Maybe a wiki entry one can point to like "See what the people behind Caddy officially think about your snake-oil" might help?