In general, worker safety has improved significantly in the 90 years since the Empire State Building was built, see here: Workplace Fatalities Fell 95% in the 20th Century. Who Deserves the Credit? - Foundation for Economic Education (fee.org)
It’s pretty common to interpret slower speed as the inevitable cost of increased safety, but looking at some notable projects the link is less than obvious to me:
-5 workers died during the construction of the Empire State Building, which was built in 11 months
-0(!) workers died during the construction of the Chrysler Building, which was built in 20 months.
-5 workers died during the construction of the Sears Tower, which was built in 4 years.
-2 workers died during the construction of One World Trade, which was built in 7 years
-60(!) workers died during the construction of the original World Trade Center, which was also built in 7 years.
It would be interesting to do a more thorough analysis, scaled to building size, but it’s not trivial to do (I did a quick check for number of deaths on some less notable buildings and they’re much harder to find if they exist at all). It’s a reasonable hypothesis, but most people suggesting I think are going off vibes rather than actual data. And it seems clear that it’s at least in-principle possible to build both quickly and safely (though you could make like, a stochastic argument against this).
The difficulty isn’t normalizing (per square foot is probably the most reasonable), it’s getting death information for individual buildings. Outside of the most famous buildings it’s not easy to track down.