Ditch WM, and funnel all four of those skill points into Ironflesh.
I did some math and the difference between the 2 builds is that the WM build hits (on average) about 4% harder at most (using 31 blunt damage from the warhammer and 60 armour). For other values, it looks about the same as that 4%. I don't know from the top of my head how much of a difference 30ish wpf makes in swing speed, but I think it's less than 0,05 seconds.
So I would basically trade 4% damage (about 1 or 2 points of raw damage) a hit for 8 HP.
Sounds very appealing and I might just do that.
Note, my other 32 build was a standard 21/15 build with 5 riding. That build has an equal 70 hp, but even with full WM it hits for about 14% damage less.
I'm going to level without WM and see how it feels. If I somehow still feel I'm not hitting fast enough (using weapon with speed 100 or higher should do the trick), I can alway get the WM later on.