Just a couple things I'd like to throw in here.
First off, that 20 ms difference is only for the COMPLETE swing (i.e if you hit them at the very very end of your swing) most people hit very early in the swing (turning into the swing + moving into it) so the actual difference will be in the lower single digit ms.
Second, the biggest penalty to your attack speed from armor comes from the acceleration penalty (movement), NOT from the wpf penalty. The wpf penalty isn't even noticeable, however your acceleration is hampered greatly from increased weight, and so you can't move into your swings AS EASILY. With good footwork though, this can still be overcome, largely by using your opponent's movements against them.
Anyway, point is: don't choose armor based on what it will do to your attack speed. Choose it based on what it does to your MOVEMENT speed (and particularly what it does to your acceleration). Footwork is crucial to every build, every weapon, and every play style. However, some need it more than others, and some people just don't use good footwork enough.
My personal recommendation: try out different armor weights and see what feels the best to you. Some people can do fine in plate, others do better in lamellar. The armor weight that's best for you is very personal to your style, and, imo, can't be chosen by number crunching, you just have to try stuff out.