Punches mostly hurt because of the mass of the fist and arm. If you’ve ever been punched by a kid you probably weren’t hurt. A skeleton weighs less than a 6-year-old, so there wouldn’t be much mass behind the punch.

But, then again, magic.