Off the top of my head:
Load/save format is a separate thing from what it uses to compute internally: http://www.imagemagick.org/Usage/basics/#hdri_formats
It most likely will affect speed. I don't think x2, however, if you bundle with IM7. A small factor, esp. if you compile IM with optimizing flags (off the top of my head, the standard optimization flags are rather tame, for portability I imagine; may have changed).
However some operations give better results going through floating point. Orthogonal (tensor) resampling filters (like standard lanczos), sigmoidal contrast, probably others. Not sure about usual colorspace transformations. I am a big fan of radical gamma mappings, and I would guess non-HDRI is asking for trouble.
Also see http://www.imagemagick.org/Usage/basics/#hdri
I never use non-HDRI if I have the choice or do anything nontrivial. But that's me.