A magnet shouldn't impact the camera in any way, or at least, I wouldn't think so.
My concern would lie in the fact that, a camera could be harmful to the device itself. A weak magnet should be fine, but a stronger one could potentially corrupt data or damage internals. Granted, that's more of a problem on hard drives, rather than flash memory (which is used in portable electronic devices). So I wouldn't freak out, but be cautious, because while unlikely, it does have the potential to be harmful to your device.