I get the feeling that adhan is a mental harm or a stress or an emotional affliction as opposed to a physical harm.
Mental as opposed to physical as I infer from :
yaḍurrūkum seems to be a physical harm and adhan is the exception.
The mental aspect of "adhan" could also be backed up from 2:196 which indicates the adhan could be mental due to the use of "rasihum" (head)
- fit'nata (29:10)
- given ultimatums, being driven out of homes (
natawakkala (trust - from 14:12)