This is becoming an issue. I have a phone and desktop with "Hey Cortana" active, and they both listen and answer.
Say "Hey Cortana! Tell me a joke.", and both of them start talking. Humorous - but,I wish there was a way to either address a specific device, or easily, quickly and temporarily...