Not sure about a ringing alarm condition, but you can have Cortana cancel an alarm by saying "Cancel (alarm name)". It will stop the recurrence of the alarm, though, so you would have to later re-enable the alarm.
I had discovered that when all my alarms were left with the default name of "alarm" that Cortana would ask which one to cancel, even though 2 of the 3 were already disabled. So I deleted the unused ones, and renamed the recurring and active alarm "Weekday Alarm". Now I can tell Cortana to "Cancel Weekday Alarm" and it does.
I just now ran a test, created a "Alarm Test", put on the Bluetooth, and waited for the test alarm to go off. When it did, I activated Cortana, asking her to cancel the "Alarm Test". She responded: "Cancel alarm . . . Okay, for which time?" I said "8:16 AM" She replied, "Sure, cancelling alarm for 8:16." She disabled the alarm, but the alarm condition persisted until I manually dismissed it. So I guess Cortana currently can't stop an occurring alarm condition.
Update: I later considered using the word "dismiss" rather than "cancel" but my subsequent test got the same result: It would again ask for the time of the alarm to be canceled (apparently naming an alarm does not help when there is more than 1 alarm stored) and while it cancelled the alarm it did not stop the alarm condition.