Yeah, Alberto has hit it exactly, if the app doesn't implement notifications 100% correctly, then I'm sure dismiss (amongst other things) is a wash, and your app won't show in the Cortana list, either.
The Skype Beta app is all kinds of messed-up, IMHO; I'm surprised they even ship it.
I can't toggle it, as mentioned, and I'm sure I could find it buried in the registry or obscure file somewhere, but it's just not useful, if you ask me. I tried un-install and re-install too, same deal.
I'm not on the latest insider build though, or app build, I sort of gave up after a ditched my 950XL, after the AU. Things got way too unstable on my phone, and I decided I didn't need that on my PC/laptop...