Why is my mail notification stuck on the lock screen?

[Windows Central Question]

On my Lumia 950, no matter what I do, the mail notification on the lock screen is always set to 2. I've switched the notifications on and off, rebooted, read all my mail etc etc. There is always a little Outlook icon on my lock screen indicating that there are 2 new mails for me.

I'm part of the Windows Insider program on the Fast track.