If you can recall, did this start happening about the time that your account was migrated to the new Outlook? That's when mine started happening a couple of weeks ago. I found a potential fix at
this link but I had mixed results with it. If I turned off get to know me and cleared the Cortana data in the cloud, I was able to get at least one more person added back to the list that stayed on the list (however I think I may have enabled "get to know me again" before adding others which may have caused the mixed results). Give that link a try and see if that works.