Not sure what caused my issue a couple of weeks ago, I was using Skype preview to send SMS and it made a real mess of my contacts both in the preview but was infecting my contacts in the People app, so I decided to break the SMS through Skype, and coincidently during that process all my contacts in the People app lost their pictures, so I whilst having a tidy up I looked in Outlook online and I had contacts from Google from years ago showing up (Many I had deleted) and it also looked a mess so I made sure I had no contacts in Google then took off the link to allow contacts showing up, somehow during that process my online and subsequently People app tidied up, I still had to add in pictures again but all has been stable since then.
Can't say if it was Skype preview, Google contacts or Outlook/live that had messed up but I am now using Skype preview for Skyping and messaging and Cortana to send/receive SMS and I am much happier with the result.