I am brand new to the smartphone world. Before my Lumia 520 phone I did not have a smartphone. As such, I am also new to this subreddit, but I have spent a few days searching for a resolution to this issue, without any good write-up, so I apologize if I overlooked a previous solution to this issue.
I recently have come across the song duplication issue on the XBOX Music app. After searching this sub, as well as the wpcentral forums, I have found a suitable solution to the issue.
(For clarification, I am using a Lumia 520 phone with Windows 8.0.)
First off: How do I delete non-functioning and duplicate tracks that are listed in my phone?
Delete duplicate tracks/artists/albums off your SD first. Now open up your phone's HD on explorer. Go to 'Music' and you'll find files under 'Albums' and 'Artists' that were either duplicates or non-functioning album or artists files. Delete them here now and they'll no longer exist on your phone. You can check right after doing this to make sure. You may need to go to Properties->::click on:: hidden files, in order to view all files that need to be deleted. I advise you keep any files you don't want deleted that are not duplicates or non-functioning.
This next step may not be necessary and I advise against it if you are trying to keep some of your files on the SD card (again, which are NOT present in duplicates or non-functioning). In fact, it may not be necessary at all in the first place, but it won't hurt unless you just really want to keep certain files
If you are exclusively using your SD card for music and videos, I might suggest you format it under settings->phone storage->::click on:: SD card->format SD card->YES. It will ask whether you'd like to store new pictures, music and videos on your SD->YES (I imagine most people are using SD card for this kind of storage.)
At this point, if you go to your XBOX Music app, you should find it is completely empty if you formatted. If you still find non-functioning or duplicates, make sure you deleted all hidden files and any strange or unknown files under ARTIST or ALBUM in your phone's HD.
Next: How do I add music without causing track duplication?
You'll need to download the Windows Phone app for desktop. Download and install it. From now on this will be the only way you'll add music or video files to your computer. If you've previously selected under settings to use your SD card as your primary storage device for pictures, music, and videos, they will automatically be saved here in the future using the desktop app.
After installation, make sure you click on 'Sync music'. (I believe at some point during installation it will ask you how you how you want to sync--I chose iTunes, but WMP and manual are options, but I can't attest to whether that will ultimately solve the duplication issue. It works for me using iTunes.)
After your desktop app/music folder/music software are synced, all your music/media folders will be listed on the desktop app.
Restart your desktop app. Restart your phone.
At this point, you'll probably want to do a trial run, particularly with an album that you know always creates duplicate tracks on your phone. I suggest using the search bar as a quick and easy way to find the artist or album you're looking to add, as the desktop app interface is very slow when browsing.
Check your album of choice and select the black 'Sync' button at the bottom right in the desktop app. It should upload only the number of files that there are tracks.
Check your XBOX Music app. You should now only have the one album you have synced, with no duplicate tracks.
Trouble Shooting:
Make sure you check on 'Hidden' under 'Properties' to view hidden files in your Music, Artist, or Album folders on your phone, to make sure you have deleted the music you no longer want. Any files under Artist or Album you do not recognize should also be deleted.
If you already had the Windows app for desktop prior to following the above procedure, it will still try to sync files you previously had on your phone. If do you not want these files, you will have to uncheck them manually on the desktop app. If you do want them, they will be synced without duplication.
Final thoughts:
This appears to be an SD-specific issue. It's my opinion that track duplication happens because your phone tries to sync files that are added to your SD card automatically. It stores this sync data on the phone's HD drive regardless of whether the files are stored on your HD or SD card. When you access these files on the XBOX music app, it displays the synced information from the HD and the information from the SD, leading to duplicate files. When you delete the files manually from the SD card in explorer, it does not delete the sync files on the HD, and so you then continue to see non-functioning tracks. By using the sync function in the Windows Phone app for desktop, it still creates these sync files on your phone's HD, but now it knows not to create duplicate tracks on the phone/music app, and it then deletes these synced files both from the SD AND the phone's HD when you remove them using the desktop app, thereby cleanly removing your tracks from your XBOX music app.
Thanks for taking the time to read this. I hope it works for you, as it took me a bit of research and then trial and error to finally get a flawless system going. If anyone has any suggestions or questions I will happy to edit or clarify to the best of my ability. As I said before I am new to the smartphone world, but the above system worked great for me, and I hope it will for you too. The most important thing is to make sure you delete all those visible AND hidden files (ALL OF THEM) within your phone's Artist and Album folders under Music, then only use the desktop app for adding files.
TL;DR: Manually delete ALL duplicate and non-functioning files on your SD AND HD. Install Windows Phone app for desktop and use exclusively for file transfers to SD. No more duplicates or non-functioning tracks. Thanks.