The biggest deciding factor I have here is cost.
The Lumia 640 will be the most cost-effective option across the board. It lacks a couple of the 650's features like tap-to-pay and the premium build for example, but it is likely to be the cheapest option. It was available on AT&T for $30 off-contract for some time and while prices online may not match that price, $50 is still a good price. It is relatively easy to repair and parts aren't pricey. ($20 for a new display and I replaced it without needing to fuss with melting glue and stuff like one some phones. You'll need some of that double-sided tape though) The 830 isn't bad either but I haven't broken an 830 to-date.
Battery life is superior on the 640 but whether it is worth it depends. The 830 and 650 are still excellent/good performers.
I don't know about the 640 XL but it's larger and with a nicer camera. Battery life is probably better but I don't recall off the top of my head.
The 830 has the most bells-and-whistles here. It has Qi charging, a camera superior to the three other options and a camera button to boot. It's a bit more compact too despite the same sized display.
The advantage of the 650 is that it is most recent and if you need to restore it using WDRT you won't be thrown back to 8.1. Also you get some stuff as mentioned above and call recording. If you want an OLED screen then this is the phone. The others have an LCD. (I prefer the latter but the former is certainly compelling to many.) But ultimately these devices aren't too different and I would see what I can get for the best price.