So for starters, there's a huge difference between megabits (Mb) and megabytes (MB). 1MB = 8Mb. Your download speed of 50 is in megabits per second or Mbps. So in reality, you're only downloading at a rate of 6.25 MBps at full speed. The fact that you're wired to a gigabit router or whatever means absolutely nothing. At a rate of 6.25 MBps, it would take you approximately one hour to download a 20GB (1GB = 1024MB). That's if there is no other load on your own network, your ISP's network, interference, packet loss, etc., etc. So two hours to download 20GB or three hours for 30GB isn't unreasonable.
As I said in a previous post, if you're installing from a disc, ensure you're in offline mode first. Otherwise, as soon as you tell it to install, it will begin downloading the game from the Xbox servers with the latest updates. It will completely ignore the disc. You should be able to cut down your install time to 10-15 minutes by installing from disc, and then spend another 10-15 minutes installing whatever updates/patches there have been for the game.