I don't have any specific recommendations for a very young child (mine are older), but I have some general suggestions. I don't know if you're familiar with the Windows 8 family safety settings, but if not (or for the benefit of any other readers who may not be), you can set your own account as the administrator for the tablet, password protected of course, and then create a child account for the child. The child account can just be a local account without a password, maybe with your child's picture or a favorite character or toy as the account picture.
In the tablet's account settings, family safety settings will take you to a web based interface where you actually have some pretty granular controls for setting access restrictions, including locking out some or pretty much all desktop apps (probably a good choice for a very young child.). Then you can go into the child's account and at the very least unpin (or even uninstall) any apps the child won't need (email for example). Go into the store app in the child's account and set it up with your information so you can get apps into the account. When you have the apps installed that you want, make sure to either set it to "Always ask for your password when buying an app" or use the "Change User" button to switch it back to the local account so the child can't accidentally buy a bunch of apps. (You can tap the button and then it'll ask for another Microsoft account; If you cancel, I think it ends up signing out your account, leaving the store app with no active MS account.) Probably also a good idea to unpin the store from the start screen.
One other thing you may find useful for a very young child is the "assigned access" feature. In your administrator account, you can use that to restrict the child account to running just a single app (a movie player or game) when the child signs in. The child then can't quit the app or bring up the charms or task switcher or start screen. It take 5 quick taps on the Windows button to sign out of the account back to the login window.