You can't delete IE, but you may try Kid's Corner. Internet Explorer is not accessible in this mode.
1. Go to Settings.
2. Find Kid's Corner.
3. Turn it on.
4. Choose your options, select apps to be available for your kid.
5. Tap "Run Kid's Corner".
You can access it via lock screen -> swipe right. Better set a password to prevent your kid from accessing your
Start Screen. Take note that your password will be applied to both your phone and Kid's Corner. (a bit hassle)
Or... Try an easier way: Give your boy a little beating whenever he touches Internet Explorer. Sounds simple?
![Stick out tongue :p :p](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)