Yes it 100% a limit in the system so its only your main calendar that will work.
I got round it buts its not simple.
Baiscally you have to create multiple google accounts. So I created myemail-2 -3 -4 etc.
Then export your extra calendars from your main account (Just google how to do it).
Rename the default calendar in your new accounts to match the names of your exported ones and import them into the defaults.
Then just share the calendars with your main google account and give yourself full control.
This way you still control all our google calendars from the one account on the web but you can setup all of the google accounts on your phone and then see all the calendars.
Like I said a lot of work but if you really need your calendars then this works.