The problem is calendars are thought just as events containers and not as a way of categorizing events (or restricting access to them). Our Calendar/Category restriction modules are only to restrict events access to people who is not allowed to create events in those categories/calendars.
The only way I can think of doing this is using the User Authorised Mode in JEvents, which allows you really fine grain permission setting per user. This obligues you to set permissions user by user or to use the Auto Create Authorised user plugin to set them automatically on user creation:
https://www.jevents.net/products-new/ad ... ised-users