Please forgive me for butting in, but with regard to caching, I've just spent about a month wrestling this issue to the ground... and I've learned more than I wanted to know about joomla caching etc.
First, and I'm sure that MANY people will tell you this, download and install Jotcache. this will solve about 50% of your issues out of the box that the built in cache does not solve.
This is not and ad for Jotcache... its free. and I have no vested interest other than to say, that after diving deep into the subject, This is THE caching solution for joomla. Don't let it scare you, its pretty complex, but works out of the box. just install it, turn off the system cache (and page cache if you're using it), and you'll notice an amazing improvement over the built in cache. As you get more familiar with it, you will discover the item specific caching features that can really help you tune Jevents performance.
I'd also recommend turning on gzip compression unless you have a reason not to. between this and Jotcache most of your issues will be solved.
Next, if you're using shared hosting, this is also probably effecting you. The situation you describe could cause literally hundreds of calls to the database for each load. Most shared hosting plans will see this, and throttle you... sad but true.
I have Jevents installed for one of my clients with 850+ employees, and they have regular online meetings that involve hundreds of people. (which is the reason I needed to become joomla cache educated). On a VPS server with 4 CPU cores allocated, and nothing else special, my page loads for Jevents pages even with events that involve hundreds of people are under 2 seconds.
Hope this is of some help to you, I know how frustrating these issues can be.
Warm Regards,
Trauncher.