I've experience this before. Last night, I cached the rigid bodies and that didn't even help. I think this is a bug and since we have the new rigid bodies, I don't think they will ever address this issue. The problem is that in a lot of cases, the new rigid body collisions don't work as well as the Bullet rigid bodies.