Yeah, of course the culture could just obliterate the empire militarily. I think only a few civilizations (pre ascension) could hold their own for awhile.

You bring up a good point about shattering the view of the game though.