There was a post on Gamasutra by one of the Civ IV devs talking about this very problem. It turns out that most players don't really want an enemy that makes perfect moves most of the time. The problem with AI is not making it hard enough, but making it feel good to win against. That's a much more difficult problem than making a good AI in many games.