Today I was reading a bit about CiV AI on the civfanatics forums. There are plenty of complaints about the AI, which I have talked about at great length before, but something came up this time that really got me understanding what a brutal problem the designers face. It is somewhat similar to the problem faced by Wizards in trying to design the next generation of DnD - that is, both are trying to design systems of rules that can literally cope with anything a person's imagination can throw into them.
As a counterexample, the AI in Starcraft 2 has a very limited set of options to take and can assume many things. It can simply follow a script to build 4 Zealots, 3 Stalkers, 1 Immortal and then rush. Nothing wrong with that build and as long as there is some land route to the enemy base everything works out fine. The SC2 AI has several different sets of approaches it uses to play a game but it does follow some simple scripts each time; this works because it always has the same buildings and units to choose from. If you changed Zealots to come out of Forges instead of Gateways the AI would fail completely and wouldn't be able to play the game at all.
This is exactly the problem CiV faces since it is a game built from the outset to be moddable. The AI cannot be assigned build orders or any specific strategies surrounding what to do because any unit, building, improvement, terrain feature or even mechanic might not be there in a particular mod and the AI still needs to be able to play. How can you possibly design a strong AI for a game where you don't even know what the rules are? You could tell the AI to build specialized cities where one city for example tries to make lots of gold by building buildings and improvements that maximize gold output. This utterly falls apart though when there aren't any buildings in the game that give a % increase to gold output or when all cities can get them. You can't even hardcode in endgame strategies like building spaceships for victory because that victory condition might be turned off or changed to something else entirely!
DnD is in a similar boat. Some people want it to be a rules system for optimizing drop in games that are mostly dungeon crawls with ad hoc rules for noncombat situations. At this task DnD 4th is great. Some people want erotically charged sexventures. At this task DnD 4th is .... lacking. No matter what the system tries to do it will fail much of the time because nobody can actually tell you what it is *supposed* to do without being contradicted by a thousand others. This really hit me when I read the design statement for DnD Next which pretty much said that they wanted to make DnD better for what people wanted. They left out what exactly it was that people wanted, and wisely so, because there just isn't any sort of useful answer and I think they know it. There is no consistent vision for what DnD is supposed to do and this leaves it in a very difficult place.
I don't have a good answer for these situations. It is great that CiV is moddable and that the AI really does play the mods quite reasonably (if you count its normal play as reasonable!). It is fantastic that so many people with different desires can play DnD and have fun. It is, however, guaranteed that because both the CiV AI and the DnD rules try to please everyone and let people's imaginations roam free that they will forever be mired in mediocrity for pretty much everyone.
Penny Arcade has a fantastic comic about this from which I stole the phrase "erotically charged sexventure". From: http://penny-arcade.com/comic/2012/01/11