{ "Part I": { "part": "Part I", "chapters": [ { "chapter": 1, "text": "1\nINTRODUCTION\nIn which we try to explain why we consider artificial intelligence to be a subject\nmost worthy of study, and in which we try to decide what exactly it is, this being a\ngood thing to decide before embarking.\nWe call ourselves Homo sapiens—man the wise—because our intelligence is so important\nINTELLIGENCE\nto us. For thousands of years, we have tried to understand how we think; that is, how a mere\nhandful of matter can perceive, understand, predict, and manipulate a world far larger and\nmore complicated than itself. The field of artificial intelligence, or AI, goes further still: it\nARTIFICIAL\nINTELLIGENCE\nattempts not just to understand but also to build intelligent entities.\nAI is one of the newest fields in science and engineering. Work started in earnest soon\nafter World War II, and the name itself was coined in 1956. Along with molecular biology,\nAI is regularly cited as the “field I would most like to be in” by scientists in other disciplines.\nA student in physics might reasonably feel that all the good ideas have already been taken by\nGalileo, Newton, Einstein, and the rest. AI, on the other hand, still has openings for several\nfull-time Einsteins and Edisons.\nAI currently encompasses a huge variety of subfields, ranging from the general (learning\nand perception) to the specific, such as playing chess, proving mathematical theorems, writing\npoetry, driving a car on a crowded street, and diagnosing diseases. AI is relevant to any\nintellectual task; it is truly a universal field.\n1.1\nWHAT IS AI?\nWe have claimed that AI is exciting, but we have not said what it is. In Figure 1.1 we see\neight definitions of AI, laid out along two dimensions. The definitions on top are concerned\nwith thought processes and reasoning, whereas the ones on the bottom address behavior. The\ndefinitions on the left measure success in terms of fidelity to human performance, whereas\nthe ones on the right measure against an ideal performance measure, called rationality. A\nRATIONALITY\nsystem is rational if it does the “right thing,” given what it knows.\nHistorically, all four approaches to AI have been followed, each by different people\nwith different methods. A human-centered approach must be in part an empirical science, in-\n1\n2\nChapter\n1.\nIntroduction\nThinking Humanly\nThinking Rationally\n“The exciting new effort to make comput-\ners think . . . machines with minds, in the\nfull and literal sense.” (Haugeland, 1985)\n“The study of mental faculties through the\nuse of computational models.”\n(Charniak and McDermott, 1985)\n“[The automation of] activities that we\nassociate with human thinking, activities\nsuch as decision-making, problem solv-\ning, learning . . .” (Bellman, 1978)\n“The study of the computations that make\nit possible to perceive, reason, and act.”\n(Winston, 1992)\nActing Humanly\nActing Rationally\n“The art of creating machines that per-\nform functions that require intelligence\nwhen performed by people.” (Kurzweil,\n1990)\n“Computational Intelligence is the study\nof the design of intelligent agents.” (Poole\net al., 1998)\n“The study of how to make computers do\nthings at which, at the moment, people are\nbetter.” (Rich and Knight, 1991)\n“AI . . . is concerned with intelligent be-\nhavior in artifacts.” (Nilsson, 1998)\nFigure 1.1\nSome definitions of artificial intelligence, organized into four categories.\nvolving observations and hypotheses about human behavior. A rationalist1 approach involves\na combination of mathematics and engineering. The various group have both disparaged and\nhelped each other. Let us look at the four approaches in more detail.\n1.1.1\nActing humanly: The Turing Test approach\nThe Turing Test, proposed by Alan Turing (1950), was designed to provide a satisfactory\nTURING TEST\noperational definition of intelligence. A computer passes the test if a human interrogator, after\nposing some written questions, cannot tell whether the written responses come from a person\nor from a computer. Chapter 26 discusses the details of the test and whether a computer would\nreally be intelligent if it passed. For now, we note that programming a computer to pass a\nrigorously applied test provides plenty to work on. The computer would need to possess the\nfollowing capabilities:\n• natural language processing to enable it to communicate successfully in English;\nNATURAL LANGUAGE\nPROCESSING\n• knowledge representation to store what it knows or hears;\nKNOWLEDGE\nREPRESENTATION\n• automated reasoning to use the stored information to answer questions and to draw\nAUTOMATED\nREASONING\nnew conclusions;\n• machine learning to adapt to new circumstances and to detect and extrapolate patterns.\nMACHINE LEARNING\n1 By distinguishing between human and rational behavior, we are not suggesting that humans are necessarily\n“irrational” in the sense of “emotionally unstable” or “insane.” One merely need note that we are not perfect:\nnot all chess players are grandmasters; and, unfortunately, not everyone gets an A on the exam. Some systematic\nerrors in human reasoning are cataloged by Kahneman et al. (1982).\nSection 1.1.\nWhat Is AI?\n3\nTuring’s test deliberately avoided direct physical interaction between the interrogator and the\ncomputer, because physical simulation of a person is unnecessary for intelligence. However,\nthe so-called total Turing Test includes a video signal so that the interrogator can test the\nTOTAL TURING TEST\nsubject’s perceptual abilities, as well as the opportunity for the interrogator to pass physical\nobjects “through the hatch.” To pass the total Turing Test, the computer will need\n• computer vision to perceive objects, and\nCOMPUTER VISION\n• robotics to manipulate objects and move about.\nROBOTICS\nThese six disciplines compose most of AI, and Turing deserves credit for designing a test\nthat remains relevant 60 years later. Yet AI researchers have devoted little effort to passing\nthe Turing Test, believing that it is more important to study the underlying principles of in-\ntelligence than to duplicate an exemplar. The quest for “artificial flight” succeeded when the\nWright brothers and others stopped imitating birds and started using wind tunnels and learn-\ning about aerodynamics. Aeronautical engineering texts do not define the goal of their field\nas making “machines that fly so exactly like pigeons that they can fool even other pigeons.”\n1.1.2\nThinking humanly: The cognitive modeling approach\nIf we are going to say that a given program thinks like a human, we must have some way of\ndetermining how humans think. We need to get inside the actual workings of human minds.\nThere are three ways to do this: through introspection—trying to catch our own thoughts as\nthey go by; through psychological experiments—observing a person in action; and through\nbrain imaging—observing the brain in action. Once we have a sufficiently precise theory of\nthe mind, it becomes possible to express the theory as a computer program. If the program’s\ninput–output behavior matches corresponding human behavior, that is evidence that some of\nthe program’s mechanisms could also be operating in humans. For example, Allen Newell\nand Herbert Simon, who developed GPS, the “General Problem Solver” (Newell and Simon,\n1961), were not content merely to have their program solve problems correctly. They were\nmore concerned with comparing the trace of its reasoning steps to traces of human subjects\nsolving the same problems. The interdisciplinary field of cognitive science brings together\nCOGNITIVE SCIENCE\ncomputer models from AI and experimental techniques from psychology to construct precise\nand testable theories of the human mind.\nCognitive science is a fascinating field in itself, worthy of several textbooks and at least\none encyclopedia (Wilson and Keil, 1999). We will occasionally comment on similarities or\ndifferences between AI techniques and human cognition. Real cognitive science, however, is\nnecessarily based on experimental investigation of actual humans or animals. We will leave\nthat for other books, as we assume the reader has only a computer for experimentation.\nIn the early days of AI there was often confusion between the approaches: an author\nwould argue that an algorithm performs well on a task and that it is therefore a good model\nof human performance, or vice versa. Modern authors separate the two kinds of claims;\nthis distinction has allowed both AI and cognitive science to develop more rapidly. The two\nfields continue to fertilize each other, most notably in computer vision, which incorporates\nneurophysiological evidence into computational models.\n4\nChapter\n1.\nIntroduction\n1.1.3\nThinking rationally: The “laws of thought” approach\nThe Greek philosopher Aristotle was one of the first to attempt to codify “right thinking,” that\nis, irrefutable reasoning processes. His syllogisms provided patterns for argument structures\nSYLLOGISM\nthat always yielded correct conclusions when given correct premises—for example, “Socrates\nis a man; all men are mortal; therefore, Socrates is mortal.” These laws of thought were\nsupposed to govern the operation of the mind; their study initiated the field called logic.\nLOGIC\nLogicians in the 19th century developed a precise notation for statements about all kinds\nof objects in the world and the relations among them. (Contrast this with ordinary arithmetic\nnotation, which provides only for statements about numbers.) By 1965, programs existed\nthat could, in principle, solve any solvable problem described in logical notation. (Although\nif no solution exists, the program might loop forever.) The so-called logicist tradition within\nLOGICIST\nartificial intelligence hopes to build on such programs to create intelligent systems.\nThere are two main obstacles to this approach. First, it is not easy to take informal\nknowledge and state it in the formal terms required by logical notation, particularly when\nthe knowledge is less than 100% certain. Second, there is a big difference between solving\na problem “in principle” and solving it in practice. Even problems with just a few hundred\nfacts can exhaust the computational resources of any computer unless it has some guidance\nas to which reasoning steps to try first. Although both of these obstacles apply to any attempt\nto build computational reasoning systems, they appeared first in the logicist tradition.\n1.1.4\nActing rationally: The rational agent approach\nAn agent is just something that acts (agent comes from the Latin agere, to do). Of course,\nAGENT\nall computer programs do something, but computer agents are expected to do more: operate\nautonomously, perceive their environment, persist over a prolonged time period, adapt to\nchange, and create and pursue goals. A rational agent is one that acts so as to achieve the\nRATIONAL AGENT\nbest outcome or, when there is uncertainty, the best expected outcome.\nIn the “laws of thought” approach to AI, the emphasis was on correct inferences. Mak-\ning correct inferences is sometimes part of being a rational agent, because one way to act\nrationally is to reason logically to the conclusion that a given action will achieve one’s goals\nand then to act on that conclusion. On the other hand, correct inference is not all of ration-\nality; in some situations, there is no provably correct thing to do, but something must still be\ndone. There are also ways of acting rationally that cannot be said to involve inference. For\nexample, recoiling from a hot stove is a reflex action that is usually more successful than a\nslower action taken after careful deliberation.\nAll the skills needed for the Turing Test also allow an agent to act rationally. Knowledge\nrepresentation and reasoning enable agents to reach good decisions. We need to be able to\ngenerate comprehensible sentences in natural language to get by in a complex society. We\nneed learning not only for erudition, but also because it improves our ability to generate\neffective behavior.\nThe rational-agent approach has two advantages over the other approaches. First, it\nis more general than the “laws of thought” approach because correct inference is just one\nof several possible mechanisms for achieving rationality. Second, it is more amenable to\nSection 1.2.\nThe Foundations of Artificial Intelligence\n5\nscientific development than are approaches based on human behavior or human thought. The\nstandard of rationality is mathematically well defined and completely general, and can be\n“unpacked” to generate agent designs that provably achieve it. Human behavior, on the other\nhand, is well adapted for one specific environment and is defined by, well, the sum total\nof all the things that humans do. This book therefore concentrates on general principles\nof rational agents and on components for constructing them. We will see that despite the\napparent simplicity with which the problem can be stated, an enormous variety of issues\ncome up when we try to solve it. Chapter 2 outlines some of these issues in more detail.\nOne important point to keep in mind: We will see before too long that achieving perfect\nrationality—always doing the right thing—is not feasible in complicated environments. The\ncomputational demands are just too high. For most of the book, however, we will adopt the\nworking hypothesis that perfect rationality is a good starting point for analysis. It simplifies\nthe problem and provides the appropriate setting for most of the foundational material in\nthe field. Chapters 5 and 17 deal explicitly with the issue of limited rationality—acting\nLIMITED\nRATIONALITY\nappropriately when there is not enough time to do all the computations one might like.\n1.2\nTHE FOUNDATIONS OF ARTIFICIAL INTELLIGENCE\nIn this section, we provide a brief history of the disciplines that contributed ideas, viewpoints,\nand techniques to AI. Like any history, this one is forced to concentrate on a small number\nof people, events, and ideas and to ignore others that also were important. We organize the\nhistory around a series of questions. We certainly would not wish to give the impression that\nthese questions are the only ones the disciplines address or that the disciplines have all been\nworking toward AI as their ultimate fruition.\n1.2.1\nPhilosophy\n• Can formal rules be used to draw valid conclusions?\n• How does the mind arise from a physical brain?\n• Where does knowledge come from?\n• How does knowledge lead to action?\nAristotle (384–322 B.C.), whose bust appears on the front cover of this book, was the first\nto formulate a precise set of laws governing the rational part of the mind. He developed an\ninformal system of syllogisms for proper reasoning, which in principle allowed one to gener-\nate conclusions mechanically, given initial premises. Much later, Ramon Lull (d. 1315) had\nthe idea that useful reasoning could actually be carried out by a mechanical artifact. Thomas\nHobbes (1588–1679) proposed that reasoning was like numerical computation, that “we add\nand subtract in our silent thoughts.” The automation of computation itself was already well\nunder way. Around 1500, Leonardo da Vinci (1452–1519) designed but did not build a me-\nchanical calculator; recent reconstructions have shown the design to be functional. The first\nknown calculating machine was constructed around 1623 by the German scientist Wilhelm\nSchickard (1592–1635), although the Pascaline, built in 1642 by Blaise Pascal (1623–1662),\n6\nChapter\n1.\nIntroduction\nis more famous. Pascal wrote that “the arithmetical machine produces effects which appear\nnearer to thought than all the actions of animals.” Gottfried Wilhelm Leibniz (1646–1716)\nbuilt a mechanical device intended to carry out operations on concepts rather than numbers,\nbut its scope was rather limited. Leibniz did surpass Pascal by building a calculator that\ncould add, subtract, multiply, and take roots, whereas the Pascaline could only add and sub-\ntract. Some speculated that machines might not just do calculations but actually be able to\nthink and act on their own. In his 1651 book Leviathan, Thomas Hobbes suggested the idea\nof an “artificial animal,” arguing “For what is the heart but a spring; and the nerves, but so\nmany strings; and the joints, but so many wheels.”\nIt’s one thing to say that the mind operates, at least in part, according to logical rules, and\nto build physical systems that emulate some of those rules; it’s another to say that the mind\nitself is such a physical system. Ren´e Descartes (1596–1650) gave the first clear discussion\nof the distinction between mind and matter and of the problems that arise. One problem with\na purely physical conception of the mind is that it seems to leave little room for free will:\nif the mind is governed entirely by physical laws, then it has no more free will than a rock\n“deciding” to fall toward the center of the earth. Descartes was a strong advocate of the power\nof reasoning in understanding the world, a philosophy now called rationalism, and one that\nRATIONALISM\ncounts Aristotle and Leibnitz as members. But Descartes was also a proponent of dualism.\nDUALISM\nHe held that there is a part of the human mind (or soul or spirit) that is outside of nature,\nexempt from physical laws. Animals, on the other hand, did not possess this dual quality;\nthey could be treated as machines. An alternative to dualism is materialism, which holds\nMATERIALISM\nthat the brain’s operation according to the laws of physics constitutes the mind. Free will is\nsimply the way that the perception of available choices appears to the choosing entity.\nGiven a physical mind that manipulates knowledge, the next problem is to establish\nthe source of knowledge. The empiricism movement, starting with Francis Bacon’s (1561–\nEMPIRICISM\n1626) Novum Organum,2 is characterized by a dictum of John Locke (1632–1704): “Nothing\nis in the understanding, which was not first in the senses.” David Hume’s (1711–1776) A\nTreatise of Human Nature (Hume, 1739) proposed what is now known as the principle of\ninduction: that general rules are acquired by exposure to repeated associations between their\nINDUCTION\nelements. Building on the work of Ludwig Wittgenstein (1889–1951) and Bertrand Russell\n(1872–1970), the famous Vienna Circle, led by Rudolf Carnap (1891–1970), developed the\ndoctrine of logical positivism. This doctrine holds that all knowledge can be characterized by\nLOGICAL POSITIVISM\nlogical theories connected, ultimately, to observation sentences that correspond to sensory\nOBSERVATION\nSENTENCES\ninputs; thus logical positivism combines rationalism and empiricism.3 The confirmation the-\nory of Carnap and Carl Hempel (1905–1997) attempted to analyze the acquisition of knowl-\nCONFIRMATION\nTHEORY\nedge from experience. Carnap’s book The Logical Structure of the World (1928) defined an\nexplicit computational procedure for extracting knowledge from elementary experiences. It\nwas probably the first theory of mind as a computational process.\n2 The Novum Organum is an update of Aristotle’s Organon, or instrument of thought. Thus Aristotle can be\nseen as both an empiricist and a rationalist.\n3 In this picture, all meaningful statements can be verified or falsified either by experimentation or by analysis\nof the meaning of the words. Because this rules out most of metaphysics, as was the intention, logical positivism\nwas unpopular in some circles.\nSection 1.2.\nThe Foundations of Artificial Intelligence\n7\nThe final element in the philosophical picture of the mind is the connection between\nknowledge and action. This question is vital to AI because intelligence requires action as well\nas reasoning. Moreover, only by understanding how actions are justified can we understand\nhow to build an agent whose actions are justifiable (or rational). Aristotle argued (in De Motu\nAnimalium) that actions are justified by a logical connection between goals and knowledge of\nthe action’s outcome (the last part of this extract also appears on the front cover of this book,\nin the original Greek):\nBut how does it happen that thinking is sometimes accompanied by action and sometimes\nnot, sometimes by motion, and sometimes not? It looks as if almost the same thing\nhappens as in the case of reasoning and making inferences about unchanging objects. But\nin that case the end is a speculative proposition . . . whereas here the conclusion which\nresults from the two premises is an action. . . . I need covering; a cloak is a covering. I\nneed a cloak. What I need, I have to make; I need a cloak. I have to make a cloak. And\nthe conclusion, the “I have to make a cloak,” is an action.\nIn the Nicomachean Ethics (Book III. 3, 1112b), Aristotle further elaborates on this topic,\nsuggesting an algorithm:\nWe deliberate not about ends, but about means. For a doctor does not deliberate whether\nhe shall heal, nor an orator whether he shall persuade, . . . They assume the end and\nconsider how and by what means it is attained, and if it seems easily and best produced\nthereby; while if it is achieved by one means only they consider how it will be achieved\nby this and by what means this will be achieved, till they come to the first cause, . . . and\nwhat is last in the order of analysis seems to be first in the order of becoming. And if we\ncome on an impossibility, we give up the search, e.g., if we need money and this cannot\nbe got; but if a thing appears possible we try to do it.\nAristotle’s algorithm was implemented 2300 years later by Newell and Simon in their GPS\nprogram. We would now call it a regression planning system (see Chapter 10).\nGoal-based analysis is useful, but does not say what to do when several actions will\nachieve the goal or when no action will achieve it completely. Antoine Arnauld (1612–1694)\ncorrectly described a quantitative formula for deciding what action to take in cases like this\n(see Chapter 16). John Stuart Mill’s (1806–1873) book Utilitarianism (Mill, 1863) promoted\nthe idea of rational decision criteria in all spheres of human activity. The more formal theory\nof decisions is discussed in the following section.\n1.2.2\nMathematics\n• What are the formal rules to draw valid conclusions?\n• What can be computed?\n• How do we reason with uncertain information?\nPhilosophers staked out some of the fundamental ideas of AI, but the leap to a formal science\nrequired a level of mathematical formalization in three fundamental areas: logic, computa-\ntion, and probability.\nThe idea of formal logic can be traced back to the philosophers of ancient Greece, but\nits mathematical development really began with the work of George Boole (1815–1864), who\n8\nChapter\n1.\nIntroduction\nworked out the details of propositional, or Boolean, logic (Boole, 1847). In 1879, Gottlob\nFrege (1848–1925) extended Boole’s logic to include objects and relations, creating the first-\norder logic that is used today.4 Alfred Tarski (1902–1983) introduced a theory of reference\nthat shows how to relate the objects in a logic to objects in the real world.\nThe next step was to determine the limits of what could be done with logic and com-\nputation. The first nontrivial algorithm is thought to be Euclid’s algorithm for computing\nALGORITHM\ngreatest common divisors. The word algorithm (and the idea of studying them) comes from\nal-Khowarazmi, a Persian mathematician of the 9th century, whose writings also introduced\nArabic numerals and algebra to Europe. Boole and others discussed algorithms for logical\ndeduction, and, by the late 19th century, efforts were under way to formalize general mathe-\nmatical reasoning as logical deduction. In 1930, Kurt G¨odel (1906–1978) showed that there\nexists an effective procedure to prove any true statement in the first-order logic of Frege and\nRussell, but that first-order logic could not capture the principle of mathematical induction\nneeded to characterize the natural numbers. In 1931, G¨odel showed that limits on deduc-\ntion do exist. His incompleteness theorem showed that in any formal theory as strong as\nINCOMPLETENESS\nTHEOREM\nPeano arithmetic (the elementary theory of natural numbers), there are true statements that\nare undecidable in the sense that they have no proof within the theory.\nThis fundamental result can also be interpreted as showing that some functions on the\nintegers cannot be represented by an algorithm—that is, they cannot be computed. This\nmotivated Alan Turing (1912–1954) to try to characterize exactly which functions are com-\nputable—capable of being computed. This notion is actually slightly problematic because\nCOMPUTABLE\nthe notion of a computation or effective procedure really cannot be given a formal definition.\nHowever, the Church–Turing thesis, which states that the Turing machine (Turing, 1936) is\ncapable of computing any computable function, is generally accepted as providing a sufficient\ndefinition. Turing also showed that there were some functions that no Turing machine can\ncompute. For example, no machine can tell in general whether a given program will return\nan answer on a given input or run forever.\nAlthough decidability and computability are important to an understanding of computa-\ntion, the notion of tractability has had an even greater impact. Roughly speaking, a problem\nTRACTABILITY\nis called intractable if the time required to solve instances of the problem grows exponentially\nwith the size of the instances. The distinction between polynomial and exponential growth\nin complexity was first emphasized in the mid-1960s (Cobham, 1964; Edmonds, 1965). It is\nimportant because exponential growth means that even moderately large instances cannot be\nsolved in any reasonable time. Therefore, one should strive to divide the overall problem of\ngenerating intelligent behavior into tractable subproblems rather than intractable ones.\nHow can one recognize an intractable problem? The theory of NP-completeness, pio-\nNP-COMPLETENESS\nneered by Steven Cook (1971) and Richard Karp (1972), provides a method. Cook and Karp\nshowed the existence of large classes of canonical combinatorial search and reasoning prob-\nlems that are NP-complete. Any problem class to which the class of NP-complete problems\ncan be reduced is likely to be intractable. (Although it has not been proved that NP-complete\n4 Frege’s proposed notation for first-order logic—an arcane combination of textual and geometric features—\nnever became popular.\nSection 1.2.\nThe Foundations of Artificial Intelligence\n9\nproblems are necessarily intractable, most theoreticians believe it.) These results contrast\nwith the optimism with which the popular press greeted the first computers—“Electronic\nSuper-Brains” that were “Faster than Einstein!” Despite the increasing speed of computers,\ncareful use of resources will characterize intelligent systems. Put crudely, the world is an\nextremely large problem instance! Work in AI has helped explain why some instances of\nNP-complete problems are hard, yet others are easy (Cheeseman et al., 1991).\nBesides logic and computation, the third great contribution of mathematics to AI is the\ntheory of probability. The Italian Gerolamo Cardano (1501–1576) first framed the idea of\nPROBABILITY\nprobability, describing it in terms of the possible outcomes of gambling events. In 1654,\nBlaise Pascal (1623–1662), in a letter to Pierre Fermat (1601–1665), showed how to pre-\ndict the future of an unfinished gambling game and assign average payoffs to the gamblers.\nProbability quickly became an invaluable part of all the quantitative sciences, helping to deal\nwith uncertain measurements and incomplete theories. James Bernoulli (1654–1705), Pierre\nLaplace (1749–1827), and others advanced the theory and introduced new statistical meth-\nods. Thomas Bayes (1702–1761), who appears on the front cover of this book, proposed\na rule for updating probabilities in the light of new evidence. Bayes’ rule underlies most\nmodern approaches to uncertain reasoning in AI systems.\n1.2.3\nEconomics\n• How should we make decisions so as to maximize payoff?\n• How should we do this when others may not go along?\n• How should we do this when the payoff may be far in the future?\nThe science of economics got its start in 1776, when Scottish philosopher Adam Smith\n(1723–1790) published An Inquiry into the Nature and Causes of the Wealth of Nations.\nWhile the ancient Greeks and others had made contributions to economic thought, Smith was\nthe first to treat it as a science, using the idea that economies can be thought of as consist-\ning of individual agents maximizing their own economic well-being. Most people think of\neconomics as being about money, but economists will say that they are really studying how\npeople make choices that lead to preferred outcomes. When McDonald’s offers a hamburger\nfor a dollar, they are asserting that they would prefer the dollar and hoping that customers will\nprefer the hamburger. The mathematical treatment of “preferred outcomes” or utility was\nUTILITY\nfirst formalized by L´eon Walras (pronounced “Valrasse”) (1834-1910) and was improved by\nFrank Ramsey (1931) and later by John von Neumann and Oskar Morgenstern in their book\nThe Theory of Games and Economic Behavior (1944).\nDecision theory, which combines probability theory with utility theory, provides a for-\nDECISION THEORY\nmal and complete framework for decisions (economic or otherwise) made under uncertainty—\nthat is, in cases where probabilistic descriptions appropriately capture the decision maker’s\nenvironment. This is suitable for “large” economies where each agent need pay no attention\nto the actions of other agents as individuals. For “small” economies, the situation is much\nmore like a game: the actions of one player can significantly affect the utility of another\n(either positively or negatively). Von Neumann and Morgenstern’s development of game\ntheory (see also Luce and Raiffa, 1957) included the surprising result that, for some games,\nGAME THEORY\n10\nChapter\n1.\nIntroduction\na rational agent should adopt policies that are (or least appear to be) randomized. Unlike de-\ncision theory, game theory does not offer an unambiguous prescription for selecting actions.\nFor the most part, economists did not address the third question listed above, namely,\nhow to make rational decisions when payoffs from actions are not immediate but instead re-\nsult from several actions taken in sequence. This topic was pursued in the field of operations\nresearch, which emerged in World War II from efforts in Britain to optimize radar installa-\nOPERATIONS\nRESEARCH\ntions, and later found civilian applications in complex management decisions. The work of\nRichard Bellman (1957) formalized a class of sequential decision problems called Markov\ndecision processes, which we study in Chapters 17 and 21.\nWork in economics and operations research has contributed much to our notion of ra-\ntional agents, yet for many years AI research developed along entirely separate paths. One\nreason was the apparent complexity of making rational decisions. The pioneering AI re-\nsearcher Herbert Simon (1916–2001) won the Nobel Prize in economics in 1978 for his early\nwork showing that models based on satisficing—making decisions that are “good enough,”\nSATISFICING\nrather than laboriously calculating an optimal decision—gave a better description of actual\nhuman behavior (Simon, 1947). Since the 1990s, there has been a resurgence of interest in\ndecision-theoretic techniques for agent systems (Wellman, 1995).\n1.2.4\nNeuroscience\n• How do brains process information?\nNeuroscience is the study of the nervous system, particularly the brain. Although the exact\nNEUROSCIENCE\nway in which the brain enables thought is one of the great mysteries of science, the fact that it\ndoes enable thought has been appreciated for thousands of years because of the evidence that\nstrong blows to the head can lead to mental incapacitation. It has also long been known that\nhuman brains are somehow different; in about 335 B.C. Aristotle wrote, “Of all the animals,\nman has the largest brain in proportion to his size.”5 Still, it was not until the middle of the\n18th century that the brain was widely recognized as the seat of consciousness. Before then,\ncandidate locations included the heart and the spleen.\nPaul Broca’s (1824–1880) study of aphasia (speech deficit) in brain-damaged patients\nin 1861 demonstrated the existence of localized areas of the brain responsible for specific\ncognitive functions. In particular, he showed that speech production was localized to the\nportion of the left hemisphere now called Broca’s area.6 By that time, it was known that\nthe brain consisted of nerve cells, or neurons, but it was not until 1873 that Camillo Golgi\nNEURON\n(1843–1926) developed a staining technique allowing the observation of individual neurons\nin the brain (see Figure 1.2). This technique was used by Santiago Ramon y Cajal (1852–\n1934) in his pioneering studies of the brain’s neuronal structures.7 Nicolas Rashevsky (1936,\n1938) was the first to apply mathematical models to the study of the nervous sytem.\n5 Since then, it has been discovered that the tree shrew (Scandentia) has a higher ratio of brain to body mass.\n6 Many cite Alexander Hood (1824) as a possible prior source.\n7 Golgi persisted in his belief that the brain’s functions were carried out primarily in a continuous medium in\nwhich neurons were embedded, whereas Cajal propounded the “neuronal doctrine.” The two shared the Nobel\nprize in 1906 but gave mutually antagonistic acceptance speeches.\nSection 1.2.\nThe Foundations of Artificial Intelligence\n11\nAxon\nCell body or Soma\nNucleus\nDendrite\nSynapses\nAxonal arborization\nAxon from another cell\nSynapse\nFigure 1.2\nThe parts of a nerve cell or neuron. Each neuron consists of a cell body,\nor soma, that contains a cell nucleus. Branching out from the cell body are a number of\nfibers called dendrites and a single long fiber called the axon. The axon stretches out for a\nlong distance, much longer than the scale in this diagram indicates. Typically, an axon is\n1 cm long (100 times the diameter of the cell body), but can reach up to 1 meter. A neuron\nmakes connections with 10 to 100,000 other neurons at junctions called synapses. Signals are\npropagated from neuron to neuron by a complicated electrochemical reaction. The signals\ncontrol brain activity in the short term and also enable long-term changes in the connectivity\nof neurons. These mechanisms are thought to form the basis for learning in the brain. Most\ninformation processing goes on in the cerebral cortex, the outer layer of the brain. The basic\norganizational unit appears to be a column of tissue about 0.5 mm in diameter, containing\nabout 20,000 neurons and extending the full depth of the cortex about 4 mm in humans).\nWe now have some data on the mapping between areas of the brain and the parts of the\nbody that they control or from which they receive sensory input. Such mappings are able to\nchange radically over the course of a few weeks, and some animals seem to have multiple\nmaps. Moreover, we do not fully understand how other areas can take over functions when\none area is damaged. There is almost no theory on how an individual memory is stored.\nThe measurement of intact brain activity began in 1929 with the invention by Hans\nBerger of the electroencephalograph (EEG). The recent development of functional magnetic\nresonance imaging (fMRI) (Ogawa et al., 1990; Cabeza and Nyberg, 2001) is giving neu-\nroscientists unprecedentedly detailed images of brain activity, enabling measurements that\ncorrespond in interesting ways to ongoing cognitive processes. These are augmented by\nadvances in single-cell recording of neuron activity. Individual neurons can be stimulated\nelectrically, chemically, or even optically (Han and Boyden, 2007), allowing neuronal input–\noutput relationships to be mapped. Despite these advances, we are still a long way from\nunderstanding how cognitive processes actually work.\nThe truly amazing conclusion is that a collection of simple cells can lead to thought,\naction, and consciousness or, in the pithy words of John Searle (1992), brains cause minds.\n12\nChapter\n1.\nIntroduction\nSupercomputer\nPersonal Computer\nHuman Brain\nComputational units\n104 CPUs, 1012 transistors 4 CPUs, 109 transistors 1011 neurons\nStorage units\n1014 bits RAM\n1011 bits RAM\n1011 neurons\n1015 bits disk\n1013 bits disk\n1014 synapses\nCycle time\n10−9 sec\n10−9 sec\n10−3 sec\nOperations/sec\n1015\n1010\n1017\nMemory updates/sec\n1014\n1010\n1014\nFigure 1.3\nA crude comparison of the raw computational resources available to the IBM\nBLUE GENE supercomputer, a typical personal computer of 2008, and the human brain. The\nbrain’s numbers are essentially fixed, whereas the supercomputer’s numbers have been in-\ncreasing by a factor of 10 every 5 years or so, allowing it to achieve rough parity with the\nbrain. The personal computer lags behind on all metrics except cycle time.\nThe only real alternative theory is mysticism: that minds operate in some mystical realm that\nis beyond physical science.\nBrains and digital computers have somewhat different properties. Figure 1.3 shows that\ncomputers have a cycle time that is a million times faster than a brain. The brain makes up\nfor that with far more storage and interconnection than even a high-end personal computer,\nalthough the largest supercomputers have a capacity that is similar to the brain’s. (It should\nbe noted, however, that the brain does not seem to use all of its neurons simultaneously.)\nFuturists make much of these numbers, pointing to an approaching singularity at which\nSINGULARITY\ncomputers reach a superhuman level of performance (Vinge, 1993; Kurzweil, 2005), but the\nraw comparisons are not especially informative. Even with a computer of virtually unlimited\ncapacity, we still would not know how to achieve the brain’s level of intelligence.\n1.2.5\nPsychology\n• How do humans and animals think and act?\nThe origins of scientific psychology are usually traced to the work of the German physi-\ncist Hermann von Helmholtz (1821–1894) and his student Wilhelm Wundt (1832–1920).\nHelmholtz applied the scientific method to the study of human vision, and his Handbook\nof Physiological Optics is even now described as “the single most important treatise on the\nphysics and physiology of human vision” (Nalwa, 1993, p.15). In 1879, Wundt opened the\nfirst laboratory of experimental psychology, at the University of Leipzig. Wundt insisted\non carefully controlled experiments in which his workers would perform a perceptual or as-\nsociative task while introspecting on their thought processes. The careful controls went a\nlong way toward making psychology a science, but the subjective nature of the data made\nit unlikely that an experimenter would ever disconfirm his or her own theories. Biologists\nstudying animal behavior, on the other hand, lacked introspective data and developed an ob-\njective methodology, as described by H. S. Jennings (1906) in his influential work Behavior of\nthe Lower Organisms. Applying this viewpoint to humans, the behaviorism movement, led\nBEHAVIORISM\nby John Watson (1878–1958), rejected any theory involving mental processes on the grounds\nSection 1.2.\nThe Foundations of Artificial Intelligence\n13\nthat introspection could not provide reliable evidence. Behaviorists insisted on studying only\nobjective measures of the percepts (or stimulus) given to an animal and its resulting actions\n(or response). Behaviorism discovered a lot about rats and pigeons but had less success at\nunderstanding humans.\nCognitive psychology, which views the brain as an information-processing device,\nCOGNITIVE\nPSYCHOLOGY\ncan be traced back at least to the works of William James (1842–1910). Helmholtz also\ninsisted that perception involved a form of unconscious logical inference. The cognitive\nviewpoint was largely eclipsed by behaviorism in the United States, but at Cambridge’s Ap-\nplied Psychology Unit, directed by Frederic Bartlett (1886–1969), cognitive modeling was\nable to flourish. The Nature of Explanation, by Bartlett’s student and successor Kenneth\nCraik (1943), forcefully reestablished the legitimacy of such “mental” terms as beliefs and\ngoals, arguing that they are just as scientific as, say, using pressure and temperature to talk\nabout gases, despite their being made of molecules that have neither. Craik specified the\nthree key steps of a knowledge-based agent: (1) the stimulus must be translated into an inter-\nnal representation, (2) the representation is manipulated by cognitive processes to derive new\ninternal representations, and (3) these are in turn retranslated back into action. He clearly\nexplained why this was a good design for an agent:\nIf the organism carries a “small-scale model” of external reality and of its own possible\nactions within its head, it is able to try out various alternatives, conclude which is the best\nof them, react to future situations before they arise, utilize the knowledge of past events\nin dealing with the present and future, and in every way to react in a much fuller, safer,\nand more competent manner to the emergencies which face it. (Craik, 1943)\nAfter Craik’s death in a bicycle accident in 1945, his work was continued by Donald Broad-\nbent, whose book Perception and Communication (1958) was one of the first works to model\npsychological phenomena as information processing. Meanwhile, in the United States, the\ndevelopment of computer modeling led to the creation of the field of cognitive science. The\nfield can be said to have started at a workshop in September 1956 at MIT. (We shall see that\nthis is just two months after the conference at which AI itself was “born.”) At the workshop,\nGeorge Miller presented The Magic Number Seven, Noam Chomsky presented Three Models\nof Language, and Allen Newell and Herbert Simon presented The Logic Theory Machine.\nThese three influential papers showed how computer models could be used to address the\npsychology of memory, language, and logical thinking, respectively. It is now a common\n(although far from universal) view among psychologists that “a cognitive theory should be\nlike a computer program” (Anderson, 1980); that is, it should describe a detailed information-\nprocessing mechanism whereby some cognitive function might be implemented.\n1.2.6\nComputer engineering\n• How can we build an efficient computer?\nFor artificial intelligence to succeed, we need two things: intelligence and an artifact. The\ncomputer has been the artifact of choice. The modern digital electronic computer was in-\nvented independently and almost simultaneously by scientists in three countries embattled in\n14\nChapter\n1.\nIntroduction\nWorld War II. The first operational computer was the electromechanical Heath Robinson,8\nbuilt in 1940 by Alan Turing’s team for a single purpose: deciphering German messages. In\n1943, the same group developed the Colossus, a powerful general-purpose machine based\non vacuum tubes.9 The first operational programmable computer was the Z-3, the inven-\ntion of Konrad Zuse in Germany in 1941. Zuse also invented floating-point numbers and the\nfirst high-level programming language, Plankalk¨ul. The first electronic computer, the ABC,\nwas assembled by John Atanasoff and his student Clifford Berry between 1940 and 1942\nat Iowa State University. Atanasoff’s research received little support or recognition; it was\nthe ENIAC, developed as part of a secret military project at the University of Pennsylvania\nby a team including John Mauchly and John Eckert, that proved to be the most influential\nforerunner of modern computers.\nSince that time, each generation of computer hardware has brought an increase in speed\nand capacity and a decrease in price. Performance doubled every 18 months or so until around\n2005, when power dissipation problems led manufacturers to start multiplying the number of\nCPU cores rather than the clock speed. Current expectations are that future increases in power\nwill come from massive parallelism—a curious convergence with the properties of the brain.\nOf course, there were calculating devices before the electronic computer. The earliest\nautomated machines, dating from the 17th century, were discussed on page 6. The first pro-\ngrammable machine was a loom, devised in 1805 by Joseph Marie Jacquard (1752–1834),\nthat used punched cards to store instructions for the pattern to be woven. In the mid-19th\ncentury, Charles Babbage (1792–1871) designed two machines, neither of which he com-\npleted. The Difference Engine was intended to compute mathematical tables for engineering\nand scientific projects. It was finally built and shown to work in 1991 at the Science Museum\nin London (Swade, 2000). Babbage’s Analytical Engine was far more ambitious: it included\naddressable memory, stored programs, and conditional jumps and was the first artifact capa-\nble of universal computation. Babbage’s colleague Ada Lovelace, daughter of the poet Lord\nByron, was perhaps the world’s first programmer. (The programming language Ada is named\nafter her.) She wrote programs for the unfinished Analytical Engine and even speculated that\nthe machine could play chess or compose music.\nAI also owes a debt to the software side of computer science, which has supplied the\noperating systems, programming languages, and tools needed to write modern programs (and\npapers about them). But this is one area where the debt has been repaid: work in AI has pio-\nneered many ideas that have made their way back to mainstream computer science, including\ntime sharing, interactive interpreters, personal computers with windows and mice, rapid de-\nvelopment environments, the linked list data type, automatic storage management, and key\nconcepts of symbolic, functional, declarative, and object-oriented programming.\n8 Heath Robinson was a cartoonist famous for his depictions of whimsical and absurdly complicated contrap-\ntions for everyday tasks such as buttering toast.\n9 In the postwar period, Turing wanted to use these computers for AI research—for example, one of the first\nchess programs (Turing et al., 1953). His efforts were blocked by the British government.\nSection 1.2.\nThe Foundations of Artificial Intelligence\n15\n1.2.7\nControl theory and cybernetics\n• How can artifacts operate under their own control?\nKtesibios of Alexandria (c. 250 B.C.) built the first self-controlling machine: a water clock\nwith a regulator that maintained a constant flow rate. This invention changed the definition\nof what an artifact could do. Previously, only living things could modify their behavior in\nresponse to changes in the environment. Other examples of self-regulating feedback control\nsystems include the steam engine governor, created by James Watt (1736–1819), and the\nthermostat, invented by Cornelis Drebbel (1572–1633), who also invented the submarine.\nThe mathematical theory of stable feedback systems was developed in the 19th century.\nThe central figure in the creation of what is now called control theory was Norbert\nCONTROL THEORY\nWiener (1894–1964). Wiener was a brilliant mathematician who worked with Bertrand Rus-\nsell, among others, before developing an interest in biological and mechanical control systems\nand their connection to cognition. Like Craik (who also used control systems as psychological\nmodels), Wiener and his colleagues Arturo Rosenblueth and Julian Bigelow challenged the\nbehaviorist orthodoxy (Rosenblueth et al., 1943). They viewed purposive behavior as aris-\ning from a regulatory mechanism trying to minimize “error”—the difference between current\nstate and goal state. In the late 1940s, Wiener, along with Warren McCulloch, Walter Pitts,\nand John von Neumann, organized a series of influential conferences that explored the new\nmathematical and computational models of cognition. Wiener’s book Cybernetics (1948) be-\nCYBERNETICS\ncame a bestseller and awoke the public to the possibility of artificially intelligent machines.\nMeanwhile, in Britain, W. Ross Ashby (Ashby, 1940) pioneered similar ideas. Ashby, Alan\nTuring, Grey Walter, and others formed the Ratio Club for “those who had Wiener’s ideas\nbefore Wiener’s book appeared.” Ashby’s Design for a Brain (1948, 1952) elaborated on his\nidea that intelligence could be created by the use of homeostatic devices containing appro-\nHOMEOSTATIC\npriate feedback loops to achieve stable adaptive behavior.\nModern control theory, especially the branch known as stochastic optimal control, has\nas its goal the design of systems that maximize an objective function over time. This roughly\nOBJECTIVE\nFUNCTION\nmatches our view of AI: designing systems that behave optimally. Why, then, are AI and\ncontrol theory two different fields, despite the close connections among their founders? The\nanswer lies in the close coupling between the mathematical techniques that were familiar to\nthe participants and the corresponding sets of problems that were encompassed in each world\nview. Calculus and matrix algebra, the tools of control theory, lend themselves to systems that\nare describable by fixed sets of continuous variables, whereas AI was founded in part as a way\nto escape from the these perceived limitations. The tools of logical inference and computation\nallowed AI researchers to consider problems such as language, vision, and planning that fell\ncompletely outside the control theorist’s purview.\n1.2.8\nLinguistics\n• How does language relate to thought?\nIn 1957, B. F. Skinner published Verbal Behavior. This was a comprehensive, detailed ac-\ncount of the behaviorist approach to language learning, written by the foremost expert in\n16\nChapter\n1.\nIntroduction\nthe field. But curiously, a review of the book became as well known as the book itself, and\nserved to almost kill off interest in behaviorism. The author of the review was the linguist\nNoam Chomsky, who had just published a book on his own theory, Syntactic Structures.\nChomsky pointed out that the behaviorist theory did not address the notion of creativity in\nlanguage—it did not explain how a child could understand and make up sentences that he or\nshe had never heard before. Chomsky’s theory—based on syntactic models going back to the\nIndian linguist Panini (c. 350 B.C.)—could explain this, and unlike previous theories, it was\nformal enough that it could in principle be programmed.\nModern linguistics and AI, then, were “born” at about the same time, and grew up\ntogether, intersecting in a hybrid field called computational linguistics or natural language\nCOMPUTATIONAL\nLINGUISTICS\nprocessing. The problem of understanding language soon turned out to be considerably more\ncomplex than it seemed in 1957. Understanding language requires an understanding of the\nsubject matter and context, not just an understanding of the structure of sentences. This might\nseem obvious, but it was not widely appreciated until the 1960s. Much of the early work in\nknowledge representation (the study of how to put knowledge into a form that a computer\ncan reason with) was tied to language and informed by research in linguistics, which was\nconnected in turn to decades of work on the philosophical analysis of language.\n1.3\nTHE HISTORY OF ARTIFICIAL INTELLIGENCE\nWith the background material behind us, we are ready to cover the development of AI itself.\n1.3.1\nThe gestation of artificial intelligence (1943–1955)\nThe first work that is now generally recognized as AI was done by Warren McCulloch and\nWalter Pitts (1943). They drew on three sources: knowledge of the basic physiology and\nfunction of neurons in the brain; a formal analysis of propositional logic due to Russell and\nWhitehead; and Turing’s theory of computation. They proposed a model of artificial neurons\nin which each neuron is characterized as being “on” or “off,” with a switch to “on” occurring\nin response to stimulation by a sufficient number of neighboring neurons. The state of a\nneuron was conceived of as “factually equivalent to a proposition which proposed its adequate\nstimulus.” They showed, for example, that any computable function could be computed by\nsome network of connected neurons, and that all the logical connectives (and, or, not, etc.)\ncould be implemented by simple net structures. McCulloch and Pitts also suggested that\nsuitably defined networks could learn. Donald Hebb (1949) demonstrated a simple updating\nrule for modifying the connection strengths between neurons. His rule, now called Hebbian\nlearning, remains an influential model to this day.\nHEBBIAN LEARNING\nTwo undergraduate students at Harvard, Marvin Minsky and Dean Edmonds, built the\nfirst neural network computer in 1950. The SNARC, as it was called, used 3000 vacuum\ntubes and a surplus automatic pilot mechanism from a B-24 bomber to simulate a network of\n40 neurons. Later, at Princeton, Minsky studied universal computation in neural networks.\nHis Ph.D. committee was skeptical about whether this kind of work should be considered\nSection 1.3.\nThe History of Artificial Intelligence\n17\nmathematics, but von Neumann reportedly said, “If it isn’t now, it will be someday.” Minsky\nwas later to prove influential theorems showing the limitations of neural network research.\nThere were a number of early examples of work that can be characterized as AI, but\nAlan Turing’s vision was perhaps the most influential. He gave lectures on the topic as early\nas 1947 at the London Mathematical Society and articulated a persuasive agenda in his 1950\narticle “Computing Machinery and Intelligence.” Therein, he introduced the Turing Test,\nmachine learning, genetic algorithms, and reinforcement learning. He proposed the Child\nProgramme idea, explaining “Instead of trying to produce a programme to simulate the adult\nmind, why not rather try to produce one which simulated the child’s?”\n1.3.2\nThe birth of artificial intelligence (1956)\nPrinceton was home to another influential figure in AI, John McCarthy. After receiving his\nPhD there in 1951 and working for two years as an instructor, McCarthy moved to Stan-\nford and then to Dartmouth College, which was to become the official birthplace of the field.\nMcCarthy convinced Minsky, Claude Shannon, and Nathaniel Rochester to help him bring\ntogether U.S. researchers interested in automata theory, neural nets, and the study of intel-\nligence. They organized a two-month workshop at Dartmouth in the summer of 1956. The\nproposal states:10\nWe propose that a 2 month, 10 man study of artificial intelligence be carried\nout during the summer of 1956 at Dartmouth College in Hanover, New Hamp-\nshire. The study is to proceed on the basis of the conjecture that every aspect of\nlearning or any other feature of intelligence can in principle be so precisely de-\nscribed that a machine can be made to simulate it. An attempt will be made to find\nhow to make machines use language, form abstractions and concepts, solve kinds\nof problems now reserved for humans, and improve themselves. We think that a\nsignificant advance can be made in one or more of these problems if a carefully\nselected group of scientists work on it together for a summer.\nThere were 10 attendees in all, including Trenchard More from Princeton, Arthur Samuel\nfrom IBM, and Ray Solomonoff and Oliver Selfridge from MIT.\nTwo researchers from Carnegie Tech,11 Allen Newell and Herbert Simon, rather stole\nthe show. Although the others had ideas and in some cases programs for particular appli-\ncations such as checkers, Newell and Simon already had a reasoning program, the Logic\nTheorist (LT), about which Simon claimed, “We have invented a computer program capable\nof thinking non-numerically, and thereby solved the venerable mind–body problem.”12 Soon\nafter the workshop, the program was able to prove most of the theorems in Chapter 2 of Rus-\n10 This was the first official usage of McCarthy’s term artificial intelligence. Perhaps “computational rationality”\nwould have been more precise and less threatening, but “AI” has stuck. At the 50th anniversary of the Dartmouth\nconference, McCarthy stated that he resisted the terms “computer” or “computational” in deference to Norbert\nWeiner, who was promoting analog cybernetic devices rather than digital computers.\n11 Now Carnegie Mellon University (CMU).\n12 Newell and Simon also invented a list-processing language, IPL, to write LT. They had no compiler and\ntranslated it into machine code by hand. To avoid errors, they worked in parallel, calling out binary numbers to\neach other as they wrote each instruction to make sure they agreed.\n18\nChapter\n1.\nIntroduction\nsell and Whitehead’s Principia Mathematica. Russell was reportedly delighted when Simon\nshowed him that the program had come up with a proof for one theorem that was shorter than\nthe one in Principia. The editors of the Journal of Symbolic Logic were less impressed; they\nrejected a paper coauthored by Newell, Simon, and Logic Theorist.\nThe Dartmouth workshop did not lead to any new breakthroughs, but it did introduce\nall the major figures to each other. For the next 20 years, the field would be dominated by\nthese people and their students and colleagues at MIT, CMU, Stanford, and IBM.\nLooking at the proposal for the Dartmouth workshop (McCarthy et al., 1955), we can\nsee why it was necessary for AI to become a separate field. Why couldn’t all the work done\nin AI have taken place under the name of control theory or operations research or decision\ntheory, which, after all, have objectives similar to those of AI? Or why isn’t AI a branch\nof mathematics? The first answer is that AI from the start embraced the idea of duplicating\nhuman faculties such as creativity, self-improvement, and language use. None of the other\nfields were addressing these issues. The second answer is methodology. AI is the only one\nof these fields that is clearly a branch of computer science (although operations research does\nshare an emphasis on computer simulations), and AI is the only field to attempt to build\nmachines that will function autonomously in complex, changing environments.\n1.3.3\nEarly enthusiasm, great expectations (1952–1969)\nThe early years of AI were full of successes—in a limited way. Given the primitive comput-\ners and programming tools of the time and the fact that only a few years earlier computers\nwere seen as things that could do arithmetic and no more, it was astonishing whenever a com-\nputer did anything remotely clever. The intellectual establishment, by and large, preferred to\nbelieve that “a machine can never do X.” (See Chapter 26 for a long list of X’s gathered\nby Turing.) AI researchers naturally responded by demonstrating one X after another. John\nMcCarthy referred to this period as the “Look, Ma, no hands!” era.\nNewell and Simon’s early success was followed up with the General Problem Solver,\nor GPS. Unlike Logic Theorist, this program was designed from the start to imitate human\nproblem-solving protocols. Within the limited class of puzzles it could handle, it turned out\nthat the order in which the program considered subgoals and possible actions was similar to\nthat in which humans approached the same problems. Thus, GPS was probably the first pro-\ngram to embody the “thinking humanly” approach. The success of GPS and subsequent pro-\ngrams as models of cognition led Newell and Simon (1976) to formulate the famous physical\nsymbol system hypothesis, which states that “a physical symbol system has the necessary and\nPHYSICAL SYMBOL\nSYSTEM\nsufficient means for general intelligent action.” What they meant is that any system (human\nor machine) exhibiting intelligence must operate by manipulating data structures composed\nof symbols. We will see later that this hypothesis has been challenged from many directions.\nAt IBM, Nathaniel Rochester and his colleagues produced some of the first AI pro-\ngrams.\nHerbert Gelernter (1959) constructed the Geometry Theorem Prover, which was\nable to prove theorems that many students of mathematics would find quite tricky. Starting\nin 1952, Arthur Samuel wrote a series of programs for checkers (draughts) that eventually\nlearned to play at a strong amateur level. Along the way, he disproved the idea that comput-\nSection 1.3.\nThe History of Artificial Intelligence\n19\ners can do only what they are told to: his program quickly learned to play a better game than\nits creator. The program was demonstrated on television in February 1956, creating a strong\nimpression. Like Turing, Samuel had trouble finding computer time. Working at night, he\nused machines that were still on the testing floor at IBM’s manufacturing plant. Chapter 5\ncovers game playing, and Chapter 21 explains the learning techniques used by Samuel.\nJohn McCarthy moved from Dartmouth to MIT and there made three crucial contribu-\ntions in one historic year: 1958. In MIT AI Lab Memo No. 1, McCarthy defined the high-level\nlanguage Lisp, which was to become the dominant AI programming language for the next 30\nLISP\nyears. With Lisp, McCarthy had the tool he needed, but access to scarce and expensive com-\nputing resources was also a serious problem. In response, he and others at MIT invented time\nsharing. Also in 1958, McCarthy published a paper entitled Programs with Common Sense,\nin which he described the Advice Taker, a hypothetical program that can be seen as the first\ncomplete AI system. Like the Logic Theorist and Geometry Theorem Prover, McCarthy’s\nprogram was designed to use knowledge to search for solutions to problems. But unlike the\nothers, it was to embody general knowledge of the world. For example, he showed how\nsome simple axioms would enable the program to generate a plan to drive to the airport. The\nprogram was also designed to accept new axioms in the normal course of operation, thereby\nallowing it to achieve competence in new areas without being reprogrammed. The Advice\nTaker thus embodied the central principles of knowledge representation and reasoning: that\nit is useful to have a formal, explicit representation of the world and its workings and to be\nable to manipulate that representation with deductive processes. It is remarkable how much\nof the 1958 paper remains relevant today.\n1958 also marked the year that Marvin Minsky moved to MIT. His initial collaboration\nwith McCarthy did not last, however. McCarthy stressed representation and reasoning in for-\nmal logic, whereas Minsky was more interested in getting programs to work and eventually\ndeveloped an anti-logic outlook. In 1963, McCarthy started the AI lab at Stanford. His plan\nto use logic to build the ultimate Advice Taker was advanced by J. A. Robinson’s discov-\nery in 1965 of the resolution method (a complete theorem-proving algorithm for first-order\nlogic; see Chapter 9). Work at Stanford emphasized general-purpose methods for logical\nreasoning. Applications of logic included Cordell Green’s question-answering and planning\nsystems (Green, 1969b) and the Shakey robotics project at the Stanford Research Institute\n(SRI). The latter project, discussed further in Chapter 25, was the first to demonstrate the\ncomplete integration of logical reasoning and physical activity.\nMinsky supervised a series of students who chose limited problems that appeared to\nrequire intelligence to solve. These limited domains became known as microworlds. James\nMICROWORLD\nSlagle’s SAINT program (1963) was able to solve closed-form calculus integration problems\ntypical of first-year college courses. Tom Evans’s ANALOGY program (1968) solved geo-\nmetric analogy problems that appear in IQ tests. Daniel Bobrow’s STUDENT program (1967)\nsolved algebra story problems, such as the following:\nIf the number of customers Tom gets is twice the square of 20 percent of the number\nof advertisements he runs, and the number of advertisements he runs is 45, what is the\nnumber of customers Tom gets?\n20\nChapter\n1.\nIntroduction\nRed\nGreen\nRed\nGreen\nGreen\nBlue\nBlue\nRed\nFigure 1.4\nA scene from the blocks world. SHRDLU (Winograd, 1972) has just completed\nthe command “Find a block which is taller than the one you are holding and put it in the box.”\nThe most famous microworld was the blocks world, which consists of a set of solid blocks\nplaced on a tabletop (or more often, a simulation of a tabletop), as shown in Figure 1.4.\nA typical task in this world is to rearrange the blocks in a certain way, using a robot hand\nthat can pick up one block at a time. The blocks world was home to the vision project of\nDavid Huffman (1971), the vision and constraint-propagation work of David Waltz (1975),\nthe learning theory of Patrick Winston (1970), the natural-language-understanding program\nof Terry Winograd (1972), and the planner of Scott Fahlman (1974).\nEarly work building on the neural networks of McCulloch and Pitts also flourished.\nThe work of Winograd and Cowan (1963) showed how a large number of elements could\ncollectively represent an individual concept, with a corresponding increase in robustness and\nparallelism. Hebb’s learning methods were enhanced by Bernie Widrow (Widrow and Hoff,\n1960; Widrow, 1962), who called his networks adalines, and by Frank Rosenblatt (1962)\nwith his perceptrons. The perceptron convergence theorem (Block et al., 1962) says that\nthe learning algorithm can adjust the connection strengths of a perceptron to match any input\ndata, provided such a match exists. These topics are covered in Chapter 20.\n1.3.4\nA dose of reality (1966–1973)\nFrom the beginning, AI researchers were not shy about making predictions of their coming\nsuccesses. The following statement by Herbert Simon in 1957 is often quoted:\nIt is not my aim to surprise or shock you—but the simplest way I can summarize is to say\nthat there are now in the world machines that think, that learn and that create. Moreover,\nSection 1.3.\nThe History of Artificial Intelligence\n21\ntheir ability to do these things is going to increase rapidly until—in a visible future—the\nrange of problems they can handle will be coextensive with the range to which the human\nmind has been applied.\nTerms such as “visible future” can be interpreted in various ways, but Simon also made\nmore concrete predictions: that within 10 years a computer would be chess champion, and\na significant mathematical theorem would be proved by machine. These predictions came\ntrue (or approximately true) within 40 years rather than 10. Simon’s overconfidence was due\nto the promising performance of early AI systems on simple examples. In almost all cases,\nhowever, these early systems turned out to fail miserably when tried out on wider selections\nof problems and on more difficult problems.\nThe first kind of difficulty arose because most early programs knew nothing of their\nsubject matter; they succeeded by means of simple syntactic manipulations. A typical story\noccurred in early machine translation efforts, which were generously funded by the U.S. Na-\ntional Research Council in an attempt to speed up the translation of Russian scientific papers\nin the wake of the Sputnik launch in 1957. It was thought initially that simple syntactic trans-\nformations based on the grammars of Russian and English, and word replacement from an\nelectronic dictionary, would suffice to preserve the exact meanings of sentences. The fact is\nthat accurate translation requires background knowledge in order to resolve ambiguity and\nestablish the content of the sentence. The famous retranslation of “the spirit is willing but\nthe flesh is weak” as “the vodka is good but the meat is rotten” illustrates the difficulties en-\ncountered. In 1966, a report by an advisory committee found that “there has been no machine\ntranslation of general scientific text, and none is in immediate prospect.” All U.S. government\nfunding for academic translation projects was canceled. Today, machine translation is an im-\nperfect but widely used tool for technical, commercial, government, and Internet documents.\nThe second kind of difficulty was the intractability of many of the problems that AI was\nattempting to solve. Most of the early AI programs solved problems by trying out different\ncombinations of steps until the solution was found. This strategy worked initially because\nmicroworlds contained very few objects and hence very few possible actions and very short\nsolution sequences. Before the theory of computational complexity was developed, it was\nwidely thought that “scaling up” to larger problems was simply a matter of faster hardware\nand larger memories. The optimism that accompanied the development of resolution theorem\nproving, for example, was soon dampened when researchers failed to prove theorems involv-\ning more than a few dozen facts. The fact that a program can find a solution in principle does\nnot mean that the program contains any of the mechanisms needed to find it in practice.\nThe illusion of unlimited computational power was not confined to problem-solving\nprograms. Early experiments in machine evolution (now called genetic algorithms) (Fried-\nMACHINE EVOLUTION\nGENETIC\nALGORITHM\nberg, 1958; Friedberg et al., 1959) were based on the undoubtedly correct belief that by\nmaking an appropriate series of small mutations to a machine-code program, one can gen-\nerate a program with good performance for any particular task. The idea, then, was to try\nrandom mutations with a selection process to preserve mutations that seemed useful. De-\nspite thousands of hours of CPU time, almost no progress was demonstrated. Modern genetic\nalgorithms use better representations and have shown more success.\n22\nChapter\n1.\nIntroduction\nFailure to come to grips with the “combinatorial explosion” was one of the main criti-\ncisms of AI contained in the Lighthill report (Lighthill, 1973), which formed the basis for the\ndecision by the British government to end support for AI research in all but two universities.\n(Oral tradition paints a somewhat different and more colorful picture, with political ambitions\nand personal animosities whose description is beside the point.)\nA third difficulty arose because of some fundamental limitations on the basic structures\nbeing used to generate intelligent behavior. For example, Minsky and Papert’s book Percep-\ntrons (1969) proved that, although perceptrons (a simple form of neural network) could be\nshown to learn anything they were capable of representing, they could represent very little. In\nparticular, a two-input perceptron (restricted to be simpler than the form Rosenblatt originally\nstudied) could not be trained to recognize when its two inputs were different. Although their\nresults did not apply to more complex, multilayer networks, research funding for neural-net\nresearch soon dwindled to almost nothing. Ironically, the new back-propagation learning al-\ngorithms for multilayer networks that were to cause an enormous resurgence in neural-net\nresearch in the late 1980s were actually discovered first in 1969 (Bryson and Ho, 1969).\n1.3.5\nKnowledge-based systems: The key to power? (1969–1979)\nThe picture of problem solving that had arisen during the first decade of AI research was of\na general-purpose search mechanism trying to string together elementary reasoning steps to\nfind complete solutions. Such approaches have been called weak methods because, although\nWEAK METHOD\ngeneral, they do not scale up to large or difficult problem instances. The alternative to weak\nmethods is to use more powerful, domain-specific knowledge that allows larger reasoning\nsteps and can more easily handle typically occurring cases in narrow areas of expertise. One\nmight say that to solve a hard problem, you have to almost know the answer already.\nThe DENDRAL program (Buchanan et al., 1969) was an early example of this approach.\nIt was developed at Stanford, where Ed Feigenbaum (a former student of Herbert Simon),\nBruce Buchanan (a philosopher turned computer scientist), and Joshua Lederberg (a Nobel\nlaureate geneticist) teamed up to solve the problem of inferring molecular structure from the\ninformation provided by a mass spectrometer. The input to the program consists of the ele-\nmentary formula of the molecule (e.g., C6H13NO2) and the mass spectrum giving the masses\nof the various fragments of the molecule generated when it is bombarded by an electron beam.\nFor example, the mass spectrum might contain a peak at m = 15, corresponding to the mass\nof a methyl (CH3) fragment.\nThe naive version of the program generated all possible structures consistent with the\nformula, and then predicted what mass spectrum would be observed for each, comparing this\nwith the actual spectrum. As one might expect, this is intractable for even moderate-sized\nmolecules. The DENDRAL researchers consulted analytical chemists and found that they\nworked by looking for well-known patterns of peaks in the spectrum that suggested common\nsubstructures in the molecule. For example, the following rule is used to recognize a ketone\n(C=O) subgroup (which weighs 28):\nif there are two peaks at x1 and x2 such that\n(a) x1 + x2 = M + 28 (M is the mass of the whole molecule);\nSection 1.3.\nThe History of Artificial Intelligence\n23\n(b) x1 −28 is a high peak;\n(c) x2 −28 is a high peak;\n(d) At least one of x1 and x2 is high.\nthen there is a ketone subgroup\nRecognizing that the molecule contains a particular substructure reduces the number of pos-\nsible candidates enormously. DENDRAL was powerful because\nAll the relevant theoretical knowledge to solve these problems has been mapped over from\nits general form in the [spectrum prediction component] (“first principles”) to efficient\nspecial forms (“cookbook recipes”). (Feigenbaum et al., 1971)\nThe significance of DENDRAL was that it was the first successful knowledge-intensive sys-\ntem: its expertise derived from large numbers of special-purpose rules. Later systems also\nincorporated the main theme of McCarthy’s Advice Taker approach—the clean separation of\nthe knowledge (in the form of rules) from the reasoning component.\nWith this lesson in mind, Feigenbaum and others at Stanford began the Heuristic Pro-\ngramming Project (HPP) to investigate the extent to which the new methodology of expert\nsystems could be applied to other areas of human expertise. The next major effort was in\nEXPERT SYSTEMS\nthe area of medical diagnosis. Feigenbaum, Buchanan, and Dr. Edward Shortliffe developed\nMYCIN to diagnose blood infections. With about 450 rules, MYCIN was able to perform\nas well as some experts, and considerably better than junior doctors. It also contained two\nmajor differences from DENDRAL. First, unlike the DENDRAL rules, no general theoretical\nmodel existed from which the MYCIN rules could be deduced. They had to be acquired from\nextensive interviewing of experts, who in turn acquired them from textbooks, other experts,\nand direct experience of cases. Second, the rules had to reflect the uncertainty associated with\nmedical knowledge. MYCIN incorporated a calculus of uncertainty called certainty factors\nCERTAINTY FACTOR\n(see Chapter 14), which seemed (at the time) to fit well with how doctors assessed the impact\nof evidence on the diagnosis.\nThe importance of domain knowledge was also apparent in the area of understanding\nnatural language. Although Winograd’s SHRDLU system for understanding natural language\nhad engendered a good deal of excitement, its dependence on syntactic analysis caused some\nof the same problems as occurred in the early machine translation work. It was able to\novercome ambiguity and understand pronoun references, but this was mainly because it was\ndesigned specifically for one area—the blocks world. Several researchers, including Eugene\nCharniak, a fellow graduate student of Winograd’s at MIT, suggested that robust language\nunderstanding would require general knowledge about the world and a general method for\nusing that knowledge.\nAt Yale, linguist-turned-AI-researcher Roger Schank emphasized this point, claiming,\n“There is no such thing as syntax,” which upset a lot of linguists but did serve to start a useful\ndiscussion. Schank and his students built a series of programs (Schank and Abelson, 1977;\nWilensky, 1978; Schank and Riesbeck, 1981; Dyer, 1983) that all had the task of under-\nstanding natural language. The emphasis, however, was less on language per se and more on\nthe problems of representing and reasoning with the knowledge required for language under-\nstanding. The problems included representing stereotypical situations (Cullingford, 1981),\n24\nChapter\n1.\nIntroduction\ndescribing human memory organization (Rieger, 1976; Kolodner, 1983), and understanding\nplans and goals (Wilensky, 1983).\nThe widespread growth of applications to real-world problems caused a concurrent in-\ncrease in the demands for workable knowledge representation schemes. A large number\nof different representation and reasoning languages were developed. Some were based on\nlogic—for example, the Prolog language became popular in Europe, and the PLANNER fam-\nily in the United States. Others, following Minsky’s idea of frames (1975), adopted a more\nFRAMES\nstructured approach, assembling facts about particular object and event types and arranging\nthe types into a large taxonomic hierarchy analogous to a biological taxonomy.\n1.3.6\nAI becomes an industry (1980–present)\nThe first successful commercial expert system, R1, began operation at the Digital Equipment\nCorporation (McDermott, 1982). The program helped configure orders for new computer\nsystems; by 1986, it was saving the company an estimated $40 million a year. By 1988,\nDEC’s AI group had 40 expert systems deployed, with more on the way. DuPont had 100 in\nuse and 500 in development, saving an estimated $10 million a year. Nearly every major U.S.\ncorporation had its own AI group and was either using or investigating expert systems.\nIn 1981, the Japanese announced the “Fifth Generation” project, a 10-year plan to build\nintelligent computers running Prolog. In response, the United States formed the Microelec-\ntronics and Computer Technology Corporation (MCC) as a research consortium designed to\nassure national competitiveness. In both cases, AI was part of a broad effort, including chip\ndesign and human-interface research. In Britain, the Alvey report reinstated the funding that\nwas cut by the Lighthill report.13 In all three countries, however, the projects never met their\nambitious goals.\nOverall, the AI industry boomed from a few million dollars in 1980 to billions of dollars\nin 1988, including hundreds of companies building expert systems, vision systems, robots,\nand software and hardware specialized for these purposes. Soon after that came a period\ncalled the “AI Winter,” in which many companies fell by the wayside as they failed to deliver\non extravagant promises.\n1.3.7\nThe return of neural networks (1986–present)\nIn the mid-1980s at least four different groups reinvented the back-propagation learning\nBACK-PROPAGATION\nalgorithm first found in 1969 by Bryson and Ho. The algorithm was applied to many learn-\ning problems in computer science and psychology, and the widespread dissemination of the\nresults in the collection Parallel Distributed Processing (Rumelhart and McClelland, 1986)\ncaused great excitement.\nThese so-called connectionist models of intelligent systems were seen by some as di-\nCONNECTIONIST\nrect competitors both to the symbolic models promoted by Newell and Simon and to the\nlogicist approach of McCarthy and others (Smolensky, 1988). It might seem obvious that\nat some level humans manipulate symbols—in fact, Terrence Deacon’s book The Symbolic\n13 To save embarrassment, a new field called IKBS (Intelligent Knowledge-Based Systems) was invented because\nArtificial Intelligence had been officially canceled.\nSection 1.3.\nThe History of Artificial Intelligence\n25\nSpecies (1997) suggests that this is the defining characteristic of humans—but the most ar-\ndent connectionists questioned whether symbol manipulation had any real explanatory role in\ndetailed models of cognition. This question remains unanswered, but the current view is that\nconnectionist and symbolic approaches are complementary, not competing. As occurred with\nthe separation of AI and cognitive science, modern neural network research has bifurcated\ninto two fields, one concerned with creating effective network architectures and algorithms\nand understanding their mathematical properties, the other concerned with careful modeling\nof the empirical properties of actual neurons and ensembles of neurons.\n1.3.8\nAI adopts the scientific method (1987–present)\nRecent years have seen a revolution in both the content and the methodology of work in\nartificial intelligence.14 It is now more common to build on existing theories than to propose\nbrand-new ones, to base claims on rigorous theorems or hard experimental evidence rather\nthan on intuition, and to show relevance to real-world applications rather than toy examples.\nAI was founded in part as a rebellion against the limitations of existing fields like control\ntheory and statistics, but now it is embracing those fields. As David McAllester (1998) put it:\nIn the early period of AI it seemed plausible that new forms of symbolic computation,\ne.g., frames and semantic networks, made much of classical theory obsolete. This led to\na form of isolationism in which AI became largely separated from the rest of computer\nscience. This isolationism is currently being abandoned. There is a recognition that\nmachine learning should not be isolated from information theory, that uncertain reasoning\nshould not be isolated from stochastic modeling, that search should not be isolated from\nclassical optimization and control, and that automated reasoning should not be isolated\nfrom formal methods and static analysis.\nIn terms of methodology, AI has finally come firmly under the scientific method. To be ac-\ncepted, hypotheses must be subjected to rigorous empirical experiments, and the results must\nbe analyzed statistically for their importance (Cohen, 1995). It is now possible to replicate\nexperiments by using shared repositories of test data and code.\nThe field of speech recognition illustrates the pattern. In the 1970s, a wide variety of\ndifferent architectures and approaches were tried. Many of these were rather ad hoc and\nfragile, and were demonstrated on only a few specially selected examples. In recent years,\napproaches based on hidden Markov models (HMMs) have come to dominate the area. Two\nHIDDEN MARKOV\nMODELS\naspects of HMMs are relevant. First, they are based on a rigorous mathematical theory. This\nhas allowed speech researchers to build on several decades of mathematical results developed\nin other fields. Second, they are generated by a process of training on a large corpus of\nreal speech data. This ensures that the performance is robust, and in rigorous blind tests the\nHMMs have been improving their scores steadily. Speech technology and the related field of\nhandwritten character recognition are already making the transition to widespread industrial\n14 Some have characterized this change as a victory of the neats—those who think that AI theories should be\ngrounded in mathematical rigor—over the scruffies—those who would rather try out lots of ideas, write some\nprograms, and then assess what seems to be working. Both approaches are important. A shift toward neatness\nimplies that the field has reached a level of stability and maturity. Whether that stability will be disrupted by a\nnew scruffy idea is another question.\n26\nChapter\n1.\nIntroduction\nand consumer applications. Note that there is no scientific claim that humans use HMMs to\nrecognize speech; rather, HMMs provide a mathematical framework for understanding the\nproblem and support the engineering claim that they work well in practice.\nMachine translation follows the same course as speech recognition. In the 1950s there\nwas initial enthusiasm for an approach based on sequences of words, with models learned\naccording to the principles of information theory. That approach fell out of favor in the\n1960s, but returned in the late 1990s and now dominates the field.\nNeural networks also fit this trend. Much of the work on neural nets in the 1980s was\ndone in an attempt to scope out what could be done and to learn how neural nets differ from\n“traditional” techniques. Using improved methodology and theoretical frameworks, the field\narrived at an understanding in which neural nets can now be compared with corresponding\ntechniques from statistics, pattern recognition, and machine learning, and the most promising\ntechnique can be applied to each application. As a result of these developments, so-called\ndata mining technology has spawned a vigorous new industry.\nDATA MINING\nJudea Pearl’s (1988) Probabilistic Reasoning in Intelligent Systems led to a new accep-\ntance of probability and decision theory in AI, following a resurgence of interest epitomized\nby Peter Cheeseman’s (1985) article “In Defense of Probability.” The Bayesian network\nBAYESIAN NETWORK\nformalism was invented to allow efficient representation of, and rigorous reasoning with,\nuncertain knowledge. This approach largely overcomes many problems of the probabilistic\nreasoning systems of the 1960s and 1970s; it now dominates AI research on uncertain reason-\ning and expert systems. The approach allows for learning from experience, and it combines\nthe best of classical AI and neural nets. Work by Judea Pearl (1982a) and by Eric Horvitz and\nDavid Heckerman (Horvitz and Heckerman, 1986; Horvitz et al., 1986) promoted the idea of\nnormative expert systems: ones that act rationally according to the laws of decision theory\nand do not try to imitate the thought steps of human experts. The WindowsTM operating sys-\ntem includes several normative diagnostic expert systems for correcting problems. Chapters\n13 to 16 cover this area.\nSimilar gentle revolutions have occurred in robotics, computer vision, and knowledge\nrepresentation. A better understanding of the problems and their complexity properties, com-\nbined with increased mathematical sophistication, has led to workable research agendas and\nrobust methods. Although increased formalization and specialization led fields such as vision\nand robotics to become somewhat isolated from “mainstream” AI in the 1990s, this trend has\nreversed in recent years as tools from machine learning in particular have proved effective for\nmany problems. The process of reintegration is already yielding significant benefits\n1.3.9\nThe emergence of intelligent agents (1995–present)\nPerhaps encouraged by the progress in solving the subproblems of AI, researchers have also\nstarted to look at the “whole agent” problem again. The work of Allen Newell, John Laird,\nand Paul Rosenbloom on SOAR (Newell, 1990; Laird et al., 1987) is the best-known example\nof a complete agent architecture. One of the most important environments for intelligent\nagents is the Internet. AI systems have become so common in Web-based applications that\nthe “-bot” suffix has entered everyday language. Moreover, AI technologies underlie many\nSection 1.3.\nThe History of Artificial Intelligence\n27\nInternet tools, such as search engines, recommender systems, and Web site aggregators.\nOne consequence of trying to build complete agents is the realization that the previously\nisolated subfields of AI might need to be reorganized somewhat when their results are to be\ntied together. In particular, it is now widely appreciated that sensory systems (vision, sonar,\nspeech recognition, etc.) cannot deliver perfectly reliable information about the environment.\nHence, reasoning and planning systems must be able to handle uncertainty. A second major\nconsequence of the agent perspective is that AI has been drawn into much closer contact\nwith other fields, such as control theory and economics, that also deal with agents. Recent\nprogress in the control of robotic cars has derived from a mixture of approaches ranging from\nbetter sensors, control-theoretic integration of sensing, localization and mapping, as well as\na degree of high-level planning.\nDespite these successes, some influential founders of AI, including John McCarthy\n(2007), Marvin Minsky (2007), Nils Nilsson (1995, 2005) and Patrick Winston (Beal and\nWinston, 2009), have expressed discontent with the progress of AI. They think that AI should\nput less emphasis on creating ever-improved versions of applications that are good at a spe-\ncific task, such as driving a car, playing chess, or recognizing speech. Instead, they believe\nAI should return to its roots of striving for, in Simon’s words, “machines that think, that learn\nand that create.” They call the effort human-level AI or HLAI; their first symposium was in\nHUMAN-LEVEL AI\n2004 (Minsky et al., 2004). The effort will require very large knowledge bases; Hendler et al.\n(1995) discuss where these knowledge bases might come from.\nA related idea is the subfield of Artificial General Intelligence or AGI (Goertzel and\nARTIFICIAL GENERAL\nINTELLIGENCE\nPennachin, 2007), which held its first conference and organized the Journal of Artificial Gen-\neral Intelligence in 2008. AGI looks for a universal algorithm for learning and acting in\nany environment, and has its roots in the work of Ray Solomonoff (1964), one of the atten-\ndees of the original 1956 Dartmouth conference. Guaranteeing that what we create is really\nFriendly AI is also a concern (Yudkowsky, 2008; Omohundro, 2008), one we will return to\nFRIENDLY AI\nin Chapter 26.\n1.3.10\nThe availability of very large data sets (2001–present)\nThroughout the 60-year history of computer science, the emphasis has been on the algorithm\nas the main subject of study. But some recent work in AI suggests that for many problems, it\nmakes more sense to worry about the data and be less picky about what algorithm to apply.\nThis is true because of the increasing availability of very large data sources: for example,\ntrillions of words of English and billions of images from the Web (Kilgarriff and Grefenstette,\n2006); or billions of base pairs of genomic sequences (Collins et al., 2003).\nOne influential paper in this line was Yarowsky’s (1995) work on word-sense disam-\nbiguation: given the use of the word “plant” in a sentence, does that refer to flora or factory?\nPrevious approaches to the problem had relied on human-labeled examples combined with\nmachine learning algorithms. Yarowsky showed that the task can be done, with accuracy\nabove 96%, with no labeled examples at all. Instead, given a very large corpus of unanno-\ntated text and just the dictionary definitions of the two senses—“works, industrial plant” and\n“flora, plant life”—one can label examples in the corpus, and from there bootstrap to learn\n28\nChapter\n1.\nIntroduction\nnew patterns that help label new examples. Banko and Brill (2001) show that techniques\nlike this perform even better as the amount of available text goes from a million words to a\nbillion and that the increase in performance from using more data exceeds any difference in\nalgorithm choice; a mediocre algorithm with 100 million words of unlabeled training data\noutperforms the best known algorithm with 1 million words.\nAs another example, Hays and Efros (2007) discuss the problem of filling in holes in a\nphotograph. Suppose you use Photoshop to mask out an ex-friend from a group photo, but\nnow you need to fill in the masked area with something that matches the background. Hays\nand Efros defined an algorithm that searches through a collection of photos to find something\nthat will match. They found the performance of their algorithm was poor when they used\na collection of only ten thousand photos, but crossed a threshold into excellent performance\nwhen they grew the collection to two million photos.\nWork like this suggests that the “knowledge bottleneck” in AI—the problem of how to\nexpress all the knowledge that a system needs—may be solved in many applications by learn-\ning methods rather than hand-coded knowledge engineering, provided the learning algorithms\nhave enough data to go on (Halevy et al., 2009). Reporters have noticed the surge of new ap-\nplications and have written that “AI Winter” may be yielding to a new Spring (Havenstein,\n2005). As Kurzweil (2005) writes, “today, many thousands of AI applications are deeply\nembedded in the infrastructure of every industry.”\n1.4\nTHE STATE OF THE ART\nWhat can AI do today? A concise answer is difficult because there are so many activities in\nso many subfields. Here we sample a few applications; others appear throughout the book.\nRobotic vehicles: A driverless robotic car named STANLEY sped through the rough\nterrain of the Mojave dessert at 22 mph, finishing the 132-mile course first to win the 2005\nDARPA Grand Challenge. STANLEY is a Volkswagen Touareg outfitted with cameras, radar,\nand laser rangefinders to sense the environment and onboard software to command the steer-\ning, braking, and acceleration (Thrun, 2006). The following year CMU’s BOSS won the Ur-\nban Challenge, safely driving in traffic through the streets of a closed Air Force base, obeying\ntraffic rules and avoiding pedestrians and other vehicles.\nSpeech recognition: A traveler calling United Airlines to book a flight can have the en-\ntire conversation guided by an automated speech recognition and dialog management system.\nAutonomous planning and scheduling: A hundred million miles from Earth, NASA’s\nRemote Agent program became the first on-board autonomous planning program to control\nthe scheduling of operations for a spacecraft (Jonsson et al., 2000). REMOTE AGENT gen-\nerated plans from high-level goals specified from the ground and monitored the execution of\nthose plans—detecting, diagnosing, and recovering from problems as they occurred. Succes-\nsor program MAPGEN (Al-Chang et al., 2004) plans the daily operations for NASA’s Mars\nExploration Rovers, and MEXAR2 (Cesta et al., 2007) did mission planning—both logistics\nand science planning—for the European Space Agency’s Mars Express mission in 2008.\nSection 1.5.\nSummary\n29\nGame playing: IBM’s DEEP BLUE became the first computer program to defeat the\nworld champion in a chess match when it bested Garry Kasparov by a score of 3.5 to 2.5 in\nan exhibition match (Goodman and Keene, 1997). Kasparov said that he felt a “new kind of\nintelligence” across the board from him. Newsweek magazine described the match as “The\nbrain’s last stand.” The value of IBM’s stock increased by $18 billion. Human champions\nstudied Kasparov’s loss and were able to draw a few matches in subsequent years, but the\nmost recent human-computer matches have been won convincingly by the computer.\nSpam fighting: Each day, learning algorithms classify over a billion messages as spam,\nsaving the recipient from having to waste time deleting what, for many users, could comprise\n80% or 90% of all messages, if not classified away by algorithms. Because the spammers are\ncontinually updating their tactics, it is difficult for a static programmed approach to keep up,\nand learning algorithms work best (Sahami et al., 1998; Goodman and Heckerman, 2004).\nLogistics planning: During the Persian Gulf crisis of 1991, U.S. forces deployed a\nDynamic Analysis and Replanning Tool, DART (Cross and Walker, 1994), to do automated\nlogistics planning and scheduling for transportation. This involved up to 50,000 vehicles,\ncargo, and people at a time, and had to account for starting points, destinations, routes, and\nconflict resolution among all parameters. The AI planning techniques generated in hours\na plan that would have taken weeks with older methods. The Defense Advanced Research\nProject Agency (DARPA) stated that this single application more than paid back DARPA’s\n30-year investment in AI.\nRobotics: The iRobot Corporation has sold over two million Roomba robotic vacuum\ncleaners for home use. The company also deploys the more rugged PackBot to Iraq and\nAfghanistan, where it is used to handle hazardous materials, clear explosives, and identify\nthe location of snipers.\nMachine Translation: A computer program automatically translates from Arabic to\nEnglish, allowing an English speaker to see the headline “Ardogan Confirms That Turkey\nWould Not Accept Any Pressure, Urging Them to Recognize Cyprus.” The program uses a\nstatistical model built from examples of Arabic-to-English translations and from examples of\nEnglish text totaling two trillion words (Brants et al., 2007). None of the computer scientists\non the team speak Arabic, but they do understand statistics and machine learning algorithms.\nThese are just a few examples of artificial intelligence systems that exist today. Not\nmagic or science fiction—but rather science, engineering, and mathematics, to which this\nbook provides an introduction.\n1.5\nSUMMARY\nThis chapter defines AI and establishes the cultural background against which it has devel-\noped. Some of the important points are as follows:\n• Different people approach AI with different goals in mind. Two important questions to\nask are: Are you concerned with thinking or behavior? Do you want to model humans\nor work from an ideal standard?\n30\nChapter\n1.\nIntroduction\n• In this book, we adopt the view that intelligence is concerned mainly with rational\naction. Ideally, an intelligent agent takes the best possible action in a situation. We\nstudy the problem of building agents that are intelligent in this sense.\n• Philosophers (going back to 400 B.C.) made AI conceivable by considering the ideas\nthat the mind is in some ways like a machine, that it operates on knowledge encoded in\nsome internal language, and that thought can be used to choose what actions to take.\n• Mathematicians provided the tools to manipulate statements of logical certainty as well\nas uncertain, probabilistic statements. They also set the groundwork for understanding\ncomputation and reasoning about algorithms.\n• Economists formalized the problem of making decisions that maximize the expected\noutcome to the decision maker.\n• Neuroscientists discovered some facts about how the brain works and the ways in which\nit is similar to and different from computers.\n• Psychologists adopted the idea that humans and animals can be considered information-\nprocessing machines. Linguists showed that language use fits into this model.\n• Computer engineers provided the ever-more-powerful machines that make AI applica-\ntions possible.\n• Control theory deals with designing devices that act optimally on the basis of feedback\nfrom the environment. Initially, the mathematical tools of control theory were quite\ndifferent from AI, but the fields are coming closer together.\n• The history of AI has had cycles of success, misplaced optimism, and resulting cutbacks\nin enthusiasm and funding. There have also been cycles of introducing new creative\napproaches and systematically refining the best ones.\n• AI has advanced more rapidly in the past decade because of greater use of the scientific\nmethod in experimenting with and comparing approaches.\n• Recent progress in understanding the theoretical basis for intelligence has gone hand in\nhand with improvements in the capabilities of real systems. The subfields of AI have\nbecome more integrated, and AI has found common ground with other disciplines.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nThe methodological status of artificial intelligence is investigated in The Sciences of the Artifi-\ncial, by Herb Simon (1981), which discusses research areas concerned with complex artifacts.\nIt explains how AI can be viewed as both science and mathematics. Cohen (1995) gives an\noverview of experimental methodology within AI.\nThe Turing Test (Turing, 1950) is discussed by Shieber (1994), who severely criticizes\nthe usefulness of its instantiation in the Loebner Prize competition, and by Ford and Hayes\n(1995), who argue that the test itself is not helpful for AI. Bringsjord (2008) gives advice for\na Turing Test judge. Shieber (2004) and Epstein et al. (2008) collect a number of essays on\nthe Turing Test. Artificial Intelligence: The Very Idea, by John Haugeland (1985), gives a\nExercises\n31\nreadable account of the philosophical and practical problems of AI. Significant early papers\nin AI are anthologized in the collections by Webber and Nilsson (1981) and by Luger (1995).\nThe Encyclopedia of AI (Shapiro, 1992) contains survey articles on almost every topic in\nAI, as does Wikipedia. These articles usually provide a good entry point into the research\nliterature on each topic. An insightful and comprehensive history of AI is given by Nils\nNillson (2009), one of the early pioneers of the field.\nThe most recent work appears in the proceedings of the major AI conferences: the bi-\nennial International Joint Conference on AI (IJCAI), the annual European Conference on AI\n(ECAI), and the National Conference on AI, more often known as AAAI, after its sponsoring\norganization. The major journals for general AI are Artificial Intelligence, Computational\nIntelligence, the IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE In-\ntelligent Systems, and the electronic Journal of Artificial Intelligence Research. There are also\nmany conferences and journals devoted to specific areas, which we cover in the appropriate\nchapters. The main professional societies for AI are the American Association for Artificial\nIntelligence (AAAI), the ACM Special Interest Group in Artificial Intelligence (SIGART),\nand the Society for Artificial Intelligence and Simulation of Behaviour (AISB). AAAI’s AI\nMagazine contains many topical and tutorial articles, and its Web site, aaai.org, contains\nnews, tutorials, and background information.\nEXERCISES\nThese exercises are intended to stimulate discussion, and some might be set as term projects.\nAlternatively, preliminary attempts can be made now, and these attempts can be reviewed\nafter the completion of the book.\n1.1\nDefine in your own words: (a) intelligence, (b) artificial intelligence, (c) agent, (d)\nrationality, (e) logical reasoning.\n1.2\nRead Turing’s original paper on AI (Turing, 1950). In the paper, he discusses several\nobjections to his proposed enterprise and his test for intelligence. Which objections still carry\nweight? Are his refutations valid? Can you think of new objections arising from develop-\nments since he wrote the paper? In the paper, he predicts that, by the year 2000, a computer\nwill have a 30% chance of passing a five-minute Turing Test with an unskilled interrogator.\nWhat chance do you think a computer would have today? In another 50 years?\n1.3\nAre reflex actions (such as flinching from a hot stove) rational? Are they intelligent?\n1.4\nSuppose we extend Evans’s ANALOGY program so that it can score 200 on a standard\nIQ test. Would we then have a program more intelligent than a human? Explain.\n1.5\nThe neural structure of the sea slug Aplysia has been widely studied (first by Nobel\nLaureate Eric Kandel) because it has only about 20,000 neurons, most of them large and\neasily manipulated. Assuming that the cycle time for an Aplysia neuron is roughly the same\nas for a human neuron, how does the computational power, in terms of memory updates per\nsecond, compare with the high-end computer described in Figure 1.3?\n32\nChapter\n1.\nIntroduction\n1.6\nHow could introspection—reporting on one’s inner thoughts—be inaccurate? Could I\nbe wrong about what I’m thinking? Discuss.\n1.7\nTo what extent are the following computer systems instances of artificial intelligence:\n• Supermarket bar code scanners.\n• Web search engines.\n• Voice-activated telephone menus.\n• Internet routing algorithms that respond dynamically to the state of the network.\n1.8\nMany of the computational models of cognitive activities that have been proposed in-\nvolve quite complex mathematical operations, such as convolving an image with a Gaussian\nor finding a minimum of the entropy function. Most humans (and certainly all animals) never\nlearn this kind of mathematics at all, almost no one learns it before college, and almost no\none can compute the convolution of a function with a Gaussian in their head. What sense\ndoes it make to say that the “vision system” is doing this kind of mathematics, whereas the\nactual person has no idea how to do it?\n1.9\nWhy would evolution tend to result in systems that act rationally? What goals are such\nsystems designed to achieve?\n1.10\nIs AI a science, or is it engineering? Or neither or both? Explain.\n1.11\n“Surely computers cannot be intelligent—they can do only what their programmers\ntell them.” Is the latter statement true, and does it imply the former?\n1.12\n“Surely animals cannot be intelligent—they can do only what their genes tell them.”\nIs the latter statement true, and does it imply the former?\n1.13\n“Surely animals, humans, and computers cannot be intelligent—they can do only what\ntheir constituent atoms are told to do by the laws of physics.” Is the latter statement true, and\ndoes it imply the former?\n1.14\nExamine the AI literature to discover whether the following tasks can currently be\nsolved by computers:\na. Playing a decent game of table tennis (Ping-Pong).\nb. Driving in the center of Cairo, Egypt.\nc. Driving in Victorville, California.\nd. Buying a week’s worth of groceries at the market.\ne. Buying a week’s worth of groceries on the Web.\nf. Playing a decent game of bridge at a competitive level.\ng. Discovering and proving new mathematical theorems.\nh. Writing an intentionally funny story.\ni. Giving competent legal advice in a specialized area of law.\nj. Translating spoken English into spoken Swedish in real time.\nk. Performing a complex surgical operation.\nExercises\n33\nFor the currently infeasible tasks, try to find out what the difficulties are and predict when, if\never, they will be overcome.\n1.15\nVarious subfields of AI have held contests by defining a standard task and inviting re-\nsearchers to do their best. Examples include the DARPA Grand Challenge for robotic cars,\nThe International Planning Competition, the Robocup robotic soccer league, the TREC infor-\nmation retrieval event, and contests in machine translation, speech recognition. Investigate\nfive of these contests, and describe the progress made over the years. To what degree have the\ncontests advanced toe state of the art in AI? Do what degree do they hurt the field by drawing\nenergy away from new ideas?\n" }, { "chapter": 2, "text": "2\nINTELLIGENT AGENTS\nIn which we discuss the nature of agents, perfect or otherwise, the diversity of\nenvironments, and the resulting menagerie of agent types.\nChapter 1 identified the concept of rational agents as central to our approach to artificial\nintelligence. In this chapter, we make this notion more concrete. We will see that the concept\nof rationality can be applied to a wide variety of agents operating in any imaginable environ-\nment. Our plan in this book is to use this concept to develop a small set of design principles\nfor building successful agents—systems that can reasonably be called intelligent.\nWe begin by examining agents, environments, and the coupling between them. The\nobservation that some agents behave better than others leads naturally to the idea of a rational\nagent—one that behaves as well as possible. How well an agent can behave depends on\nthe nature of the environment; some environments are more difficult than others. We give a\ncrude categorization of environments and show how properties of an environment influence\nthe design of suitable agents for that environment. We describe a number of basic “skeleton”\nagent designs, which we flesh out in the rest of the book.\n2.1\nAGENTS AND ENVIRONMENTS\nAn agent is anything that can be viewed as perceiving its environment through sensors and\nENVIRONMENT\nSENSOR\nacting upon that environment through actuators. This simple idea is illustrated in Figure 2.1.\nACTUATOR\nA human agent has eyes, ears, and other organs for sensors and hands, legs, vocal tract, and so\non for actuators. A robotic agent might have cameras and infrared range finders for sensors\nand various motors for actuators. A software agent receives keystrokes, file contents, and\nnetwork packets as sensory inputs and acts on the environment by displaying on the screen,\nwriting files, and sending network packets.\nWe use the term percept to refer to the agent’s perceptual inputs at any given instant. An\nPERCEPT\nagent’s percept sequence is the complete history of everything the agent has ever perceived.\nPERCEPT SEQUENCE\nIn general, an agent’s choice of action at any given instant can depend on the entire percept\nsequence observed to date, but not on anything it hasn’t perceived. By specifying the agent’s\nchoice of action for every possible percept sequence, we have said more or less everything\n34\nSection 2.1.\nAgents and Environments\n35\nAgent\nSensors\nActuators\nEnvironment\nPercepts\nActions\n?\nFigure 2.1\nAgents interact with environments through sensors and actuators.\nthere is to say about the agent. Mathematically speaking, we say that an agent’s behavior is\ndescribed by the agent function that maps any given percept sequence to an action.\nAGENT FUNCTION\nWe can imagine tabulating the agent function that describes any given agent; for most\nagents, this would be a very large table—infinite, in fact, unless we place a bound on the\nlength of percept sequences we want to consider. Given an agent to experiment with, we can,\nin principle, construct this table by trying out all possible percept sequences and recording\nwhich actions the agent does in response.1 The table is, of course, an external characterization\nof the agent. Internally, the agent function for an artificial agent will be implemented by an\nagent program. It is important to keep these two ideas distinct. The agent function is an\nAGENT PROGRAM\nabstract mathematical description; the agent program is a concrete implementation, running\nwithin some physical system.\nTo illustrate these ideas, we use a very simple example—the vacuum-cleaner world\nshown in Figure 2.2. This world is so simple that we can describe everything that happens;\nit’s also a made-up world, so we can invent many variations. This particular world has just two\nlocations: squares A and B. The vacuum agent perceives which square it is in and whether\nthere is dirt in the square. It can choose to move left, move right, suck up the dirt, or do\nnothing. One very simple agent function is the following: if the current square is dirty, then\nsuck; otherwise, move to the other square. A partial tabulation of this agent function is shown\nin Figure 2.3 and an agent program that implements it appears in Figure 2.8 on page 48.\nLooking at Figure 2.3, we see that various vacuum-world agents can be defined simply\nby filling in the right-hand column in various ways. The obvious question, then, is this: What\nis the right way to fill out the table? In other words, what makes an agent good or bad,\nintelligent or stupid? We answer these questions in the next section.\n1 If the agent uses some randomization to choose its actions, then we would have to try each sequence many\ntimes to identify the probability of each action. One might imagine that acting randomly is rather silly, but we\nshow later in this chapter that it can be very intelligent.\n36\nChapter\n2.\nIntelligent Agents\nA\nB\nFigure 2.2\nA vacuum-cleaner world with just two locations.\nPercept sequence\nAction\n[A, Clean]\nRight\n[A, Dirty]\nSuck\n[B, Clean]\nLeft\n[B, Dirty]\nSuck\n[A, Clean], [A, Clean]\nRight\n[A, Clean], [A, Dirty]\nSuck\n...\n...\n[A, Clean], [A, Clean], [A, Clean]\nRight\n[A, Clean], [A, Clean], [A, Dirty]\nSuck\n...\n...\nFigure 2.3\nPartial tabulation of a simple agent function for the vacuum-cleaner world\nshown in Figure 2.2.\nBefore closing this section, we should emphasize that the notion of an agent is meant to\nbe a tool for analyzing systems, not an absolute characterization that divides the world into\nagents and non-agents. One could view a hand-held calculator as an agent that chooses the\naction of displaying “4” when given the percept sequence “2 + 2 =,” but such an analysis\nwould hardly aid our understanding of the calculator. In a sense, all areas of engineering can\nbe seen as designing artifacts that interact with the world; AI operates at (what the authors\nconsider to be) the most interesting end of the spectrum, where the artifacts have significant\ncomputational resources and the task environment requires nontrivial decision making.\n2.2\nGOOD BEHAVIOR: THE CONCEPT OF RATIONALITY\nA rational agent is one that does the right thing—conceptually speaking, every entry in the\nRATIONAL AGENT\ntable for the agent function is filled out correctly. Obviously, doing the right thing is better\nthan doing the wrong thing, but what does it mean to do the right thing?\nSection 2.2.\nGood Behavior: The Concept of Rationality\n37\nWe answer this age-old question in an age-old way: by considering the consequences\nof the agent’s behavior. When an agent is plunked down in an environment, it generates a\nsequence of actions according to the percepts it receives. This sequence of actions causes the\nenvironment to go through a sequence of states. If the sequence is desirable, then the agent\nhas performed well. This notion of desirability is captured by a performance measure that\nPERFORMANCE\nMEASURE\nevaluates any given sequence of environment states.\nNotice that we said environment states, not agent states. If we define success in terms\nof agent’s opinion of its own performance, an agent could achieve perfect rationality simply\nby deluding itself that its performance was perfect. Human agents in particular are notorious\nfor “sour grapes”—believing they did not really want something (e.g., a Nobel Prize) after\nnot getting it.\nObviously, there is not one fixed performance measure for all tasks and agents; typically,\na designer will devise one appropriate to the circumstances. This is not as easy as it sounds.\nConsider, for example, the vacuum-cleaner agent from the preceding section. We might\npropose to measure performance by the amount of dirt cleaned up in a single eight-hour shift.\nWith a rational agent, of course, what you ask for is what you get. A rational agent can\nmaximize this performance measure by cleaning up the dirt, then dumping it all on the floor,\nthen cleaning it up again, and so on. A more suitable performance measure would reward the\nagent for having a clean floor. For example, one point could be awarded for each clean square\nat each time step (perhaps with a penalty for electricity consumed and noise generated). As\na general rule, it is better to design performance measures according to what one actually\nwants in the environment, rather than according to how one thinks the agent should behave.\nEven when the obvious pitfalls are avoided, there remain some knotty issues to untangle.\nFor example, the notion of “clean floor” in the preceding paragraph is based on average\ncleanliness over time. Yet the same average cleanliness can be achieved by two different\nagents, one of which does a mediocre job all the time while the other cleans energetically but\ntakes long breaks. Which is preferable might seem to be a fine point of janitorial science, but\nin fact it is a deep philosophical question with far-reaching implications. Which is better—\na reckless life of highs and lows, or a safe but humdrum existence? Which is better—an\neconomy where everyone lives in moderate poverty, or one in which some live in plenty\nwhile others are very poor? We leave these questions as an exercise for the diligent reader.\n2.2.1\nRationality\nWhat is rational at any given time depends on four things:\n• The performance measure that defines the criterion of success.\n• The agent’s prior knowledge of the environment.\n• The actions that the agent can perform.\n• The agent’s percept sequence to date.\nThis leads to a definition of a rational agent:\nDEFINITION OF A\nRATIONAL AGENT\nFor each possible percept sequence, a rational agent should select an action that is ex-\npected to maximize its performance measure, given the evidence provided by the percept\nsequence and whatever built-in knowledge the agent has.\n38\nChapter\n2.\nIntelligent Agents\nConsider the simple vacuum-cleaner agent that cleans a square if it is dirty and moves to the\nother square if not; this is the agent function tabulated in Figure 2.3. Is this a rational agent?\nThat depends! First, we need to say what the performance measure is, what is known about\nthe environment, and what sensors and actuators the agent has. Let us assume the following:\n• The performance measure awards one point for each clean square at each time step,\nover a “lifetime” of 1000 time steps.\n• The “geography” of the environment is known a priori (Figure 2.2) but the dirt distri-\nbution and the initial location of the agent are not. Clean squares stay clean and sucking\ncleans the current square. The Left and Right actions move the agent left and right\nexcept when this would take the agent outside the environment, in which case the agent\nremains where it is.\n• The only available actions are Left, Right, and Suck.\n• The agent correctly perceives its location and whether that location contains dirt.\nWe claim that under these circumstances the agent is indeed rational; its expected perfor-\nmance is at least as high as any other agent’s. Exercise 2.2 asks you to prove this.\nOne can see easily that the same agent would be irrational under different circum-\nstances. For example, once all the dirt is cleaned up, the agent will oscillate needlessly back\nand forth; if the performance measure includes a penalty of one point for each movement left\nor right, the agent will fare poorly. A better agent for this case would do nothing once it is\nsure that all the squares are clean. If clean squares can become dirty again, the agent should\noccasionally check and re-clean them if needed. If the geography of the environment is un-\nknown, the agent will need to explore it rather than stick to squares A and B. Exercise 2.2\nasks you to design agents for these cases.\n2.2.2\nOmniscience, learning, and autonomy\nWe need to be careful to distinguish between rationality and omniscience. An omniscient\nOMNISCIENCE\nagent knows the actual outcome of its actions and can act accordingly; but omniscience is\nimpossible in reality. Consider the following example: I am walking along the Champs\nElys´ees one day and I see an old friend across the street. There is no traffic nearby and I’m\nnot otherwise engaged, so, being rational, I start to cross the street. Meanwhile, at 33,000\nfeet, a cargo door falls off a passing airliner,2 and before I make it to the other side of the\nstreet I am flattened. Was I irrational to cross the street? It is unlikely that my obituary would\nread “Idiot attempts to cross street.”\nThis example shows that rationality is not the same as perfection. Rationality max-\nimizes expected performance, while perfection maximizes actual performance. Retreating\nfrom a requirement of perfection is not just a question of being fair to agents. The point is\nthat if we expect an agent to do what turns out to be the best action after the fact, it will be\nimpossible to design an agent to fulfill this specification—unless we improve the performance\nof crystal balls or time machines.\n2 See N. Henderson, “New door latches urged for Boeing 747 jumbo jets,” Washington Post, August 24, 1989.\nSection 2.2.\nGood Behavior: The Concept of Rationality\n39\nOur definition of rationality does not require omniscience, then, because the rational\nchoice depends only on the percept sequence to date. We must also ensure that we haven’t\ninadvertently allowed the agent to engage in decidedly underintelligent activities. For exam-\nple, if an agent does not look both ways before crossing a busy road, then its percept sequence\nwill not tell it that there is a large truck approaching at high speed. Does our definition of\nrationality say that it’s now OK to cross the road? Far from it! First, it would not be rational\nto cross the road given this uninformative percept sequence: the risk of accident from cross-\ning without looking is too great. Second, a rational agent should choose the “looking” action\nbefore stepping into the street, because looking helps maximize the expected performance.\nDoing actions in order to modify future percepts—sometimes called information gather-\ning—is an important part of rationality and is covered in depth in Chapter 16. A second\nINFORMATION\nGATHERING\nexample of information gathering is provided by the exploration that must be undertaken by\nEXPLORATION\na vacuum-cleaning agent in an initially unknown environment.\nOur definition requires a rational agent not only to gather information but also to learn\nLEARNING\nas much as possible from what it perceives. The agent’s initial configuration could reflect\nsome prior knowledge of the environment, but as the agent gains experience this may be\nmodified and augmented. There are extreme cases in which the environment is completely\nknown a priori. In such cases, the agent need not perceive or learn; it simply acts correctly.\nOf course, such agents are fragile. Consider the lowly dung beetle. After digging its nest and\nlaying its eggs, it fetches a ball of dung from a nearby heap to plug the entrance. If the ball of\ndung is removed from its grasp en route, the beetle continues its task and pantomimes plug-\nging the nest with the nonexistent dung ball, never noticing that it is missing. Evolution has\nbuilt an assumption into the beetle’s behavior, and when it is violated, unsuccessful behavior\nresults. Slightly more intelligent is the sphex wasp. The female sphex will dig a burrow, go\nout and sting a caterpillar and drag it to the burrow, enter the burrow again to check all is\nwell, drag the caterpillar inside, and lay its eggs. The caterpillar serves as a food source when\nthe eggs hatch. So far so good, but if an entomologist moves the caterpillar a few inches\naway while the sphex is doing the check, it will revert to the “drag” step of its plan and will\ncontinue the plan without modification, even after dozens of caterpillar-moving interventions.\nThe sphex is unable to learn that its innate plan is failing, and thus will not change it.\nTo the extent that an agent relies on the prior knowledge of its designer rather than\non its own percepts, we say that the agent lacks autonomy. A rational agent should be\nAUTONOMY\nautonomous—it should learn what it can to compensate for partial or incorrect prior knowl-\nedge. For example, a vacuum-cleaning agent that learns to foresee where and when additional\ndirt will appear will do better than one that does not. As a practical matter, one seldom re-\nquires complete autonomy from the start: when the agent has had little or no experience, it\nwould have to act randomly unless the designer gave some assistance. So, just as evolution\nprovides animals with enough built-in reflexes to survive long enough to learn for themselves,\nit would be reasonable to provide an artificial intelligent agent with some initial knowledge\nas well as an ability to learn. After sufficient experience of its environment, the behavior\nof a rational agent can become effectively independent of its prior knowledge. Hence, the\nincorporation of learning allows one to design a single rational agent that will succeed in a\nvast variety of environments.\n40\nChapter\n2.\nIntelligent Agents\n2.3\nTHE NATURE OF ENVIRONMENTS\nNow that we have a definition of rationality, we are almost ready to think about building\nrational agents. First, however, we must think about task environments, which are essen-\nTASK ENVIRONMENT\ntially the “problems” to which rational agents are the “solutions.” We begin by showing how\nto specify a task environment, illustrating the process with a number of examples. We then\nshow that task environments come in a variety of flavors. The flavor of the task environment\ndirectly affects the appropriate design for the agent program.\n2.3.1\nSpecifying the task environment\nIn our discussion of the rationality of the simple vacuum-cleaner agent, we had to specify\nthe performance measure, the environment, and the agent’s actuators and sensors. We group\nall these under the heading of the task environment. For the acronymically minded, we call\nthis the PEAS (Performance, Environment, Actuators, Sensors) description. In designing an\nPEAS\nagent, the first step must always be to specify the task environment as fully as possible.\nThe vacuum world was a simple example; let us consider a more complex problem: an\nautomated taxi driver. We should point out, before the reader becomes alarmed, that a fully\nautomated taxi is currently somewhat beyond the capabilities of existing technology. (page 28\ndescribes an existing driving robot.) The full driving task is extremely open-ended. There is\nno limit to the novel combinations of circumstances that can arise—another reason we chose\nit as a focus for discussion. Figure 2.4 summarizes the PEAS description for the taxi’s task\nenvironment. We discuss each element in more detail in the following paragraphs.\nAgent Type\nPerformance\nMeasure\nEnvironment\nActuators\nSensors\nTaxi driver\nSafe, fast, legal,\ncomfortable trip,\nmaximize profits\nRoads, other\ntraffic,\npedestrians,\ncustomers\nSteering,\naccelerator,\nbrake, signal,\nhorn, display\nCameras, sonar,\nspeedometer,\nGPS, odometer,\naccelerometer,\nengine sensors,\nkeyboard\nFigure 2.4\nPEAS description of the task environment for an automated taxi.\nFirst, what is the performance measure to which we would like our automated driver\nto aspire? Desirable qualities include getting to the correct destination; minimizing fuel con-\nsumption and wear and tear; minimizing the trip time or cost; minimizing violations of traffic\nlaws and disturbances to other drivers; maximizing safety and passenger comfort; maximiz-\ning profits. Obviously, some of these goals conflict, so tradeoffs will be required.\nNext, what is the driving environment that the taxi will face? Any taxi driver must\ndeal with a variety of roads, ranging from rural lanes and urban alleys to 12-lane freeways.\nThe roads contain other traffic, pedestrians, stray animals, road works, police cars, puddles,\nSection 2.3.\nThe Nature of Environments\n41\nand potholes. The taxi must also interact with potential and actual passengers. There are also\nsome optional choices. The taxi might need to operate in Southern California, where snow\nis seldom a problem, or in Alaska, where it seldom is not. It could always be driving on the\nright, or we might want it to be flexible enough to drive on the left when in Britain or Japan.\nObviously, the more restricted the environment, the easier the design problem.\nThe actuators for an automated taxi include those available to a human driver: control\nover the engine through the accelerator and control over steering and braking. In addition, it\nwill need output to a display screen or voice synthesizer to talk back to the passengers, and\nperhaps some way to communicate with other vehicles, politely or otherwise.\nThe basic sensors for the taxi will include one or more controllable video cameras so\nthat it can see the road; it might augment these with infrared or sonar sensors to detect dis-\ntances to other cars and obstacles. To avoid speeding tickets, the taxi should have a speedome-\nter, and to control the vehicle properly, especially on curves, it should have an accelerometer.\nTo determine the mechanical state of the vehicle, it will need the usual array of engine, fuel,\nand electrical system sensors. Like many human drivers, it might want a global positioning\nsystem (GPS) so that it doesn’t get lost. Finally, it will need a keyboard or microphone for\nthe passenger to request a destination.\nIn Figure 2.5, we have sketched the basic PEAS elements for a number of additional\nagent types. Further examples appear in Exercise 2.4. It may come as a surprise to some read-\ners that our list of agent types includes some programs that operate in the entirely artificial\nenvironment defined by keyboard input and character output on a screen. “Surely,” one might\nsay, “this is not a real environment, is it?” In fact, what matters is not the distinction between\n“real” and “artificial” environments, but the complexity of the relationship among the behav-\nior of the agent, the percept sequence generated by the environment, and the performance\nmeasure. Some “real” environments are actually quite simple. For example, a robot designed\nto inspect parts as they come by on a conveyor belt can make use of a number of simplifying\nassumptions: that the lighting is always just so, that the only thing on the conveyor belt will\nbe parts of a kind that it knows about, and that only two actions (accept or reject) are possible.\nIn contrast, some software agents (or software robots or softbots) exist in rich, unlim-\nSOFTWARE AGENT\nSOFTBOT\nited domains. Imagine a softbot Web site operator designed to scan Internet news sources and\nshow the interesting items to its users, while selling advertising space to generate revenue.\nTo do well, that operator will need some natural language processing abilities, it will need\nto learn what each user and advertiser is interested in, and it will need to change its plans\ndynamically—for example, when the connection for one news source goes down or when a\nnew one comes online. The Internet is an environment whose complexity rivals that of the\nphysical world and whose inhabitants include many artificial and human agents.\n2.3.2\nProperties of task environments\nThe range of task environments that might arise in AI is obviously vast. We can, however,\nidentify a fairly small number of dimensions along which task environments can be catego-\nrized. These dimensions determine, to a large extent, the appropriate agent design and the\napplicability of each of the principal families of techniques for agent implementation. First,\n42\nChapter\n2.\nIntelligent Agents\nAgent Type\nPerformance\nMeasure\nEnvironment\nActuators\nSensors\nMedical\ndiagnosis system\nHealthy patient,\nreduced costs\nPatient, hospital,\nstaff\nDisplay of\nquestions, tests,\ndiagnoses,\ntreatments,\nreferrals\nKeyboard entry\nof symptoms,\nfindings, patient’s\nanswers\nSatellite image\nanalysis system\nCorrect image\ncategorization\nDownlink from\norbiting satellite\nDisplay of scene\ncategorization\nColor pixel\narrays\nPart-picking\nrobot\nPercentage of\nparts in correct\nbins\nConveyor belt\nwith parts; bins\nJointed arm and\nhand\nCamera, joint\nangle sensors\nRefinery\ncontroller\nPurity, yield,\nsafety\nRefinery,\noperators\nValves, pumps,\nheaters, displays\nTemperature,\npressure,\nchemical sensors\nInteractive\nEnglish tutor\nStudent’s score\non test\nSet of students,\ntesting agency\nDisplay of\nexercises,\nsuggestions,\ncorrections\nKeyboard entry\nFigure 2.5\nExamples of agent types and their PEAS descriptions.\nwe list the dimensions, then we analyze several task environments to illustrate the ideas. The\ndefinitions here are informal; later chapters provide more precise statements and examples of\neach kind of environment.\nFully observable vs. partially observable: If an agent’s sensors give it access to the\nFULLY OBSERVABLE\nPARTIALLY\nOBSERVABLE\ncomplete state of the environment at each point in time, then we say that the task environ-\nment is fully observable. A task environment is effectively fully observable if the sensors\ndetect all aspects that are relevant to the choice of action; relevance, in turn, depends on the\nperformance measure. Fully observable environments are convenient because the agent need\nnot maintain any internal state to keep track of the world. An environment might be partially\nobservable because of noisy and inaccurate sensors or because parts of the state are simply\nmissing from the sensor data—for example, a vacuum agent with only a local dirt sensor\ncannot tell whether there is dirt in other squares, and an automated taxi cannot see what other\ndrivers are thinking. If the agent has no sensors at all then the environment is unobserv-\nable. One might think that in such cases the agent’s plight is hopeless, but, as we discuss in\nUNOBSERVABLE\nChapter 4, the agent’s goals may still be achievable, sometimes with certainty.\nSingle agent vs. multiagent: The distinction between single-agent and multiagent en-\nSINGLE AGENT\nMULTIAGENT\nSection 2.3.\nThe Nature of Environments\n43\nvironments may seem simple enough. For example, an agent solving a crossword puzzle by\nitself is clearly in a single-agent environment, whereas an agent playing chess is in a two-\nagent environment. There are, however, some subtle issues. First, we have described how an\nentity may be viewed as an agent, but we have not explained which entities must be viewed\nas agents. Does an agent A (the taxi driver for example) have to treat an object B (another\nvehicle) as an agent, or can it be treated merely as an object behaving according to the laws of\nphysics, analogous to waves at the beach or leaves blowing in the wind? The key distinction\nis whether B’s behavior is best described as maximizing a performance measure whose value\ndepends on agent A’s behavior. For example, in chess, the opponent entity B is trying to\nmaximize its performance measure, which, by the rules of chess, minimizes agent A’s per-\nformance measure. Thus, chess is a competitive multiagent environment. In the taxi-driving\nCOMPETITIVE\nenvironment, on the other hand, avoiding collisions maximizes the performance measure of\nall agents, so it is a partially cooperative multiagent environment. It is also partially com-\nCOOPERATIVE\npetitive because, for example, only one car can occupy a parking space. The agent-design\nproblems in multiagent environments are often quite different from those in single-agent en-\nvironments; for example, communication often emerges as a rational behavior in multiagent\nenvironments; in some competitive environments, randomized behavior is rational because\nit avoids the pitfalls of predictability.\nDeterministic vs. stochastic. If the next state of the environment is completely deter-\nDETERMINISTIC\nSTOCHASTIC\nmined by the current state and the action executed by the agent, then we say the environment\nis deterministic; otherwise, it is stochastic. In principle, an agent need not worry about uncer-\ntainty in a fully observable, deterministic environment. (In our definition, we ignore uncer-\ntainty that arises purely from the actions of other agents in a multiagent environment; thus,\na game can be deterministic even though each agent may be unable to predict the actions of\nthe others.) If the environment is partially observable, however, then it could appear to be\nstochastic. Most real situations are so complex that it is impossible to keep track of all the\nunobserved aspects; for practical purposes, they must be treated as stochastic. Taxi driving is\nclearly stochastic in this sense, because one can never predict the behavior of traffic exactly;\nmoreover, one’s tires blow out and one’s engine seizes up without warning. The vacuum\nworld as we described it is deterministic, but variations can include stochastic elements such\nas randomly appearing dirt and an unreliable suction mechanism (Exercise 2.13). We say an\nenvironment is uncertain if it is not fully observable or not deterministic. One final note:\nUNCERTAIN\nour use of the word “stochastic” generally implies that uncertainty about outcomes is quan-\ntified in terms of probabilities; a nondeterministic environment is one in which actions are\nNONDETERMINISTIC\ncharacterized by their possible outcomes, but no probabilities are attached to them. Nonde-\nterministic environment descriptions are usually associated with performance measures that\nrequire the agent to succeed for all possible outcomes of its actions.\nEpisodic vs. sequential: In an episodic task environment, the agent’s experience is\nEPISODIC\nSEQUENTIAL\ndivided into atomic episodes. In each episode the agent receives a percept and then performs\na single action. Crucially, the next episode does not depend on the actions taken in previous\nepisodes. Many classification tasks are episodic. For example, an agent that has to spot\ndefective parts on an assembly line bases each decision on the current part, regardless of\nprevious decisions; moreover, the current decision doesn’t affect whether the next part is\n44\nChapter\n2.\nIntelligent Agents\ndefective. In sequential environments, on the other hand, the current decision could affect\nall future decisions.3 Chess and taxi driving are sequential: in both cases, short-term actions\ncan have long-term consequences. Episodic environments are much simpler than sequential\nenvironments because the agent does not need to think ahead.\nStatic vs. dynamic: If the environment can change while an agent is deliberating, then\nSTATIC\nDYNAMIC\nwe say the environment is dynamic for that agent; otherwise, it is static. Static environments\nare easy to deal with because the agent need not keep looking at the world while it is deciding\non an action, nor need it worry about the passage of time. Dynamic environments, on the\nother hand, are continuously asking the agent what it wants to do; if it hasn’t decided yet,\nthat counts as deciding to do nothing. If the environment itself does not change with the\npassage of time but the agent’s performance score does, then we say the environment is\nsemidynamic. Taxi driving is clearly dynamic: the other cars and the taxi itself keep moving\nSEMIDYNAMIC\nwhile the driving algorithm dithers about what to do next. Chess, when played with a clock,\nis semidynamic. Crossword puzzles are static.\nDiscrete vs. continuous: The discrete/continuous distinction applies to the state of the\nDISCRETE\nCONTINUOUS\nenvironment, to the way time is handled, and to the percepts and actions of the agent. For\nexample, the chess environment has a finite number of distinct states (excluding the clock).\nChess also has a discrete set of percepts and actions. Taxi driving is a continuous-state and\ncontinuous-time problem: the speed and location of the taxi and of the other vehicles sweep\nthrough a range of continuous values and do so smoothly over time. Taxi-driving actions are\nalso continuous (steering angles, etc.). Input from digital cameras is discrete, strictly speak-\ning, but is typically treated as representing continuously varying intensities and locations.\nKnown vs. unknown: Strictly speaking, this distinction refers not to the environment\nKNOWN\nUNKNOWN\nitself but to the agent’s (or designer’s) state of knowledge about the “laws of physics” of\nthe environment. In a known environment, the outcomes (or outcome probabilities if the\nenvironment is stochastic) for all actions are given. Obviously, if the environment is unknown,\nthe agent will have to learn how it works in order to make good decisions. Note that the\ndistinction between known and unknown environments is not the same as the one between\nfully and partially observable environments. It is quite possible for a known environment\nto be partially observable—for example, in solitaire card games, I know the rules but am\nstill unable to see the cards that have not yet been turned over. Conversely, an unknown\nenvironment can be fully observable—in a new video game, the screen may show the entire\ngame state but I still don’t know what the buttons do until I try them.\nAs one might expect, the hardest case is partially observable, multiagent, stochastic,\nsequential, dynamic, continuous, and unknown. Taxi driving is hard in all these senses, except\nthat for the most part the driver’s environment is known. Driving a rented car in a new country\nwith unfamiliar geography and traffic laws is a lot more exciting.\nFigure 2.6 lists the properties of a number of familiar environments. Note that the\nanswers are not always cut and dried. For example, we describe the part-picking robot as\nepisodic, because it normally considers each part in isolation. But if one day there is a large\n3 The word “sequential” is also used in computer science as the antonym of “parallel.” The two meanings are\nlargely unrelated.\nSection 2.3.\nThe Nature of Environments\n45\nTask Environment\nObservable Agents Deterministic\nEpisodic\nStatic\nDiscrete\nCrossword puzzle\nFully\nSingle Deterministic Sequential\nStatic\nDiscrete\nChess with a clock\nFully\nMulti\nDeterministic Sequential\nSemi\nDiscrete\nPoker\nPartially\nMulti\nStochastic\nSequential\nStatic\nDiscrete\nBackgammon\nFully\nMulti\nStochastic\nSequential\nStatic\nDiscrete\nTaxi driving\nPartially\nMulti\nStochastic\nSequential Dynamic Continuous\nMedical diagnosis\nPartially\nSingle\nStochastic\nSequential Dynamic Continuous\nImage analysis\nFully\nSingle Deterministic\nEpisodic\nSemi\nContinuous\nPart-picking robot\nPartially\nSingle\nStochastic\nEpisodic\nDynamic Continuous\nRefinery controller\nPartially\nSingle\nStochastic\nSequential Dynamic Continuous\nInteractive English tutor\nPartially\nMulti\nStochastic\nSequential Dynamic\nDiscrete\nFigure 2.6\nExamples of task environments and their characteristics.\nbatch of defective parts, the robot should learn from several observations that the distribution\nof defects has changed, and should modify its behavior for subsequent parts. We have not\nincluded a “known/unknown” column because, as explained earlier, this is not strictly a prop-\nerty of the environment. For some environments, such as chess and poker, it is quite easy to\nsupply the agent with full knowledge of the rules, but it is nonetheless interesting to consider\nhow an agent might learn to play these games without such knowledge.\nSeveral of the answers in the table depend on how the task environment is defined. We\nhave listed the medical-diagnosis task as single-agent because the disease process in a patient\nis not profitably modeled as an agent; but a medical-diagnosis system might also have to\ndeal with recalcitrant patients and skeptical staff, so the environment could have a multiagent\naspect. Furthermore, medical diagnosis is episodic if one conceives of the task as selecting a\ndiagnosis given a list of symptoms; the problem is sequential if the task can include proposing\na series of tests, evaluating progress over the course of treatment, and so on. Also, many\nenvironments are episodic at higher levels than the agent’s individual actions. For example,\na chess tournament consists of a sequence of games; each game is an episode because (by\nand large) the contribution of the moves in one game to the agent’s overall performance is\nnot affected by the moves in its previous game. On the other hand, decision making within a\nsingle game is certainly sequential.\nThe code repository associated with this book (aima.cs.berkeley.edu) includes imple-\nmentations of a number of environments, together with a general-purpose environment simu-\nlator that places one or more agents in a simulated environment, observes their behavior over\ntime, and evaluates them according to a given performance measure. Such experiments are\noften carried out not for a single environment but for many environments drawn from an en-\nvironment class. For example, to evaluate a taxi driver in simulated traffic, we would want to\nENVIRONMENT\nCLASS\nrun many simulations with different traffic, lighting, and weather conditions. If we designed\nthe agent for a single scenario, we might be able to take advantage of specific properties\nof the particular case but might not identify a good design for driving in general. For this\n46\nChapter\n2.\nIntelligent Agents\nreason, the code repository also includes an environment generator for each environment\nENVIRONMENT\nGENERATOR\nclass that selects particular environments (with certain likelihoods) in which to run the agent.\nFor example, the vacuum environment generator initializes the dirt pattern and agent location\nrandomly. We are then interested in the agent’s average performance over the environment\nclass. A rational agent for a given environment class maximizes this average performance.\nExercises 2.8 to 2.13 take you through the process of developing an environment class and\nevaluating various agents therein.\n2.4\nTHE STRUCTURE OF AGENTS\nSo far we have talked about agents by describing behavior—the action that is performed after\nany given sequence of percepts. Now we must bite the bullet and talk about how the insides\nwork. The job of AI is to design an agent program that implements the agent function—\nAGENT PROGRAM\nthe mapping from percepts to actions. We assume this program will run on some sort of\ncomputing device with physical sensors and actuators—we call this the architecture:\nARCHITECTURE\nagent = architecture + program .\nObviously, the program we choose has to be one that is appropriate for the architecture. If the\nprogram is going to recommend actions like Walk, the architecture had better have legs. The\narchitecture might be just an ordinary PC, or it might be a robotic car with several onboard\ncomputers, cameras, and other sensors. In general, the architecture makes the percepts from\nthe sensors available to the program, runs the program, and feeds the program’s action choices\nto the actuators as they are generated. Most of this book is about designing agent programs,\nalthough Chapters 24 and 25 deal directly with the sensors and actuators.\n2.4.1\nAgent programs\nThe agent programs that we design in this book all have the same skeleton: they take the\ncurrent percept as input from the sensors and return an action to the actuators.4 Notice the\ndifference between the agent program, which takes the current percept as input, and the agent\nfunction, which takes the entire percept history. The agent program takes just the current\npercept as input because nothing more is available from the environment; if the agent’s actions\nneed to depend on the entire percept sequence, the agent will have to remember the percepts.\nWe describe the agent programs in the simple pseudocode language that is defined in\nAppendix B. (The online code repository contains implementations in real programming\nlanguages.) For example, Figure 2.7 shows a rather trivial agent program that keeps track of\nthe percept sequence and then uses it to index into a table of actions to decide what to do.\nThe table—an example of which is given for the vacuum world in Figure 2.3—represents\nexplicitly the agent function that the agent program embodies. To build a rational agent in\n4 There are other choices for the agent program skeleton; for example, we could have the agent programs be\ncoroutines that run asynchronously with the environment. Each such coroutine has an input and output port and\nconsists of a loop that reads the input port for percepts and writes actions to the output port.\nSection 2.4.\nThe Structure of Agents\n47\nfunction TABLE-DRIVEN-AGENT(percept) returns an action\npersistent: percepts, a sequence, initially empty\ntable, a table of actions, indexed by percept sequences, initially fully specified\nappend percept to the end of percepts\naction ←LOOKUP(percepts,table)\nreturn action\nFigure 2.7\nThe TABLE-DRIVEN-AGENT program is invoked for each new percept and\nreturns an action each time. It retains the complete percept sequence in memory.\nthis way, we as designers must construct a table that contains the appropriate action for every\npossible percept sequence.\nIt is instructive to consider why the table-driven approach to agent construction is\ndoomed to failure. Let P be the set of possible percepts and let T be the lifetime of the\nagent (the total number of percepts it will receive). The lookup table will contain \u0002T\nt = 1 |P|t\nentries. Consider the automated taxi: the visual input from a single camera comes in at the\nrate of roughly 27 megabytes per second (30 frames per second, 640 × 480 pixels with 24\nbits of color information). This gives a lookup table with over 10250,000,000,000 entries for an\nhour’s driving. Even the lookup table for chess—a tiny, well-behaved fragment of the real\nworld—would have at least 10150 entries. The daunting size of these tables (the number of\natoms in the observable universe is less than 1080) means that (a) no physical agent in this\nuniverse will have the space to store the table, (b) the designer would not have time to create\nthe table, (c) no agent could ever learn all the right table entries from its experience, and (d)\neven if the environment is simple enough to yield a feasible table size, the designer still has\nno guidance about how to fill in the table entries.\nDespite all this, TABLE-DRIVEN-AGENT does do what we want: it implements the\ndesired agent function. The key challenge for AI is to find out how to write programs that,\nto the extent possible, produce rational behavior from a smallish program rather than from\na vast table. We have many examples showing that this can be done successfully in other\nareas: for example, the huge tables of square roots used by engineers and schoolchildren prior\nto the 1970s have now been replaced by a five-line program for Newton’s method running\non electronic calculators. The question is, can AI do for general intelligent behavior what\nNewton did for square roots? We believe the answer is yes.\nIn the remainder of this section, we outline four basic kinds of agent programs that\nembody the principles underlying almost all intelligent systems:\n• Simple reflex agents;\n• Model-based reflex agents;\n• Goal-based agents; and\n• Utility-based agents.\nEach kind of agent program combines particular components in particular ways to generate\nactions. Section 2.4.6 explains in general terms how to convert all these agents into learning\n48\nChapter\n2.\nIntelligent Agents\nfunction REFLEX-VACUUM-AGENT([location,status]) returns an action\nif status = Dirty then return Suck\nelse if location = A then return Right\nelse if location = B then return Left\nFigure 2.8\nThe agent program for a simple reflex agent in the two-state vacuum environ-\nment. This program implements the agent function tabulated in Figure 2.3.\nagents that can improve the performance of their components so as to generate better actions.\nFinally, Section 2.4.7 describes the variety of ways in which the components themselves can\nbe represented within the agent. This variety provides a major organizing principle for the\nfield and for the book itself.\n2.4.2\nSimple reflex agents\nThe simplest kind of agent is the simple reflex agent. These agents select actions on the basis\nSIMPLE REFLEX\nAGENT\nof the current percept, ignoring the rest of the percept history. For example, the vacuum agent\nwhose agent function is tabulated in Figure 2.3 is a simple reflex agent, because its decision\nis based only on the current location and on whether that location contains dirt. An agent\nprogram for this agent is shown in Figure 2.8.\nNotice that the vacuum agent program is very small indeed compared to the correspond-\ning table. The most obvious reduction comes from ignoring the percept history, which cuts\ndown the number of possibilities from 4T to just 4. A further, small reduction comes from\nthe fact that when the current square is dirty, the action does not depend on the location.\nSimple reflex behaviors occur even in more complex environments. Imagine yourself\nas the driver of the automated taxi. If the car in front brakes and its brake lights come on, then\nyou should notice this and initiate braking. In other words, some processing is done on the\nvisual input to establish the condition we call “The car in front is braking.” Then, this triggers\nsome established connection in the agent program to the action “initiate braking.” We call\nsuch a connection a condition–action rule,5 written as\nCONDITION–ACTION\nRULE\nif car-in-front-is-braking then initiate-braking.\nHumans also have many such connections, some of which are learned responses (as for driv-\ning) and some of which are innate reflexes (such as blinking when something approaches the\neye). In the course of the book, we show several different ways in which such connections\ncan be learned and implemented.\nThe program in Figure 2.8 is specific to one particular vacuum environment. A more\ngeneral and flexible approach is first to build a general-purpose interpreter for condition–\naction rules and then to create rule sets for specific task environments. Figure 2.9 gives the\nstructure of this general program in schematic form, showing how the condition–action rules\nallow the agent to make the connection from percept to action. (Do not worry if this seems\n5 Also called situation–action rules, productions, or if–then rules.\nSection 2.4.\nThe Structure of Agents\n49\nAgent\nEnvironment\nSensors\nWhat action I\nshould do now\nCondition-action rules\nActuators\nWhat the world\nis like now\nFigure 2.9\nSchematic diagram of a simple reflex agent.\nfunction SIMPLE-REFLEX-AGENT(percept) returns an action\npersistent: rules, a set of condition–action rules\nstate ←INTERPRET-INPUT(percept)\nrule ←RULE-MATCH(state,rules)\naction ←rule.ACTION\nreturn action\nFigure 2.10\nA simple reflex agent. It acts according to a rule whose condition matches\nthe current state, as defined by the percept.\ntrivial; it gets more interesting shortly.) We use rectangles to denote the current internal state\nof the agent’s decision process, and ovals to represent the background information used in\nthe process. The agent program, which is also very simple, is shown in Figure 2.10. The\nINTERPRET-INPUT function generates an abstracted description of the current state from the\npercept, and the RULE-MATCH function returns the first rule in the set of rules that matches\nthe given state description. Note that the description in terms of “rules” and “matching” is\npurely conceptual; actual implementations can be as simple as a collection of logic gates\nimplementing a Boolean circuit.\nSimple reflex agents have the admirable property of being simple, but they turn out to be\nof limited intelligence. The agent in Figure 2.10 will work only if the correct decision can be\nmade on the basis of only the current percept—that is, only if the environment is fully observ-\nable. Even a little bit of unobservability can cause serious trouble. For example, the braking\nrule given earlier assumes that the condition car-in-front-is-braking can be determined from\nthe current percept—a single frame of video. This works if the car in front has a centrally\nmounted brake light. Unfortunately, older models have different configurations of taillights,\n50\nChapter\n2.\nIntelligent Agents\nbrake lights, and turn-signal lights, and it is not always possible to tell from a single image\nwhether the car is braking. A simple reflex agent driving behind such a car would either brake\ncontinuously and unnecessarily, or, worse, never brake at all.\nWe can see a similar problem arising in the vacuum world. Suppose that a simple reflex\nvacuum agent is deprived of its location sensor and has only a dirt sensor. Such an agent\nhas just two possible percepts: [Dirty] and [Clean]. It can Suck in response to [Dirty]; what\nshould it do in response to [Clean]? Moving Left fails (forever) if it happens to start in square\nA, and moving Right fails (forever) if it happens to start in square B. Infinite loops are often\nunavoidable for simple reflex agents operating in partially observable environments.\nEscape from infinite loops is possible if the agent can randomize its actions. For ex-\nRANDOMIZATION\nample, if the vacuum agent perceives [Clean], it might flip a coin to choose between Left and\nRight. It is easy to show that the agent will reach the other square in an average of two steps.\nThen, if that square is dirty, the agent will clean it and the task will be complete. Hence, a\nrandomized simple reflex agent might outperform a deterministic simple reflex agent.\nWe mentioned in Section 2.3 that randomized behavior of the right kind can be rational\nin some multiagent environments. In single-agent environments, randomization is usually not\nrational. It is a useful trick that helps a simple reflex agent in some situations, but in most\ncases we can do much better with more sophisticated deterministic agents.\n2.4.3\nModel-based reflex agents\nThe most effective way to handle partial observability is for the agent to keep track of the\npart of the world it can’t see now. That is, the agent should maintain some sort of internal\nstate that depends on the percept history and thereby reflects at least some of the unobserved\nINTERNAL STATE\naspects of the current state. For the braking problem, the internal state is not too extensive—\njust the previous frame from the camera, allowing the agent to detect when two red lights at\nthe edge of the vehicle go on or off simultaneously. For other driving tasks such as changing\nlanes, the agent needs to keep track of where the other cars are if it can’t see them all at once.\nAnd for any driving to be possible at all, the agent needs to keep track of where its keys are.\nUpdating this internal state information as time goes by requires two kinds of knowl-\nedge to be encoded in the agent program. First, we need some information about how the\nworld evolves independently of the agent—for example, that an overtaking car generally will\nbe closer behind than it was a moment ago. Second, we need some information about how\nthe agent’s own actions affect the world—for example, that when the agent turns the steering\nwheel clockwise, the car turns to the right, or that after driving for five minutes northbound\non the freeway, one is usually about five miles north of where one was five minutes ago. This\nknowledge about “how the world works”—whether implemented in simple Boolean circuits\nor in complete scientific theories—is called a model of the world. An agent that uses such a\nmodel is called a model-based agent.\nMODEL-BASED\nAGENT\nFigure 2.11 gives the structure of the model-based reflex agent with internal state, show-\ning how the current percept is combined with the old internal state to generate the updated\ndescription of the current state, based on the agent’s model of how the world works. The agent\nprogram is shown in Figure 2.12. The interesting part is the function UPDATE-STATE, which\nSection 2.4.\nThe Structure of Agents\n51\nAgent\nEnvironment\nSensors\nState\nHow the world evolves\nWhat my actions do\nCondition-action rules\nActuators\nWhat the world\nis like now\nWhat action I\nshould do now\nFigure 2.11\nA model-based reflex agent.\nfunction MODEL-BASED-REFLEX-AGENT(percept) returns an action\npersistent: state, the agent’s current conception of the world state\nmodel, a description of how the next state depends on current state and action\nrules, a set of condition–action rules\naction, the most recent action, initially none\nstate ←UPDATE-STATE(state,action,percept,model)\nrule ←RULE-MATCH(state,rules)\naction ←rule.ACTION\nreturn action\nFigure 2.12\nA model-based reflex agent. It keeps track of the current state of the world,\nusing an internal model. It then chooses an action in the same way as the reflex agent.\nis responsible for creating the new internal state description. The details of how models and\nstates are represented vary widely depending on the type of environment and the particular\ntechnology used in the agent design. Detailed examples of models and updating algorithms\nappear in Chapters 4, 12, 11, 15, 17, and 25.\nRegardless of the kind of representation used, it is seldom possible for the agent to\ndetermine the current state of a partially observable environment exactly. Instead, the box\nlabeled “what the world is like now” (Figure 2.11) represents the agent’s “best guess” (or\nsometimes best guesses). For example, an automated taxi may not be able to see around the\nlarge truck that has stopped in front of it and can only guess about what may be causing the\nhold-up. Thus, uncertainty about the current state may be unavoidable, but the agent still has\nto make a decision.\nA perhaps less obvious point about the internal “state” maintained by a model-based\nagent is that it does not have to describe “what the world is like now” in a literal sense. For\n52\nChapter\n2.\nIntelligent Agents\nAgent\nEnvironment\nSensors\nWhat action I\nshould do now\nState\nHow the world evolves\nWhat my actions do\nActuators\nWhat the world\nis like now\nWhat it will be like\n if I do action A\nGoals\nFigure 2.13\nA model-based, goal-based agent. It keeps track of the world state as well as\na set of goals it is trying to achieve, and chooses an action that will (eventually) lead to the\nachievement of its goals.\nexample, the taxi may be driving back home, and it may have a rule telling it to fill up with\ngas on the way home unless it has at least half a tank. Although “driving back home” may\nseem to an aspect of the world state, the fact of the taxi’s destination is actually an aspect of\nthe agent’s internal state. If you find this puzzling, consider that the taxi could be in exactly\nthe same place at the same time, but intending to reach a different destination.\n2.4.4\nGoal-based agents\nKnowing something about the current state of the environment is not always enough to decide\nwhat to do. For example, at a road junction, the taxi can turn left, turn right, or go straight\non. The correct decision depends on where the taxi is trying to get to. In other words, as well\nas a current state description, the agent needs some sort of goal information that describes\nGOAL\nsituations that are desirable—for example, being at the passenger’s destination. The agent\nprogram can combine this with the model (the same information as was used in the model-\nbased reflex agent) to choose actions that achieve the goal. Figure 2.13 shows the goal-based\nagent’s structure.\nSometimes goal-based action selection is straightforward—for example, when goal sat-\nisfaction results immediately from a single action. Sometimes it will be more tricky—for\nexample, when the agent has to consider long sequences of twists and turns in order to find a\nway to achieve the goal. Search (Chapters 3 to 5) and planning (Chapters 10 and 11) are the\nsubfields of AI devoted to finding action sequences that achieve the agent’s goals.\nNotice that decision making of this kind is fundamentally different from the condition–\naction rules described earlier, in that it involves consideration of the future—both “What will\nhappen if I do such-and-such?” and “Will that make me happy?” In the reflex agent designs,\nthis information is not explicitly represented, because the built-in rules map directly from\nSection 2.4.\nThe Structure of Agents\n53\npercepts to actions. The reflex agent brakes when it sees brake lights. A goal-based agent, in\nprinciple, could reason that if the car in front has its brake lights on, it will slow down. Given\nthe way the world usually evolves, the only action that will achieve the goal of not hitting\nother cars is to brake.\nAlthough the goal-based agent appears less efficient, it is more flexible because the\nknowledge that supports its decisions is represented explicitly and can be modified. If it starts\nto rain, the agent can update its knowledge of how effectively its brakes will operate; this will\nautomatically cause all of the relevant behaviors to be altered to suit the new conditions.\nFor the reflex agent, on the other hand, we would have to rewrite many condition–action\nrules. The goal-based agent’s behavior can easily be changed to go to a different destination,\nsimply by specifying that destination as the goal. The reflex agent’s rules for when to turn\nand when to go straight will work only for a single destination; they must all be replaced to\ngo somewhere new.\n2.4.5\nUtility-based agents\nGoals alone are not enough to generate high-quality behavior in most environments. For\nexample, many action sequences will get the taxi to its destination (thereby achieving the\ngoal) but some are quicker, safer, more reliable, or cheaper than others. Goals just provide a\ncrude binary distinction between “happy” and “unhappy” states. A more general performance\nmeasure should allow a comparison of different world states according to exactly how happy\nthey would make the agent. Because “happy” does not sound very scientific, economists and\ncomputer scientists use the term utility instead.6\nUTILITY\nWe have already seen that a performance measure assigns a score to any given sequence\nof environment states, so it can easily distinguish between more and less desirable ways of\ngetting to the taxi’s destination. An agent’s utility function is essentially an internalization\nUTILITY FUNCTION\nof the performance measure. If the internal utility function and the external performance\nmeasure are in agreement, then an agent that chooses actions to maximize its utility will be\nrational according to the external performance measure.\nLet us emphasize again that this is not the only way to be rational—we have already\nseen a rational agent program for the vacuum world (Figure 2.8) that has no idea what its\nutility function is—but, like goal-based agents, a utility-based agent has many advantages in\nterms of flexibility and learning. Furthermore, in two kinds of cases, goals are inadequate but\na utility-based agent can still make rational decisions. First, when there are conflicting goals,\nonly some of which can be achieved (for example, speed and safety), the utility function\nspecifies the appropriate tradeoff. Second, when there are several goals that the agent can\naim for, none of which can be achieved with certainty, utility provides a way in which the\nlikelihood of success can be weighed against the importance of the goals.\nPartial observability and stochasticity are ubiquitous in the real world, and so, therefore,\nis decision making under uncertainty. Technically speaking, a rational utility-based agent\nchooses the action that maximizes the expected utility of the action outcomes—that is, the\nEXPECTED UTILITY\nutility the agent expects to derive, on average, given the probabilities and utilities of each\n6 The word “utility” here refers to “the quality of being useful,” not to the electric company or waterworks.\n54\nChapter\n2.\nIntelligent Agents\nAgent\nEnvironment\nSensors\nHow happy I will be\nin such a state\nState\nHow the world evolves\nWhat my actions do\nUtility\nActuators\nWhat action I\nshould do now\nWhat it will be like\nif I do action A\nWhat the world\nis like now\nFigure 2.14\nA model-based, utility-based agent. It uses a model of the world, along with\na utility function that measures its preferences among states of the world. Then it chooses the\naction that leads to the best expected utility, where expected utility is computed by averaging\nover all possible outcome states, weighted by the probability of the outcome.\noutcome. (Appendix A defines expectation more precisely.) In Chapter 16, we show that any\nrational agent must behave as if it possesses a utility function whose expected value it tries\nto maximize. An agent that possesses an explicit utility function can make rational decisions\nwith a general-purpose algorithm that does not depend on the specific utility function being\nmaximized. In this way, the “global” definition of rationality—designating as rational those\nagent functions that have the highest performance—is turned into a “local” constraint on\nrational-agent designs that can be expressed in a simple program.\nThe utility-based agent structure appears in Figure 2.14. Utility-based agent programs\nappear in Part IV, where we design decision-making agents that must handle the uncertainty\ninherent in stochastic or partially observable environments.\nAt this point, the reader may be wondering, “Is it that simple? We just build agents that\nmaximize expected utility, and we’re done?” It’s true that such agents would be intelligent,\nbut it’s not simple. A utility-based agent has to model and keep track of its environment,\ntasks that have involved a great deal of research on perception, representation, reasoning,\nand learning. The results of this research fill many of the chapters of this book. Choosing\nthe utility-maximizing course of action is also a difficult task, requiring ingenious algorithms\nthat fill several more chapters. Even with these algorithms, perfect rationality is usually\nunachievable in practice because of computational complexity, as we noted in Chapter 1.\n2.4.6\nLearning agents\nWe have described agent programs with various methods for selecting actions. We have\nnot, so far, explained how the agent programs come into being. In his famous early paper,\nTuring (1950) considers the idea of actually programming his intelligent machines by hand.\nSection 2.4.\nThe Structure of Agents\n55\nPerformance standard\nAgent\nEnvironment\nSensors\nPerformance\nelement\nchanges\nknowledge\nlearning\n goals\nProblem\ngenerator\nfeedback\n Learning\nelement\nCritic\nActuators\nFigure 2.15\nA general learning agent.\nHe estimates how much work this might take and concludes “Some more expeditious method\nseems desirable.” The method he proposes is to build learning machines and then to teach\nthem. In many areas of AI, this is now the preferred method for creating state-of-the-art\nsystems. Learning has another advantage, as we noted earlier: it allows the agent to operate\nin initially unknown environments and to become more competent than its initial knowledge\nalone might allow. In this section, we briefly introduce the main ideas of learning agents.\nThroughout the book, we comment on opportunities and methods for learning in particular\nkinds of agents. Part V goes into much more depth on the learning algorithms themselves.\nA learning agent can be divided into four conceptual components, as shown in Fig-\nure 2.15. The most important distinction is between the learning element, which is re-\nLEARNING ELEMENT\nsponsible for making improvements, and the performance element, which is responsible for\nPERFORMANCE\nELEMENT\nselecting external actions. The performance element is what we have previously considered\nto be the entire agent: it takes in percepts and decides on actions. The learning element uses\nfeedback from the critic on how the agent is doing and determines how the performance\nCRITIC\nelement should be modified to do better in the future.\nThe design of the learning element depends very much on the design of the performance\nelement. When trying to design an agent that learns a certain capability, the first question is\nnot “How am I going to get it to learn this?” but “What kind of performance element will my\nagent need to do this once it has learned how?” Given an agent design, learning mechanisms\ncan be constructed to improve every part of the agent.\nThe critic tells the learning element how well the agent is doing with respect to a fixed\nperformance standard. The critic is necessary because the percepts themselves provide no\nindication of the agent’s success. For example, a chess program could receive a percept\nindicating that it has checkmated its opponent, but it needs a performance standard to know\nthat this is a good thing; the percept itself does not say so. It is important that the performance\n56\nChapter\n2.\nIntelligent Agents\nstandard be fixed. Conceptually, one should think of it as being outside the agent altogether\nbecause the agent must not modify it to fit its own behavior.\nThe last component of the learning agent is the problem generator. It is responsible\nPROBLEM\nGENERATOR\nfor suggesting actions that will lead to new and informative experiences. The point is that\nif the performance element had its way, it would keep doing the actions that are best, given\nwhat it knows. But if the agent is willing to explore a little and do some perhaps suboptimal\nactions in the short run, it might discover much better actions for the long run. The problem\ngenerator’s job is to suggest these exploratory actions. This is what scientists do when they\ncarry out experiments. Galileo did not think that dropping rocks from the top of a tower in\nPisa was valuable in itself. He was not trying to break the rocks or to modify the brains of\nunfortunate passers-by. His aim was to modify his own brain by identifying a better theory\nof the motion of objects.\nTo make the overall design more concrete, let us return to the automated taxi example.\nThe performance element consists of whatever collection of knowledge and procedures the\ntaxi has for selecting its driving actions. The taxi goes out on the road and drives, using\nthis performance element. The critic observes the world and passes information along to the\nlearning element. For example, after the taxi makes a quick left turn across three lanes of traf-\nfic, the critic observes the shocking language used by other drivers. From this experience, the\nlearning element is able to formulate a rule saying this was a bad action, and the performance\nelement is modified by installation of the new rule. The problem generator might identify\ncertain areas of behavior in need of improvement and suggest experiments, such as trying out\nthe brakes on different road surfaces under different conditions.\nThe learning element can make changes to any of the “knowledge” components shown\nin the agent diagrams (Figures 2.9, 2.11, 2.13, and 2.14). The simplest cases involve learning\ndirectly from the percept sequence. Observation of pairs of successive states of the environ-\nment can allow the agent to learn “How the world evolves,” and observation of the results of\nits actions can allow the agent to learn “What my actions do.” For example, if the taxi exerts\na certain braking pressure when driving on a wet road, then it will soon find out how much\ndeceleration is actually achieved. Clearly, these two learning tasks are more difficult if the\nenvironment is only partially observable.\nThe forms of learning in the preceding paragraph do not need to access the external\nperformance standard—in a sense, the standard is the universal one of making predictions\nthat agree with experiment. The situation is slightly more complex for a utility-based agent\nthat wishes to learn utility information. For example, suppose the taxi-driving agent receives\nno tips from passengers who have been thoroughly shaken up during the trip. The external\nperformance standard must inform the agent that the loss of tips is a negative contribution to\nits overall performance; then the agent might be able to learn that violent maneuvers do not\ncontribute to its own utility. In a sense, the performance standard distinguishes part of the\nincoming percept as a reward (or penalty) that provides direct feedback on the quality of the\nagent’s behavior. Hard-wired performance standards such as pain and hunger in animals can\nbe understood in this way. This issue is discussed further in Chapter 21.\nIn summary, agents have a variety of components, and those components can be repre-\nsented in many ways within the agent program, so there appears to be great variety among\nSection 2.4.\nThe Structure of Agents\n57\nlearning methods. There is, however, a single unifying theme. Learning in intelligent agents\ncan be summarized as a process of modification of each component of the agent to bring the\ncomponents into closer agreement with the available feedback information, thereby improv-\ning the overall performance of the agent.\n2.4.7\nHow the components of agent programs work\nWe have described agent programs (in very high-level terms) as consisting of various compo-\nnents, whose function it is to answer questions such as: “What is the world like now?” “What\naction should I do now?” “What do my actions do?” The next question for a student of AI\nis, “How on earth do these components work?” It takes about a thousand pages to begin to\nanswer that question properly, but here we want to draw the reader’s attention to some basic\ndistinctions among the various ways that the components can represent the environment that\nthe agent inhabits.\nRoughly speaking, we can place the representations along an axis of increasing com-\nplexity and expressive power—atomic, factored, and structured. To illustrate these ideas,\nit helps to consider a particular agent component, such as the one that deals with “What my\nactions do.” This component describes the changes that might occur in the environment as\nthe result of taking an action, and Figure 2.16 provides schematic depictions of how those\ntransitions might be represented.\nB\nC\n(a) Atomic\n(b) Factored\n(b) Structured\nB\nC\nFigure 2.16\nThree ways to represent states and the transitions between them. (a) Atomic\nrepresentation: a state (such as B or C) is a black box with no internal structure; (b) Factored\nrepresentation: a state consists of a vector of attribute values; values can be Boolean, real-\nvalued, or one of a fixed set of symbols. (c) Structured representation: a state includes\nobjects, each of which may have attributes of its own as well as relationships to other objects.\nIn an atomic representation each state of the world is indivisible—it has no internal\nATOMIC\nREPRESENTATION\nstructure. Consider the problem of finding a driving route from one end of a country to the\nother via some sequence of cities (we address this problem in Figure 3.2 on page 68). For the\npurposes of solving this problem, it may suffice to reduce the state of world to just the name\nof the city we are in—a single atom of knowledge; a “black box” whose only discernible\nproperty is that of being identical to or different from another black box. The algorithms\n58\nChapter\n2.\nIntelligent Agents\nunderlying search and game-playing (Chapters 3–5), Hidden Markov models (Chapter 15),\nand Markov decision processes (Chapter 17) all work with atomic representations—or, at\nleast, they treat representations as if they were atomic.\nNow consider a higher-fidelity description for the same problem, where we need to be\nconcerned with more than just atomic location in one city or another; we might need to pay\nattention to how much gas is in the tank, our current GPS coordinates, whether or not the oil\nwarning light is working, how much spare change we have for toll crossings, what station is\non the radio, and so on. A factored representation splits up each state into a fixed set of\nFACTORED\nREPRESENTATION\nvariables or attributes, each of which can have a value. While two different atomic states\nVARIABLE\nATTRIBUTE\nVALUE\nhave nothing in common—they are just different black boxes—two different factored states\ncan share some attributes (such as being at some particular GPS location) and not others (such\nas having lots of gas or having no gas); this makes it much easier to work out how to turn\none state into another. With factored representations, we can also represent uncertainty—for\nexample, ignorance about the amount of gas in the tank can be represented by leaving that\nattribute blank. Many important areas of AI are based on factored representations, including\nconstraint satisfaction algorithms (Chapter 6), propositional logic (Chapter 7), planning\n(Chapters 10 and 11), Bayesian networks (Chapters 13–16), and the machine learning al-\ngorithms in Chapters 18, 20, and 21.\nFor many purposes, we need to understand the world as having things in it that are\nrelated to each other, not just variables with values. For example, we might notice that a\nlarge truck ahead of us is reversing into the driveway of a dairy farm but a cow has got loose\nand is blocking the truck’s path. A factored representation is unlikely to be pre-equipped\nwith the attribute TruckAheadBackingIntoDairyFarmDrivewayBlockedByLooseCow with\nvalue true or false. Instead, we would need a structured representation, in which ob-\nSTRUCTURED\nREPRESENTATION\njects such as cows and trucks and their various and varying relationships can be described\nexplicitly. (See Figure 2.16(c).) Structured representations underlie relational databases\nand first-order logic (Chapters 8, 9, and 12), first-order probability models (Chapter 14),\nknowledge-based learning (Chapter 19) and much of natural language understanding\n(Chapters 22 and 23). In fact, almost everything that humans express in natural language\nconcerns objects and their relationships.\nAs we mentioned earlier, the axis along which atomic, factored, and structured repre-\nsentations lie is the axis of increasing expressiveness. Roughly speaking, a more expressive\nEXPRESSIVENESS\nrepresentation can capture, at least as concisely, everything a less expressive one can capture,\nplus some more. Often, the more expressive language is much more concise; for example, the\nrules of chess can be written in a page or two of a structured-representation language such\nas first-order logic but require thousands of pages when written in a factored-representation\nlanguage such as propositional logic. On the other hand, reasoning and learning become\nmore complex as the expressive power of the representation increases. To gain the benefits\nof expressive representations while avoiding their drawbacks, intelligent systems for the real\nworld may need to operate at all points along the axis simultaneously.\nSection 2.5.\nSummary\n59\n2.5\nSUMMARY\nThis chapter has been something of a whirlwind tour of AI, which we have conceived of as\nthe science of agent design. The major points to recall are as follows:\n• An agent is something that perceives and acts in an environment. The agent function\nfor an agent specifies the action taken by the agent in response to any percept sequence.\n• The performance measure evaluates the behavior of the agent in an environment. A\nrational agent acts so as to maximize the expected value of the performance measure,\ngiven the percept sequence it has seen so far.\n• A task environment specification includes the performance measure, the external en-\nvironment, the actuators, and the sensors. In designing an agent, the first step must\nalways be to specify the task environment as fully as possible.\n• Task environments vary along several significant dimensions. They can be fully or\npartially observable, single-agent or multiagent, deterministic or stochastic, episodic or\nsequential, static or dynamic, discrete or continuous, and known or unknown.\n• The agent program implements the agent function. There exists a variety of basic\nagent-program designs reflecting the kind of information made explicit and used in the\ndecision process. The designs vary in efficiency, compactness, and flexibility. The\nappropriate design of the agent program depends on the nature of the environment.\n• Simple reflex agents respond directly to percepts, whereas model-based reflex agents\nmaintain internal state to track aspects of the world that are not evident in the current\npercept. Goal-based agents act to achieve their goals, and utility-based agents try to\nmaximize their own expected “happiness.”\n• All agents can improve their performance through learning.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nThe central role of action in intelligence—the notion of practical reasoning—goes back at\nleast as far as Aristotle’s Nicomachean Ethics. Practical reasoning was also the subject of\nMcCarthy’s (1958) influential paper “Programs with Common Sense.” The fields of robotics\nand control theory are, by their very nature, concerned principally with physical agents. The\nconcept of a controller in control theory is identical to that of an agent in AI. Perhaps sur-\nCONTROLLER\nprisingly, AI has concentrated for most of its history on isolated components of agents—\nquestion-answering systems, theorem-provers, vision systems, and so on—rather than on\nwhole agents. The discussion of agents in the text by Genesereth and Nilsson (1987) was an\ninfluential exception. The whole-agent view is now widely accepted and is a central theme in\nrecent texts (Poole et al., 1998; Nilsson, 1998; Padgham and Winikoff, 2004; Jones, 2007).\nChapter 1 traced the roots of the concept of rationality in philosophy and economics. In\nAI, the concept was of peripheral interest until the mid-1980s, when it began to suffuse many\n60\nChapter\n2.\nIntelligent Agents\ndiscussions about the proper technical foundations of the field. A paper by Jon Doyle (1983)\npredicted that rational agent design would come to be seen as the core mission of AI, while\nother popular topics would spin off to form new disciplines.\nCareful attention to the properties of the environment and their consequences for ra-\ntional agent design is most apparent in the control theory tradition—for example, classical\ncontrol systems (Dorf and Bishop, 2004; Kirk, 2004) handle fully observable, deterministic\nenvironments; stochastic optimal control (Kumar and Varaiya, 1986; Bertsekas and Shreve,\n2007) handles partially observable, stochastic environments; and hybrid control (Henzinger\nand Sastry, 1998; Cassandras and Lygeros, 2006) deals with environments containing both\ndiscrete and continuous elements. The distinction between fully and partially observable en-\nvironments is also central in the dynamic programming literature developed in the field of\noperations research (Puterman, 1994), which we discuss in Chapter 17.\nReflex agents were the primary model for psychological behaviorists such as Skinner\n(1953), who attempted to reduce the psychology of organisms strictly to input/output or stim-\nulus/response mappings. The advance from behaviorism to functionalism in psychology,\nwhich was at least partly driven by the application of the computer metaphor to agents (Put-\nnam, 1960; Lewis, 1966), introduced the internal state of the agent into the picture. Most\nwork in AI views the idea of pure reflex agents with state as too simple to provide much\nleverage, but work by Rosenschein (1985) and Brooks (1986) questioned this assumption\n(see Chapter 25). In recent years, a great deal of work has gone into finding efficient algo-\nrithms for keeping track of complex environments (Hamscher et al., 1992; Simon, 2006). The\nRemote Agent program (described on page 28) that controlled the Deep Space One spacecraft\nis a particularly impressive example (Muscettola et al., 1998; Jonsson et al., 2000).\nGoal-based agents are presupposed in everything from Aristotle’s view of practical rea-\nsoning to McCarthy’s early papers on logical AI. Shakey the Robot (Fikes and Nilsson,\n1971; Nilsson, 1984) was the first robotic embodiment of a logical, goal-based agent. A\nfull logical analysis of goal-based agents appeared in Genesereth and Nilsson (1987), and a\ngoal-based programming methodology called agent-oriented programming was developed by\nShoham (1993). The agent-based approach is now extremely popular in software engineer-\ning (Ciancarini and Wooldridge, 2001). It has also infiltrated the area of operating systems,\nwhere autonomic computing refers to computer systems and networks that monitor and con-\nAUTONOMIC\nCOMPUTING\ntrol themselves with a perceive–act loop and machine learning methods (Kephart and Chess,\n2003). Noting that a collection of agent programs designed to work well together in a true\nmultiagent environment necessarily exhibits modularity—the programs share no internal state\nand communicate with each other only through the environment—it is common within the\nfield of multiagent systems to design the agent program of a single agent as a collection of\nMULTIAGENT\nSYSTEMS\nautonomous sub-agents. In some cases, one can even prove that the resulting system gives\nthe same optimal solutions as a monolithic design.\nThe goal-based view of agents also dominates the cognitive psychology tradition in the\narea of problem solving, beginning with the enormously influential Human Problem Solv-\ning (Newell and Simon, 1972) and running through all of Newell’s later work (Newell, 1990).\nGoals, further analyzed as desires (general) and intentions (currently pursued), are central to\nthe theory of agents developed by Bratman (1987). This theory has been influential both in\nExercises\n61\nnatural language understanding and multiagent systems.\nHorvitz et al. (1988) specifically suggest the use of rationality conceived as the maxi-\nmization of expected utility as a basis for AI. The text by Pearl (1988) was the first in AI to\ncover probability and utility theory in depth; its exposition of practical methods for reasoning\nand decision making under uncertainty was probably the single biggest factor in the rapid\nshift towards utility-based agents in the 1990s (see Part IV).\nThe general design for learning agents portrayed in Figure 2.15 is classic in the machine\nlearning literature (Buchanan et al., 1978; Mitchell, 1997). Examples of the design, as em-\nbodied in programs, go back at least as far as Arthur Samuel’s (1959, 1967) learning program\nfor playing checkers. Learning agents are discussed in depth in Part V.\nInterest in agents and in agent design has risen rapidly in recent years, partly because of\nthe growth of the Internet and the perceived need for automated and mobile softbot (Etzioni\nand Weld, 1994). Relevant papers are collected in Readings in Agents (Huhns and Singh,\n1998) and Foundations of Rational Agency (Wooldridge and Rao, 1999). Texts on multiagent\nsystems usually provide a good introduction to many aspects of agent design (Weiss, 2000a;\nWooldridge, 2002). Several conference series devoted to agents began in the 1990s, including\nthe International Workshop on Agent Theories, Architectures, and Languages (ATAL), the\nInternational Conference on Autonomous Agents (AGENTS), and the International Confer-\nence on Multi-Agent Systems (ICMAS). In 2002, these three merged to form the International\nJoint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS). The journal\nAutonomous Agents and Multi-Agent Systems was founded in 1998. Finally, Dung Beetle\nEcology (Hanski and Cambefort, 1991) provides a wealth of interesting information on the\nbehavior of dung beetles. YouTube features inspiring video recordings of their activities.\nEXERCISES\n2.1\nSuppose that the performance measure is concerned with just the first T time steps of\nthe environment and ignores everything thereafter. Show that a rational agent’s action may\ndepend not just on the state of the environment but also on the time step it has reached.\n2.2\nLet us examine the rationality of various vacuum-cleaner agent functions.\na. Show that the simple vacuum-cleaner agent function described in Figure 2.3 is indeed\nrational under the assumptions listed on page 38.\nb. Describe a rational agent function for the case in which each movement costs one point.\nDoes the corresponding agent program require internal state?\nc. Discuss possible agent designs for the cases in which clean squares can become dirty\nand the geography of the environment is unknown. Does it make sense for the agent to\nlearn from its experience in these cases? If so, what should it learn? If not, why not?\n2.3\nFor each of the following assertions, say whether it is true or false and support your\nanswer with examples or counterexamples where appropriate.\na. An agent that senses only partial information about the state cannot be perfectly rational.\n62\nChapter\n2.\nIntelligent Agents\nb. There exist task environments in which no pure reflex agent can behave rationally.\nc. There exists a task environment in which every agent is rational.\nd. The input to an agent program is the same as the input to the agent function.\ne. Every agent function is implementable by some program/machine combination.\nf. Suppose an agent selects its action uniformly at random from the set of possible actions.\nThere exists a deterministic task environment in which this agent is rational.\ng. It is possible for a given agent to be perfectly rational in two distinct task environments.\nh. Every agent is rational in an unobservable environment.\ni. A perfectly rational poker-playing agent never loses.\n2.4\nFor each of the following activities, give a PEAS description of the task environment\nand characterize it in terms of the properties listed in Section 2.3.2.\n• Playing soccer.\n• Exploring the subsurface oceans of Titan.\n• Shopping for used AI books on the Internet.\n• Playing a tennis match.\n• Practicing tennis against a wall.\n• Performing a high jump.\n• Knitting a sweater.\n• Bidding on an item at an auction.\n2.5\nDefine in your own words the following terms: agent, agent function, agent program,\nrationality, autonomy, reflex agent, model-based agent, goal-based agent, utility-based agent,\nlearning agent.\n2.6\nThis exercise explores the differences between agent functions and agent programs.\na. Can there be more than one agent program that implements a given agent function?\nGive an example, or show why one is not possible.\nb. Are there agent functions that cannot be implemented by any agent program?\nc. Given a fixed machine architecture, does each agent program implement exactly one\nagent function?\nd. Given an architecture with n bits of storage, how many different possible agent pro-\ngrams are there?\ne. Suppose we keep the agent program fixed but speed up the machine by a factor of two.\nDoes that change the agent function?\n2.7\nWrite pseudocode agent programs for the goal-based and utility-based agents.\nThe following exercises all concern the implementation of environments and agents for the\nvacuum-cleaner world.\nExercises\n63\n2.8\nImplement a performance-measuring environment simulator for the vacuum-cleaner\nworld depicted in Figure 2.2 and specified on page 38. Your implementation should be modu-\nlar so that the sensors, actuators, and environment characteristics (size, shape, dirt placement,\netc.) can be changed easily. (Note: for some choices of programming language and operating\nsystem there are already implementations in the online code repository.)\n2.9\nImplement a simple reflex agent for the vacuum environment in Exercise 2.8. Run the\nenvironment with this agent for all possible initial dirt configurations and agent locations.\nRecord the performance score for each configuration and the overall average score.\n2.10\nConsider a modified version of the vacuum environment in Exercise 2.8, in which the\nagent is penalized one point for each movement.\na. Can a simple reflex agent be perfectly rational for this environment? Explain.\nb. What about a reflex agent with state? Design such an agent.\nc. How do your answers to a and b change if the agent’s percepts give it the clean/dirty\nstatus of every square in the environment?\n2.11\nConsider a modified version of the vacuum environment in Exercise 2.8, in which the\ngeography of the environment—its extent, boundaries, and obstacles—is unknown, as is the\ninitial dirt configuration. (The agent can go Up and Down as well as Left and Right.)\na. Can a simple reflex agent be perfectly rational for this environment? Explain.\nb. Can a simple reflex agent with a randomized agent function outperform a simple reflex\nagent? Design such an agent and measure its performance on several environments.\nc. Can you design an environment in which your randomized agent will perform poorly?\nShow your results.\nd. Can a reflex agent with state outperform a simple reflex agent? Design such an agent\nand measure its performance on several environments. Can you design a rational agent\nof this type?\n2.12\nRepeat Exercise 2.11 for the case in which the location sensor is replaced with a\n“bump” sensor that detects the agent’s attempts to move into an obstacle or to cross the\nboundaries of the environment. Suppose the bump sensor stops working; how should the\nagent behave?\n2.13\nThe vacuum environments in the preceding exercises have all been deterministic. Dis-\ncuss possible agent programs for each of the following stochastic versions:\na. Murphy’s law: twenty-five percent of the time, the Suck action fails to clean the floor if\nit is dirty and deposits dirt onto the floor if the floor is clean. How is your agent program\naffected if the dirt sensor gives the wrong answer 10% of the time?\nb. Small children: At each time step, each clean square has a 10% chance of becoming\ndirty. Can you come up with a rational agent design for this case?\n" } ] }, "Part II": { "part": "Part II", "chapters": [ { "chapter": 3, "text": "3\nSOLVING PROBLEMS BY\nSEARCHING\nIn which we see how an agent can find a sequence of actions that achieves its\ngoals when no single action will do.\nThe simplest agents discussed in Chapter 2 were the reflex agents, which base their actions on\na direct mapping from states to actions. Such agents cannot operate well in environments for\nwhich this mapping would be too large to store and would take too long to learn. Goal-based\nagents, on the other hand, consider future actions and the desirability of their outcomes.\nThis chapter describes one kind of goal-based agent called a problem-solving agent.\nPROBLEM-SOLVING\nAGENT\nProblem-solving agents use atomic representations, as described in Section 2.4.7—that is,\nstates of the world are considered as wholes, with no internal structure visible to the problem-\nsolving algorithms. Goal-based agents that use more advanced factored or structured rep-\nresentations are usually called planning agents and are discussed in Chapters 7 and 10.\nOur discussion of problem solving begins with precise definitions of problems and their\nsolutions and give several examples to illustrate these definitions. We then describe several\ngeneral-purpose search algorithms that can be used to solve these problems. We will see\nseveral uninformed search algorithms—algorithms that are given no information about the\nproblem other than its definition. Although some of these algorithms can solve any solvable\nproblem, none of them can do so efficiently. Informed search algorithms, on the other hand,\ncan do quite well given some guidance on where to look for solutions.\nIn this chapter, we limit ourselves to the simplest kind of task environment, for which\nthe solution to a problem is always a fixed sequence of actions. The more general case—where\nthe agent’s future actions may vary depending on future percepts—is handled in Chapter 4.\nThis chapter uses the concepts of asymptotic complexity (that is, O() notation) and\nNP-completeness. Readers unfamiliar with these concepts should consult Appendix A.\n3.1\nPROBLEM-SOLVING AGENTS\nIntelligent agents are supposed to maximize their performance measure. As we mentioned\nin Chapter 2, achieving this is sometimes simplified if the agent can adopt a goal and aim at\nsatisfying it. Let us first look at why and how an agent might do this.\n64\nSection 3.1.\nProblem-Solving Agents\n65\nImagine an agent in the city of Arad, Romania, enjoying a touring holiday. The agent’s\nperformance measure contains many factors: it wants to improve its suntan, improve its Ro-\nmanian, take in the sights, enjoy the nightlife (such as it is), avoid hangovers, and so on. The\ndecision problem is a complex one involving many tradeoffs and careful reading of guide-\nbooks. Now, suppose the agent has a nonrefundable ticket to fly out of Bucharest the follow-\ning day. In that case, it makes sense for the agent to adopt the goal of getting to Bucharest.\nCourses of action that don’t reach Bucharest on time can be rejected without further consid-\neration and the agent’s decision problem is greatly simplified. Goals help organize behavior\nby limiting the objectives that the agent is trying to achieve and hence the actions it needs\nto consider. Goal formulation, based on the current situation and the agent’s performance\nGOAL FORMULATION\nmeasure, is the first step in problem solving.\nWe will consider a goal to be a set of world states—exactly those states in which the\ngoal is satisfied. The agent’s task is to find out how to act, now and in the future, so that it\nreaches a goal state. Before it can do this, it needs to decide (or we need to decide on its\nbehalf) what sorts of actions and states it should consider. If it were to consider actions at\nthe level of “move the left foot forward an inch” or “turn the steering wheel one degree left,”\nthe agent would probably never find its way out of the parking lot, let alone to Bucharest,\nbecause at that level of detail there is too much uncertainty in the world and there would be\ntoo many steps in a solution. Problem formulation is the process of deciding what actions\nPROBLEM\nFORMULATION\nand states to consider, given a goal. We discuss this process in more detail later. For now, let\nus assume that the agent will consider actions at the level of driving from one major town to\nanother. Each state therefore corresponds to being in a particular town.\nOur agent has now adopted the goal of driving to Bucharest and is considering where\nto go from Arad. Three roads lead out of Arad, one toward Sibiu, one to Timisoara, and one\nto Zerind. None of these achieves the goal, so unless the agent is familiar with the geography\nof Romania, it will not know which road to follow.1 In other words, the agent will not know\nwhich of its possible actions is best, because it does not yet know enough about the state\nthat results from taking each action. If the agent has no additional information—i.e., if the\nenvironment is unknown in the sense defined in Section 2.3—then it is has no choice but to\ntry one of the actions at random. This sad situation is discussed in Chapter 4.\nBut suppose the agent has a map of Romania. The point of a map is to provide the\nagent with information about the states it might get itself into and the actions it can take. The\nagent can use this information to consider subsequent stages of a hypothetical journey via\neach of the three towns, trying to find a journey that eventually gets to Bucharest. Once it has\nfound a path on the map from Arad to Bucharest, it can achieve its goal by carrying out the\ndriving actions that correspond to the legs of the journey. In general, an agent with several\nimmediate options of unknown value can decide what to do by first examining future actions\nthat eventually lead to states of known value.\nTo be more specific about what we mean by “examining future actions,” we have to\nbe more specific about properties of the environment, as defined in Section 2.3. For now,\n1 We are assuming that most readers are in the same position and can easily imagine themselves to be as clueless\nas our agent. We apologize to Romanian readers who are unable to take advantage of this pedagogical device.\n66\nChapter\n3.\nSolving Problems by Searching\nwe assume that the environment is observable, so the agent always knows the current state.\nFor the agent driving in Romania, it’s reasonable to suppose that each city on the map has a\nsign indicating its presence to arriving drivers. We also assume the environment is discrete,\nso at any given state there are only finitely many actions to choose from. This is true for\nnavigating in Romania because each city is connected to a small number of other cities. We\nwill assume the environment is known, so the agent knows which states are reached by each\naction. (Having an accurate map suffices to meet this condition for navigation problems.)\nFinally, we assume that the environment is deterministic, so each action has exactly one\noutcome. Under ideal conditions, this is true for the agent in Romania—it means that if it\nchooses to drive from Arad to Sibiu, it does end up in Sibiu. Of course, conditions are not\nalways ideal, as we show in Chapter 4.\nUnder these assumptions, the solution to any problem is a fixed sequence of actions.\n“Of course!” one might say, “What else could it be?” Well, in general it could be a branching\nstrategy that recommends different actions in the future depending on what percepts arrive.\nFor example, under less than ideal conditions, the agent might plan to drive from Arad to\nSibiu and then to Rimnicu Vilcea but may also need to have a contingency plan in case it\narrives by accident in Zerind instead of Sibiu. Fortunately, if the agent knows the initial state\nand the environment is known and deterministic, it knows exactly where it will be after the\nfirst action and what it will perceive. Since only one percept is possible after the first action,\nthe solution can specify only one possible second action, and so on.\nThe process of looking for a sequence of actions that reaches the goal is called search.\nSEARCH\nA search algorithm takes a problem as input and returns a solution in the form of an action\nSOLUTION\nsequence. Once a solution is found, the actions it recommends can be carried out. This\nis called the execution phase. Thus, we have a simple “formulate, search, execute” design\nEXECUTION\nfor the agent, as shown in Figure 3.1. After formulating a goal and a problem to solve,\nthe agent calls a search procedure to solve it. It then uses the solution to guide its actions,\ndoing whatever the solution recommends as the next thing to do—typically, the first action of\nthe sequence—and then removing that step from the sequence. Once the solution has been\nexecuted, the agent will formulate a new goal.\nNotice that while the agent is executing the solution sequence it ignores its percepts\nwhen choosing an action because it knows in advance what they will be. An agent that\ncarries out its plans with its eyes closed, so to speak, must be quite certain of what is going\non. Control theorists call this an open-loop system, because ignoring the percepts breaks the\nOPEN-LOOP\nloop between agent and environment.\nWe first describe the process of problem formulation, and then devote the bulk of the\nchapter to various algorithms for the SEARCH function. We do not discuss the workings of\nthe UPDATE-STATE and FORMULATE-GOAL functions further in this chapter.\n3.1.1\nWell-defined problems and solutions\nA problem can be defined formally by five components:\nPROBLEM\n• The initial state that the agent starts in. For example, the initial state for our agent in\nINITIAL STATE\nRomania might be described as In(Arad).\nSection 3.1.\nProblem-Solving Agents\n67\nfunction SIMPLE-PROBLEM-SOLVING-AGENT(percept) returns an action\npersistent: seq, an action sequence, initially empty\nstate, some description of the current world state\ngoal, a goal, initially null\nproblem, a problem formulation\nstate ←UPDATE-STATE(state,percept)\nif seq is empty then\ngoal ←FORMULATE-GOAL(state)\nproblem ←FORMULATE-PROBLEM(state,goal)\nseq ←SEARCH(problem)\nif seq = failure then return a null action\naction ←FIRST(seq)\nseq ←REST(seq)\nreturn action\nFigure 3.1\nA simple problem-solving agent. It first formulates a goal and a problem,\nsearches for a sequence of actions that would solve the problem, and then executes the actions\none at a time. When this is complete, it formulates another goal and starts over.\n• A description of the possible actions available to the agent. Given a particular state s,\nACTIONS\nACTIONS(s) returns the set of actions that can be executed in s. We say that each of\nthese actions is applicable in s. For example, from the state In(Arad), the applicable\nAPPLICABLE\nactions are {Go(Sibiu), Go(Timisoara), Go(Zerind)}.\n• A description of what each action does; the formal name for this is the transition\nmodel, specified by a function RESULT(s, a) that returns the state that results from\nTRANSITION MODEL\ndoing action a in state s. We also use the term successor to refer to any state reachable\nSUCCESSOR\nfrom a given state by a single action.2 For example, we have\nRESULT(In(Arad), Go(Zerind)) = In(Zerind) .\nTogether, the initial state, actions, and transition model implicitly define the state space\nSTATE SPACE\nof the problem—the set of all states reachable from the initial state by any sequence\nof actions. The state space forms a directed network or graph in which the nodes\nGRAPH\nare states and the links between nodes are actions. (The map of Romania shown in\nFigure 3.2 can be interpreted as a state-space graph if we view each road as standing\nfor two driving actions, one in each direction.) A path in the state space is a sequence\nPATH\nof states connected by a sequence of actions.\n• The goal test, which determines whether a given state is a goal state. Sometimes there\nGOAL TEST\nis an explicit set of possible goal states, and the test simply checks whether the given\nstate is one of them. The agent’s goal in Romania is the singleton set {In(Bucharest)}.\n2 Many treatments of problem solving, including previous editions of this book, use a successor function, which\nreturns the set of all successors, instead of separate ACTIONS and RESULT functions. The successor function\nmakes it difficult to describe an agent that knows what actions it can try but not what they achieve. Also, note\nsome author use RESULT(a, s) instead of RESULT(s, a), and some use DO instead of RESULT.\n68\nChapter\n3.\nSolving Problems by Searching\nGiurgiu\nUrziceni\nHirsova\nEforie\nNeamt\nOradea\nZerind\nArad\nTimisoara\nLugoj\nMehadia\nDrobeta\nCraiova\nSibiu\nFagaras\nPitesti\nVaslui\nIasi\nRimnicu Vilcea\nBucharest\n71\n75\n118\n111\n70\n75\n120\n151\n140\n99\n80\n97\n101\n211\n138\n146\n85\n90\n98\n142\n92\n87\n86\nFigure 3.2\nA simplified road map of part of Romania.\nSometimes the goal is specified by an abstract property rather than an explicitly enumer-\nated set of states. For example, in chess, the goal is to reach a state called “checkmate,”\nwhere the opponent’s king is under attack and can’t escape.\n• A path cost function that assigns a numeric cost to each path. The problem-solving\nPATH COST\nagent chooses a cost function that reflects its own performance measure. For the agent\ntrying to get to Bucharest, time is of the essence, so the cost of a path might be its length\nin kilometers. In this chapter, we assume that the cost of a path can be described as the\nsum of the costs of the individual actions along the path.3 The step cost of taking action\nSTEP COST\na in state s to reach state s′ is denoted by c(s, a, s′). The step costs for Romania are\nshown in Figure 3.2 as route distances. We assume that step costs are nonnegative.4\nThe preceding elements define a problem and can be gathered into a single data structure\nthat is given as input to a problem-solving algorithm. A solution to a problem is an action\nsequence that leads from the initial state to a goal state. Solution quality is measured by the\npath cost function, and an optimal solution has the lowest path cost among all solutions.\nOPTIMAL SOLUTION\n3.1.2\nFormulating problems\nIn the preceding section we proposed a formulation of the problem of getting to Bucharest in\nterms of the initial state, actions, transition model, goal test, and path cost. This formulation\nseems reasonable, but it is still a model—an abstract mathematical description—and not the\n3 This assumption is algorithmically convenient but also theoretically justifiable—see page 649 in Chapter 17.\n4 The implications of negative costs are explored in Exercise 3.8.\nSection 3.2.\nExample Problems\n69\nreal thing. Compare the simple state description we have chosen, In(Arad), to an actual cross-\ncountry trip, where the state of the world includes so many things: the traveling companions,\nthe current radio program, the scenery out of the window, the proximity of law enforcement\nofficers, the distance to the next rest stop, the condition of the road, the weather, and so on.\nAll these considerations are left out of our state descriptions because they are irrelevant to the\nproblem of finding a route to Bucharest. The process of removing detail from a representation\nis called abstraction.\nABSTRACTION\nIn addition to abstracting the state description, we must abstract the actions themselves.\nA driving action has many effects. Besides changing the location of the vehicle and its oc-\ncupants, it takes up time, consumes fuel, generates pollution, and changes the agent (as they\nsay, travel is broadening). Our formulation takes into account only the change in location.\nAlso, there are many actions that we omit altogether: turning on the radio, looking out of\nthe window, slowing down for law enforcement officers, and so on. And of course, we don’t\nspecify actions at the level of “turn steering wheel to the left by one degree.”\nCan we be more precise about defining the appropriate level of abstraction? Think of the\nabstract states and actions we have chosen as corresponding to large sets of detailed world\nstates and detailed action sequences. Now consider a solution to the abstract problem: for\nexample, the path from Arad to Sibiu to Rimnicu Vilcea to Pitesti to Bucharest. This abstract\nsolution corresponds to a large number of more detailed paths. For example, we could drive\nwith the radio on between Sibiu and Rimnicu Vilcea, and then switch it off for the rest of\nthe trip. The abstraction is valid if we can expand any abstract solution into a solution in the\nmore detailed world; a sufficient condition is that for every detailed state that is “in Arad,”\nthere is a detailed path to some state that is “in Sibiu,” and so on.5 The abstraction is useful\nif carrying out each of the actions in the solution is easier than the original problem; in this\ncase they are easy enough that they can be carried out without further search or planning by\nan average driving agent. The choice of a good abstraction thus involves removing as much\ndetail as possible while retaining validity and ensuring that the abstract actions are easy to\ncarry out. Were it not for the ability to construct useful abstractions, intelligent agents would\nbe completely swamped by the real world.\n3.2\nEXAMPLE PROBLEMS\nThe problem-solving approach has been applied to a vast array of task environments. We\nlist some of the best known here, distinguishing between toy and real-world problems. A\ntoy problem is intended to illustrate or exercise various problem-solving methods. It can be\nTOY PROBLEM\ngiven a concise, exact description and hence is usable by different researchers to compare the\nperformance of algorithms. A real-world problem is one whose solutions people actually\nREAL-WORLD\nPROBLEM\ncare about. Such problems tend not to have a single agreed-upon description, but we can give\nthe general flavor of their formulations.\n5 See Section 11.2 for a more complete set of definitions and algorithms.\n70\nChapter\n3.\nSolving Problems by Searching\nR\nL\nS\nS\nS\nS\nR\nL\nR\nL\nR\nL\nS\nS\nS\nS\nL\nL\nL\nL\nR\nR\nR\nR\nFigure 3.3\nThe state space for the vacuum world. Links denote actions: L = Left, R =\nRight, S = Suck.\n3.2.1\nToy problems\nThe first example we examine is the vacuum world first introduced in Chapter 2. (See\nFigure 2.2.) This can be formulated as a problem as follows:\n• States: The state is determined by both the agent location and the dirt locations. The\nagent is in one of two locations, each of which might or might not contain dirt. Thus,\nthere are 2 × 22 = 8 possible world states. A larger environment with n locations has\nn · 2n states.\n• Initial state: Any state can be designated as the initial state.\n• Actions: In this simple environment, each state has just three actions: Left, Right, and\nSuck. Larger environments might also include Up and Down.\n• Transition model: The actions have their expected effects, except that moving Left in\nthe leftmost square, moving Right in the rightmost square, and Sucking in a clean square\nhave no effect. The complete state space is shown in Figure 3.3.\n• Goal test: This checks whether all the squares are clean.\n• Path cost: Each step costs 1, so the path cost is the number of steps in the path.\nCompared with the real world, this toy problem has discrete locations, discrete dirt, reliable\ncleaning, and it never gets any dirtier. Chapter 4 relaxes some of these assumptions.\nThe 8-puzzle, an instance of which is shown in Figure 3.4, consists of a 3×3 board with\n8-PUZZLE\neight numbered tiles and a blank space. A tile adjacent to the blank space can slide into the\nspace. The object is to reach a specified goal state, such as the one shown on the right of the\nfigure. The standard formulation is as follows:\nSection 3.2.\nExample Problems\n71\n2\nStart State\nGoal State\n1\n3\n4\n6\n7\n5\n1\n2\n3\n4\n6\n7\n8\n5\n8\nFigure 3.4\nA typical instance of the 8-puzzle.\n• States: A state description specifies the location of each of the eight tiles and the blank\nin one of the nine squares.\n• Initial state: Any state can be designated as the initial state. Note that any given goal\ncan be reached from exactly half of the possible initial states (Exercise 3.4).\n• Actions: The simplest formulation defines the actions as movements of the blank space\nLeft, Right, Up, or Down. Different subsets of these are possible depending on where\nthe blank is.\n• Transition model: Given a state and action, this returns the resulting state; for example,\nif we apply Left to the start state in Figure 3.4, the resulting state has the 5 and the blank\nswitched.\n• Goal test: This checks whether the state matches the goal configuration shown in Fig-\nure 3.4. (Other goal configurations are possible.)\n• Path cost: Each step costs 1, so the path cost is the number of steps in the path.\nWhat abstractions have we included here? The actions are abstracted to their beginning and\nfinal states, ignoring the intermediate locations where the block is sliding. We have abstracted\naway actions such as shaking the board when pieces get stuck and ruled out extracting the\npieces with a knife and putting them back again. We are left with a description of the rules of\nthe puzzle, avoiding all the details of physical manipulations.\nThe 8-puzzle belongs to the family of sliding-block puzzles, which are often used as\nSLIDING-BLOCK\nPUZZLES\ntest problems for new search algorithms in AI. This family is known to be NP-complete,\nso one does not expect to find methods significantly better in the worst case than the search\nalgorithms described in this chapter and the next. The 8-puzzle has 9!/2 = 181, 440 reachable\nstates and is easily solved. The 15-puzzle (on a 4×4 board) has around 1.3 trillion states, and\nrandom instances can be solved optimally in a few milliseconds by the best search algorithms.\nThe 24-puzzle (on a 5 × 5 board) has around 1025 states, and random instances take several\nhours to solve optimally.\nThe goal of the 8-queens problem is to place eight queens on a chessboard such that\n8-QUEENS PROBLEM\nno queen attacks any other. (A queen attacks any piece in the same row, column or diago-\nnal.) Figure 3.5 shows an attempted solution that fails: the queen in the rightmost column is\nattacked by the queen at the top left.\n72\nChapter\n3.\nSolving Problems by Searching\nFigure 3.5\nAlmost a solution to the 8-queens problem. (Solution is left as an exercise.)\nAlthough efficient special-purpose algorithms exist for this problem and for the whole\nn-queens family, it remains a useful test problem for search algorithms. There are two main\nkinds of formulation. An incremental formulation involves operators that augment the state\nINCREMENTAL\nFORMULATION\ndescription, starting with an empty state; for the 8-queens problem, this means that each\naction adds a queen to the state. A complete-state formulation starts with all 8 queens on\nCOMPLETE-STATE\nFORMULATION\nthe board and moves them around. In either case, the path cost is of no interest because only\nthe final state counts. The first incremental formulation one might try is the following:\n• States: Any arrangement of 0 to 8 queens on the board is a state.\n• Initial state: No queens on the board.\n• Actions: Add a queen to any empty square.\n• Transition model: Returns the board with a queen added to the specified square.\n• Goal test: 8 queens are on the board, none attacked.\nIn this formulation, we have 64 · 63 · · · 57 ≈1.8 × 1014 possible sequences to investigate. A\nbetter formulation would prohibit placing a queen in any square that is already attacked:\n• States: All possible arrangements of n queens (0 ≤n ≤8), one per column in the\nleftmost n columns, with no queen attacking another.\n• Actions: Add a queen to any square in the leftmost empty column such that it is not\nattacked by any other queen.\nThis formulation reduces the 8-queens state space from 1.8 × 1014 to just 2,057, and solutions\nare easy to find. On the other hand, for 100 queens the reduction is from roughly 10400 states\nto about 1052 states (Exercise 3.5)—a big improvement, but not enough to make the problem\ntractable. Section 4.1 describes the complete-state formulation, and Chapter 6 gives a simple\nalgorithm that solves even the million-queens problem with ease.\nSection 3.2.\nExample Problems\n73\nOur final toy problem was devised by Donald Knuth (1964) and illustrates how infinite\nstate spaces can arise. Knuth conjectured that, starting with the number 4, a sequence of fac-\ntorial, square root, and floor operations will reach any desired positive integer. For example,\nwe can reach 5 from 4 as follows:\n\u0003\n\u0004\n\u0005\n\u0005\n\u0006\n\u0007\b\t\n(4!)!\n\u000b\n= 5 .\nThe problem definition is very simple:\n• States: Positive numbers.\n• Initial state: 4.\n• Actions: Apply factorial, square root, or floor operation (factorial for integers only).\n• Transition model: As given by the mathematical definitions of the operations.\n• Goal test: State is the desired positive integer.\nTo our knowledge there is no bound on how large a number might be constructed in the pro-\ncess of reaching a given target—for example, the number 620,448,401,733,239,439,360,000\nis generated in the expression for 5—so the state space for this problem is infinite. Such\nstate spaces arise frequently in tasks involving the generation of mathematical expressions,\ncircuits, proofs, programs, and other recursively defined objects.\n3.2.2\nReal-world problems\nWe have already seen how the route-finding problem is defined in terms of specified loca-\nROUTE-FINDING\nPROBLEM\ntions and transitions along links between them. Route-finding algorithms are used in a variety\nof applications. Some, such as Web sites and in-car systems that provide driving directions,\nare relatively straightforward extensions of the Romania example. Others, such as routing\nvideo streams in computer networks, military operations planning, and airline travel-planning\nsystems, involve much more complex specifications. Consider the airline travel problems that\nmust be solved by a travel-planning Web site:\n• States: Each state obviously includes a location (e.g., an airport) and the current time.\nFurthermore, because the cost of an action (a flight segment) may depend on previous\nsegments, their fare bases, and their status as domestic or international, the state must\nrecord extra information about these “historical” aspects.\n• Initial state: This is specified by the user’s query.\n• Actions: Take any flight from the current location, in any seat class, leaving after the\ncurrent time, leaving enough time for within-airport transfer if needed.\n• Transition model: The state resulting from taking a flight will have the flight’s desti-\nnation as the current location and the flight’s arrival time as the current time.\n• Goal test: Are we at the final destination specified by the user?\n• Path cost: This depends on monetary cost, waiting time, flight time, customs and im-\nmigration procedures, seat quality, time of day, type of airplane, frequent-flyer mileage\nawards, and so on.\n74\nChapter\n3.\nSolving Problems by Searching\nCommercial travel advice systems use a problem formulation of this kind, with many addi-\ntional complications to handle the byzantine fare structures that airlines impose. Any sea-\nsoned traveler knows, however, that not all air travel goes according to plan. A really good\nsystem should include contingency plans—such as backup reservations on alternate flights—\nto the extent that these are justified by the cost and likelihood of failure of the original plan.\nTouring problems are closely related to route-finding problems, but with an impor-\nTOURING PROBLEM\ntant difference. Consider, for example, the problem “Visit every city in Figure 3.2 at least\nonce, starting and ending in Bucharest.”\nAs with route finding, the actions correspond\nto trips between adjacent cities. The state space, however, is quite different. Each state\nmust include not just the current location but also the set of cities the agent has visited.\nSo the initial state would be In(Bucharest), Visited({Bucharest}), a typical intermedi-\nate state would be In(Vaslui), Visited({Bucharest, Urziceni, Vaslui}), and the goal test\nwould check whether the agent is in Bucharest and all 20 cities have been visited.\nThe traveling salesperson problem (TSP) is a touring problem in which each city\nTRAVELING\nSALESPERSON\nPROBLEM\nmust be visited exactly once. The aim is to find the shortest tour. The problem is known to\nbe NP-hard, but an enormous amount of effort has been expended to improve the capabilities\nof TSP algorithms. In addition to planning trips for traveling salespersons, these algorithms\nhave been used for tasks such as planning movements of automatic circuit-board drills and of\nstocking machines on shop floors.\nA VLSI layout problem requires positioning millions of components and connections\nVLSI LAYOUT\non a chip to minimize area, minimize circuit delays, minimize stray capacitances, and max-\nimize manufacturing yield. The layout problem comes after the logical design phase and is\nusually split into two parts: cell layout and channel routing. In cell layout, the primitive\ncomponents of the circuit are grouped into cells, each of which performs some recognized\nfunction. Each cell has a fixed footprint (size and shape) and requires a certain number of\nconnections to each of the other cells. The aim is to place the cells on the chip so that they do\nnot overlap and so that there is room for the connecting wires to be placed between the cells.\nChannel routing finds a specific route for each wire through the gaps between the cells. These\nsearch problems are extremely complex, but definitely worth solving. Later in this chapter,\nwe present some algorithms capable of solving them.\nRobot navigation is a generalization of the route-finding problem described earlier.\nROBOT NAVIGATION\nRather than following a discrete set of routes, a robot can move in a continuous space with\n(in principle) an infinite set of possible actions and states. For a circular robot moving on a\nflat surface, the space is essentially two-dimensional. When the robot has arms and legs or\nwheels that must also be controlled, the search space becomes many-dimensional. Advanced\ntechniques are required just to make the search space finite. We examine some of these\nmethods in Chapter 25. In addition to the complexity of the problem, real robots must also\ndeal with errors in their sensor readings and motor controls.\nAutomatic assembly sequencing of complex objects by a robot was first demonstrated\nAUTOMATIC\nASSEMBLY\nSEQUENCING\nby FREDDY (Michie, 1972). Progress since then has been slow but sure, to the point where\nthe assembly of intricate objects such as electric motors is economically feasible. In assembly\nproblems, the aim is to find an order in which to assemble the parts of some object. If the\nwrong order is chosen, there will be no way to add some part later in the sequence without\nSection 3.3.\nSearching for Solutions\n75\nundoing some of the work already done. Checking a step in the sequence for feasibility is a\ndifficult geometrical search problem closely related to robot navigation. Thus, the generation\nof legal actions is the expensive part of assembly sequencing. Any practical algorithm must\navoid exploring all but a tiny fraction of the state space. Another important assembly problem\nis protein design, in which the goal is to find a sequence of amino acids that will fold into a\nPROTEIN DESIGN\nthree-dimensional protein with the right properties to cure some disease.\n3.3\nSEARCHING FOR SOLUTIONS\nHaving formulated some problems, we now need to solve them. A solution is an action\nsequence, so search algorithms work by considering various possible action sequences. The\npossible action sequences starting at the initial state form a search tree with the initial state\nSEARCH TREE\nat the root; the branches are actions and the nodes correspond to states in the state space of\nNODE\nthe problem. Figure 3.6 shows the first few steps in growing the search tree for finding a route\nfrom Arad to Bucharest. The root node of the tree corresponds to the initial state, In(Arad).\nThe first step is to test whether this is a goal state. (Clearly it is not, but it is important to\ncheck so that we can solve trick problems like “starting in Arad, get to Arad.”) Then we\nneed to consider taking various actions. We do this by expanding the current state; that is,\nEXPANDING\napplying each legal action to the current state, thereby generating a new set of states. In\nGENERATING\nthis case, we add three branches from the parent node In(Arad) leading to three new child\nPARENT NODE\nnodes: In(Sibiu), In(Timisoara), and In(Zerind). Now we must choose which of these three\nCHILD NODE\npossibilities to consider further.\nThis is the essence of search—following up one option now and putting the others aside\nfor later, in case the first choice does not lead to a solution. Suppose we choose Sibiu first.\nWe check to see whether it is a goal state (it is not) and then expand it to get In(Arad),\nIn(Fagaras), In(Oradea), and In(RimnicuVilcea). We can then choose any of these four or go\nback and choose Timisoara or Zerind. Each of these six nodes is a leaf node, that is, a node\nLEAF NODE\nwith no children in the tree. The set of all leaf nodes available for expansion at any given\npoint is called the frontier. (Many authors call it the open list, which is both geographically\nFRONTIER\nOPEN LIST\nless evocative and less accurate, because other data structures are better suited than a list.) In\nFigure 3.6, the frontier of each tree consists of those nodes with bold outlines.\nThe process of expanding nodes on the frontier continues until either a solution is found\nor there are no more states to expand. The general TREE-SEARCH algorithm is shown infor-\nmally in Figure 3.7. Search algorithms all share this basic structure; they vary primarily\naccording to how they choose which state to expand next—the so-called search strategy.\nSEARCH STRATEGY\nThe eagle-eyed reader will notice one peculiar thing about the search tree shown in Fig-\nure 3.6: it includes the path from Arad to Sibiu and back to Arad again! We say that In(Arad)\nis a repeated state in the search tree, generated in this case by a loopy path. Considering\nREPEATED STATE\nLOOPY PATH\nsuch loopy paths means that the complete search tree for Romania is infinite because there\nis no limit to how often one can traverse a loop. On the other hand, the state space—the\nmap shown in Figure 3.2—has only 20 states. As we discuss in Section 3.4, loops can cause\n76\nChapter\n3.\nSolving Problems by Searching\ncertain algorithms to fail, making otherwise solvable problems unsolvable. Fortunately, there\nis no need to consider loopy paths. We can rely on more than intuition for this: because path\ncosts are additive and step costs are nonnegative, a loopy path to any given state is never\nbetter than the same path with the loop removed.\nLoopy paths are a special case of the more general concept of redundant paths, which\nREDUNDANT PATH\nexist whenever there is more than one way to get from one state to another. Consider the paths\nArad–Sibiu (140 km long) and Arad–Zerind–Oradea–Sibiu (297 km long). Obviously, the\nsecond path is redundant—it’s just a worse way to get to the same state. If you are concerned\nabout reaching the goal, there’s never any reason to keep more than one path to any given\nstate, because any goal state that is reachable by extending one path is also reachable by\nextending the other.\nIn some cases, it is possible to define the problem itself so as to eliminate redundant\npaths. For example, if we formulate the 8-queens problem (page 71) so that a queen can be\nplaced in any column, then each state with n queens can be reached by n! different paths; but\nif we reformulate the problem so that each new queen is placed in the leftmost empty column,\nthen each state can be reached only through one path.\n(a) The initial state\n(b) After expanding Arad\n(c) After expanding Sibiu\nRimnicu Vilcea\nLugoj\nArad\nFagaras\nOradea\nArad\nArad\nOradea\nRimnicu Vilcea\nLugoj\nZerind\nSibiu\nArad\nFagaras\nOradea\nTimisoara\nArad\nArad\nOradea\nLugoj\nArad\nArad\nOradea\nZerind\nArad\nSibiu\nTimisoara\nArad\nRimnicu Vilcea\nZerind\nArad\nSibiu\nArad\nFagaras\nOradea\nTimisoara\nFigure 3.6\nPartial search trees for finding a route from Arad to Bucharest. Nodes that\nhave been expanded are shaded; nodes that have been generated but not yet expanded are\noutlined in bold; nodes that have not yet been generated are shown in faint dashed lines.\nSection 3.3.\nSearching for Solutions\n77\nfunction TREE-SEARCH(problem) returns a solution, or failure\ninitialize the frontier using the initial state of problem\nloop do\nif the frontier is empty then return failure\nchoose a leaf node and remove it from the frontier\nif the node contains a goal state then return the corresponding solution\nexpand the chosen node, adding the resulting nodes to the frontier\nfunction GRAPH-SEARCH(problem) returns a solution, or failure\ninitialize the frontier using the initial state of problem\ninitialize the explored set to be empty\nloop do\nif the frontier is empty then return failure\nchoose a leaf node and remove it from the frontier\nif the node contains a goal state then return the corresponding solution\nadd the node to the explored set\nexpand the chosen node, adding the resulting nodes to the frontier\nonly if not in the frontier or explored set\nFigure 3.7\nAn informal description of the general tree-search and graph-search algo-\nrithms. The parts of GRAPH-SEARCH marked in bold italic are the additions needed to\nhandle repeated states.\nIn other cases, redundant paths are unavoidable.\nThis includes all problems where\nthe actions are reversible, such as route-finding problems and sliding-block puzzles. Route-\nfinding on a rectangular grid (like the one used later for Figure 3.9) is a particularly impor-\nRECTANGULAR GRID\ntant example in computer games. In such a grid, each state has four successors, so a search\ntree of depth d that includes repeated states has 4d leaves; but there are only about 2d2 distinct\nstates within d steps of any given state. For d = 20, this means about a trillion nodes but only\nabout 800 distinct states. Thus, following redundant paths can cause a tractable problem to\nbecome intractable. This is true even for algorithms that know how to avoid infinite loops.\nAs the saying goes, algorithms that forget their history are doomed to repeat it.\nThe\nway to avoid exploring redundant paths is to remember where one has been. To do this, we\naugment the TREE-SEARCH algorithm with a data structure called the explored set (also\nEXPLORED SET\nknown as the closed list), which remembers every expanded node. Newly generated nodes\nCLOSED LIST\nthat match previously generated nodes—ones in the explored set or the frontier—can be dis-\ncarded instead of being added to the frontier. The new algorithm, called GRAPH-SEARCH, is\nshown informally in Figure 3.7. The specific algorithms in this chapter draw on this general\ndesign.\nClearly, the search tree constructed by the GRAPH-SEARCH algorithm contains at most\none copy of each state, so we can think of it as growing a tree directly on the state-space graph,\nas shown in Figure 3.8. The algorithm has another nice property: the frontier separates the\nSEPARATOR\nstate-space graph into the explored region and the unexplored region, so that every path from\n78\nChapter\n3.\nSolving Problems by Searching\nFigure 3.8\nA sequence of search trees generated by a graph search on the Romania prob-\nlem of Figure 3.2. At each stage, we have extended each path by one step. Notice that at the\nthird stage, the northernmost city (Oradea) has become a dead end: both of its successors are\nalready explored via other paths.\n(c)\n(b)\n(a)\nFigure 3.9\nThe separation property of GRAPH-SEARCH, illustrated on a rectangular-grid\nproblem. The frontier (white nodes) always separates the explored region of the state space\n(black nodes) from the unexplored region (gray nodes). In (a), just the root has been ex-\npanded. In (b), one leaf node has been expanded. In (c), the remaining successors of the root\nhave been expanded in clockwise order.\nthe initial state to an unexplored state has to pass through a state in the frontier. (If this\nseems completely obvious, try Exercise 3.13 now.) This property is illustrated in Figure 3.9.\nAs every step moves a state from the frontier into the explored region while moving some\nstates from the unexplored region into the frontier, we see that the algorithm is systematically\nexamining the states in the state space, one by one, until it finds a solution.\n3.3.1\nInfrastructure for search algorithms\nSearch algorithms require a data structure to keep track of the search tree that is being con-\nstructed. For each node n of the tree, we have a structure that contains four components:\n• n.STATE: the state in the state space to which the node corresponds;\n• n.PARENT: the node in the search tree that generated this node;\n• n.ACTION: the action that was applied to the parent to generate the node;\n• n.PATH-COST: the cost, traditionally denoted by g(n), of the path from the initial state\nto the node, as indicated by the parent pointers.\nSection 3.3.\nSearching for Solutions\n79\n1\n2\n3\n4\n5\n6\n7\n8\n1\n2\n3\n4\n5\n6\n7\n8\nNode\nSTATE\nPARENT\nACTION = Right\nPATH-COST = 6\nFigure 3.10\nNodes are the data structures from which the search tree is constructed. Each\nhas a parent, a state, and various bookkeeping fields. Arrows point from child to parent.\nGiven the components for a parent node, it is easy to see how to compute the necessary\ncomponents for a child node. The function CHILD-NODE takes a parent node and an action\nand returns the resulting child node:\nfunction CHILD-NODE(problem,parent,action) returns a node\nreturn a node with\nSTATE = problem.RESULT(parent.STATE,action),\nPARENT = parent, ACTION = action,\nPATH-COST = parent.PATH-COST + problem.STEP-COST(parent.STATE,action)\nThe node data structure is depicted in Figure 3.10. Notice how the PARENT pointers\nstring the nodes together into a tree structure. These pointers also allow the solution path to be\nextracted when a goal node is found; we use the SOLUTION function to return the sequence\nof actions obtained by following parent pointers back to the root.\nUp to now, we have not been very careful to distinguish between nodes and states, but in\nwriting detailed algorithms it’s important to make that distinction. A node is a bookkeeping\ndata structure used to represent the search tree. A state corresponds to a configuration of the\nworld. Thus, nodes are on particular paths, as defined by PARENT pointers, whereas states\nare not. Furthermore, two different nodes can contain the same world state if that state is\ngenerated via two different search paths.\nNow that we have nodes, we need somewhere to put them. The frontier needs to be\nstored in such a way that the search algorithm can easily choose the next node to expand\naccording to its preferred strategy. The appropriate data structure for this is a queue. The\nQUEUE\noperations on a queue are as follows:\n• EMPTY?(queue) returns true only if there are no more elements in the queue.\n• POP(queue) removes the first element of the queue and returns it.\n• INSERT(element, queue) inserts an element and returns the resulting queue.\n80\nChapter\n3.\nSolving Problems by Searching\nQueues are characterized by the order in which they store the inserted nodes. Three common\nvariants are the first-in, first-out or FIFO queue, which pops the oldest element of the queue;\nFIFO QUEUE\nthe last-in, first-out or LIFO queue (also known as a stack), which pops the newest element\nLIFO QUEUE\nof the queue; and the priority queue, which pops the element of the queue with the highest\nPRIORITY QUEUE\npriority according to some ordering function.\nThe explored set can be implemented with a hash table to allow efficient checking for\nrepeated states. With a good implementation, insertion and lookup can be done in roughly\nconstant time no matter how many states are stored. One must take care to implement the\nhash table with the right notion of equality between states. For example, in the traveling\nsalesperson problem (page 74), the hash table needs to know that the set of visited cities\n{Bucharest,Urziceni,Vaslui} is the same as {Urziceni,Vaslui,Bucharest}. Sometimes this can\nbe achieved most easily by insisting that the data structures for states be in some canonical\nform; that is, logically equivalent states should map to the same data structure. In the case\nCANONICAL FORM\nof states described by sets, for example, a bit-vector representation or a sorted list without\nrepetition would be canonical, whereas an unsorted list would not.\n3.3.2\nMeasuring problem-solving performance\nBefore we get into the design of specific search algorithms, we need to consider the criteria\nthat might be used to choose among them. We can evaluate an algorithm’s performance in\nfour ways:\n• Completeness: Is the algorithm guaranteed to find a solution when there is one?\nCOMPLETENESS\n• Optimality: Does the strategy find the optimal solution, as defined on page 68?\nOPTIMALITY\n• Time complexity: How long does it take to find a solution?\nTIME COMPLEXITY\n• Space complexity: How much memory is needed to perform the search?\nSPACE COMPLEXITY\nTime and space complexity are always considered with respect to some measure of the prob-\nlem difficulty. In theoretical computer science, the typical measure is the size of the state\nspace graph, |V | + |E|, where V is the set of vertices (nodes) of the graph and E is the set\nof edges (links). This is appropriate when the graph is an explicit data structure that is input\nto the search program. (The map of Romania is an example of this.) In AI, the graph is often\nrepresented implicitly by the initial state, actions, and transition model and is frequently infi-\nnite. For these reasons, complexity is expressed in terms of three quantities: b, the branching\nfactor or maximum number of successors of any node; d, the depth of the shallowest goal\nBRANCHING FACTOR\nDEPTH\nnode (i.e., the number of steps along the path from the root); and m, the maximum length of\nany path in the state space. Time is often measured in terms of the number of nodes generated\nduring the search, and space in terms of the maximum number of nodes stored in memory.\nFor the most part, we describe time and space complexity for search on a tree; for a graph,\nthe answer depends on how “redundant” the paths in the state space are.\nTo assess the effectiveness of a search algorithm, we can consider just the search cost—\nSEARCH COST\nwhich typically depends on the time complexity but can also include a term for memory\nusage—or we can use the total cost, which combines the search cost and the path cost of the\nTOTAL COST\nsolution found. For the problem of finding a route from Arad to Bucharest, the search cost\nis the amount of time taken by the search and the solution cost is the total length of the path\nSection 3.4.\nUninformed Search Strategies\n81\nin kilometers. Thus, to compute the total cost, we have to add milliseconds and kilometers.\nThere is no “official exchange rate” between the two, but it might be reasonable in this case to\nconvert kilometers into milliseconds by using an estimate of the car’s average speed (because\ntime is what the agent cares about). This enables the agent to find an optimal tradeoff point\nat which further computation to find a shorter path becomes counterproductive. The more\ngeneral problem of tradeoffs between different goods is taken up in Chapter 16.\n3.4\nUNINFORMED SEARCH STRATEGIES\nThis section covers several search strategies that come under the heading of uninformed\nsearch (also called blind search). The term means that the strategies have no additional\nUNINFORMED\nSEARCH\nBLIND SEARCH\ninformation about states beyond that provided in the problem definition. All they can do is\ngenerate successors and distinguish a goal state from a non-goal state. All search strategies\nare distinguished by the order in which nodes are expanded. Strategies that know whether\none non-goal state is “more promising” than another are called informed search or heuristic\nINFORMED SEARCH\nsearch strategies; they are covered in Section 3.5.\nHEURISTIC SEARCH\n3.4.1\nBreadth-first search\nBreadth-first search is a simple strategy in which the root node is expanded first, then all the\nBREADTH-FIRST\nSEARCH\nsuccessors of the root node are expanded next, then their successors, and so on. In general,\nall the nodes are expanded at a given depth in the search tree before any nodes at the next\nlevel are expanded.\nBreadth-first search is an instance of the general graph-search algorithm (Figure 3.7) in\nwhich the shallowest unexpanded node is chosen for expansion. This is achieved very simply\nby using a FIFO queue for the frontier. Thus, new nodes (which are always deeper than their\nparents) go to the back of the queue, and old nodes, which are shallower than the new nodes,\nget expanded first. There is one slight tweak on the general graph-search algorithm, which is\nthat the goal test is applied to each node when it is generated rather than when it is selected for\nexpansion. This decision is explained below, where we discuss time complexity. Note also\nthat the algorithm, following the general template for graph search, discards any new path to\na state already in the frontier or explored set; it is easy to see that any such path must be at\nleast as deep as the one already found. Thus, breadth-first search always has the shallowest\npath to every node on the frontier.\nPseudocode is given in Figure 3.11. Figure 3.12 shows the progress of the search on a\nsimple binary tree.\nHow does breadth-first search rate according to the four criteria from the previous sec-\ntion? We can easily see that it is complete—if the shallowest goal node is at some finite depth\nd, breadth-first search will eventually find it after generating all shallower nodes (provided\nthe branching factor b is finite). Note that as soon as a goal node is generated, we know it\nis the shallowest goal node because all shallower nodes must have been generated already\nand failed the goal test. Now, the shallowest goal node is not necessarily the optimal one;\n82\nChapter\n3.\nSolving Problems by Searching\nfunction BREADTH-FIRST-SEARCH(problem) returns a solution, or failure\nnode ←a node with STATE = problem.INITIAL-STATE, PATH-COST = 0\nif problem.GOAL-TEST(node.STATE) then return SOLUTION(node)\nfrontier ←a FIFO queue with node as the only element\nexplored ←an empty set\nloop do\nif EMPTY?(frontier) then return failure\nnode ←POP(frontier) /* chooses the shallowest node in frontier */\nadd node.STATE to explored\nfor each action in problem.ACTIONS(node.STATE) do\nchild ←CHILD-NODE(problem,node,action)\nif child.STATE is not in explored or frontier then\nif problem.GOAL-TEST(child.STATE) then return SOLUTION(child)\nfrontier ←INSERT(child,frontier)\nFigure 3.11\nBreadth-first search on a graph.\ntechnically, breadth-first search is optimal if the path cost is a nondecreasing function of the\ndepth of the node. The most common such scenario is that all actions have the same cost.\nSo far, the news about breadth-first search has been good. The news about time and\nspace is not so good. Imagine searching a uniform tree where every state has b successors.\nThe root of the search tree generates b nodes at the first level, each of which generates b more\nnodes, for a total of b2 at the second level. Each of these generates b more nodes, yielding b3\nnodes at the third level, and so on. Now suppose that the solution is at depth d. In the worst\ncase, it is the last node generated at that level. Then the total number of nodes generated is\nb + b2 + b3 + · · · + bd = O(bd) .\n(If the algorithm were to apply the goal test to nodes when selected for expansion, rather than\nwhen generated, the whole layer of nodes at depth d would be expanded before the goal was\ndetected and the time complexity would be O(bd+1).)\nAs for space complexity: for any kind of graph search, which stores every expanded\nnode in the explored set, the space complexity is always within a factor of b of the time\ncomplexity. For breadth-first graph search in particular, every node generated remains in\nmemory. There will be O(bd−1) nodes in the explored set and O(bd) nodes in the frontier,\nA\nB\nC\nE\nF\nG\nD\nA\nB\nD\nE\nF\nG\nC\nA\nC\nD\nE\nF\nG\nB\nB\nC\nD\nE\nF\nG\nA\nFigure 3.12\nBreadth-first search on a simple binary tree. At each stage, the node to be\nexpanded next is indicated by a marker.\nSection 3.4.\nUninformed Search Strategies\n83\nso the space complexity is O(bd), i.e., it is dominated by the size of the frontier. Switching\nto a tree search would not save much space, and in a state space with many redundant paths,\nswitching could cost a great deal of time.\nAn exponential complexity bound such as O(bd) is scary. Figure 3.13 shows why. It\nlists, for various values of the solution depth d, the time and memory required for a breadth-\nfirst search with branching factor b = 10. The table assumes that 1 million nodes can be\ngenerated per second and that a node requires 1000 bytes of storage. Many search problems\nfit roughly within these assumptions (give or take a factor of 100) when run on a modern\npersonal computer.\nDepth\nNodes\nTime\nMemory\n2\n110\n.11 milliseconds\n107 kilobytes\n4\n11,110\n11 milliseconds\n10.6 megabytes\n6\n106\n1.1 seconds\n1 gigabyte\n8\n108\n2 minutes\n103 gigabytes\n10\n1010\n3 hours\n10 terabytes\n12\n1012\n13 days\n1 petabyte\n14\n1014\n3.5 years\n99 petabytes\n16\n1016\n350 years\n10 exabytes\nFigure 3.13\nTime and memory requirements for breadth-first search. The numbers shown\nassume branching factor b = 10; 1 million nodes/second; 1000 bytes/node.\nTwo lessons can be learned from Figure 3.13. First, the memory requirements are a\nbigger problem for breadth-first search than is the execution time. One might wait 13 days\nfor the solution to an important problem with search depth 12, but no personal computer has\nthe petabyte of memory it would take. Fortunately, other strategies require less memory.\nThe second lesson is that time is still a major factor. If your problem has a solution at\ndepth 16, then (given our assumptions) it will take about 350 years for breadth-first search (or\nindeed any uninformed search) to find it. In general, exponential-complexity search problems\ncannot be solved by uninformed methods for any but the smallest instances.\n3.4.2\nUniform-cost search\nWhen all step costs are equal, breadth-first search is optimal because it always expands the\nshallowest unexpanded node. By a simple extension, we can find an algorithm that is optimal\nwith any step-cost function. Instead of expanding the shallowest node, uniform-cost search\nUNIFORM-COST\nSEARCH\nexpands the node n with the lowest path cost g(n). This is done by storing the frontier as a\npriority queue ordered by g. The algorithm is shown in Figure 3.14.\nIn addition to the ordering of the queue by path cost, there are two other significant\ndifferences from breadth-first search. The first is that the goal test is applied to a node when\nit is selected for expansion (as in the generic graph-search algorithm shown in Figure 3.7)\nrather than when it is first generated. The reason is that the first goal node that is generated\n84\nChapter\n3.\nSolving Problems by Searching\nfunction UNIFORM-COST-SEARCH(problem) returns a solution, or failure\nnode ←a node with STATE = problem.INITIAL-STATE, PATH-COST = 0\nfrontier ←a priority queue ordered by PATH-COST, with node as the only element\nexplored ←an empty set\nloop do\nif EMPTY?(frontier) then return failure\nnode ←POP(frontier) /* chooses the lowest-cost node in frontier */\nif problem.GOAL-TEST(node.STATE) then return SOLUTION(node)\nadd node.STATE to explored\nfor each action in problem.ACTIONS(node.STATE) do\nchild ←CHILD-NODE(problem,node,action)\nif child.STATE is not in explored or frontier then\nfrontier ←INSERT(child,frontier)\nelse if child.STATE is in frontier with higher PATH-COST then\nreplace that frontier node with child\nFigure 3.14\nUniform-cost search on a graph. The algorithm is identical to the general\ngraph search algorithm in Figure 3.7, except for the use of a priority queue and the addition\nof an extra check in case a shorter path to a frontier state is discovered. The data structure for\nfrontier needs to support efficient membership testing, so it should combine the capabilities\nof a priority queue and a hash table.\nSibiu\nFagaras\nPitesti\nRimnicu Vilcea\nBucharest\n99\n80\n97\n101\n211\nFigure 3.15\nPart of the Romania state space, selected to illustrate uniform-cost search.\nmay be on a suboptimal path. The second difference is that a test is added in case a better\npath is found to a node currently on the frontier.\nBoth of these modifications come into play in the example shown in Figure 3.15, where\nthe problem is to get from Sibiu to Bucharest. The successors of Sibiu are Rimnicu Vilcea and\nFagaras, with costs 80 and 99, respectively. The least-cost node, Rimnicu Vilcea, is expanded\nnext, adding Pitesti with cost 80 + 97 = 177. The least-cost node is now Fagaras, so it is\nexpanded, adding Bucharest with cost 99 + 211 = 310. Now a goal node has been generated,\nbut uniform-cost search keeps going, choosing Pitesti for expansion and adding a second path\nSection 3.4.\nUninformed Search Strategies\n85\nto Bucharest with cost 80+97+101 = 278. Now the algorithm checks to see if this new path\nis better than the old one; it is, so the old one is discarded. Bucharest, now with g-cost 278,\nis selected for expansion and the solution is returned.\nIt is easy to see that uniform-cost search is optimal in general. First, we observe that\nwhenever uniform-cost search selects a node n for expansion, the optimal path to that node\nhas been found. (Were this not the case, there would have to be another frontier node n′ on\nthe optimal path from the start node to n, by the graph separation property of Figure 3.9;\nby definition, n′ would have lower g-cost than n and would have been selected first.) Then,\nbecause step costs are nonnegative, paths never get shorter as nodes are added. These two\nfacts together imply that uniform-cost search expands nodes in order of their optimal path\ncost. Hence, the first goal node selected for expansion must be the optimal solution.\nUniform-cost search does not care about the number of steps a path has, but only about\ntheir total cost. Therefore, it will get stuck in an infinite loop if there is a path with an infinite\nsequence of zero-cost actions—for example, a sequence of NoOp actions.6 Completeness is\nguaranteed provided the cost of every step exceeds some small positive constant ϵ.\nUniform-cost search is guided by path costs rather than depths, so its complexity is not\neasily characterized in terms of b and d. Instead, let C∗be the cost of the optimal solution,7\nand assume that every action costs at least ϵ. Then the algorithm’s worst-case time and space\ncomplexity is O(b1+⌊C∗/ϵ⌋), which can be much greater than bd. This is because uniform-\ncost search can explore large trees of small steps before exploring paths involving large and\nperhaps useful steps. When all step costs are equal, b1+⌊C∗/ϵ⌋is just bd+1. When all step\ncosts are the same, uniform-cost search is similar to breadth-first search, except that the latter\nstops as soon as it generates a goal, whereas uniform-cost search examines all the nodes at\nthe goal’s depth to see if one has a lower cost; thus uniform-cost search does strictly more\nwork by expanding nodes at depth d unnecessarily.\n3.4.3\nDepth-first search\nDepth-first search always expands the deepest node in the current frontier of the search tree.\nDEPTH-FIRST\nSEARCH\nThe progress of the search is illustrated in Figure 3.16. The search proceeds immediately\nto the deepest level of the search tree, where the nodes have no successors. As those nodes\nare expanded, they are dropped from the frontier, so then the search “backs up” to the next\ndeepest node that still has unexplored successors.\nThe depth-first search algorithm is an instance of the graph-search algorithm in Fig-\nure 3.7; whereas breadth-first-search uses a FIFO queue, depth-first search uses a LIFO queue.\nA LIFO queue means that the most recently generated node is chosen for expansion. This\nmust be the deepest unexpanded node because it is one deeper than its parent—which, in turn,\nwas the deepest unexpanded node when it was selected.\nAs an alternative to the GRAPH-SEARCH-style implementation, it is common to im-\nplement depth-first search with a recursive function that calls itself on each of its children in\nturn. (A recursive depth-first algorithm incorporating a depth limit is shown in Figure 3.17.)\n6 NoOp, or “no operation,” is the name of an assembly language instruction that does nothing.\n7 Here, and throughout the book, the “star” in C∗means an optimal value for C.\n86\nChapter\n3.\nSolving Problems by Searching\nA\nC\nF\nG\nM\nN\nO\nA\nC\nF\nG\nL\nM\nN\nO\nA\nC\nF\nG\nL\nM\nN\nO\nC\nF\nG\nL\nM\nN\nO\nA\nB\nC\nE\nF\nG\nK\nL\nM\nN\nO\nA\nC\nE\nF\nG\nJ\nK\nL\nM\nN\nO\nA\nC\nE\nF\nG\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nFigure 3.16\nDepth-first search on a binary tree. The unexplored region is shown in light\ngray. Explored nodes with no descendants in the frontier are removed from memory. Nodes\nat depth 3 have no successors and M is the only goal node.\nThe properties of depth-first search depend strongly on whether the graph-search or\ntree-search version is used. The graph-search version, which avoids repeated states and re-\ndundant paths, is complete in finite state spaces because it will eventually expand every node.\nThe tree-search version, on the other hand, is not complete—for example, in Figure 3.6 the\nalgorithm will follow the Arad–Sibiu–Arad–Sibiu loop forever. Depth-first tree search can be\nmodified at no extra memory cost so that it checks new states against those on the path from\nthe root to the current node; this avoids infinite loops in finite state spaces but does not avoid\nthe proliferation of redundant paths. In infinite state spaces, both versions fail if an infinite\nnon-goal path is encountered. For example, in Knuth’s 4 problem, depth-first search would\nkeep applying the factorial operator forever.\nFor similar reasons, both versions are nonoptimal. For example, in Figure 3.16, depth-\nfirst search will explore the entire left subtree even if node C is a goal node. If node J were\nalso a goal node, then depth-first search would return it as a solution instead of C, which\nwould be a better solution; hence, depth-first search is not optimal.\nSection 3.4.\nUninformed Search Strategies\n87\nThe time complexity of depth-first graph search is bounded by the size of the state space\n(which may be infinite, of course). A depth-first tree search, on the other hand, may generate\nall of the O(bm) nodes in the search tree, where m is the maximum depth of any node; this\ncan be much greater than the size of the state space. Note that m itself can be much larger\nthan d (the depth of the shallowest solution) and is infinite if the tree is unbounded.\nSo far, depth-first search seems to have no clear advantage over breadth-first search,\nso why do we include it? The reason is the space complexity. For a graph search, there is\nno advantage, but a depth-first tree search needs to store only a single path from the root\nto a leaf node, along with the remaining unexpanded sibling nodes for each node on the\npath. Once a node has been expanded, it can be removed from memory as soon as all its\ndescendants have been fully explored. (See Figure 3.16.) For a state space with branching\nfactor b and maximum depth m, depth-first search requires storage of only O(bm) nodes.\nUsing the same assumptions as for Figure 3.13 and assuming that nodes at the same depth as\nthe goal node have no successors, we find that depth-first search would require 156 kilobytes\ninstead of 10 exabytes at depth d = 16, a factor of 7 trillion times less space. This has\nled to the adoption of depth-first tree search as the basic workhorse of many areas of AI,\nincluding constraint satisfaction (Chapter 6), propositional satisfiability (Chapter 7), and logic\nprogramming (Chapter 9). For the remainder of this section, we focus primarily on the tree-\nsearch version of depth-first search.\nA variant of depth-first search called backtracking search uses still less memory. (See\nBACKTRACKING\nSEARCH\nChapter 6 for more details.) In backtracking, only one successor is generated at a time rather\nthan all successors; each partially expanded node remembers which successor to generate\nnext. In this way, only O(m) memory is needed rather than O(bm). Backtracking search\nfacilitates yet another memory-saving (and time-saving) trick: the idea of generating a suc-\ncessor by modifying the current state description directly rather than copying it first. This\nreduces the memory requirements to just one state description and O(m) actions. For this to\nwork, we must be able to undo each modification when we go back to generate the next suc-\ncessor. For problems with large state descriptions, such as robotic assembly, these techniques\nare critical to success.\n3.4.4\nDepth-limited search\nThe embarrassing failure of depth-first search in infinite state spaces can be alleviated by\nsupplying depth-first search with a predetermined depth limit ℓ. That is, nodes at depth ℓare\ntreated as if they have no successors. This approach is called depth-limited search. The\nDEPTH-LIMITED\nSEARCH\ndepth limit solves the infinite-path problem. Unfortunately, it also introduces an additional\nsource of incompleteness if we choose ℓ< d, that is, the shallowest goal is beyond the depth\nlimit. (This is likely when d is unknown.) Depth-limited search will also be nonoptimal if\nwe choose ℓ> d. Its time complexity is O(bℓ) and its space complexity is O(bℓ). Depth-first\nsearch can be viewed as a special case of depth-limited search with ℓ= ∞.\nSometimes, depth limits can be based on knowledge of the problem. For example, on\nthe map of Romania there are 20 cities. Therefore, we know that if there is a solution, it must\nbe of length 19 at the longest, so ℓ= 19 is a possible choice. But in fact if we studied the\n88\nChapter\n3.\nSolving Problems by Searching\nfunction DEPTH-LIMITED-SEARCH(problem,limit) returns a solution, or failure/cutoff\nreturn RECURSIVE-DLS(MAKE-NODE(problem.INITIAL-STATE),problem,limit)\nfunction RECURSIVE-DLS(node,problem,limit) returns a solution, or failure/cutoff\nif problem.GOAL-TEST(node.STATE) then return SOLUTION(node)\nelse if limit = 0 then return cutoff\nelse\ncutoffoccurred? ←false\nfor each action in problem.ACTIONS(node.STATE) do\nchild ←CHILD-NODE(problem,node,action)\nresult ←RECURSIVE-DLS(child,problem,limit −1)\nif result = cutoffthen cutoffoccurred? ←true\nelse if result ̸= failure then return result\nif cutoffoccurred? then return cutoffelse return failure\nFigure 3.17\nA recursive implementation of depth-limited tree search.\nmap carefully, we would discover that any city can be reached from any other city in at most\n9 steps. This number, known as the diameter of the state space, gives us a better depth limit,\nDIAMETER\nwhich leads to a more efficient depth-limited search. For most problems, however, we will\nnot know a good depth limit until we have solved the problem.\nDepth-limited search can be implemented as a simple modification to the general tree-\nor graph-search algorithm. Alternatively, it can be implemented as a simple recursive al-\ngorithm as shown in Figure 3.17. Notice that depth-limited search can terminate with two\nkinds of failure: the standard failure value indicates no solution; the cutoffvalue indicates\nno solution within the depth limit.\n3.4.5\nIterative deepening depth-first search\nIterative deepening search (or iterative deepening depth-first search) is a general strategy,\nITERATIVE\nDEEPENING SEARCH\noften used in combination with depth-first tree search, that finds the best depth limit. It does\nthis by gradually increasing the limit—first 0, then 1, then 2, and so on—until a goal is found.\nThis will occur when the depth limit reaches d, the depth of the shallowest goal node. The\nalgorithm is shown in Figure 3.18. Iterative deepening combines the benefits of depth-first\nand breadth-first search. Like depth-first search, its memory requirements are modest: O(bd)\nto be precise. Like breadth-first search, it is complete when the branching factor is finite and\noptimal when the path cost is a nondecreasing function of the depth of the node. Figure 3.19\nshows four iterations of ITERATIVE-DEEPENING-SEARCH on a binary search tree, where the\nsolution is found on the fourth iteration.\nIterative deepening search may seem wasteful because states are generated multiple\ntimes. It turns out this is not too costly. The reason is that in a search tree with the same (or\nnearly the same) branching factor at each level, most of the nodes are in the bottom level,\nso it does not matter much that the upper levels are generated multiple times. In an iterative\ndeepening search, the nodes on the bottom level (depth d) are generated once, those on the\nSection 3.4.\nUninformed Search Strategies\n89\nfunction ITERATIVE-DEEPENING-SEARCH(problem) returns a solution, or failure\nfor depth = 0 to ∞do\nresult ←DEPTH-LIMITED-SEARCH(problem,depth)\nif result ̸= cutoff then return result\nFigure 3.18\nThe iterative deepening search algorithm, which repeatedly applies depth-\nlimited search with increasing limits. It terminates when a solution is found or if the depth-\nlimited search returns failure, meaning that no solution exists.\nLimit = 3\nLimit = 2\nLimit = 1\nLimit = 0\nA\nA\nA\nB\nC\nA\nB\nC\nA\nB\nC\nA\nB\nC\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nA\nB\nC\nD\nE\nF\nG\nH\nJ\nK\nL\nM\nN\nO\nI\nA\nB\nC\nD\nE\nF\nG\nH\nI\nJ\nK\nL\nM\nN\nO\nFigure 3.19\nFour iterations of iterative deepening search on a binary tree.\n90\nChapter\n3.\nSolving Problems by Searching\nnext-to-bottom level are generated twice, and so on, up to the children of the root, which are\ngenerated d times. So the total number of nodes generated in the worst case is\nN(IDS) = (d)b + (d −1)b2 + · · · + (1)bd ,\nwhich gives a time complexity of O(bd)—asymptotically the same as breadth-first search.\nThere is some extra cost for generating the upper levels multiple times, but it is not large. For\nexample, if b = 10 and d = 5, the numbers are\nN(IDS) = 50 + 400 + 3, 000 + 20, 000 + 100, 000 = 123, 450\nN(BFS) = 10 + 100 + 1, 000 + 10, 000 + 100, 000 = 111, 110 .\nIf you are really concerned about repeating the repetition, you can use a hybrid approach\nthat runs breadth-first search until almost all the available memory is consumed, and then\nruns iterative deepening from all the nodes in the frontier. In general, iterative deepening is\nthe preferred uninformed search method when the search space is large and the depth of the\nsolution is not known.\nIterative deepening search is analogous to breadth-first search in that it explores a com-\nplete layer of new nodes at each iteration before going on to the next layer. It would seem\nworthwhile to develop an iterative analog to uniform-cost search, inheriting the latter algo-\nrithm’s optimality guarantees while avoiding its memory requirements. The idea is to use\nincreasing path-cost limits instead of increasing depth limits. The resulting algorithm, called\niterative lengthening search, is explored in Exercise 3.17. It turns out, unfortunately, that\nITERATIVE\nLENGTHENING\nSEARCH\niterative lengthening incurs substantial overhead compared to uniform-cost search.\n3.4.6\nBidirectional search\nThe idea behind bidirectional search is to run two simultaneous searches—one forward from\nthe initial state and the other backward from the goal—hoping that the two searches meet in\nthe middle (Figure 3.20). The motivation is that bd/2 + bd/2 is much less than bd, or in the\nfigure, the area of the two small circles is less than the area of one big circle centered on the\nstart and reaching to the goal.\nBidirectional search is implemented by replacing the goal test with a check to see\nwhether the frontiers of the two searches intersect; if they do, a solution has been found.\n(It is important to realize that the first such solution found may not be optimal, even if the\ntwo searches are both breadth-first; some additional search is required to make sure there\nisn’t another short-cut across the gap.) The check can be done when each node is generated\nor selected for expansion and, with a hash table, will take constant time. For example, if a\nproblem has solution depth d = 6, and each direction runs breadth-first search one node at a\ntime, then in the worst case the two searches meet when they have generated all of the nodes\nat depth 3. For b = 10, this means a total of 2,220 node generations, compared with 1,111,110\nfor a standard breadth-first search. Thus, the time complexity of bidirectional search using\nbreadth-first searches in both directions is O(bd/2). The space complexity is also O(bd/2).\nWe can reduce this by roughly half if one of the two searches is done by iterative deepening,\nbut at least one of the frontiers must be kept in memory so that the intersection check can be\ndone. This space requirement is the most significant weakness of bidirectional search.\nSection 3.4.\nUninformed Search Strategies\n91\nGoal\nStart\nFigure 3.20\nA schematic view of a bidirectional search that is about to succeed when a\nbranch from the start node meets a branch from the goal node.\nThe reduction in time complexity makes bidirectional search attractive, but how do we\nsearch backward? This is not as easy as it sounds. Let the predecessors of a state x be all\nPREDECESSOR\nthose states that have x as a successor. Bidirectional search requires a method for computing\npredecessors. When all the actions in the state space are reversible, the predecessors of x are\njust its successors. Other cases may require substantial ingenuity.\nConsider the question of what we mean by “the goal” in searching “backward from the\ngoal.” For the 8-puzzle and for finding a route in Romania, there is just one goal state, so the\nbackward search is very much like the forward search. If there are several explicitly listed\ngoal states—for example, the two dirt-free goal states in Figure 3.3—then we can construct a\nnew dummy goal state whose immediate predecessors are all the actual goal states. But if the\ngoal is an abstract description, such as the goal that “no queen attacks another queen” in the\nn-queens problem, then bidirectional search is difficult to use.\n3.4.7\nComparing uninformed search strategies\nFigure 3.21 compares search strategies in terms of the four evaluation criteria set forth in\nSection 3.3.2. This comparison is for tree-search versions. For graph searches, the main\ndifferences are that depth-first search is complete for finite state spaces and that the space and\ntime complexities are bounded by the size of the state space.\nCriterion\nBreadth-\nUniform-\nDepth-\nDepth-\nIterative\nBidirectional\nFirst\nCost\nFirst\nLimited\nDeepening\n(if applicable)\nComplete?\nYesa\nYesa,b\nNo\nNo\nYesa\nYesa,d\nTime\nO(bd)\nO(b1+⌊C∗/ϵ⌋)\nO(bm)\nO(bℓ)\nO(bd)\nO(bd/2)\nSpace\nO(bd)\nO(b1+⌊C∗/ϵ⌋)\nO(bm)\nO(bℓ)\nO(bd)\nO(bd/2)\nOptimal?\nYesc\nYes\nNo\nNo\nYesc\nYesc,d\nFigure 3.21\nEvaluation of tree-search strategies. b is the branching factor; d is the depth\nof the shallowest solution; m is the maximum depth of the search tree; l is the depth limit.\nSuperscript caveats are as follows: a complete if b is finite; b complete if step costs ≥ϵ for\npositive ϵ; c optimal if step costs are all identical; d if both directions use breadth-first search.\n92\nChapter\n3.\nSolving Problems by Searching\n3.5\nINFORMED (HEURISTIC) SEARCH STRATEGIES\nThis section shows how an informed search strategy—one that uses problem-specific knowl-\nINFORMED SEARCH\nedge beyond the definition of the problem itself—can find solutions more efficiently than can\nan uninformed strategy.\nThe general approach we consider is called best-first search. Best-first search is an\nBEST-FIRST SEARCH\ninstance of the general TREE-SEARCH or GRAPH-SEARCH algorithm in which a node is\nselected for expansion based on an evaluation function, f(n). The evaluation function is\nEVALUATION\nFUNCTION\nconstrued as a cost estimate, so the node with the lowest evaluation is expanded first. The\nimplementation of best-first graph search is identical to that for uniform-cost search (Fig-\nure 3.14), except for the use of f instead of g to order the priority queue.\nThe choice of f determines the search strategy. (For example, as Exercise 3.21 shows,\nbest-first tree search includes depth-first search as a special case.) Most best-first algorithms\ninclude as a component of f a heuristic function, denoted h(n):\nHEURISTIC\nFUNCTION\nh(n) = estimated cost of the cheapest path from the state at node n to a goal state.\n(Notice that h(n) takes a node as input, but, unlike g(n), it depends only on the state at that\nnode.) For example, in Romania, one might estimate the cost of the cheapest path from Arad\nto Bucharest via the straight-line distance from Arad to Bucharest.\nHeuristic functions are the most common form in which additional knowledge of the\nproblem is imparted to the search algorithm. We study heuristics in more depth in Section 3.6.\nFor now, we consider them to be arbitrary, nonnegative, problem-specific functions, with one\nconstraint: if n is a goal node, then h(n) = 0. The remainder of this section covers two ways\nto use heuristic information to guide search.\n3.5.1\nGreedy best-first search\nGreedy best-first search8 tries to expand the node that is closest to the goal, on the grounds\nGREEDY BEST-FIRST\nSEARCH\nthat this is likely to lead to a solution quickly. Thus, it evaluates nodes by using just the\nheuristic function; that is, f(n) = h(n).\nLet us see how this works for route-finding problems in Romania; we use the straight-\nline distance heuristic, which we will call hSLD.\nIf the goal is Bucharest, we need to\nSTRAIGHT-LINE\nDISTANCE\nknow the straight-line distances to Bucharest, which are shown in Figure 3.22. For exam-\nple, hSLD(In(Arad)) = 366. Notice that the values of hSLD cannot be computed from the\nproblem description itself. Moreover, it takes a certain amount of experience to know that\nhSLD is correlated with actual road distances and is, therefore, a useful heuristic.\nFigure 3.23 shows the progress of a greedy best-first search using hSLD to find a path\nfrom Arad to Bucharest. The first node to be expanded from Arad will be Sibiu because it\nis closer to Bucharest than either Zerind or Timisoara. The next node to be expanded will\nbe Fagaras because it is closest. Fagaras in turn generates Bucharest, which is the goal. For\nthis particular problem, greedy best-first search using hSLD finds a solution without ever\n8 Our first edition called this greedy search; other authors have called it best-first search. Our more general\nusage of the latter term follows Pearl (1984).\nSection 3.5.\nInformed (Heuristic) Search Strategies\n93\nUrziceni\nNeamt\nOradea\nZerind\nTimisoara\nMehadia\nSibiu\nPitesti\nRimnicu Vilcea\nVaslui\nBucharest\nGiurgiu\nHirsova\nEforie\nArad\nLugoj\nDrobeta\nCraiova\nFagaras\nIasi\n 0\n160\n242\n161\n77\n151\n366\n244\n226\n176\n241\n253\n329\n80\n199\n380\n234\n374\n100\n193\nFigure 3.22\nValues of hSLD—straight-line distances to Bucharest.\nexpanding a node that is not on the solution path; hence, its search cost is minimal. It is\nnot optimal, however: the path via Sibiu and Fagaras to Bucharest is 32 kilometers longer\nthan the path through Rimnicu Vilcea and Pitesti. This shows why the algorithm is called\n“greedy”—at each step it tries to get as close to the goal as it can.\nGreedy best-first tree search is also incomplete even in a finite state space, much like\ndepth-first search. Consider the problem of getting from Iasi to Fagaras. The heuristic sug-\ngests that Neamt be expanded first because it is closest to Fagaras, but it is a dead end. The\nsolution is to go first to Vaslui—a step that is actually farther from the goal according to\nthe heuristic—and then to continue to Urziceni, Bucharest, and Fagaras. The algorithm will\nnever find this solution, however, because expanding Neamt puts Iasi back into the frontier,\nIasi is closer to Fagaras than Vaslui is, and so Iasi will be expanded again, leading to an infi-\nnite loop. (The graph search version is complete in finite spaces, but not in infinite ones.) The\nworst-case time and space complexity for the tree version is O(bm), where m is the maximum\ndepth of the search space. With a good heuristic function, however, the complexity can be\nreduced substantially. The amount of the reduction depends on the particular problem and on\nthe quality of the heuristic.\n3.5.2\nA* search: Minimizing the total estimated solution cost\nThe most widely known form of best-first search is called A∗search (pronounced “A-star\nA∗SEARCH\nsearch”). It evaluates nodes by combining g(n), the cost to reach the node, and h(n), the cost\nto get from the node to the goal:\nf(n) = g(n) + h(n) .\nSince g(n) gives the path cost from the start node to node n, and h(n) is the estimated cost\nof the cheapest path from n to the goal, we have\nf(n) = estimated cost of the cheapest solution through n .\nThus, if we are trying to find the cheapest solution, a reasonable thing to try first is the\nnode with the lowest value of g(n) + h(n). It turns out that this strategy is more than just\nreasonable: provided that the heuristic function h(n) satisfies certain conditions, A∗search is\nboth complete and optimal. The algorithm is identical to UNIFORM-COST-SEARCH except\nthat A∗uses g + h instead of g.\n94\nChapter\n3.\nSolving Problems by Searching\nRimnicu Vilcea\nZerind\nArad\nSibiu\nArad\nFagaras\nOradea\nTimisoara\nSibiu\nBucharest\n329\n374\n366\n380\n193\n253\n0\nRimnicu Vilcea\nArad\nSibiu\nArad\nFagaras\nOradea\nTimisoara\n329\nZerind\n374\n366\n176\n380\n193\nZerind\nArad\nSibiu\nTimisoara\n253\n329\n374\nArad\n366\n(a) The initial state\n(b) After expanding Arad\n(c) After expanding Sibiu\n(d) After expanding Fagaras\nFigure 3.23\nStages in a greedy best-first tree search for Bucharest with the straight-line\ndistance heuristic hSLD. Nodes are labeled with their h-values.\nConditions for optimality: Admissibility and consistency\nThe first condition we require for optimality is that h(n) be an admissible heuristic. An\nADMISSIBLE\nHEURISTIC\nadmissible heuristic is one that never overestimates the cost to reach the goal. Because g(n)\nis the actual cost to reach n along the current path, and f(n) = g(n) + h(n), we have as an\nimmediate consequence that f(n) never overestimates the true cost of a solution along the\ncurrent path through n.\nAdmissible heuristics are by nature optimistic because they think the cost of solving\nthe problem is less than it actually is. An obvious example of an admissible heuristic is the\nstraight-line distance hSLD that we used in getting to Bucharest. Straight-line distance is\nadmissible because the shortest path between any two points is a straight line, so the straight\nSection 3.5.\nInformed (Heuristic) Search Strategies\n95\nline cannot be an overestimate. In Figure 3.24, we show the progress of an A∗tree search for\nBucharest. The values of g are computed from the step costs in Figure 3.2, and the values of\nhSLD are given in Figure 3.22. Notice in particular that Bucharest first appears on the frontier\nat step (e), but it is not selected for expansion because its f-cost (450) is higher than that of\nPitesti (417). Another way to say this is that there might be a solution through Pitesti whose\ncost is as low as 417, so the algorithm will not settle for a solution that costs 450.\nA second, slightly stronger condition called consistency (or sometimes monotonicity)\nCONSISTENCY\nMONOTONICITY\nis required only for applications of A∗to graph search.9 A heuristic h(n) is consistent if, for\nevery node n and every successor n′ of n generated by any action a, the estimated cost of\nreaching the goal from n is no greater than the step cost of getting to n′ plus the estimated\ncost of reaching the goal from n′:\nh(n) ≤c(n, a, n′) + h(n′) .\nThis is a form of the general triangle inequality, which stipulates that each side of a triangle\nTRIANGLE\nINEQUALITY\ncannot be longer than the sum of the other two sides. Here, the triangle is formed by n, n′,\nand the goal Gn closest to n. For an admissible heuristic, the inequality makes perfect sense:\nif there were a route from n to Gn via n′ that was cheaper than h(n), that would violate the\nproperty that h(n) is a lower bound on the cost to reach Gn.\nIt is fairly easy to show (Exercise 3.29) that every consistent heuristic is also admissible.\nConsistency is therefore a stricter requirement than admissibility, but one has to work quite\nhard to concoct heuristics that are admissible but not consistent. All the admissible heuristics\nwe discuss in this chapter are also consistent. Consider, for example, hSLD. We know that\nthe general triangle inequality is satisfied when each side is measured by the straight-line\ndistance and that the straight-line distance between n and n′ is no greater than c(n, a, n′).\nHence, hSLD is a consistent heuristic.\nOptimality of A*\nAs we mentioned earlier, A∗has the following properties: the tree-search version of A∗is\noptimal if h(n) is admissible, while the graph-search version is optimal if h(n) is consistent.\nWe show the second of these two claims since it is more useful. The argument es-\nsentially mirrors the argument for the optimality of uniform-cost search, with g replaced by\nf—just as in the A∗algorithm itself.\nThe first step is to establish the following: if h(n) is consistent, then the values of\nf(n) along any path are nondecreasing. The proof follows directly from the definition of\nconsistency. Suppose n′ is a successor of n; then g(n′) = g(n) + c(n, a, n′) for some action\na, and we have\nf(n′) = g(n′) + h(n′) = g(n) + c(n, a, n′) + h(n′) ≥g(n) + h(n) = f(n) .\nThe next step is to prove that whenever A∗selects a node n for expansion, the optimal path\nto that node has been found. Were this not the case, there would have to be another frontier\nnode n′ on the optimal path from the start node to n, by the graph separation property of\n9 With an admissible but inconsistent heuristic, A∗requires some extra bookkeeping to ensure optimality.\n96\nChapter\n3.\nSolving Problems by Searching\n(a) The initial state\n(b) After expanding Arad\n(c) After expanding Sibiu\nArad\nSibiu\nTimisoara\n447=118+329\nZerind\n449=75+374\n393=140+253\nArad\n366=0+366\n(d) After expanding Rimnicu Vilcea\n(e) After expanding Fagaras\n(f) After expanding Pitesti\nZerind\nArad\nSibiu\nArad\nTimisoara\nRimnicu Vilcea\nFagaras\nOradea\n447=118+329\n449=75+374\n646=280+366\n413=220+193\n415=239+176 671=291+380\nZerind\nArad\nSibiu\nTimisoara\n447=118+329\n449=75+374\nRimnicu Vilcea\nCraiova\nPitesti\nSibiu\n526=366+160\n553=300+253\n417=317+100\nZerind\nArad\nSibiu\nArad\nTimisoara\nSibiu\nBucharest\nFagaras\nOradea\nCraiova\nPitesti\nSibiu\n447=118+329\n449=75+374\n646=280+366\n591=338+253\n450=450+0\n526=366+160\n553=300+253\n417=317+100\n671=291+380\nZerind\nArad\nSibiu\nArad\nTimisoara\nSibiu\nBucharest\nOradea\nCraiova\nPitesti\nSibiu\nBucharest\nCraiova\nRimnicu Vilcea\n418=418+0\n447=118+329\n449=75+374\n646=280+366\n591=338+253\n450=450+0\n526=366+160\n553=300+253\n615=455+160 607=414+193\n671=291+380\nRimnicu Vilcea\nFagaras\nRimnicu Vilcea\nArad\nFagaras\nOradea\n646=280+366 415=239+176 671=291+380\nFigure 3.24\nStages in an A∗search for Bucharest. Nodes are labeled with f = g + h. The\nh values are the straight-line distances to Bucharest taken from Figure 3.22.\nSection 3.5.\nInformed (Heuristic) Search Strategies\n97\nO\nZ\nA\nT\nL\nM\nD\nC\nR\nF\nP\nG\nB\nU\nH\nE\nV\nI\nN\n380\n400\n420\nS\nFigure 3.25\nMap of Romania showing contours at f = 380, f = 400, and f = 420, with\nArad as the start state. Nodes inside a given contour have f-costs less than or equal to the\ncontour value.\nFigure 3.9; because f is nondecreasing along any path, n′ would have lower f-cost than n\nand would have been selected first.\nFrom the two preceding observations, it follows that the sequence of nodes expanded\nby A∗using GRAPH-SEARCH is in nondecreasing order of f(n). Hence, the first goal node\nselected for expansion must be an optimal solution because f is the true cost for goal nodes\n(which have h = 0) and all later goal nodes will be at least as expensive.\nThe fact that f-costs are nondecreasing along any path also means that we can draw\ncontours in the state space, just like the contours in a topographic map. Figure 3.25 shows\nCONTOUR\nan example. Inside the contour labeled 400, all nodes have f(n) less than or equal to 400,\nand so on. Then, because A∗expands the frontier node of lowest f-cost, we can see that an\nA∗search fans out from the start node, adding nodes in concentric bands of increasing f-cost.\nWith uniform-cost search (A∗search using h(n) = 0), the bands will be “circular”\naround the start state. With more accurate heuristics, the bands will stretch toward the goal\nstate and become more narrowly focused around the optimal path. If C∗is the cost of the\noptimal solution path, then we can say the following:\n• A∗expands all nodes with f(n) < C∗.\n• A∗might then expand some of the nodes right on the “goal contour” (where f(n) = C∗)\nbefore selecting a goal node.\nCompleteness requires that there be only finitely many nodes with cost less than or equal to\nC∗, a condition that is true if all step costs exceed some finite ϵ and if b is finite.\nNotice that A∗expands no nodes with f(n) > C∗—for example, Timisoara is not\nexpanded in Figure 3.24 even though it is a child of the root. We say that the subtree below\n98\nChapter\n3.\nSolving Problems by Searching\nTimisoara is pruned; because hSLD is admissible, the algorithm can safely ignore this subtree\nPRUNING\nwhile still guaranteeing optimality. The concept of pruning—eliminating possibilities from\nconsideration without having to examine them—is important for many areas of AI.\nOne final observation is that among optimal algorithms of this type—algorithms that\nextend search paths from the root and use the same heuristic information—A∗is optimally\nefficient for any given consistent heuristic. That is, no other optimal algorithm is guaran-\nOPTIMALLY\nEFFICIENT\nteed to expand fewer nodes than A∗(except possibly through tie-breaking among nodes with\nf(n) = C∗). This is because any algorithm that does not expand all nodes with f(n) < C∗\nruns the risk of missing the optimal solution.\nThat A∗search is complete, optimal, and optimally efficient among all such algorithms\nis rather satisfying. Unfortunately, it does not mean that A∗is the answer to all our searching\nneeds. The catch is that, for most problems, the number of states within the goal contour\nsearch space is still exponential in the length of the solution. The details of the analysis are\nbeyond the scope of this book, but the basic results are as follows. For problems with constant\nstep costs, the growth in run time as a function of the optimal solution depth d is analyzed in\nterms of the the absolute error or the relative error of the heuristic. The absolute error is\nABSOLUTE ERROR\nRELATIVE ERROR\ndefined as Δ ≡h∗−h, where h∗is the actual cost of getting from the root to the goal, and\nthe relative error is defined as ϵ ≡(h∗−h)/h∗.\nThe complexity results depend very strongly on the assumptions made about the state\nspace. The simplest model studied is a state space that has a single goal and is essentially a\ntree with reversible actions. (The 8-puzzle satisfies the first and third of these assumptions.)\nIn this case, the time complexity of A∗is exponential in the maximum absolute error, that is,\nO(bΔ). For constant step costs, we can write this as O(bϵd), where d is the solution depth.\nFor almost all heuristics in practical use, the absolute error is at least proportional to the path\ncost h∗, so ϵ is constant or growing and the time complexity is exponential in d. We can\nalso see the effect of a more accurate heuristic: O(bϵd) = O((bϵ)d), so the effective branching\nfactor (defined more formally in the next section) is bϵ.\nWhen the state space has many goal states—particularly near-optimal goal states—the\nsearch process can be led astray from the optimal path and there is an extra cost proportional\nto the number of goals whose cost is within a factor ϵ of the optimal cost. Finally, in the\ngeneral case of a graph, the situation is even worse. There can be exponentially many states\nwith f(n) < C∗even if the absolute error is bounded by a constant. For example, consider\na version of the vacuum world where the agent can clean up any square for unit cost without\neven having to visit it: in that case, squares can be cleaned in any order. With N initially dirty\nsquares, there are 2N states where some subset has been cleaned and all of them are on an\noptimal solution path—and hence satisfy f(n) < C∗—even if the heuristic has an error of 1.\nThe complexity of A∗often makes it impractical to insist on finding an optimal solution.\nOne can use variants of A∗that find suboptimal solutions quickly, or one can sometimes\ndesign heuristics that are more accurate but not strictly admissible. In any case, the use of a\ngood heuristic still provides enormous savings compared to the use of an uninformed search.\nIn Section 3.6, we look at the question of designing good heuristics.\nComputation time is not, however, A∗’s main drawback. Because it keeps all generated\nnodes in memory (as do all GRAPH-SEARCH algorithms), A∗usually runs out of space long\nSection 3.5.\nInformed (Heuristic) Search Strategies\n99\nfunction RECURSIVE-BEST-FIRST-SEARCH(problem) returns a solution, or failure\nreturn RBFS(problem, MAKE-NODE(problem.INITIAL-STATE),∞)\nfunction RBFS(problem,node,f limit) returns a solution, or failure and a new f-cost limit\nif problem.GOAL-TEST(node.STATE) then return SOLUTION(node)\nsuccessors ←[ ]\nfor each action in problem.ACTIONS(node.STATE) do\nadd CHILD-NODE(problem,node,action) into successors\nif successors is empty then return failure, ∞\nfor each s in successors do /* update f with value from previous search, if any */\ns.f ←max(s.g + s.h, node.f ))\nloop do\nbest ←the lowest f-value node in successors\nif best.f > f limit then return failure, best.f\nalternative ←the second-lowest f-value among successors\nresult,best.f ←RBFS(problem,best,min(f limit, alternative))\nif result ̸= failure then return result\nFigure 3.26\nThe algorithm for recursive best-first search.\nbefore it runs out of time. For this reason, A∗is not practical for many large-scale prob-\nlems. There are, however, algorithms that overcome the space problem without sacrificing\noptimality or completeness, at a small cost in execution time. We discuss these next.\n3.5.3\nMemory-bounded heuristic search\nThe simplest way to reduce memory requirements for A∗is to adapt the idea of iterative\ndeepening to the heuristic search context, resulting in the iterative-deepening A∗(IDA∗) al-\nITERATIVE-\nDEEPENING\nA∗\ngorithm. The main difference between IDA∗and standard iterative deepening is that the cutoff\nused is the f-cost (g +h) rather than the depth; at each iteration, the cutoff value is the small-\nest f-cost of any node that exceeded the cutoff on the previous iteration. IDA∗is practical\nfor many problems with unit step costs and avoids the substantial overhead associated with\nkeeping a sorted queue of nodes. Unfortunately, it suffers from the same difficulties with real-\nvalued costs as does the iterative version of uniform-cost search described in Exercise 3.17.\nThis section briefly examines two other memory-bounded algorithms, called RBFS and MA∗.\nRecursive best-first search (RBFS) is a simple recursive algorithm that attempts to\nRECURSIVE\nBEST-FIRST SEARCH\nmimic the operation of standard best-first search, but using only linear space. The algorithm\nis shown in Figure 3.26. Its structure is similar to that of a recursive depth-first search, but\nrather than continuing indefinitely down the current path, it uses the f limit variable to keep\ntrack of the f-value of the best alternative path available from any ancestor of the current\nnode. If the current node exceeds this limit, the recursion unwinds back to the alternative\npath. As the recursion unwinds, RBFS replaces the f-value of each node along the path\nwith a backed-up value—the best f-value of its children. In this way, RBFS remembers the\nBACKED-UP VALUE\nf-value of the best leaf in the forgotten subtree and can therefore decide whether it’s worth\n100\nChapter\n3.\nSolving Problems by Searching\nZerind\nArad\nSibiu\nArad\nFagaras\nOradea\nCraiova\nSibiu\nBucharest\nCraiova\nRimnicu Vilcea\nZerind\nArad\nSibiu\nArad\nSibiu\nBucharest\nRimnicu Vilcea\nOradea\nZerind\nArad\nSibiu\nArad\nTimisoara\nTimisoara\nTimisoara\nFagaras\nOradea\nRimnicu Vilcea\nCraiova\nPitesti\nSibiu\n646\n415\n671\n526\n553\n646\n671\n450\n591\n646\n671\n526\n553\n418\n615\n607\n447\n449\n447\n447\n449\n449\n366\n393\n366\n393\n413\n413 417\n415\n366\n393\n415 450\n417\nRimnicu Vilcea\nFagaras\n447\n415\n447\n447\n417\n(a) After expanding Arad, Sibiu, \n and Rimnicu Vilcea\n(c) After switching back to Rimnicu Vilcea\n and expanding Pitesti\n(b) After unwinding back to Sibiu \n and expanding Fagaras\n447\n447\n∞\n∞\n∞\n417\n417\nPitesti\nFigure 3.27\nStages in an RBFS search for the shortest route to Bucharest. The f-limit\nvalue for each recursive call is shown on top of each current node, and every node is labeled\nwith its f-cost. (a) The path via Rimnicu Vilcea is followed until the current best leaf (Pitesti)\nhas a value that is worse than the best alternative path (Fagaras). (b) The recursion unwinds\nand the best leaf value of the forgotten subtree (417) is backed up to Rimnicu Vilcea; then\nFagaras is expanded, revealing a best leaf value of 450. (c) The recursion unwinds and the\nbest leaf value of the forgotten subtree (450) is backed up to Fagaras; then Rimnicu Vilcea is\nexpanded. This time, because the best alternative path (through Timisoara) costs at least 447,\nthe expansion continues to Bucharest.\nreexpanding the subtree at some later time. Figure 3.27 shows how RBFS reaches Bucharest.\nRBFS is somewhat more efficient than IDA∗, but still suffers from excessive node re-\ngeneration. In the example in Figure 3.27, RBFS follows the path via Rimnicu Vilcea, then\nSection 3.5.\nInformed (Heuristic) Search Strategies\n101\n“changes its mind” and tries Fagaras, and then changes its mind back again. These mind\nchanges occur because every time the current best path is extended, its f-value is likely to\nincrease—h is usually less optimistic for nodes closer to the goal. When this happens, the\nsecond-best path might become the best path, so the search has to backtrack to follow it.\nEach mind change corresponds to an iteration of IDA∗and could require many reexpansions\nof forgotten nodes to recreate the best path and extend it one more node.\nLike A∗tree search, RBFS is an optimal algorithm if the heuristic function h(n) is\nadmissible. Its space complexity is linear in the depth of the deepest optimal solution, but\nits time complexity is rather difficult to characterize: it depends both on the accuracy of the\nheuristic function and on how often the best path changes as nodes are expanded.\nIDA∗and RBFS suffer from using too little memory. Between iterations, IDA∗retains\nonly a single number: the current f-cost limit. RBFS retains more information in memory,\nbut it uses only linear space: even if more memory were available, RBFS has no way to make\nuse of it. Because they forget most of what they have done, both algorithms may end up reex-\npanding the same states many times over. Furthermore, they suffer the potentially exponential\nincrease in complexity associated with redundant paths in graphs (see Section 3.3).\nIt seems sensible, therefore, to use all available memory. Two algorithms that do this\nare MA∗(memory-bounded A∗) and SMA∗(simplified MA∗). SMA∗is—well—simpler, so\nMA*\nSMA*\nwe will describe it. SMA∗proceeds just like A∗, expanding the best leaf until memory is full.\nAt this point, it cannot add a new node to the search tree without dropping an old one. SMA∗\nalways drops the worst leaf node—the one with the highest f-value. Like RBFS, SMA∗\nthen backs up the value of the forgotten node to its parent. In this way, the ancestor of a\nforgotten subtree knows the quality of the best path in that subtree. With this information,\nSMA∗regenerates the subtree only when all other paths have been shown to look worse than\nthe path it has forgotten. Another way of saying this is that, if all the descendants of a node n\nare forgotten, then we will not know which way to go from n, but we will still have an idea\nof how worthwhile it is to go anywhere from n.\nThe complete algorithm is too complicated to reproduce here,10 but there is one subtlety\nworth mentioning. We said that SMA∗expands the best leaf and deletes the worst leaf. What\nif all the leaf nodes have the same f-value? To avoid selecting the same node for deletion\nand expansion, SMA∗expands the newest best leaf and deletes the oldest worst leaf. These\ncoincide when there is only one leaf, but in that case, the current search tree must be a single\npath from root to leaf that fills all of memory. If the leaf is not a goal node, then even if it is on\nan optimal solution path, that solution is not reachable with the available memory. Therefore,\nthe node can be discarded exactly as if it had no successors.\nSMA∗is complete if there is any reachable solution—that is, if d, the depth of the\nshallowest goal node, is less than the memory size (expressed in nodes). It is optimal if any\noptimal solution is reachable; otherwise, it returns the best reachable solution. In practical\nterms, SMA∗is a fairly robust choice for finding optimal solutions, particularly when the state\nspace is a graph, step costs are not uniform, and node generation is expensive compared to\nthe overhead of maintaining the frontier and the explored set.\n10 A rough sketch appeared in the first edition of this book.\n102\nChapter\n3.\nSolving Problems by Searching\nOn very hard problems, however, it will often be the case that SMA∗is forced to switch\nback and forth continually among many candidate solution paths, only a small subset of which\ncan fit in memory. (This resembles the problem of thrashing in disk paging systems.) Then\nTHRASHING\nthe extra time required for repeated regeneration of the same nodes means that problems\nthat would be practically solvable by A∗, given unlimited memory, become intractable for\nSMA∗. That is to say, memory limitations can make a problem intractable from the point\nof view of computation time. Although no current theory explains the tradeoff between time\nand memory, it seems that this is an inescapable problem. The only way out is to drop the\noptimality requirement.\n3.5.4\nLearning to search better\nWe have presented several fixed strategies—breadth-first, greedy best-first, and so on—that\nhave been designed by computer scientists. Could an agent learn how to search better? The\nanswer is yes, and the method rests on an important concept called the metalevel state space.\nMETALEVEL STATE\nSPACE\nEach state in a metalevel state space captures the internal (computational) state of a program\nthat is searching in an object-level state space such as Romania. For example, the internal\nOBJECT-LEVEL STATE\nSPACE\nstate of the A∗algorithm consists of the current search tree. Each action in the metalevel state\nspace is a computation step that alters the internal state; for example, each computation step\nin A∗expands a leaf node and adds its successors to the tree. Thus, Figure 3.24, which shows\na sequence of larger and larger search trees, can be seen as depicting a path in the metalevel\nstate space where each state on the path is an object-level search tree.\nNow, the path in Figure 3.24 has five steps, including one step, the expansion of Fagaras,\nthat is not especially helpful. For harder problems, there will be many such missteps, and a\nmetalevel learning algorithm can learn from these experiences to avoid exploring unpromis-\nMETALEVEL\nLEARNING\ning subtrees. The techniques used for this kind of learning are described in Chapter 21. The\ngoal of learning is to minimize the total cost of problem solving, trading off computational\nexpense and path cost.\n3.6\nHEURISTIC FUNCTIONS\nIn this section, we look at heuristics for the 8-puzzle, in order to shed light on the nature of\nheuristics in general.\nThe 8-puzzle was one of the earliest heuristic search problems. As mentioned in Sec-\ntion 3.2, the object of the puzzle is to slide the tiles horizontally or vertically into the empty\nspace until the configuration matches the goal configuration (Figure 3.28).\nThe average solution cost for a randomly generated 8-puzzle instance is about 22 steps.\nThe branching factor is about 3. (When the empty tile is in the middle, four moves are\npossible; when it is in a corner, two; and when it is along an edge, three.) This means\nthat an exhaustive tree search to depth 22 would look at about 322 ≈3.1 × 1010 states.\nA graph search would cut this down by a factor of about 170,000 because only 9!/2 =\n181, 440 distinct states are reachable. (See Exercise 3.4.) This is a manageable number, but\nSection 3.6.\nHeuristic Functions\n103\n2\nStart State\nGoal State\n1\n3\n4\n6\n7\n5\n1\n2\n3\n4\n6\n7\n8\n5\n8\nFigure 3.28\nA typical instance of the 8-puzzle. The solution is 26 steps long.\nthe corresponding number for the 15-puzzle is roughly 1013, so the next order of business is\nto find a good heuristic function. If we want to find the shortest solutions by using A∗, we\nneed a heuristic function that never overestimates the number of steps to the goal. There is a\nlong history of such heuristics for the 15-puzzle; here are two commonly used candidates:\n• h1 = the number of misplaced tiles. For Figure 3.28, all of the eight tiles are out of\nposition, so the start state would have h1 = 8. h1 is an admissible heuristic because it\nis clear that any tile that is out of place must be moved at least once.\n• h2 = the sum of the distances of the tiles from their goal positions. Because tiles\ncannot move along diagonals, the distance we will count is the sum of the horizontal\nand vertical distances. This is sometimes called the city block distance or Manhattan\ndistance. h2 is also admissible because all any move can do is move one tile one step\nMANHATTAN\nDISTANCE\ncloser to the goal. Tiles 1 to 8 in the start state give a Manhattan distance of\nh2 = 3 + 1 + 2 + 2 + 2 + 3 + 3 + 2 = 18 .\nAs expected, neither of these overestimates the true solution cost, which is 26.\n3.6.1\nThe effect of heuristic accuracy on performance\nOne way to characterize the quality of a heuristic is the effective branching factor b∗. If the\nEFFECTIVE\nBRANCHING FACTOR\ntotal number of nodes generated by A∗for a particular problem is N and the solution depth is\nd, then b∗is the branching factor that a uniform tree of depth d would have to have in order\nto contain N + 1 nodes. Thus,\nN + 1 = 1 + b∗+ (b∗)2 + · · · + (b∗)d .\nFor example, if A∗finds a solution at depth 5 using 52 nodes, then the effective branching\nfactor is 1.92. The effective branching factor can vary across problem instances, but usually\nit is fairly constant for sufficiently hard problems. (The existence of an effective branching\nfactor follows from the result, mentioned earlier, that the number of nodes expanded by A∗\ngrows exponentially with solution depth.) Therefore, experimental measurements of b∗on a\nsmall set of problems can provide a good guide to the heuristic’s overall usefulness. A well-\ndesigned heuristic would have a value of b∗close to 1, allowing fairly large problems to be\nsolved at reasonable computational cost.\n104\nChapter\n3.\nSolving Problems by Searching\nTo test the heuristic functions h1 and h2, we generated 1200 random problems with\nsolution lengths from 2 to 24 (100 for each even number) and solved them with iterative\ndeepening search and with A∗tree search using both h1 and h2. Figure 3.29 gives the average\nnumber of nodes generated by each strategy and the effective branching factor. The results\nsuggest that h2 is better than h1, and is far better than using iterative deepening search. Even\nfor small problems with d = 12, A∗with h2 is 50,000 times more efficient than uninformed\niterative deepening search.\nSearch Cost (nodes generated)\nEffective Branching Factor\nd\nIDS\nA∗(h1)\nA∗(h2)\nIDS\nA∗(h1)\nA∗(h2)\n2\n10\n6\n6\n2.45\n1.79\n1.79\n4\n112\n13\n12\n2.87\n1.48\n1.45\n6\n680\n20\n18\n2.73\n1.34\n1.30\n8\n6384\n39\n25\n2.80\n1.33\n1.24\n10\n47127\n93\n39\n2.79\n1.38\n1.22\n12\n3644035\n227\n73\n2.78\n1.42\n1.24\n14\n–\n539\n113\n–\n1.44\n1.23\n16\n–\n1301\n211\n–\n1.45\n1.25\n18\n–\n3056\n363\n–\n1.46\n1.26\n20\n–\n7276\n676\n–\n1.47\n1.27\n22\n–\n18094\n1219\n–\n1.48\n1.28\n24\n–\n39135\n1641\n–\n1.48\n1.26\nFigure 3.29\nComparison of the search costs and effective branching factors for the\nITERATIVE-DEEPENING-SEARCH and A∗algorithms with h1, h2. Data are averaged over\n100 instances of the 8-puzzle for each of various solution lengths d.\nOne might ask whether h2 is always better than h1. The answer is “Essentially, yes.” It\nis easy to see from the definitions of the two heuristics that, for any node n, h2(n) ≥h1(n).\nWe thus say that h2 dominates h1. Domination translates directly into efficiency: A∗using\nDOMINATION\nh2 will never expand more nodes than A∗using h1 (except possibly for some nodes with\nf(n) = C∗). The argument is simple. Recall the observation on page 97 that every node\nwith f(n) < C∗will surely be expanded. This is the same as saying that every node with\nh(n) < C∗−g(n) will surely be expanded. But because h2 is at least as big as h1 for all\nnodes, every node that is surely expanded by A∗search with h2 will also surely be expanded\nwith h1, and h1 might cause other nodes to be expanded as well. Hence, it is generally\nbetter to use a heuristic function with higher values, provided it is consistent and that the\ncomputation time for the heuristic is not too long.\n3.6.2\nGenerating admissible heuristics from relaxed problems\nWe have seen that both h1 (misplaced tiles) and h2 (Manhattan distance) are fairly good\nheuristics for the 8-puzzle and that h2 is better. How might one have come up with h2? Is it\npossible for a computer to invent such a heuristic mechanically?\nh1 and h2 are estimates of the remaining path length for the 8-puzzle, but they are also\nperfectly accurate path lengths for simplified versions of the puzzle. If the rules of the puzzle\nSection 3.6.\nHeuristic Functions\n105\nwere changed so that a tile could move anywhere instead of just to the adjacent empty square,\nthen h1 would give the exact number of steps in the shortest solution. Similarly, if a tile could\nmove one square in any direction, even onto an occupied square, then h2 would give the exact\nnumber of steps in the shortest solution. A problem with fewer restrictions on the actions is\ncalled a relaxed problem. The state-space graph of the relaxed problem is a supergraph of\nRELAXED PROBLEM\nthe original state space because the removal of restrictions creates added edges in the graph.\nBecause the relaxed problem adds edges to the state space, any optimal solution in the\noriginal problem is, by definition, also a solution in the relaxed problem; but the relaxed\nproblem may have better solutions if the added edges provide short cuts. Hence, the cost of\nan optimal solution to a relaxed problem is an admissible heuristic for the original problem.\nFurthermore, because the derived heuristic is an exact cost for the relaxed problem, it must\nobey the triangle inequality and is therefore consistent (see page 95).\nIf a problem definition is written down in a formal language, it is possible to construct\nrelaxed problems automatically.11 For example, if the 8-puzzle actions are described as\nA tile can move from square A to square B if\nA is horizontally or vertically adjacent to B and B is blank,\nwe can generate three relaxed problems by removing one or both of the conditions:\n(a) A tile can move from square A to square B if A is adjacent to B.\n(b) A tile can move from square A to square B if B is blank.\n(c) A tile can move from square A to square B.\nFrom (a), we can derive h2 (Manhattan distance). The reasoning is that h2 would be the\nproper score if we moved each tile in turn to its destination. The heuristic derived from (b) is\ndiscussed in Exercise 3.31. From (c), we can derive h1 (misplaced tiles) because it would be\nthe proper score if tiles could move to their intended destination in one step. Notice that it is\ncrucial that the relaxed problems generated by this technique can be solved essentially without\nsearch, because the relaxed rules allow the problem to be decomposed into eight independent\nsubproblems. If the relaxed problem is hard to solve, then the values of the corresponding\nheuristic will be expensive to obtain.12\nA program called ABSOLVER can generate heuristics automatically from problem def-\ninitions, using the “relaxed problem” method and various other techniques (Prieditis, 1993).\nABSOLVER generated a new heuristic for the 8-puzzle that was better than any preexisting\nheuristic and found the first useful heuristic for the famous Rubik’s Cube puzzle.\nOne problem with generating new heuristic functions is that one often fails to get a\nsingle “clearly best” heuristic. If a collection of admissible heuristics h1 . . . hm is available\nfor a problem and none of them dominates any of the others, which should we choose? As it\nturns out, we need not make a choice. We can have the best of all worlds, by defining\nh(n) = max{h1(n), . . . , hm(n)} .\n11 In Chapters 8 and 10, we describe formal languages suitable for this task; with formal descriptions that can be\nmanipulated, the construction of relaxed problems can be automated. For now, we use English.\n12 Note that a perfect heuristic can be obtained simply by allowing h to run a full breadth-first search “on the\nsly.” Thus, there is a tradeoff between accuracy and computation time for heuristic functions.\n106\nChapter\n3.\nSolving Problems by Searching\nStart State\nGoal State\n1\n2\n3\n4\n6\n8\n5\n2\n1\n3\n6\n7\n8\n54\nFigure 3.30\nA subproblem of the 8-puzzle instance given in Figure 3.28. The task is to\nget tiles 1, 2, 3, and 4 into their correct positions, without worrying about what happens to\nthe other tiles.\nThis composite heuristic uses whichever function is most accurate on the node in question.\nBecause the component heuristics are admissible, h is admissible; it is also easy to prove that\nh is consistent. Furthermore, h dominates all of its component heuristics.\n3.6.3\nGenerating admissible heuristics from subproblems: Pattern databases\nAdmissible heuristics can also be derived from the solution cost of a subproblem of a given\nSUBPROBLEM\nproblem. For example, Figure 3.30 shows a subproblem of the 8-puzzle instance in Fig-\nure 3.28. The subproblem involves getting tiles 1, 2, 3, 4 into their correct positions. Clearly,\nthe cost of the optimal solution of this subproblem is a lower bound on the cost of the com-\nplete problem. It turns out to be more accurate than Manhattan distance in some cases.\nThe idea behind pattern databases is to store these exact solution costs for every pos-\nPATTERN DATABASE\nsible subproblem instance—in our example, every possible configuration of the four tiles\nand the blank. (The locations of the other four tiles are irrelevant for the purposes of solv-\ning the subproblem, but moves of those tiles do count toward the cost.) Then we compute\nan admissible heuristic hDB for each complete state encountered during a search simply by\nlooking up the corresponding subproblem configuration in the database. The database itself is\nconstructed by searching back13 from the goal and recording the cost of each new pattern en-\ncountered; the expense of this search is amortized over many subsequent problem instances.\nThe choice of 1-2-3-4 is fairly arbitrary; we could also construct databases for 5-6-7-8,\nfor 2-4-6-8, and so on. Each database yields an admissible heuristic, and these heuristics can\nbe combined, as explained earlier, by taking the maximum value. A combined heuristic of\nthis kind is much more accurate than the Manhattan distance; the number of nodes generated\nwhen solving random 15-puzzles can be reduced by a factor of 1000.\nOne might wonder whether the heuristics obtained from the 1-2-3-4 database and the\n5-6-7-8 could be added, since the two subproblems seem not to overlap. Would this still give\nan admissible heuristic? The answer is no, because the solutions of the 1-2-3-4 subproblem\nand the 5-6-7-8 subproblem for a given state will almost certainly share some moves—it is\n13 By working backward from the goal, the exact solution cost of every instance encountered is immediately\navailable. This is an example of dynamic programming, which we discuss further in Chapter 17.\nSection 3.6.\nHeuristic Functions\n107\nunlikely that 1-2-3-4 can be moved into place without touching 5-6-7-8, and vice versa. But\nwhat if we don’t count those moves? That is, we record not the total cost of solving the 1-2-\n3-4 subproblem, but just the number of moves involving 1-2-3-4. Then it is easy to see that\nthe sum of the two costs is still a lower bound on the cost of solving the entire problem. This\nis the idea behind disjoint pattern databases. With such databases, it is possible to solve\nDISJOINT PATTERN\nDATABASES\nrandom 15-puzzles in a few milliseconds—the number of nodes generated is reduced by a\nfactor of 10,000 compared with the use of Manhattan distance. For 24-puzzles, a speedup of\nroughly a factor of a million can be obtained.\nDisjoint pattern databases work for sliding-tile puzzles because the problem can be\ndivided up in such a way that each move affects only one subproblem—because only one tile\nis moved at a time. For a problem such as Rubik’s Cube, this kind of subdivision is difficult\nbecause each move affects 8 or 9 of the 26 cubies. More general ways of defining additive,\nadmissible heuristics have been proposed that do apply to Rubik’s cube (Yang et al., 2008),\nbut they have not yielded a heuristic better than the best nonadditive heuristic for the problem.\n3.6.4\nLearning heuristics from experience\nA heuristic function h(n) is supposed to estimate the cost of a solution beginning from the\nstate at node n. How could an agent construct such a function? One solution was given in\nthe preceding sections—namely, to devise relaxed problems for which an optimal solution\ncan be found easily. Another solution is to learn from experience. “Experience” here means\nsolving lots of 8-puzzles, for instance. Each optimal solution to an 8-puzzle problem provides\nexamples from which h(n) can be learned. Each example consists of a state from the solu-\ntion path and the actual cost of the solution from that point. From these examples, a learning\nalgorithm can be used to construct a function h(n) that can (with luck) predict solution costs\nfor other states that arise during search. Techniques for doing just this using neural nets, de-\ncision trees, and other methods are demonstrated in Chapter 18. (The reinforcement learning\nmethods described in Chapter 21 are also applicable.)\nInductive learning methods work best when supplied with features of a state that are\nFEATURE\nrelevant to predicting the state’s value, rather than with just the raw state description. For\nexample, the feature “number of misplaced tiles” might be helpful in predicting the actual\ndistance of a state from the goal. Let’s call this feature x1(n). We could take 100 randomly\ngenerated 8-puzzle configurations and gather statistics on their actual solution costs. We\nmight find that when x1(n) is 5, the average solution cost is around 14, and so on. Given\nthese data, the value of x1 can be used to predict h(n). Of course, we can use several features.\nA second feature x2(n) might be “number of pairs of adjacent tiles that are not adjacent in the\ngoal state.” How should x1(n) and x2(n) be combined to predict h(n)? A common approach\nis to use a linear combination:\nh(n) = c1x1(n) + c2x2(n) .\nThe constants c1 and c2 are adjusted to give the best fit to the actual data on solution costs.\nOne expects both c1 and c2 to be positive because misplaced tiles and incorrect adjacent pairs\nmake the problem harder to solve. Notice that this heuristic does satisfy the condition that\nh(n) = 0 for goal states, but it is not necessarily admissible or consistent.\n108\nChapter\n3.\nSolving Problems by Searching\n3.7\nSUMMARY\nThis chapter has introduced methods that an agent can use to select actions in environments\nthat are deterministic, observable, static, and completely known. In such cases, the agent can\nconstruct sequences of actions that achieve its goals; this process is called search.\n• Before an agent can start searching for solutions, a goal must be identified and a well-\ndefined problem must be formulated.\n• A problem consists of five parts: the initial state, a set of actions, a transition model\ndescribing the results of those actions, a goal test function, and a path cost function.\nThe environment of the problem is represented by a state space. A path through the\nstate space from the initial state to a goal state is a solution.\n• Search algorithms treat states and actions as atomic: they do not consider any internal\nstructure they might possess.\n• A general TREE-SEARCH algorithm considers all possible paths to find a solution,\nwhereas a GRAPH-SEARCH algorithm avoids consideration of redundant paths.\n• Search algorithms are judged on the basis of completeness, optimality, time complex-\nity, and space complexity. Complexity depends on b, the branching factor in the state\nspace, and d, the depth of the shallowest solution.\n• Uninformed search methods have access only to the problem definition. The basic\nalgorithms are as follows:\n– Breadth-first search expands the shallowest nodes first; it is complete, optimal\nfor unit step costs, but has exponential space complexity.\n– Uniform-cost search expands the node with lowest path cost, g(n), and is optimal\nfor general step costs.\n– Depth-first search expands the deepest unexpanded node first. It is neither com-\nplete nor optimal, but has linear space complexity. Depth-limited search adds a\ndepth bound.\n– Iterative deepening search calls depth-first search with increasing depth limits\nuntil a goal is found. It is complete, optimal for unit step costs, has time complexity\ncomparable to breadth-first search, and has linear space complexity.\n– Bidirectional search can enormously reduce time complexity, but it is not always\napplicable and may require too much space.\n• Informed search methods may have access to a heuristic function h(n) that estimates\nthe cost of a solution from n.\n– The generic best-first search algorithm selects a node for expansion according to\nan evaluation function.\n– Greedy best-first search expands nodes with minimal h(n). It is not optimal but\nis often efficient.\nBibliographical and Historical Notes\n109\n– A∗search expands nodes with minimal f(n) = g(n) + h(n). A∗is complete and\noptimal, provided that h(n) is admissible (for TREE-SEARCH) or consistent (for\nGRAPH-SEARCH). The space complexity of A∗is still prohibitive.\n– RBFS (recursive best-first search) and SMA∗(simplified memory-bounded A∗)\nare robust, optimal search algorithms that use limited amounts of memory; given\nenough time, they can solve problems that A∗cannot solve because it runs out of\nmemory.\n• The performance of heuristic search algorithms depends on the quality of the heuristic\nfunction. One can sometimes construct good heuristics by relaxing the problem defi-\nnition, by storing precomputed solution costs for subproblems in a pattern database, or\nby learning from experience with the problem class.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nThe topic of state-space search originated in more or less its current form in the early years of\nAI. Newell and Simon’s work on the Logic Theorist (1957) and GPS (1961) led to the estab-\nlishment of search algorithms as the primary weapons in the armory of 1960s AI researchers\nand to the establishment of problem solving as the canonical AI task. Work in operations\nresearch by Richard Bellman (1957) showed the importance of additive path costs in sim-\nplifying optimization algorithms. The text on Automated Problem Solving by Nils Nilsson\n(1971) established the area on a solid theoretical footing.\nMost of the state-space search problems analyzed in this chapter have a long history\nin the literature and are less trivial than they might seem. The missionaries and cannibals\nproblem used in Exercise 3.9 was analyzed in detail by Amarel (1968). It had been consid-\nered earlier—in AI by Simon and Newell (1961) and in operations research by Bellman and\nDreyfus (1962).\nThe 8-puzzle is a smaller cousin of the 15-puzzle, whose history is recounted at length\nby Slocum and Sonneveld (2006). It was widely believed to have been invented by the fa-\nmous American game designer Sam Loyd, based on his claims to that effect from 1891 on-\nward (Loyd, 1959). Actually it was invented by Noyes Chapman, a postmaster in Canastota,\nNew York, in the mid-1870s. (Chapman was unable to patent his invention, as a generic\npatent covering sliding blocks with letters, numbers, or pictures was granted to Ernest Kinsey\nin 1878.) It quickly attracted the attention of the public and of mathematicians (Johnson and\nStory, 1879; Tait, 1880). The editors of the American Journal of Mathematics stated, “The\n‘15’ puzzle for the last few weeks has been prominently before the American public, and may\nsafely be said to have engaged the attention of nine out of ten persons of both sexes and all\nages and conditions of the community.” Ratner and Warmuth (1986) showed that the general\nn × n version of the 15-puzzle belongs to the class of NP-complete problems.\nThe 8-queens problem was first published anonymously in the German chess maga-\nzine Schach in 1848; it was later attributed to one Max Bezzel. It was republished in 1850\nand at that time drew the attention of the eminent mathematician Carl Friedrich Gauss, who\n110\nChapter\n3.\nSolving Problems by Searching\nattempted to enumerate all possible solutions; initially he found only 72, but eventually he\nfound the correct answer of 92, although Nauck published all 92 solutions first, in 1850.\nNetto (1901) generalized the problem to n queens, and Abramson and Yung (1989) found an\nO(n) algorithm.\nEach of the real-world search problems listed in the chapter has been the subject of a\ngood deal of research effort. Methods for selecting optimal airline flights remain proprietary\nfor the most part, but Carl de Marcken (personal communication) has shown that airline ticket\npricing and restrictions have become so convoluted that the problem of selecting an optimal\nflight is formally undecidable. The traveling-salesperson problem is a standard combinato-\nrial problem in theoretical computer science (Lawler et al., 1992). Karp (1972) proved the\nTSP to be NP-hard, but effective heuristic approximation methods were developed (Lin and\nKernighan, 1973). Arora (1998) devised a fully polynomial approximation scheme for Eu-\nclidean TSPs. VLSI layout methods are surveyed by Shahookar and Mazumder (1991), and\nmany layout optimization papers appear in VLSI journals. Robotic navigation and assembly\nproblems are discussed in Chapter 25.\nUninformed search algorithms for problem solving are a central topic of classical com-\nputer science (Horowitz and Sahni, 1978) and operations research (Dreyfus, 1969). Breadth-\nfirst search was formulated for solving mazes by Moore (1959). The method of dynamic\nprogramming (Bellman, 1957; Bellman and Dreyfus, 1962), which systematically records\nsolutions for all subproblems of increasing lengths, can be seen as a form of breadth-first\nsearch on graphs. The two-point shortest-path algorithm of Dijkstra (1959) is the origin\nof uniform-cost search. These works also introduced the idea of explored and frontier sets\n(closed and open lists).\nA version of iterative deepening designed to make efficient use of the chess clock was\nfirst used by Slate and Atkin (1977) in the CHESS 4.5 game-playing program. Martelli’s\nalgorithm B (1977) includes an iterative deepening aspect and also dominates A∗’s worst-case\nperformance with admissible but inconsistent heuristics. The iterative deepening technique\ncame to the fore in work by Korf (1985a). Bidirectional search, which was introduced by\nPohl (1971), can also be effective in some cases.\nThe use of heuristic information in problem solving appears in an early paper by Simon\nand Newell (1958), but the phrase “heuristic search” and the use of heuristic functions that\nestimate the distance to the goal came somewhat later (Newell and Ernst, 1965; Lin, 1965).\nDoran and Michie (1966) conducted extensive experimental studies of heuristic search. Al-\nthough they analyzed path length and “penetrance” (the ratio of path length to the total num-\nber of nodes examined so far), they appear to have ignored the information provided by the\npath cost g(n). The A∗algorithm, incorporating the current path cost into heuristic search,\nwas developed by Hart, Nilsson, and Raphael (1968), with some later corrections (Hart et al.,\n1972). Dechter and Pearl (1985) demonstrated the optimal efficiency of A∗.\nThe original A∗paper introduced the consistency condition on heuristic functions. The\nmonotone condition was introduced by Pohl (1977) as a simpler replacement, but Pearl (1984)\nshowed that the two were equivalent.\nPohl (1977) pioneered the study of the relationship between the error in heuristic func-\ntions and the time complexity of A∗. Basic results were obtained for tree search with unit step\nBibliographical and Historical Notes\n111\ncosts and a single goal node (Pohl, 1977; Gaschnig, 1979; Huyn et al., 1980; Pearl, 1984) and\nwith multiple goal nodes (Dinh et al., 2007). The “effective branching factor” was proposed\nby Nilsson (1971) as an empirical measure of the efficiency; it is equivalent to assuming a\ntime cost of O((b∗)d). For tree search applied to a graph, Korf et al. (2001) argue that the time\ncost is better modeled as O(bd−k), where k depends on the heuristic accuracy; this analysis\nhas elicited some controversy, however. For graph search, Helmert and R¨oger (2008) noted\nthat several well-known problems contained exponentially many nodes on optimal solution\npaths, implying exponential time complexity for A∗even with constant absolute error in h.\nThere are many variations on the A∗algorithm. Pohl (1973) proposed the use of dynamic\nweighting, which uses a weighted sum fw(n) = wgg(n) + whh(n) of the current path length\nand the heuristic function as an evaluation function, rather than the simple sum f(n) = g(n)+\nh(n) used in A∗. The weights wg and wh are adjusted dynamically as the search progresses.\nPohl’s algorithm can be shown to be ϵ-admissible—that is, guaranteed to find solutions within\na factor 1 + ϵ of the optimal solution, where ϵ is a parameter supplied to the algorithm. The\nsame property is exhibited by the A∗\nϵ algorithm (Pearl, 1984), which can select any node from\nthe frontier provided its f-cost is within a factor 1+ ϵ of the lowest-f-cost frontier node. The\nselection can be done so as to minimize search cost.\nBidirectional versions of A∗have been investigated; a combination of bidirectional A∗\nand known landmarks was used to efficiently find driving routes for Microsoft’s online map\nservice (Goldberg et al., 2006). After caching a set of paths between landmarks, the algorithm\ncan find an optimal path between any pair of points in a 24 million point graph of the United\nStates, searching less than 0.1% of the graph. Others approaches to bidirectional search\ninclude a breadth-first search backward from the goal up to a fixed depth, followed by a\nforward IDA∗search (Dillenburg and Nelson, 1994; Manzini, 1995).\nA∗and other state-space search algorithms are closely related to the branch-and-bound\ntechniques that are widely used in operations research\n(Lawler and Wood, 1966).\nThe\nrelationships between state-space search and branch-and-bound have been investigated in\ndepth (Kumar and Kanal, 1983; Nau et al., 1984; Kumar et al., 1988). Martelli and Monta-\nnari (1978) demonstrate a connection between dynamic programming (see Chapter 17) and\ncertain types of state-space search. Kumar and Kanal (1988) attempt a “grand unification” of\nheuristic search, dynamic programming, and branch-and-bound techniques under the name\nof CDP—the “composite decision process.”\nBecause computers in the late 1950s and early 1960s had at most a few thousand words\nof main memory, memory-bounded heuristic search was an early research topic. The Graph\nTraverser (Doran and Michie, 1966), one of the earliest search programs, commits to an\noperator after searching best-first up to the memory limit. IDA∗(Korf, 1985a, 1985b) was the\nfirst widely used optimal, memory-bounded heuristic search algorithm, and a large number\nof variants have been developed. An analysis of the efficiency of IDA∗and of its difficulties\nwith real-valued heuristics appears in Patrick et al. (1992).\nRBFS (Korf, 1993) is actually somewhat more complicated than the algorithm shown\nin Figure 3.26, which is closer to an independently developed algorithm called iterative ex-\npansion (Russell, 1992). RBFS uses a lower bound as well as the upper bound; the two al-\nITERATIVE\nEXPANSION\ngorithms behave identically with admissible heuristics, but RBFS expands nodes in best-first\n112\nChapter\n3.\nSolving Problems by Searching\norder even with an inadmissible heuristic. The idea of keeping track of the best alternative\npath appeared earlier in Bratko’s (1986) elegant Prolog implementation of A∗and in the DTA∗\nalgorithm (Russell and Wefald, 1991). The latter work also discusses metalevel state spaces\nand metalevel learning.\nThe MA∗algorithm appeared in Chakrabarti et al. (1989). SMA∗, or Simplified MA∗,\nemerged from an attempt to implement MA∗as a comparison algorithm for IE (Russell, 1992).\nKaindl and Khorsand (1994) have applied SMA∗to produce a bidirectional search algorithm\nthat is substantially faster than previous algorithms. Korf and Zhang (2000) describe a divide-\nand-conquer approach, and Zhou and Hansen (2002) introduce memory-bounded A∗graph\nsearch and a strategy for switching to breadth-first search to increase memory-efficiency\n(Zhou and Hansen, 2006). Korf (1995) surveys memory-bounded search techniques.\nThe idea that admissible heuristics can be derived by problem relaxation appears in the\nseminal paper by Held and Karp (1970), who used the minimum-spanning-tree heuristic to\nsolve the TSP. (See Exercise 3.30.)\nThe automation of the relaxation process was implemented successfully by Priedi-\ntis (1993), building on earlier work with Mostow (Mostow and Prieditis, 1989). Holte and\nHernadvolgyi (2001) describe more recent steps towards automating the process. The use of\npattern databases to derive admissible heuristics is due to Gasser (1995) and Culberson and\nSchaeffer (1996, 1998); disjoint pattern databases are described by Korf and Felner (2002);\na similar method using symbolic patterns is due to Edelkamp (2009). Felner et al. (2007)\nshow how to compress pattern databases to save space. The probabilistic interpretation of\nheuristics was investigated in depth by Pearl (1984) and Hansson and Mayer (1989).\nBy far the most comprehensive source on heuristics and heuristic search algorithms\nis Pearl’s (1984) Heuristics text. This book provides especially good coverage of the wide\nvariety of offshoots and variations of A∗, including rigorous proofs of their formal properties.\nKanal and Kumar (1988) present an anthology of important articles on heuristic search, and\nRayward-Smith et al. (1996) cover approaches from Operations Research. Papers about new\nsearch algorithms—which, remarkably, continue to be discovered—appear in journals such\nas Artificial Intelligence and Journal of the ACM.\nThe topic of parallel search algorithms was not covered in the chapter, partly because\nPARALLEL SEARCH\nit requires a lengthy discussion of parallel computer architectures. Parallel search became a\npopular topic in the 1990s in both AI and theoretical computer science (Mahanti and Daniels,\n1993; Grama and Kumar, 1995; Crauser et al., 1998) and is making a comeback in the era\nof new multicore and cluster architectures (Ralphs et al., 2004; Korf and Schultze, 2005).\nAlso of increasing importance are search algorithms for very large graphs that require disk\nstorage (Korf, 2008).\nEXERCISES\n3.1\nExplain why problem formulation must follow goal formulation.\n3.2\nYour goal is to navigate a robot out of a maze. The robot starts in the center of the maze\nExercises\n113\nfacing north. You can turn the robot to face north, east, south, or west. You can direct the\nrobot to move forward a certain distance, although it will stop before hitting a wall.\na. Formulate this problem. How large is the state space?\nb. In navigating a maze, the only place we need to turn is at the intersection of two or\nmore corridors. Reformulate this problem using this observation. How large is the state\nspace now?\nc. From each point in the maze, we can move in any of the four directions until we reach a\nturning point, and this is the only action we need to do. Reformulate the problem using\nthese actions. Do we need to keep track of the robot’s orientation now?\nd. In our initial description of the problem we already abstracted from the real world,\nrestricting actions and removing details. List three such simplifications we made.\n3.3\nSuppose two friends live in different cities on a map, such as the Romania map shown\nin Figure 3.2. On every turn, we can simultaneously move each friend to a neighboring city\non the map. The amount of time needed to move from city i to neighbor j is equal to the road\ndistance d(i, j) between the cities, but on each turn the friend that arrives first must wait until\nthe other one arrives (and calls the first on his/her cell phone) before the next turn can begin.\nWe want the two friends to meet as quickly as possible.\na. Write a detailed formulation for this search problem. (You will find it helpful to define\nsome formal notation here.)\nb. Let D(i, j) be the straight-line distance between cities i and j. Which of the following\nheuristic functions are admissible? (i) D(i, j); (ii) 2 · D(i, j); (iii) D(i, j)/2.\nc. Are there completely connected maps for which no solution exists?\nd. Are there maps in which all solutions require one friend to visit the same city twice?\n3.4\nShow that the 8-puzzle states are divided into two disjoint sets, such that any state is\nreachable from any other state in the same set, while no state is reachable from any state in\nthe other set. (Hint: See Berlekamp et al. (1982).) Devise a procedure to decide which set a\ngiven state is in, and explain why this is useful for generating random states.\n3.5\nConsider the n-queens problem using the “efficient” incremental formulation given on\npage 72. Explain why the state space has at least\n3√\nn! states and estimate the largest n for\nwhich exhaustive exploration is feasible. (Hint: Derive a lower bound on the branching factor\nby considering the maximum number of squares that a queen can attack in any column.)\n3.6\nGive a complete problem formulation for each of the following. Choose a formulation\nthat is precise enough to be implemented.\na. Using only four colors, you have to color a planar map in such a way that no two\nadjacent regions have the same color.\nb. A 3-foot-tall monkey is in a room where some bananas are suspended from the 8-foot\nceiling. He would like to get the bananas. The room contains two stackable, movable,\nclimbable 3-foot-high crates.\n114\nChapter\n3.\nSolving Problems by Searching\nS\nG\nFigure 3.31\nA scene with polygonal obstacles. S and G are the start and goal states.\nc. You have a program that outputs the message “illegal input record” when fed a certain\nfile of input records. You know that processing of each record is independent of the\nother records. You want to discover what record is illegal.\nd. You have three jugs, measuring 12 gallons, 8 gallons, and 3 gallons, and a water faucet.\nYou can fill the jugs up or empty them out from one to another or onto the ground. You\nneed to measure out exactly one gallon.\n3.7\nConsider the problem of finding the shortest path between two points on a plane that has\nconvex polygonal obstacles as shown in Figure 3.31. This is an idealization of the problem\nthat a robot has to solve to navigate in a crowded environment.\na. Suppose the state space consists of all positions (x, y) in the plane. How many states\nare there? How many paths are there to the goal?\nb. Explain briefly why the shortest path from one polygon vertex to any other in the scene\nmust consist of straight-line segments joining some of the vertices of the polygons.\nDefine a good state space now. How large is this state space?\nc. Define the necessary functions to implement the search problem, including an ACTIONS\nfunction that takes a vertex as input and returns a set of vectors, each of which maps the\ncurrent vertex to one of the vertices that can be reached in a straight line. (Do not forget\nthe neighbors on the same polygon.) Use the straight-line distance for the heuristic\nfunction.\nd. Apply one or more of the algorithms in this chapter to solve a range of problems in the\ndomain, and comment on their performance.\n3.8\nOn page 68, we said that we would not consider problems with negative path costs. In\nthis exercise, we explore this decision in more depth.\na. Suppose that actions can have arbitrarily large negative costs; explain why this possi-\nbility would force any optimal algorithm to explore the entire state space.\nExercises\n115\nb. Does it help if we insist that step costs must be greater than or equal to some negative\nconstant c? Consider both trees and graphs.\nc. Suppose that a set of actions forms a loop in the state space such that executing the set in\nsome order results in no net change to the state. If all of these actions have negative cost,\nwhat does this imply about the optimal behavior for an agent in such an environment?\nd. One can easily imagine actions with high negative cost, even in domains such as route\nfinding. For example, some stretches of road might have such beautiful scenery as to\nfar outweigh the normal costs in terms of time and fuel. Explain, in precise terms,\nwithin the context of state-space search, why humans do not drive around scenic loops\nindefinitely, and explain how to define the state space and actions for route finding so\nthat artificial agents can also avoid looping.\ne. Can you think of a real domain in which step costs are such as to cause looping?\n3.9\nThe missionaries and cannibals problem is usually stated as follows. Three mission-\naries and three cannibals are on one side of a river, along with a boat that can hold one or\ntwo people. Find a way to get everyone to the other side without ever leaving a group of mis-\nsionaries in one place outnumbered by the cannibals in that place. This problem is famous in\nAI because it was the subject of the first paper that approached problem formulation from an\nanalytical viewpoint (Amarel, 1968).\na. Formulate the problem precisely, making only those distinctions necessary to ensure a\nvalid solution. Draw a diagram of the complete state space.\nb. Implement and solve the problem optimally using an appropriate search algorithm. Is it\na good idea to check for repeated states?\nc. Why do you think people have a hard time solving this puzzle, given that the state space\nis so simple?\n3.10\nDefine in your own words the following terms: state, state space, search tree, search\nnode, goal, action, transition model, and branching factor.\n3.11\nWhat’s the difference between a world state, a state description, and a search node?\nWhy is this distinction useful?\n3.12\nAn action such as Go(Sibiu) really consists of a long sequence of finer-grained actions:\nturn on the car, release the brake, accelerate forward, etc. Having composite actions of this\nkind reduces the number of steps in a solution sequence, thereby reducing the search time.\nSuppose we take this to the logical extreme, by making super-composite actions out of every\npossible sequence of Go actions. Then every problem instance is solved by a single super-\ncomposite action, such as Go(Sibiu)Go(Rimnicu Vilcea)Go(Pitesti)Go(Bucharest). Explain\nhow search would work in this formulation. Is this a practical approach for speeding up\nproblem solving?\n3.13\nProve that GRAPH-SEARCH satisfies the graph separation property illustrated in Fig-\nure 3.9. (Hint: Begin by showing that the property holds at the start, then show that if it holds\nbefore an iteration of the algorithm, it holds afterwards.) Describe a search algorithm that\nviolates the property.\n116\nChapter\n3.\nSolving Problems by Searching\nx 12\nx 16\nx 2\nx 2\nFigure 3.32\nThe track pieces in a wooden railway set; each is labeled with the number of\ncopies in the set. Note that curved pieces and “fork” pieces (“switches” or “points”) can be\nflipped over so they can curve in either direction. Each curve subtends 45 degrees.\n3.14\nWhich of the following are true and which are false? Explain your answers.\na. Depth-first search always expands at least as many nodes as A∗search with an admissi-\nble heuristic.\nb. h(n) = 0 is an admissible heuristic for the 8-puzzle.\nc. A∗is of no use in robotics because percepts, states, and actions are continuous.\nd. Breadth-first search is complete even if zero step costs are allowed.\ne. Assume that a rook can move on a chessboard any number of squares in a straight line,\nvertically or horizontally, but cannot jump over other pieces. Manhattan distance is an\nadmissible heuristic for the problem of moving the rook from square A to square B in\nthe smallest number of moves.\n3.15\nConsider a state space where the start state is number 1 and each state k has two\nsuccessors: numbers 2k and 2k + 1.\na. Draw the portion of the state space for states 1 to 15.\nb. Suppose the goal state is 11. List the order in which nodes will be visited for breadth-\nfirst search, depth-limited search with limit 3, and iterative deepening search.\nc. How well would bidirectional search work on this problem? What is the branching\nfactor in each direction of the bidirectional search?\nd. Does the answer to (c) suggest a reformulation of the problem that would allow you to\nsolve the problem of getting from state 1 to a given goal state with almost no search?\ne. Call the action going from k to 2k Left, and the action going to 2k + 1 Right. Can you\nfind an algorithm that outputs the solution to this problem without any search at all?\n3.16\nA basic wooden railway set contains the pieces shown in Figure 3.32. The task is to\nconnect these pieces into a railway that has no overlapping tracks and no loose ends where a\ntrain could run off onto the floor.\na. Suppose that the pieces fit together exactly with no slack. Give a precise formulation of\nthe task as a search problem.\nb. Identify a suitable uninformed search algorithm for this task and explain your choice.\nc. Explain why removing any one of the “fork” pieces makes the problem unsolvable.\nExercises\n117\nd. Give an upper bound on the total size of the state space defined by your formulation.\n(Hint: think about the maximum branching factor for the construction process and the\nmaximum depth, ignoring the problem of overlapping pieces and loose ends. Begin by\npretending that every piece is unique.)\n3.17\nOn page 90, we mentioned iterative lengthening search, an iterative analog of uni-\nform cost search. The idea is to use increasing limits on path cost. If a node is generated\nwhose path cost exceeds the current limit, it is immediately discarded. For each new itera-\ntion, the limit is set to the lowest path cost of any node discarded in the previous iteration.\na. Show that this algorithm is optimal for general path costs.\nb. Consider a uniform tree with branching factor b, solution depth d, and unit step costs.\nHow many iterations will iterative lengthening require?\nc. Now consider step costs drawn from the continuous range [ϵ, 1], where 0 < ϵ < 1. How\nmany iterations are required in the worst case?\nd. Implement the algorithm and apply it to instances of the 8-puzzle and traveling sales-\nperson problems. Compare the algorithm’s performance to that of uniform-cost search,\nand comment on your results.\n3.18\nDescribe a state space in which iterative deepening search performs much worse than\ndepth-first search (for example, O(n2) vs. O(n)).\n3.19\nWrite a program that will take as input two Web page URLs and find a path of links\nfrom one to the other. What is an appropriate search strategy? Is bidirectional search a good\nidea? Could a search engine be used to implement a predecessor function?\n3.20\nConsider the vacuum-world problem defined in Figure 2.2.\na. Which of the algorithms defined in this chapter would be appropriate for this problem?\nShould the algorithm use tree search or graph search?\nb. Apply your chosen algorithm to compute an optimal sequence of actions for a 3 × 3\nworld whose initial state has dirt in the three top squares and the agent in the center.\nc. Construct a search agent for the vacuum world, and evaluate its performance in a set of\n3 × 3 worlds with probability 0.2 of dirt in each square. Include the search cost as well\nas path cost in the performance measure, using a reasonable exchange rate.\nd. Compare your best search agent with a simple randomized reflex agent that sucks if\nthere is dirt and otherwise moves randomly.\ne. Consider what would happen if the world were enlarged to n × n. How does the per-\nformance of the search agent and of the reflex agent vary with n?\n3.21\nProve each of the following statements, or give a counterexample:\na. Breadth-first search is a special case of uniform-cost search.\nb. Depth-first search is a special case of best-first tree search.\nc. Uniform-cost search is a special case of A∗search.\n118\nChapter\n3.\nSolving Problems by Searching\n3.22\nCompare the performance of A∗and RBFS on a set of randomly generated problems\nin the 8-puzzle (with Manhattan distance) and TSP (with MST—see Exercise 3.30) domains.\nDiscuss your results. What happens to the performance of RBFS when a small random num-\nber is added to the heuristic values in the 8-puzzle domain?\n3.23\nTrace the operation of A∗search applied to the problem of getting to Bucharest from\nLugoj using the straight-line distance heuristic. That is, show the sequence of nodes that the\nalgorithm will consider and the f, g, and h score for each node.\n3.24\nDevise a state space in which A∗using GRAPH-SEARCH returns a suboptimal solution\nwith an h(n) function that is admissible but inconsistent.\n3.25\nThe heuristic path algorithm (Pohl, 1977) is a best-first search in which the evalu-\nHEURISTIC PATH\nALGORITHM\nation function is f(n) = (2 −w)g(n) + wh(n). For what values of w is this complete?\nFor what values is it optimal, assuming that h is admissible? What kind of search does this\nperform for w = 0, w = 1, and w = 2?\n3.26\nConsider the unbounded version of the regular 2D grid shown in Figure 3.9. The start\nstate is at the origin, (0,0), and the goal state is at (x, y).\na. What is the branching factor b in this state space?\nb. How many distinct states are there at depth k (for k > 0)?\nc. What is the maximum number of nodes expanded by breadth-first tree search?\nd. What is the maximum number of nodes expanded by breadth-first graph search?\ne. Is h = |u −x| + |v −y| an admissible heuristic for a state at (u, v)? Explain.\nf. How many nodes are expanded by A∗graph search using h?\ng. Does h remain admissible if some links are removed?\nh. Does h remain admissible if some links are added between nonadjacent states?\n3.27\nn vehicles occupy squares (1, 1) through (n, 1) (i.e., the bottom row) of an n × n grid.\nThe vehicles must be moved to the top row but in reverse order; so the vehicle i that starts in\n(i, 1) must end up in (n −i + 1, n). On each time step, every one of the n vehicles can move\none square up, down, left, or right, or stay put; but if a vehicle stays put, one other adjacent\nvehicle (but not more than one) can hop over it. Two vehicles cannot occupy the same square.\na. Calculate the size of the state space as a function of n.\nb. Calculate the branching factor as a function of n.\nc. Suppose that vehicle i is at (xi, yi); write a nontrivial admissible heuristic hi for the\nnumber of moves it will require to get to its goal location (n −i + 1, n), assuming no\nother vehicles are on the grid.\nd. Which of the following heuristics are admissible for the problem of moving all n vehi-\ncles to their destinations? Explain.\n(i) \u0002n\ni = 1 hi.\n(ii) max{h1, . . . , hn}.\n(iii) min{h1, . . . , hn}.\nExercises\n119\n3.28\nInvent a heuristic function for the 8-puzzle that sometimes overestimates, and show\nhow it can lead to a suboptimal solution on a particular problem. (You can use a computer to\nhelp if you want.) Prove that if h never overestimates by more than c, A∗using h returns a\nsolution whose cost exceeds that of the optimal solution by no more than c.\n3.29\nProve that if a heuristic is consistent, it must be admissible. Construct an admissible\nheuristic that is not consistent.\n3.30\nThe traveling salesperson problem (TSP) can be solved with the minimum-spanning-\ntree (MST) heuristic, which estimates the cost of completing a tour, given that a partial tour\nhas already been constructed. The MST cost of a set of cities is the smallest sum of the link\ncosts of any tree that connects all the cities.\na. Show how this heuristic can be derived from a relaxed version of the TSP.\nb. Show that the MST heuristic dominates straight-line distance.\nc. Write a problem generator for instances of the TSP where cities are represented by\nrandom points in the unit square.\nd. Find an efficient algorithm in the literature for constructing the MST, and use it with A∗\ngraph search to solve instances of the TSP.\n3.31\nOn page 105, we defined the relaxation of the 8-puzzle in which a tile can move from\nsquare A to square B if B is blank. The exact solution of this problem defines Gaschnig’s\nheuristic (Gaschnig, 1979). Explain why Gaschnig’s heuristic is at least as accurate as h1\n(misplaced tiles), and show cases where it is more accurate than both h1 and h2 (Manhattan\ndistance). Explain how to calculate Gaschnig’s heuristic efficiently.\n3.32\nWe gave two simple heuristics for the 8-puzzle: Manhattan distance and misplaced\ntiles. Several heuristics in the literature purport to improve on this—see, for example, Nils-\nson (1971), Mostow and Prieditis (1989), and Hansson et al. (1992). Test these claims by\nimplementing the heuristics and comparing the performance of the resulting algorithms.\n" }, { "chapter": 4, "text": "4\nBEYOND CLASSICAL\nSEARCH\nIn which we relax the simplifying assumptions of the previous chapter, thereby\ngetting closer to the real world.\nChapter 3 addressed a single category of problems: observable, deterministic, known envi-\nronments where the solution is a sequence of actions. In this chapter, we look at what happens\nwhen these assumptions are relaxed. We begin with a fairly simple case: Sections 4.1 and 4.2\ncover algorithms that perform purely local search in the state space, evaluating and modify-\ning one or more current states rather than systematically exploring paths from an initial state.\nThese algorithms are suitable for problems in which all that matters is the solution state, not\nthe path cost to reach it. The family of local search algorithms includes methods inspired by\nstatistical physics (simulated annealing) and evolutionary biology (genetic algorithms).\nThen, in Sections 4.3–4.4, we examine what happens when we relax the assumptions\nof determinism and observability. The key idea is that if an agent cannot predict exactly what\npercept it will receive, then it will need to consider what to do under each contingency that\nits percepts may reveal. With partial observability, the agent will also need to keep track of\nthe states it might be in.\nFinally, Section 4.5 investigates online search, in which the agent is faced with a state\nspace that is initially unknown and must be explored.\n4.1\nLOCAL SEARCH ALGORITHMS AND OPTIMIZATION PROBLEMS\nThe search algorithms that we have seen so far are designed to explore search spaces sys-\ntematically. This systematicity is achieved by keeping one or more paths in memory and by\nrecording which alternatives have been explored at each point along the path. When a goal is\nfound, the path to that goal also constitutes a solution to the problem. In many problems, how-\never, the path to the goal is irrelevant. For example, in the 8-queens problem (see page 71),\nwhat matters is the final configuration of queens, not the order in which they are added. The\nsame general property holds for many important applications such as integrated-circuit de-\nsign, factory-floor layout, job-shop scheduling, automatic programming, telecommunications\nnetwork optimization, vehicle routing, and portfolio management.\n120\nSection 4.1.\nLocal Search Algorithms and Optimization Problems\n121\nIf the path to the goal does not matter, we might consider a different class of algo-\nrithms, ones that do not worry about paths at all. Local search algorithms operate using\nLOCAL SEARCH\na single current node (rather than multiple paths) and generally move only to neighbors\nCURRENT NODE\nof that node. Typically, the paths followed by the search are not retained. Although local\nsearch algorithms are not systematic, they have two key advantages: (1) they use very little\nmemory—usually a constant amount; and (2) they can often find reasonable solutions in large\nor infinite (continuous) state spaces for which systematic algorithms are unsuitable.\nIn addition to finding goals, local search algorithms are useful for solving pure op-\ntimization problems, in which the aim is to find the best state according to an objective\nOPTIMIZATION\nPROBLEM\nfunction. Many optimization problems do not fit the “standard” search model introduced in\nOBJECTIVE\nFUNCTION\nChapter 3. For example, nature provides an objective function—reproductive fitness—that\nDarwinian evolution could be seen as attempting to optimize, but there is no “goal test” and\nno “path cost” for this problem.\nTo understand local search, we find it useful to consider the state-space landscape (as\nSTATE-SPACE\nLANDSCAPE\nin Figure 4.1). A landscape has both “location” (defined by the state) and “elevation” (defined\nby the value of the heuristic cost function or objective function). If elevation corresponds to\ncost, then the aim is to find the lowest valley—a global minimum; if elevation corresponds\nGLOBAL MINIMUM\nto an objective function, then the aim is to find the highest peak—a global maximum. (You\nGLOBAL MAXIMUM\ncan convert from one to the other just by inserting a minus sign.) Local search algorithms\nexplore this landscape. A complete local search algorithm always finds a goal if one exists;\nan optimal algorithm always finds a global minimum/maximum.\ncurrent\nstate\nobjective function\nstate space\nglobal maximum\nlocal maximum\n“flat” local maximum\nshoulder\nFigure 4.1\nA one-dimensional state-space landscape in which elevation corresponds to the\nobjective function. The aim is to find the global maximum. Hill-climbing search modifies\nthe current state to try to improve it, as shown by the arrow. The various topographic features\nare defined in the text.\n122\nChapter\n4.\nBeyond Classical Search\nfunction HILL-CLIMBING(problem) returns a state that is a local maximum\ncurrent ←MAKE-NODE(problem.INITIAL-STATE)\nloop do\nneighbor ←a highest-valued successor of current\nif neighbor.VALUE ≤current.VALUE then return current.STATE\ncurrent ←neighbor\nFigure 4.2\nThe hill-climbing search algorithm, which is the most basic local search tech-\nnique. At each step the current node is replaced by the best neighbor; in this version, that\nmeans the neighbor with the highest VALUE, but if a heuristic cost estimate h is used, we\nwould find the neighbor with the lowest h.\n4.1.1\nHill-climbing search\nThe hill-climbing search algorithm (steepest-ascent version) is shown in Figure 4.2. It is\nHILL CLIMBING\nSTEEPEST ASCENT\nsimply a loop that continually moves in the direction of increasing value—that is, uphill. It\nterminates when it reaches a “peak” where no neighbor has a higher value. The algorithm\ndoes not maintain a search tree, so the data structure for the current node need only record\nthe state and the value of the objective function. Hill climbing does not look ahead beyond\nthe immediate neighbors of the current state. This resembles trying to find the top of Mount\nEverest in a thick fog while suffering from amnesia.\nTo illustrate hill climbing, we will use the 8-queens problem introduced on page 71.\nLocal search algorithms typically use a complete-state formulation, where each state has\n8 queens on the board, one per column. The successors of a state are all possible states\ngenerated by moving a single queen to another square in the same column (so each state has\n8 × 7 = 56 successors). The heuristic cost function h is the number of pairs of queens that\nare attacking each other, either directly or indirectly. The global minimum of this function\nis zero, which occurs only at perfect solutions. Figure 4.3(a) shows a state with h = 17. The\nfigure also shows the values of all its successors, with the best successors having h = 12.\nHill-climbing algorithms typically choose randomly among the set of best successors if there\nis more than one.\nHill climbing is sometimes called greedy local search because it grabs a good neighbor\nGREEDY LOCAL\nSEARCH\nstate without thinking ahead about where to go next. Although greed is considered one of the\nseven deadly sins, it turns out that greedy algorithms often perform quite well. Hill climbing\noften makes rapid progress toward a solution because it is usually quite easy to improve a bad\nstate. For example, from the state in Figure 4.3(a), it takes just five steps to reach the state\nin Figure 4.3(b), which has h = 1 and is very nearly a solution. Unfortunately, hill climbing\noften gets stuck for the following reasons:\n• Local maxima: a local maximum is a peak that is higher than each of its neighboring\nLOCAL MAXIMUM\nstates but lower than the global maximum. Hill-climbing algorithms that reach the\nvicinity of a local maximum will be drawn upward toward the peak but will then be\nstuck with nowhere else to go. Figure 4.1 illustrates the problem schematically. More\nSection 4.1.\nLocal Search Algorithms and Optimization Problems\n123\n14\n18\n17\n15\n14\n18\n14\n14\n14\n14\n14\n12\n16\n12\n13\n16\n17\n14\n18\n13\n14\n17\n15\n18\n15\n13\n15\n13\n12\n15\n15\n13\n15\n12\n13\n14\n14\n14\n16\n12\n14\n12\n12\n15\n16\n13\n14\n12\n14\n18\n16\n16\n16\n14\n16\n14\n(a)\n(b)\nFigure 4.3\n(a) An 8-queens state with heuristic cost estimate h = 17, showing the value of\nh for each possible successor obtained by moving a queen within its column. The best moves\nare marked. (b) A local minimum in the 8-queens state space; the state has h = 1 but every\nsuccessor has a higher cost.\nconcretely, the state in Figure 4.3(b) is a local maximum (i.e., a local minimum for the\ncost h); every move of a single queen makes the situation worse.\n• Ridges: a ridge is shown in Figure 4.4. Ridges result in a sequence of local maxima\nRIDGE\nthat is very difficult for greedy algorithms to navigate.\n• Plateaux: a plateau is a flat area of the state-space landscape. It can be a flat local\nPLATEAU\nmaximum, from which no uphill exit exists, or a shoulder, from which progress is\nSHOULDER\npossible. (See Figure 4.1.) A hill-climbing search might get lost on the plateau.\nIn each case, the algorithm reaches a point at which no progress is being made. Starting from\na randomly generated 8-queens state, steepest-ascent hill climbing gets stuck 86% of the time,\nsolving only 14% of problem instances. It works quickly, taking just 4 steps on average when\nit succeeds and 3 when it gets stuck—not bad for a state space with 88 ≈17 million states.\nThe algorithm in Figure 4.2 halts if it reaches a plateau where the best successor has\nthe same value as the current state. Might it not be a good idea to keep going—to allow a\nsideways move in the hope that the plateau is really a shoulder, as shown in Figure 4.1? The\nSIDEWAYS MOVE\nanswer is usually yes, but we must take care. If we always allow sideways moves when there\nare no uphill moves, an infinite loop will occur whenever the algorithm reaches a flat local\nmaximum that is not a shoulder. One common solution is to put a limit on the number of con-\nsecutive sideways moves allowed. For example, we could allow up to, say, 100 consecutive\nsideways moves in the 8-queens problem. This raises the percentage of problem instances\nsolved by hill climbing from 14% to 94%. Success comes at a cost: the algorithm averages\nroughly 21 steps for each successful instance and 64 for each failure.\n124\nChapter\n4.\nBeyond Classical Search\nFigure 4.4\nIllustration of why ridges cause difficulties for hill climbing. The grid of states\n(dark circles) is superimposed on a ridge rising from left to right, creating a sequence of local\nmaxima that are not directly connected to each other. From each local maximum, all the\navailable actions point downhill.\nMany variants of hill climbing have been invented. Stochastic hill climbing chooses at\nSTOCHASTIC HILL\nCLIMBING\nrandom from among the uphill moves; the probability of selection can vary with the steepness\nof the uphill move. This usually converges more slowly than steepest ascent, but in some\nstate landscapes, it finds better solutions. First-choice hill climbing implements stochastic\nFIRST-CHOICE HILL\nCLIMBING\nhill climbing by generating successors randomly until one is generated that is better than the\ncurrent state. This is a good strategy when a state has many (e.g., thousands) of successors.\nThe hill-climbing algorithms described so far are incomplete—they often fail to find\na goal when one exists because they can get stuck on local maxima. Random-restart hill\nclimbing adopts the well-known adage, “If at first you don’t succeed, try, try again.” It con-\nRANDOM-RESTART\nHILL CLIMBING\nducts a series of hill-climbing searches from randomly generated initial states,1 until a goal\nis found. It is trivially complete with probability approaching 1, because it will eventually\ngenerate a goal state as the initial state. If each hill-climbing search has a probability p of\nsuccess, then the expected number of restarts required is 1/p. For 8-queens instances with\nno sideways moves allowed, p ≈0.14, so we need roughly 7 iterations to find a goal (6 fail-\nures and 1 success). The expected number of steps is the cost of one successful iteration plus\n(1−p)/p times the cost of failure, or roughly 22 steps in all. When we allow sideways moves,\n1/0.94 ≈1.06 iterations are needed on average and (1 × 21) + (0.06/0.94) × 64 ≈25 steps.\nFor 8-queens, then, random-restart hill climbing is very effective indeed. Even for three mil-\nlion queens, the approach can find solutions in under a minute.2\n1 Generating a random state from an implicitly specified state space can be a hard problem in itself.\n2 Luby et al. (1993) prove that it is best, in some cases, to restart a randomized search algorithm after a particular,\nfixed amount of time and that this can be much more efficient than letting each search continue indefinitely.\nDisallowing or limiting the number of sideways moves is an example of this idea.\nSection 4.1.\nLocal Search Algorithms and Optimization Problems\n125\nThe success of hill climbing depends very much on the shape of the state-space land-\nscape: if there are few local maxima and plateaux, random-restart hill climbing will find a\ngood solution very quickly. On the other hand, many real problems have a landscape that\nlooks more like a widely scattered family of balding porcupines on a flat floor, with miniature\nporcupines living on the tip of each porcupine needle, ad infinitum. NP-hard problems typi-\ncally have an exponential number of local maxima to get stuck on. Despite this, a reasonably\ngood local maximum can often be found after a small number of restarts.\n4.1.2\nSimulated annealing\nA hill-climbing algorithm that never makes “downhill” moves toward states with lower value\n(or higher cost) is guaranteed to be incomplete, because it can get stuck on a local maxi-\nmum. In contrast, a purely random walk—that is, moving to a successor chosen uniformly\nat random from the set of successors—is complete but extremely inefficient. Therefore, it\nseems reasonable to try to combine hill climbing with a random walk in some way that yields\nboth efficiency and completeness. Simulated annealing is such an algorithm. In metallurgy,\nSIMULATED\nANNEALING\nannealing is the process used to temper or harden metals and glass by heating them to a\nhigh temperature and then gradually cooling them, thus allowing the material to reach a low-\nenergy crystalline state. To explain simulated annealing, we switch our point of view from\nhill climbing to gradient descent (i.e., minimizing cost) and imagine the task of getting a\nGRADIENT DESCENT\nping-pong ball into the deepest crevice in a bumpy surface. If we just let the ball roll, it will\ncome to rest at a local minimum. If we shake the surface, we can bounce the ball out of the\nlocal minimum. The trick is to shake just hard enough to bounce the ball out of local min-\nima but not hard enough to dislodge it from the global minimum. The simulated-annealing\nsolution is to start by shaking hard (i.e., at a high temperature) and then gradually reduce the\nintensity of the shaking (i.e., lower the temperature).\nThe innermost loop of the simulated-annealing algorithm (Figure 4.5) is quite similar to\nhill climbing. Instead of picking the best move, however, it picks a random move. If the move\nimproves the situation, it is always accepted. Otherwise, the algorithm accepts the move with\nsome probability less than 1. The probability decreases exponentially with the “badness” of\nthe move—the amount ΔE by which the evaluation is worsened. The probability also de-\ncreases as the “temperature” T goes down: “bad” moves are more likely to be allowed at the\nstart when T is high, and they become more unlikely as T decreases. If the schedule lowers\nT slowly enough, the algorithm will find a global optimum with probability approaching 1.\nSimulated annealing was first used extensively to solve VLSI layout problems in the\nearly 1980s. It has been applied widely to factory scheduling and other large-scale optimiza-\ntion tasks. In Exercise 4.4, you are asked to compare its performance to that of random-restart\nhill climbing on the 8-queens puzzle.\n4.1.3\nLocal beam search\nKeeping just one node in memory might seem to be an extreme reaction to the problem of\nmemory limitations. The local beam search algorithm3 keeps track of k states rather than\nLOCAL BEAM\nSEARCH\n3 Local beam search is an adaptation of beam search, which is a path-based algorithm.\n126\nChapter\n4.\nBeyond Classical Search\nfunction SIMULATED-ANNEALING(problem,schedule) returns a solution state\ninputs: problem, a problem\nschedule, a mapping from time to “temperature”\ncurrent ←MAKE-NODE(problem.INITIAL-STATE)\nfor t = 1 to ∞do\nT ←schedule(t)\nif T = 0 then return current\nnext ←a randomly selected successor of current\nΔE ←next.VALUE – current.VALUE\nif ΔE > 0 then current ←next\nelse current ←next only with probability eΔE/T\nFigure 4.5\nThe simulated annealing algorithm, a version of stochastic hill climbing where\nsome downhill moves are allowed. Downhill moves are accepted readily early in the anneal-\ning schedule and then less often as time goes on. The schedule input determines the value of\nthe temperature T as a function of time.\njust one. It begins with k randomly generated states. At each step, all the successors of all k\nstates are generated. If any one is a goal, the algorithm halts. Otherwise, it selects the k best\nsuccessors from the complete list and repeats.\nAt first sight, a local beam search with k states might seem to be nothing more than\nrunning k random restarts in parallel instead of in sequence. In fact, the two algorithms\nare quite different. In a random-restart search, each search process runs independently of\nthe others. In a local beam search, useful information is passed among the parallel search\nthreads. In effect, the states that generate the best successors say to the others, “Come over\nhere, the grass is greener!” The algorithm quickly abandons unfruitful searches and moves\nits resources to where the most progress is being made.\nIn its simplest form, local beam search can suffer from a lack of diversity among the\nk states—they can quickly become concentrated in a small region of the state space, making\nthe search little more than an expensive version of hill climbing. A variant called stochastic\nbeam search, analogous to stochastic hill climbing, helps alleviate this problem. Instead\nSTOCHASTIC BEAM\nSEARCH\nof choosing the best k from the the pool of candidate successors, stochastic beam search\nchooses k successors at random, with the probability of choosing a given successor being\nan increasing function of its value. Stochastic beam search bears some resemblance to the\nprocess of natural selection, whereby the “successors” (offspring) of a “state” (organism)\npopulate the next generation according to its “value” (fitness).\n4.1.4\nGenetic algorithms\nA genetic algorithm (or GA) is a variant of stochastic beam search in which successor states\nGENETIC\nALGORITHM\nare generated by combining two parent states rather than by modifying a single state. The\nanalogy to natural selection is the same as in stochastic beam search, except that now we are\ndealing with sexual rather than asexual reproduction.\nSection 4.1.\nLocal Search Algorithms and Optimization Problems\n127\n(a)\nInitial Population\n(b)\nFitness Function\n(c)\nSelection\n(d)\nCrossover\n(e)\nMutation\n24\n23\n20\n11\n29%\n31%\n26%\n14%\n32752411\n24748552\n32752411\n24415124\n32748552\n24752411\n32752124\n24415411\n32252124\n24752411\n32748152\n24415417\n24748552\n32752411\n24415124\n32543213\nFigure 4.6\nThe genetic algorithm, illustrated for digit strings representing 8-queens states.\nThe initial population in (a) is ranked by the fitness function in (b), resulting in pairs for\nmating in (c). They produce offspring in (d), which are subject to mutation in (e).\n+\n=\nFigure 4.7\nThe 8-queens states corresponding to the first two parents in Figure 4.6(c) and\nthe first offspring in Figure 4.6(d). The shaded columns are lost in the crossover step and the\nunshaded columns are retained.\nLike beam searches, GAs begin with a set of k randomly generated states, called the\npopulation. Each state, or individual, is represented as a string over a finite alphabet—most\nPOPULATION\nINDIVIDUAL\ncommonly, a string of 0s and 1s. For example, an 8-queens state must specify the positions of\n8 queens, each in a column of 8 squares, and so requires 8 × log2 8 = 24 bits. Alternatively,\nthe state could be represented as 8 digits, each in the range from 1 to 8. (We demonstrate later\nthat the two encodings behave differently.) Figure 4.6(a) shows a population of four 8-digit\nstrings representing 8-queens states.\nThe production of the next generation of states is shown in Figure 4.6(b)–(e). In (b),\neach state is rated by the objective function, or (in GA terminology) the fitness function. A\nFITNESS FUNCTION\nfitness function should return higher values for better states, so, for the 8-queens problem\nwe use the number of nonattacking pairs of queens, which has a value of 28 for a solution.\nThe values of the four states are 24, 23, 20, and 11. In this particular variant of the genetic\nalgorithm, the probability of being chosen for reproducing is directly proportional to the\nfitness score, and the percentages are shown next to the raw scores.\nIn (c), two pairs are selected at random for reproduction, in accordance with the prob-\n128\nChapter\n4.\nBeyond Classical Search\nabilities in (b). Notice that one individual is selected twice and one not at all.4 For each\npair to be mated, a crossover point is chosen randomly from the positions in the string. In\nCROSSOVER\nFigure 4.6, the crossover points are after the third digit in the first pair and after the fifth digit\nin the second pair.5\nIn (d), the offspring themselves are created by crossing over the parent strings at the\ncrossover point. For example, the first child of the first pair gets the first three digits from the\nfirst parent and the remaining digits from the second parent, whereas the second child gets\nthe first three digits from the second parent and the rest from the first parent. The 8-queens\nstates involved in this reproduction step are shown in Figure 4.7. The example shows that\nwhen two parent states are quite different, the crossover operation can produce a state that is\na long way from either parent state. It is often the case that the population is quite diverse\nearly on in the process, so crossover (like simulated annealing) frequently takes large steps in\nthe state space early in the search process and smaller steps later on when most individuals\nare quite similar.\nFinally, in (e), each location is subject to random mutation with a small independent\nMUTATION\nprobability. One digit was mutated in the first, third, and fourth offspring. In the 8-queens\nproblem, this corresponds to choosing a queen at random and moving it to a random square\nin its column. Figure 4.8 describes an algorithm that implements all these steps.\nLike stochastic beam search, genetic algorithms combine an uphill tendency with ran-\ndom exploration and exchange of information among parallel search threads. The primary\nadvantage, if any, of genetic algorithms comes from the crossover operation. Yet it can be\nshown mathematically that, if the positions of the genetic code are permuted initially in a\nrandom order, crossover conveys no advantage. Intuitively, the advantage comes from the\nability of crossover to combine large blocks of letters that have evolved independently to per-\nform useful functions, thus raising the level of granularity at which the search operates. For\nexample, it could be that putting the first three queens in positions 2, 4, and 6 (where they do\nnot attack each other) constitutes a useful block that can be combined with other blocks to\nconstruct a solution.\nThe theory of genetic algorithms explains how this works using the idea of a schema,\nSCHEMA\nwhich is a substring in which some of the positions can be left unspecified. For example,\nthe schema 246***** describes all 8-queens states in which the first three queens are in\npositions 2, 4, and 6, respectively. Strings that match the schema (such as 24613578) are\ncalled instances of the schema. It can be shown that if the average fitness of the instances of\nINSTANCE\na schema is above the mean, then the number of instances of the schema within the population\nwill grow over time. Clearly, this effect is unlikely to be significant if adjacent bits are totally\nunrelated to each other, because then there will be few contiguous blocks that provide a\nconsistent benefit. Genetic algorithms work best when schemata correspond to meaningful\ncomponents of a solution. For example, if the string is a representation of an antenna, then the\nschemata may represent components of the antenna, such as reflectors and deflectors. A good\n4 There are many variants of this selection rule. The method of culling, in which all individuals below a given\nthreshold are discarded, can be shown to converge faster than the random version (Baum et al., 1995).\n5 It is here that the encoding matters. If a 24-bit encoding is used instead of 8 digits, then the crossover point\nhas a 2/3 chance of being in the middle of a digit, which results in an essentially arbitrary mutation of that digit.\nSection 4.2.\nLocal Search in Continuous Spaces\n129\nfunction GENETIC-ALGORITHM(population, FITNESS-FN) returns an individual\ninputs: population, a set of individuals\nFITNESS-FN, a function that measures the fitness of an individual\nrepeat\nnew population ←empty set\nfor i = 1 to SIZE(population) do\nx ←RANDOM-SELECTION(population, FITNESS-FN)\ny ←RANDOM-SELECTION(population, FITNESS-FN)\nchild ←REPRODUCE(x,y)\nif (small random probability) then child ←MUTATE(child)\nadd child to new population\npopulation ←new population\nuntil some individual is fit enough, or enough time has elapsed\nreturn the best individual in population, according to FITNESS-FN\nfunction REPRODUCE(x,y) returns an individual\ninputs: x,y, parent individuals\nn ←LENGTH(x); c ←random number from 1 to n\nreturn APPEND(SUBSTRING(x,1,c), SUBSTRING(y,c + 1,n))\nFigure 4.8\nA genetic algorithm. The algorithm is the same as the one diagrammed in\nFigure 4.6, with one variation: in this more popular version, each mating of two parents\nproduces only one offspring, not two.\ncomponent is likely to be good in a variety of different designs. This suggests that successful\nuse of genetic algorithms requires careful engineering of the representation.\nIn practice, genetic algorithms have had a widespread impact on optimization problems,\nsuch as circuit layout and job-shop scheduling. At present, it is not clear whether the appeal\nof genetic algorithms arises from their performance or from their æsthetically pleasing origins\nin the theory of evolution. Much work remains to be done to identify the conditions under\nwhich genetic algorithms perform well.\n4.2\nLOCAL SEARCH IN CONTINUOUS SPACES\nIn Chapter 2, we explained the distinction between discrete and continuous environments,\npointing out that most real-world environments are continuous. Yet none of the algorithms\nwe have described (except for first-choice hill climbing and simulated annealing) can handle\ncontinuous state and action spaces, because they have infinite branching factors. This section\nprovides a very brief introduction to some local search techniques for finding optimal solu-\ntions in continuous spaces. The literature on this topic is vast; many of the basic techniques\n130\nChapter\n4.\nBeyond Classical Search\nEVOLUTION AND SEARCH\nThe theory of evolution was developed in Charles Darwin’s On the Origin of\nSpecies by Means of Natural Selection (1859) and independently by Alfred Russel\nWallace (1858). The central idea is simple: variations occur in reproduction and\nwill be preserved in successive generations approximately in proportion to their\neffect on reproductive fitness.\nDarwin’s theory was developed with no knowledge of how the traits of organ-\nisms can be inherited and modified. The probabilistic laws governing these pro-\ncesses were first identified by Gregor Mendel (1866), a monk who experimented\nwith sweet peas. Much later, Watson and Crick (1953) identified the structure of the\nDNA molecule and its alphabet, AGTC (adenine, guanine, thymine, cytosine). In\nthe standard model, variation occurs both by point mutations in the letter sequence\nand by “crossover” (in which the DNA of an offspring is generated by combining\nlong sections of DNA from each parent).\nThe analogy to local search algorithms has already been described; the princi-\npal difference between stochastic beam search and evolution is the use of sexual re-\nproduction, wherein successors are generated from multiple organisms rather than\njust one. The actual mechanisms of evolution are, however, far richer than most\ngenetic algorithms allow. For example, mutations can involve reversals, duplica-\ntions, and movement of large chunks of DNA; some viruses borrow DNA from one\norganism and insert it in another; and there are transposable genes that do nothing\nbut copy themselves many thousands of times within the genome. There are even\ngenes that poison cells from potential mates that do not carry the gene, thereby in-\ncreasing their own chances of replication. Most important is the fact that the genes\nthemselves encode the mechanisms whereby the genome is reproduced and trans-\nlated into an organism. In genetic algorithms, those mechanisms are a separate\nprogram that is not represented within the strings being manipulated.\nDarwinian evolution may appear inefficient, having generated blindly some\n1045 or so organisms without improving its search heuristics one iota.\nFifty\nyears before Darwin, however, the otherwise great French naturalist Jean Lamarck\n(1809) proposed a theory of evolution whereby traits acquired by adaptation dur-\ning an organism’s lifetime would be passed on to its offspring. Such a process\nwould be effective but does not seem to occur in nature. Much later, James Bald-\nwin (1896) proposed a superficially similar theory: that behavior learned during an\norganism’s lifetime could accelerate the rate of evolution. Unlike Lamarck’s, Bald-\nwin’s theory is entirely consistent with Darwinian evolution because it relies on se-\nlection pressures operating on individuals that have found local optima among the\nset of possible behaviors allowed by their genetic makeup. Computer simulations\nconfirm that the “Baldwin effect” is real, once “ordinary” evolution has created\norganisms whose internal performance measure correlates with actual fitness.\nSection 4.2.\nLocal Search in Continuous Spaces\n131\noriginated in the 17th century, after the development of calculus by Newton and Leibniz.6 We\nfind uses for these techniques at several places in the book, including the chapters on learning,\nvision, and robotics.\nWe begin with an example. Suppose we want to place three new airports anywhere\nin Romania, such that the sum of squared distances from each city on the map (Figure 3.2)\nto its nearest airport is minimized. The state space is then defined by the coordinates of\nthe airports: (x1, y1), (x2, y2), and (x3, y3). This is a six-dimensional space; we also say\nthat states are defined by six variables. (In general, states are defined by an n-dimensional\nVARIABLE\nvector of variables, x.) Moving around in this space corresponds to moving one or more of\nthe airports on the map. The objective function f(x1, y1, x2, y2, x3, y3) is relatively easy to\ncompute for any particular state once we compute the closest cities. Let Ci be the set of\ncities whose closest airport (in the current state) is airport i. Then, in the neighborhood of the\ncurrent state, where the Cis remain constant, we have\nf(x1, y1, x2, y2, x3, y3) =\n3\n\f\ni = 1\n\f\nc∈Ci\n(xi −xc)2 + (yi −yc)2 .\n(4.1)\nThis expression is correct locally, but not globally because the sets Ci are (discontinuous)\nfunctions of the state.\nOne way to avoid continuous problems is simply to discretize the neighborhood of each\nDISCRETIZATION\nstate. For example, we can move only one airport at a time in either the x or y direction by\na fixed amount ±δ. With 6 variables, this gives 12 possible successors for each state. We\ncan then apply any of the local search algorithms described previously. We could also ap-\nply stochastic hill climbing and simulated annealing directly, without discretizing the space.\nThese algorithms choose successors randomly, which can be done by generating random vec-\ntors of length δ.\nMany methods attempt to use the gradient of the landscape to find a maximum. The\nGRADIENT\ngradient of the objective function is a vector ∇f that gives the magnitude and direction of the\nsteepest slope. For our problem, we have\n∇f =\n\r ∂f\n∂x1\n, ∂f\n∂y1\n, ∂f\n∂x2\n, ∂f\n∂y2\n, ∂f\n∂x3\n, ∂f\n∂y3\n\u000e\n.\nIn some cases, we can find a maximum by solving the equation ∇f = 0. (This could be done,\nfor example, if we were placing just one airport; the solution is the arithmetic mean of all the\ncities’ coordinates.) In many cases, however, this equation cannot be solved in closed form.\nFor example, with three airports, the expression for the gradient depends on what cities are\nclosest to each airport in the current state. This means we can compute the gradient locally\n(but not globally); for example,\n∂f\n∂x1\n= 2\n\f\nc∈C1\n(xi −xc) .\n(4.2)\nGiven a locally correct expression for the gradient, we can perform steepest-ascent hill climb-\n6 A basic knowledge of multivariate calculus and vector arithmetic is useful for reading this section.\n132\nChapter\n4.\nBeyond Classical Search\ning by updating the current state according to the formula\nx ←x + α∇f(x) ,\nwhere α is a small constant often called the step size. In other cases, the objective function\nSTEP SIZE\nmight not be available in a differentiable form at all—for example, the value of a particular set\nof airport locations might be determined by running some large-scale economic simulation\npackage. In those cases, we can calculate a so-called empirical gradient by evaluating the\nEMPIRICAL\nGRADIENT\nresponse to small increments and decrements in each coordinate. Empirical gradient search\nis the same as steepest-ascent hill climbing in a discretized version of the state space.\nHidden beneath the phrase “α is a small constant” lies a huge variety of methods for\nadjusting α. The basic problem is that, if α is too small, too many steps are needed; if α\nis too large, the search could overshoot the maximum. The technique of line search tries to\nLINE SEARCH\novercome this dilemma by extending the current gradient direction—usually by repeatedly\ndoubling α—until f starts to decrease again. The point at which this occurs becomes the new\ncurrent state. There are several schools of thought about how the new direction should be\nchosen at this point.\nFor many problems, the most effective algorithm is the venerable Newton–Raphson\nNEWTON–RAPHSON\nmethod. This is a general technique for finding roots of functions—that is, solving equations\nof the form g(x) = 0. It works by computing a new estimate for the root x according to\nNewton’s formula\nx ←x −g(x)/g′(x) .\nTo find a maximum or minimum of f, we need to find x such that the gradient is zero (i.e.,\n∇f(x) = 0). Thus, g(x) in Newton’s formula becomes ∇f(x), and the update equation can\nbe written in matrix–vector form as\nx ←x −H−1\nf (x)∇f(x) ,\nwhere Hf(x) is the Hessian matrix of second derivatives, whose elements Hij are given\nHESSIAN\nby ∂2f/∂xi∂xj. For our airport example, we can see from Equation (4.2) that Hf(x) is\nparticularly simple: the off-diagonal elements are zero and the diagonal elements for airport\ni are just twice the number of cities in Ci. A moment’s calculation shows that one step of\nthe update moves airport i directly to the centroid of Ci, which is the minimum of the local\nexpression for f from Equation (4.1).7 For high-dimensional problems, however, computing\nthe n2 entries of the Hessian and inverting it may be expensive, so many approximate versions\nof the Newton–Raphson method have been developed.\nLocal search methods suffer from local maxima, ridges, and plateaux in continuous\nstate spaces just as much as in discrete spaces. Random restarts and simulated annealing can\nbe used and are often helpful. High-dimensional continuous spaces are, however, big places\nin which it is easy to get lost.\nA final topic with which a passing acquaintance is useful is constrained optimization.\nCONSTRAINED\nOPTIMIZATION\nAn optimization problem is constrained if solutions must satisfy some hard constraints on the\nvalues of the variables. For example, in our airport-siting problem, we might constrain sites\n7 In general, the Newton–Raphson update can be seen as fitting a quadratic surface to f at x and then moving\ndirectly to the minimum of that surface—which is also the minimum of f if f is quadratic.\nSection 4.3.\nSearching with Nondeterministic Actions\n133\nto be inside Romania and on dry land (rather than in the middle of lakes). The difficulty of\nconstrained optimization problems depends on the nature of the constraints and the objective\nfunction. The best-known category is that of linear programming problems, in which con-\nLINEAR\nPROGRAMMING\nstraints must be linear inequalities forming a convex set 8 and the objective function is also\nCONVEX SET\nlinear. The time complexity of linear programming is polynomial in the number of variables.\nLinear programming is probably the most widely studied and broadly useful class of\noptimization problems. It is a special case of the more general problem of convex opti-\nmization, which allows the constraint region to be any convex region and the objective to\nCONVEX\nOPTIMIZATION\nbe any function that is convex within the constraint region. Under certain conditions, convex\noptimization problems are also polynomially solvable and may be feasible in practice with\nthousands of variables. Several important problems in machine learning and control theory\ncan be formulated as convex optimization problems (see Chapter 20).\n4.3\nSEARCHING WITH NONDETERMINISTIC ACTIONS\nIn Chapter 3, we assumed that the environment is fully observable and deterministic and that\nthe agent knows what the effects of each action are. Therefore, the agent can calculate exactly\nwhich state results from any sequence of actions and always knows which state it is in. Its\npercepts provide no new information after each action, although of course they tell the agent\nthe initial state.\nWhen the environment is either partially observable or nondeterministic (or both), per-\ncepts become useful. In a partially observable environment, every percept helps narrow down\nthe set of possible states the agent might be in, thus making it easier for the agent to achieve\nits goals. When the environment is nondeterministic, percepts tell the agent which of the pos-\nsible outcomes of its actions has actually occurred. In both cases, the future percepts cannot\nbe determined in advance and the agent’s future actions will depend on those future percepts.\nSo the solution to a problem is not a sequence but a contingency plan (also known as a strat-\nCONTINGENCY PLAN\negy) that specifies what to do depending on what percepts are received. In this section, we\nSTRATEGY\nexamine the case of nondeterminism, deferring partial observability to Section 4.4.\n4.3.1\nThe erratic vacuum world\nAs an example, we use the vacuum world, first introduced in Chapter 2 and defined as a\nsearch problem in Section 3.2.1. Recall that the state space has eight states, as shown in\nFigure 4.9. There are three actions—Left, Right, and Suck—and the goal is to clean up all\nthe dirt (states 7 and 8). If the environment is observable, deterministic, and completely\nknown, then the problem is trivially solvable by any of the algorithms in Chapter 3 and the\nsolution is an action sequence. For example, if the initial state is 1, then the action sequence\n[Suck,Right,Suck] will reach a goal state, 8.\n8 A set of points S is convex if the line joining any two points in S is also contained in S. A convex function is\none for which the space “above” it forms a convex set; by definition, convex functions have no local (as opposed\nto global) minima.\n134\nChapter\n4.\nBeyond Classical Search\n1\n2\n8\n7\n5\n6\n3\n4\nFigure 4.9\nThe eight possible states of the vacuum world; states 7 and 8 are goal states.\nNow suppose that we introduce nondeterminism in the form of a powerful but erratic\nvacuum cleaner. In the erratic vacuum world, the Suck action works as follows:\nERRATIC VACUUM\nWORLD\n• When applied to a dirty square the action cleans the square and sometimes cleans up\ndirt in an adjacent square, too.\n• When applied to a clean square the action sometimes deposits dirt on the carpet.9\nTo provide a precise formulation of this problem, we need to generalize the notion of a tran-\nsition model from Chapter 3. Instead of defining the transition model by a RESULT function\nthat returns a single state, we use a RESULTS function that returns a set of possible outcome\nstates. For example, in the erratic vacuum world, the Suck action in state 1 leads to a state in\nthe set {5, 7}—the dirt in the right-hand square may or may not be vacuumed up.\nWe also need to generalize the notion of a solution to the problem. For example, if we\nstart in state 1, there is no single sequence of actions that solves the problem. Instead, we\nneed a contingency plan such as the following:\n[Suck, if State = 5 then [Right, Suck] else [ ]] .\n(4.3)\nThus, solutions for nondeterministic problems can contain nested if–then–else statements;\nthis means that they are trees rather than sequences. This allows the selection of actions\nbased on contingencies arising during execution. Many problems in the real, physical world\nare contingency problems because exact prediction is impossible. For this reason, many\npeople keep their eyes open while walking around or driving.\n9 We assume that most readers face similar problems and can sympathize with our agent. We apologize to\nowners of modern, efficient home appliances who cannot take advantage of this pedagogical device.\nSection 4.3.\nSearching with Nondeterministic Actions\n135\n4.3.2\nAND–OR search trees\nThe next question is how to find contingent solutions to nondeterministic problems. As in\nChapter 3, we begin by constructing search trees, but here the trees have a different character.\nIn a deterministic environment, the only branching is introduced by the agent’s own choices\nin each state. We call these nodes OR nodes. In the vacuum world, for example, at an OR\nOR NODE\nnode the agent chooses Left or Right or Suck. In a nondeterministic environment, branching\nis also introduced by the environment’s choice of outcome for each action. We call these\nnodes AND nodes. For example, the Suck action in state 1 leads to a state in the set {5, 7},\nAND NODE\nso the agent would need to find a plan for state 5 and for state 7. These two kinds of nodes\nalternate, leading to an AND–OR tree as illustrated in Figure 4.10.\nAND–OR TREE\nA solution for an AND–OR search problem is a subtree that (1) has a goal node at every\nleaf, (2) specifies one action at each of its OR nodes, and (3) includes every outcome branch\nat each of its AND nodes. The solution is shown in bold lines in the figure; it corresponds\nto the plan given in Equation (4.3). (The plan uses if–then–else notation to handle the AND\nbranches, but when there are more than two branches at a node, it might be better to use a case\nLeft\nSuck\nRight\nSuck\nRight\nSuck\n6 \nGOAL\n8 \nGOAL\n7 \n1 \n2 \n5 \n1 \nLOOP\n5 \nLOOP\n5 \nLOOP\nLeft\nSuck\n1 \nLOOP\nGOAL\n8 \n4 \nFigure 4.10\nThe first two levels of the search tree for the erratic vacuum world. State\nnodes are OR nodes where some action must be chosen. At the AND nodes, shown as circles,\nevery outcome must be handled, as indicated by the arc linking the outgoing branches. The\nsolution found is shown in bold lines.\n136\nChapter\n4.\nBeyond Classical Search\nfunction AND-OR-GRAPH-SEARCH(problem) returns a conditional plan, or failure\nOR-SEARCH(problem.INITIAL-STATE,problem,[ ])\nfunction OR-SEARCH(state,problem,path) returns a conditional plan, or failure\nif problem.GOAL-TEST(state) then return the empty plan\nif state is on path then return failure\nfor each action in problem.ACTIONS(state) do\nplan ←AND-SEARCH(RESULTS(state,action),problem,[state | path])\nif plan ̸= failure then return [action | plan]\nreturn failure\nfunction AND-SEARCH(states,problem,path) returns a conditional plan, or failure\nfor each si in states do\nplani ←OR-SEARCH(si,problem,path)\nif plani = failure then return failure\nreturn [if s1 then plan1 else if s2 then plan2 else . . . if sn−1 then plann−1 else plann]\nFigure 4.11\nAn algorithm for searching AND–OR graphs generated by nondeterministic\nenvironments. It returns a conditional plan that reaches a goal state in all circumstances. (The\nnotation [x | l] refers to the list formed by adding object x to the front of list l.)\nconstruct.) Modifying the basic problem-solving agent shown in Figure 3.1 to execute con-\ntingent solutions of this kind is straightforward. One may also consider a somewhat different\nagent design, in which the agent can act before it has found a guaranteed plan and deals with\nsome contingencies only as they arise during execution. This type of interleaving of search\nINTERLEAVING\nand execution is also useful for exploration problems (see Section 4.5) and for game playing\n(see Chapter 5).\nFigure 4.11 gives a recursive, depth-first algorithm for AND–OR graph search. One\nkey aspect of the algorithm is the way in which it deals with cycles, which often arise in\nnondeterministic problems (e.g., if an action sometimes has no effect or if an unintended\neffect can be corrected). If the current state is identical to a state on the path from the root,\nthen it returns with failure. This doesn’t mean that there is no solution from the current state;\nit simply means that if there is a noncyclic solution, it must be reachable from the earlier\nincarnation of the current state, so the new incarnation can be discarded. With this check, we\nensure that the algorithm terminates in every finite state space, because every path must reach\na goal, a dead end, or a repeated state. Notice that the algorithm does not check whether the\ncurrent state is a repetition of a state on some other path from the root, which is important for\nefficiency. Exercise 4.5 investigates this issue.\nAND–OR graphs can also be explored by breadth-first or best-first methods. The concept\nof a heuristic function must be modified to estimate the cost of a contingent solution rather\nthan a sequence, but the notion of admissibility carries over and there is an analog of the A∗\nalgorithm for finding optimal solutions. Pointers are given in the bibliographical notes at the\nend of the chapter.\nSection 4.3.\nSearching with Nondeterministic Actions\n137\nSuck\nRight\n6 \n1 \n2 \n5 \nRight\nFigure 4.12\nPart of the search graph for the slippery vacuum world, where we have shown\n(some) cycles explicitly. All solutions for this problem are cyclic plans because there is no\nway to move reliably.\n4.3.3\nTry, try again\nConsider the slippery vacuum world, which is identical to the ordinary (non-erratic) vac-\nuum world except that movement actions sometimes fail, leaving the agent in the same loca-\ntion. For example, moving Right in state 1 leads to the state set {1, 2}. Figure 4.12 shows\npart of the search graph; clearly, there are no longer any acyclic solutions from state 1, and\nAND-OR-GRAPH-SEARCH would return with failure. There is, however, a cyclic solution,\nCYCLIC SOLUTION\nwhich is to keep trying Right until it works. We can express this solution by adding a label to\nLABEL\ndenote some portion of the plan and using that label later instead of repeating the plan itself.\nThus, our cyclic solution is\n[Suck, L1 : Right, if State = 5 then L1 else Suck] .\n(A better syntax for the looping part of this plan would be “while State = 5 do Right.”)\nIn general a cyclic plan may be considered a solution provided that every leaf is a goal\nstate and that a leaf is reachable from every point in the plan. The modifications needed\nto AND-OR-GRAPH-SEARCH are covered in Exercise 4.6. The key realization is that a loop\nin the state space back to a state L translates to a loop in the plan back to the point where the\nsubplan for state L is executed.\nGiven the definition of a cyclic solution, an agent executing such a solution will eventu-\nally reach the goal provided that each outcome of a nondeterministic action eventually occurs.\nIs this condition reasonable? It depends on the reason for the nondeterminism. If the action\nrolls a die, then it’s reasonable to suppose that eventually a six will be rolled. If the action is\nto insert a hotel card key into the door lock, but it doesn’t work the first time, then perhaps it\nwill eventually work, or perhaps one has the wrong key (or the wrong room!). After seven or\n138\nChapter\n4.\nBeyond Classical Search\neight tries, most people will assume the problem is with the key and will go back to the front\ndesk to get a new one. One way to understand this decision is to say that the initial problem\nformulation (observable, nondeterministic) is abandoned in favor of a different formulation\n(partially observable, deterministic) where the failure is attributed to an unobservable prop-\nerty of the key. We have more to say on this issue in Chapter 13.\n4.4\nSEARCHING WITH PARTIAL OBSERVATIONS\nWe now turn to the problem of partial observability, where the agent’s percepts do not suf-\nfice to pin down the exact state. As noted at the beginning of the previous section, if the\nagent is in one of several possible states, then an action may lead to one of several possible\noutcomes—even if the environment is deterministic. The key concept required for solving\npartially observable problems is the belief state, representing the agent’s current belief about\nBELIEF STATE\nthe possible physical states it might be in, given the sequence of actions and percepts up to\nthat point. We begin with the simplest scenario for studying belief states, which is when the\nagent has no sensors at all; then we add in partial sensing as well as nondeterministic actions.\n4.4.1\nSearching with no observation\nWhen the agent’s percepts provide no information at all, we have what is called a sensor-\nless problem or sometimes a conformant problem. At first, one might think the sensorless\nSENSORLESS\nCONFORMANT\nagent has no hope of solving a problem if it has no idea what state it’s in; in fact, sensorless\nproblems are quite often solvable. Moreover, sensorless agents can be surprisingly useful,\nprimarily because they don’t rely on sensors working properly. In manufacturing systems,\nfor example, many ingenious methods have been developed for orienting parts correctly from\nan unknown initial position by using a sequence of actions with no sensing at all. The high\ncost of sensing is another reason to avoid it: for example, doctors often prescribe a broad-\nspectrum antibiotic rather than using the contingent plan of doing an expensive blood test,\nthen waiting for the results to come back, and then prescribing a more specific antibiotic and\nperhaps hospitalization because the infection has progressed too far.\nWe can make a sensorless version of the vacuum world. Assume that the agent knows\nthe geography of its world, but doesn’t know its location or the distribution of dirt. In that\ncase, its initial state could be any element of the set {1, 2, 3, 4, 5, 6, 7, 8}. Now, consider what\nhappens if it tries the action Right. This will cause it to be in one of the states {2, 4, 6, 8}—the\nagent now has more information! Furthermore, the action sequence [Right,Suck] will always\nend up in one of the states {4, 8}. Finally, the sequence [Right,Suck,Left,Suck] is guaranteed\nto reach the goal state 7 no matter what the start state. We say that the agent can coerce the\nCOERCION\nworld into state 7.\nTo solve sensorless problems, we search in the space of belief states rather than physical\nstates.10 Notice that in belief-state space, the problem is fully observable because the agent\n10 In a fully observable environment, each belief state contains one physical state. Thus, we can view the algo-\nrithms in Chapter 3 as searching in a belief-state space of singleton belief states.\nSection 4.4.\nSearching with Partial Observations\n139\nalways knows its own belief state. Furthermore, the solution (if any) is always a sequence of\nactions. This is because, as in the ordinary problems of Chapter 3, the percepts received after\neach action are completely predictable—they’re always empty! So there are no contingencies\nto plan for. This is true even if the environment is nondeterminstic.\nIt is instructive to see how the belief-state search problem is constructed. Suppose\nthe underlying physical problem P is defined by ACTIONSP , RESULTP , GOAL-TESTP, and\nSTEP-COSTP. Then we can define the corresponding sensorless problem as follows:\n• Belief states: The entire belief-state space contains every possible set of physical states.\nIf P has N states, then the sensorless problem has up to 2N states, although many may\nbe unreachable from the initial state.\n• Initial state: Typically the set of all states in P, although in some cases the agent will\nhave more knowledge than this.\n• Actions: This is slightly tricky. Suppose the agent is in belief state b = {s1, s2}, but\nACTIONSP (s1) ̸= ACTIONSP(s2); then the agent is unsure of which actions are legal.\nIf we assume that illegal actions have no effect on the environment, then it is safe to\ntake the union of all the actions in any of the physical states in the current belief state b:\nACTIONS(b) =\n\u000f\ns∈b\nACTIONSP(s) .\nOn the other hand, if an illegal action might be the end of the world, it is safer to allow\nonly the intersection, that is, the set of actions legal in all the states. For the vacuum\nworld, every state has the same legal actions, so both methods give the same result.\n• Transition model: The agent doesn’t know which state in the belief state is the right\none; so as far as it knows, it might get to any of the states resulting from applying the\naction to one of the physical states in the belief state. For deterministic actions, the set\nof states that might be reached is\nb′ = RESULT(b, a) = {s′ : s′ = RESULTP(s, a) and s ∈b} .\n(4.4)\nWith deterministic actions, b′ is never larger than b. With nondeterminism, we have\nb′ = RESULT(b, a) = {s′ : s′ ∈RESULTSP(s, a) and s ∈b}\n=\n\u000f\ns∈b\nRESULTSP(s, a) ,\nwhich may be larger than b, as shown in Figure 4.13.\nThe process of generating\nthe new belief state after the action is called the prediction step; the notation b′ =\nPREDICTION\nPREDICTP (b, a) will come in handy.\n• Goal test: The agent wants a plan that is sure to work, which means that a belief state\nsatisfies the goal only if all the physical states in it satisfy GOAL-TESTP . The agent\nmay accidentally achieve the goal earlier, but it won’t know that it has done so.\n• Path cost: This is also tricky. If the same action can have different costs in different\nstates, then the cost of taking an action in a given belief state could be one of several\nvalues. (This gives rise to a new class of problems, which we explore in Exercise 4.9.)\nFor now we assume that the cost of an action is the same in all states and so can be\ntransferred directly from the underlying physical problem.\n140\nChapter\n4.\nBeyond Classical Search\n2 \n4 \n1 \n3 \n2 \n4 \n1 \n3 \n1 \n3 \n(b)\n(a)\nFigure 4.13\n(a) Predicting the next belief state for the sensorless vacuum world with a\ndeterministic action, Right. (b) Prediction for the same belief state and action in the slippery\nversion of the sensorless vacuum world.\nFigure 4.14 shows the reachable belief-state space for the deterministic, sensorless vacuum\nworld. There are only 12 reachable belief states out of 28 = 256 possible belief states.\nThe preceding definitions enable the automatic construction of the belief-state problem\nformulation from the definition of the underlying physical problem. Once this is done, we\ncan apply any of the search algorithms of Chapter 3. In fact, we can do a little bit more\nthan that. In “ordinary” graph search, newly generated states are tested to see if they are\nidentical to existing states. This works for belief states, too; for example, in Figure 4.14, the\naction sequence [Suck,Left,Suck] starting at the initial state reaches the same belief state as\n[Right,Left,Suck], namely, {5, 7}. Now, consider the belief state reached by [Left], namely,\n{1, 3, 5, 7}. Obviously, this is not identical to {5, 7}, but it is a superset. It is easy to prove\n(Exercise 4.8) that if an action sequence is a solution for a belief state b, it is also a solution for\nany subset of b. Hence, we can discard a path reaching {1, 3, 5, 7} if {5, 7} has already been\ngenerated. Conversely, if {1, 3, 5, 7} has already been generated and found to be solvable,\nthen any subset, such as {5, 7}, is guaranteed to be solvable. This extra level of pruning may\ndramatically improve the efficiency of sensorless problem solving.\nEven with this improvement, however, sensorless problem-solving as we have described\nit is seldom feasible in practice. The difficulty is not so much the vastness of the belief-state\nspace—even though it is exponentially larger than the underlying physical state space; in\nmost cases the branching factor and solution length in the belief-state space and physical\nstate space are not so different. The real difficulty lies with the size of each belief state. For\nexample, the initial belief state for the 10 × 10 vacuum world contains 100 × 2100 or around\n1032 physical states—far too many if we use the atomic representation, which is an explicit\nlist of states.\nOne solution is to represent the belief state by some more compact description. In\nEnglish, we could say the agent knows “Nothing” in the initial state; after moving Left, we\ncould say, “Not in the rightmost column,” and so on. Chapter 7 explains how to do this in a\nformal representation scheme. Another approach is to avoid the standard search algorithms,\nwhich treat belief states as black boxes just like any other problem state. Instead, we can look\nSection 4.4.\nSearching with Partial Observations\n141\nL\nR\nS\nL\nR\nS\nL\nR\nS\nL\nR\nS\nL\nR\nS\nL\nR\nS\nL\nR\nS\n1\n1\n3\n5\n7\n2\n4\n6\n8\n2\n3\n4\n5\n6\n7\n8\n4\n5\n7\n8\n5\n3\n7\n6\n4\n8\n4\n8\n5\n7\n6\n8\n8\n7\n3\n7\nFigure 4.14\nThe reachable portion of the belief-state space for the deterministic, sensor-\nless vacuum world. Each shaded box corresponds to a single belief state. At any given point,\nthe agent is in a particular belief state but does not know which physical state it is in. The\ninitial belief state (complete ignorance) is the top center box. Actions are represented by\nlabeled links. Self-loops are omitted for clarity.\ninside the belief states and develop incremental belief-state search algorithms that build up\nINCREMENTAL\nBELIEF-STATE\nSEARCH\nthe solution one physical state at a time. For example, in the sensorless vacuum world, the\ninitial belief state is {1, 2, 3, 4, 5, 6, 7, 8}, and we have to find an action sequence that works\nin all 8 states. We can do this by first finding a solution that works for state 1; then we check\nif it works for state 2; if not, go back and find a different solution for state 1, and so on. Just\nas an AND–OR search has to find a solution for every branch at an AND node, this algorithm\nhas to find a solution for every state in the belief state; the difference is that AND–OR search\ncan find a different solution for each branch, whereas an incremental belief-state search has\nto find one solution that works for all the states.\nThe main advantage of the incremental approach is that it is typically able to detect\nfailure quickly—when a belief state is unsolvable, it is usually the case that a small subset of\nthe belief state, consisting of the first few states examined, is also unsolvable. In some cases,\n142\nChapter\n4.\nBeyond Classical Search\nthis leads to a speedup proportional to the size of the belief states, which may themselves be\nas large as the physical state space itself.\nEven the most efficient solution algorithm is not of much use when no solutions exist.\nMany things just cannot be done without sensing. For example, the sensorless 8-puzzle is\nimpossible. On the other hand, a little bit of sensing can go a long way. For example, every\n8-puzzle instance is solvable if just one square is visible—the solution involves moving each\ntile in turn into the visible square and then keeping track of its location.\n4.4.2\nSearching with observations\nFor a general partially observable problem, we have to specify how the environment generates\npercepts for the agent. For example, we might define the local-sensing vacuum world to be\none in which the agent has a position sensor and a local dirt sensor but has no sensor capable\nof detecting dirt in other squares. The formal problem specification includes a PERCEPT(s)\nfunction that returns the percept received in a given state. (If sensing is nondeterministic,\nthen we use a PERCEPTS function that returns a set of possible percepts.) For example, in the\nlocal-sensing vacuum world, the PERCEPT in state 1 is [A, Dirty]. Fully observable problems\nare a special case in which PERCEPT(s) = s for every state s, while sensorless problems are\na special case in which PERCEPT(s) = null.\nWhen observations are partial, it will usually be the case that several states could have\nproduced any given percept. For example, the percept [A, Dirty] is produced by state 3 as\nwell as by state 1. Hence, given this as the initial percept, the initial belief state for the\nlocal-sensing vacuum world will be {1, 3}. The ACTIONS, STEP-COST, and GOAL-TEST\nare constructed from the underlying physical problem just as for sensorless problems, but the\ntransition model is a bit more complicated. We can think of transitions from one belief state\nto the next for a particular action as occurring in three stages, as shown in Figure 4.15:\n• The prediction stage is the same as for sensorless problems: given the action a in belief\nstate b, the predicted belief state is ˆb = PREDICT(b, a).11\n• The observation prediction stage determines the set of percepts o that could be ob-\nserved in the predicted belief state:\nPOSSIBLE-PERCEPTS(ˆb) = {o : o = PERCEPT(s) and s ∈ˆb} .\n• The update stage determines, for each possible percept, the belief state that would\nresult from the percept. The new belief state bo is just the set of states in ˆb that could\nhave produced the percept:\nbo = UPDATE(ˆb, o) = {s : o = PERCEPT(s) and s ∈ˆb} .\nNotice that each updated belief state bo can be no larger than the predicted belief state ˆb;\nobservations can only help reduce uncertainty compared to the sensorless case. More-\nover, for deterministic sensing, the belief states for the different possible percepts will\nbe disjoint, forming a partition of the original predicted belief state.\n11 Here, and throughout the book, the “hat” in ˆb means an estimated or predicted value for b.\nSection 4.4.\nSearching with Partial Observations\n143\n2 \n4 \n4 \n1 \n2 \n4 \n1 \n3 \n2 \n1 \n3 \n3 \n(b)\n(a)\n4 \n2 \n1 \n3 \nRight\n[A,Dirty]\n[B,Dirty]\n[B,Clean]\nRight\n[B,Dirty]\n[B,Clean]\nFigure 4.15\nTwo example of transitions in local-sensing vacuum worlds. (a) In the de-\nterministic world, Right is applied in the initial belief state, resulting in a new belief state\nwith two possible physical states; for those states, the possible percepts are [B, Dirty] and\n[B, Clean], leading to two belief states, each of which is a singleton. (b) In the slippery\nworld, Right is applied in the initial belief state, giving a new belief state with four physi-\ncal states; for those states, the possible percepts are [A, Dirty], [B, Dirty], and [B, Clean],\nleading to three belief states as shown.\nPutting these three stages together, we obtain the possible belief states resulting from a given\naction and the subsequent possible percepts:\nRESULTS(b, a) = {bo : bo = UPDATE(PREDICT(b, a), o) and\no ∈POSSIBLE-PERCEPTS(PREDICT(b, a))} .\n(4.5)\nAgain, the nondeterminism in the partially observable problem comes from the inability\nto predict exactly which percept will be received after acting; underlying nondeterminism in\nthe physical environment may contribute to this inability by enlarging the belief state at the\nprediction stage, leading to more percepts at the observation stage.\n4.4.3\nSolving partially observable problems\nThe preceding section showed how to derive the RESULTS function for a nondeterministic\nbelief-state problem from an underlying physical problem and the PERCEPT function. Given\n144\nChapter\n4.\nBeyond Classical Search\n7 \n5 \n1 \n3 \n4 \n2 \nSuck\n[B,Dirty]\n[B,Clean]\nRight\n[A,Clean]\nFigure 4.16\nThe first level of the AND–OR search tree for a problem in the local-sensing\nvacuum world; Suck is the first step of the solution.\nsuch a formulation, the AND–OR search algorithm of Figure 4.11 can be applied directly to\nderive a solution. Figure 4.16 shows part of the search tree for the local-sensing vacuum\nworld, assuming an initial percept [A, Dirty]. The solution is the conditional plan\n[Suck, Right, if Bstate = {6} then Suck else [ ]] .\nNotice that, because we supplied a belief-state problem to the AND–OR search algorithm, it\nreturned a conditional plan that tests the belief state rather than the actual state. This is as it\nshould be: in a partially observable environment the agent won’t be able to execute a solution\nthat requires testing the actual state.\nAs in the case of standard search algorithms applied to sensorless problems, the AND–\nOR search algorithm treats belief states as black boxes, just like any other states. One can\nimprove on this by checking for previously generated belief states that are subsets or supersets\nof the current state, just as for sensorless problems. One can also derive incremental search\nalgorithms, analogous to those described for sensorless problems, that provide substantial\nspeedups over the black-box approach.\n4.4.4\nAn agent for partially observable environments\nThe design of a problem-solving agent for partially observable environments is quite similar\nto the simple problem-solving agent in Figure 3.1: the agent formulates a problem, calls a\nsearch algorithm (such as AND-OR-GRAPH-SEARCH) to solve it, and executes the solution.\nThere are two main differences. First, the solution to a problem will be a conditional plan\nrather than a sequence; if the first step is an if–then–else expression, the agent will need to\ntest the condition in the if-part and execute the then-part or the else-part accordingly. Second,\nthe agent will need to maintain its belief state as it performs actions and receives percepts.\nThis process resembles the prediction–observation–update process in Equation (4.5) but is\nactually simpler because the percept is given by the environment rather than calculated by the\nSection 4.4.\nSearching with Partial Observations\n145\n7 \n5 \n6 \n2 \n1 \n3 \n6 \n4 \n8 \n2 \n[B,Dirty]\nRight\n[A,Clean]\n7 \n5 \nSuck\nFigure 4.17\nTwo prediction–update cycles of belief-state maintenance in the kindergarten\nvacuum world with local sensing.\nagent. Given an initial belief state b, an action a, and a percept o, the new belief state is:\nb′ = UPDATE(PREDICT(b, a), o) .\n(4.6)\nFigure 4.17 shows the belief state being maintained in the kindergarten vacuum world with\nlocal sensing, wherein any square may become dirty at any time unless the agent is actively\ncleaning it at that moment.12\nIn partially observable environments—which include the vast majority of real-world\nenvironments—maintaining one’s belief state is a core function of any intelligent system.\nThis function goes under various names, including monitoring, filtering and state estima-\nMONITORING\nFILTERING\ntion. Equation (4.6) is called a recursive state estimator because it computes the new belief\nSTATE ESTIMATION\nRECURSIVE\nstate from the previous one rather than by examining the entire percept sequence. If the agent\nis not to “fall behind,” the computation has to happen as fast as percepts are coming in. As\nthe environment becomes more complex, the exact update computation becomes infeasible\nand the agent will have to compute an approximate belief state, perhaps focusing on the im-\nplications of the percept for the aspects of the environment that are of current interest. Most\nwork on this problem has been done for stochastic, continuous-state environments with the\ntools of probability theory, as explained in Chapter 15. Here we will show an example in a\ndiscrete environment with detrministic sensors and nondeterministic actions.\nThe example concerns a robot with the task of localization: working out where it is,\nLOCALIZATION\ngiven a map of the world and a sequence of percepts and actions. Our robot is placed in the\nmaze-like environment of Figure 4.18. The robot is equipped with four sonar sensors that\ntell whether there is an obstacle—the outer wall or a black square in the figure—in each of\nthe four compass directions. We assume that the sensors give perfectly correct data, and that\nthe robot has a correct map of the enviornment. But unfortunately the robot’s navigational\nsystem is broken, so when it executes a Move action, it moves randomly to one of the adjacent\nsquares. The robot’s task is to determine its current location.\nSuppose the robot has just been switched on, so it does not know where it is. Thus its\ninitial belief state b consists of the set of all locations. The the robot receives the percept\n12 The usual apologies to those who are unfamiliar with the effect of small children on the environment.\n146\nChapter\n4.\nBeyond Classical Search\n(a) Possible locations of robot after E1 = NSW\n(b) Possible locations of robot After E1 = NSW, E2 = NS\nFigure 4.18\nPossible positions of the robot, ⊙, (a) after one observation E1 = NSW and\n(b) after a second observation E2 = NS. When sensors are noiseless and the transition model\nis accurate, there are no other possible locations for the robot consistent with this sequence\nof two observations.\nNSW, meaning there are obstacles to the north, west, and south, and does an update using the\nequation bo = UPDATE(b), yielding the 4 locations shown in Figure 4.18(a). You can inspect\nthe maze to see that those are the only four locations that yield the percept NWS.\nNext the robot executes a Move action, but the result is nondeterministic. The new be-\nlief state, ba = PREDICT(bo, Move), contains all the locations that are one step away from the\nlocations in bo. When the second percept, NS, arrives, the robot does UPDATE(ba, NS) and\nfinds that the belief state has collapsed down to the single location shown in Figure 4.18(b).\nThat’s the only location that could be the result of\nUPDATE(PREDICT(UPDATE(b, NSW ), Move), NS) .\nWith nondetermnistic actions the PREDICT step grows the belief state, but the UPDATE step\nshrinks it back down—as long as the percepts provide some useful identifying information.\nSometimes the percepts don’t help much for localization: If there were one or more long\neast-west corridors, then a robot could receive a long sequence of NS percepts, but never\nknow where in the corridor(s) it was.\nSection 4.5.\nOnline Search Agents and Unknown Environments\n147\n4.5\nONLINE SEARCH AGENTS AND UNKNOWN ENVIRONMENTS\nSo far we have concentrated on agents that use offline search algorithms. They compute\nOFFLINE SEARCH\na complete solution before setting foot in the real world and then execute the solution. In\ncontrast, an online search13 agent interleaves computation and action: first it takes an action,\nONLINE SEARCH\nthen it observes the environment and computes the next action. Online search is a good idea\nin dynamic or semidynamic domains—domains where there is a penalty for sitting around\nand computing too long. Online search is also helpful in nondeterministic domains because\nit allows the agent to focus its computational efforts on the contingencies that actually arise\nrather than those that might happen but probably won’t. Of course, there is a tradeoff: the\nmore an agent plans ahead, the less often it will find itself up the creek without a paddle.\nOnline search is a necessary idea for unknown environments, where the agent does not\nknow what states exist or what its actions do. In this state of ignorance, the agent faces an\nexploration problem and must use its actions as experiments in order to learn enough to\nEXPLORATION\nPROBLEM\nmake deliberation worthwhile.\nThe canonical example of online search is a robot that is placed in a new building and\nmust explore it to build a map that it can use for getting from A to B. Methods for escaping\nfrom labyrinths—required knowledge for aspiring heroes of antiquity—are also examples of\nonline search algorithms. Spatial exploration is not the only form of exploration, however.\nConsider a newborn baby: it has many possible actions but knows the outcomes of none of\nthem, and it has experienced only a few of the possible states that it can reach. The baby’s\ngradual discovery of how the world works is, in part, an online search process.\n4.5.1\nOnline search problems\nAn online search problem must be solved by an agent executing actions, rather than by pure\ncomputation. We assume a deterministic and fully observable environment (Chapter 17 re-\nlaxes these assumptions), but we stipulate that the agent knows only the following:\n• ACTIONS(s), which returns a list of actions allowed in state s;\n• The step-cost function c(s, a, s′)—note that this cannot be used until the agent knows\nthat s′ is the outcome; and\n• GOAL-TEST(s).\nNote in particular that the agent cannot determine RESULT(s, a) except by actually being\nin s and doing a. For example, in the maze problem shown in Figure 4.19, the agent does\nnot know that going Up from (1,1) leads to (1,2); nor, having done that, does it know that\ngoing Down will take it back to (1,1). This degree of ignorance can be reduced in some\napplications—for example, a robot explorer might know how its movement actions work and\nbe ignorant only of the locations of obstacles.\n13 The term “online” is commonly used in computer science to refer to algorithms that must process input data\nas they are received rather than waiting for the entire input data set to become available.\n148\nChapter\n4.\nBeyond Classical Search\nG\nS\n1\n2\n3\n1\n2\n3\nFigure 4.19\nA simple maze problem. The agent starts at S and must reach G but knows\nnothing of the environment.\nS\nG\nS\nG\nA\nA\nS\nG\n(a)\n(b)\nFigure 4.20\n(a) Two state spaces that might lead an online search agent into a dead end.\nAny given agent will fail in at least one of these spaces. (b) A two-dimensional environment\nthat can cause an online search agent to follow an arbitrarily inefficient route to the goal.\nWhichever choice the agent makes, the adversary blocks that route with another long, thin\nwall, so that the path followed is much longer than the best possible path.\nFinally, the agent might have access to an admissible heuristic function h(s) that es-\ntimates the distance from the current state to a goal state. For example, in Figure 4.19, the\nagent might know the location of the goal and be able to use the Manhattan-distance heuristic.\nTypically, the agent’s objective is to reach a goal state while minimizing cost. (Another\npossible objective is simply to explore the entire environment.) The cost is the total path cost\nof the path that the agent actually travels. It is common to compare this cost with the path\ncost of the path the agent would follow if it knew the search space in advance—that is, the\nactual shortest path (or shortest complete exploration). In the language of online algorithms,\nthis is called the competitive ratio; we would like it to be as small as possible.\nCOMPETITIVE RATIO\nSection 4.5.\nOnline Search Agents and Unknown Environments\n149\nAlthough this sounds like a reasonable request, it is easy to see that the best achievable\ncompetitive ratio is infinite in some cases. For example, if some actions are irreversible—\nIRREVERSIBLE\ni.e., they lead to a state from which no action leads back to the previous state—the online\nsearch might accidentally reach a dead-end state from which no goal state is reachable. Per-\nDEAD END\nhaps the term “accidentally” is unconvincing—after all, there might be an algorithm that\nhappens not to take the dead-end path as it explores. Our claim, to be more precise, is that no\nalgorithm can avoid dead ends in all state spaces. Consider the two dead-end state spaces in\nFigure 4.20(a). To an online search algorithm that has visited states S and A, the two state\nspaces look identical, so it must make the same decision in both. Therefore, it will fail in\none of them. This is an example of an adversary argument—we can imagine an adversary\nADVERSARY\nARGUMENT\nconstructing the state space while the agent explores it and putting the goals and dead ends\nwherever it chooses.\nDead ends are a real difficulty for robot exploration—staircases, ramps, cliffs, one-way\nstreets, and all kinds of natural terrain present opportunities for irreversible actions. To make\nprogress, we simply assume that the state space is safely explorable—that is, some goal state\nSAFELY EXPLORABLE\nis reachable from every reachable state. State spaces with reversible actions, such as mazes\nand 8-puzzles, can be viewed as undirected graphs and are clearly safely explorable.\nEven in safely explorable environments, no bounded competitive ratio can be guaran-\nteed if there are paths of unbounded cost. This is easy to show in environments with irre-\nversible actions, but in fact it remains true for the reversible case as well, as Figure 4.20(b)\nshows. For this reason, it is common to describe the performance of online search algorithms\nin terms of the size of the entire state space rather than just the depth of the shallowest goal.\n4.5.2\nOnline search agents\nAfter each action, an online agent receives a percept telling it what state it has reached; from\nthis information, it can augment its map of the environment. The current map is used to\ndecide where to go next. This interleaving of planning and action means that online search\nalgorithms are quite different from the offline search algorithms we have seen previously. For\nexample, offline algorithms such as A∗can expand a node in one part of the space and then\nimmediately expand a node in another part of the space, because node expansion involves\nsimulated rather than real actions. An online algorithm, on the other hand, can discover\nsuccessors only for a node that it physically occupies. To avoid traveling all the way across\nthe tree to expand the next node, it seems better to expand nodes in a local order. Depth-first\nsearch has exactly this property because (except when backtracking) the next node expanded\nis a child of the previous node expanded.\nAn online depth-first search agent is shown in Figure 4.21. This agent stores its map\nin a table, RESULT[s, a], that records the state resulting from executing action a in state s.\nWhenever an action from the current state has not been explored, the agent tries that action.\nThe difficulty comes when the agent has tried all the actions in a state. In offline depth-first\nsearch, the state is simply dropped from the queue; in an online search, the agent has to\nbacktrack physically. In depth-first search, this means going back to the state from which the\nagent most recently entered the current state. To achieve that, the algorithm keeps a table that\n150\nChapter\n4.\nBeyond Classical Search\nfunction ONLINE-DFS-AGENT(s′) returns an action\ninputs: s′, a percept that identifies the current state\npersistent: result, a table indexed by state and action, initially empty\nuntried, a table that lists, for each state, the actions not yet tried\nunbacktracked, a table that lists, for each state, the backtracks not yet tried\ns, a, the previous state and action, initially null\nif GOAL-TEST(s′) then return stop\nif s′ is a new state (not in untried) then untried[s′] ←ACTIONS(s′)\nif s is not null then\nresult[s,a] ←s′\nadd s to the front of unbacktracked[s′]\nif untried[s′] is empty then\nif unbacktracked[s′] is empty then return stop\nelse a ←an action b such that result[s′,b] = POP(unbacktracked[s′])\nelse a ←POP(untried[s′])\ns ←s′\nreturn a\nFigure 4.21\nAn online search agent that uses depth-first exploration. The agent is appli-\ncable only in state spaces in which every action can be “undone” by some other action.\nlists, for each state, the predecessor states to which the agent has not yet backtracked. If the\nagent has run out of states to which it can backtrack, then its search is complete.\nWe recommend that the reader trace through the progress of O NLINE-DFS-AGENT\nwhen applied to the maze given in Figure 4.19. It is fairly easy to see that the agent will, in\nthe worst case, end up traversing every link in the state space exactly twice. For exploration,\nthis is optimal; for finding a goal, on the other hand, the agent’s competitive ratio could be\narbitrarily bad if it goes off on a long excursion when there is a goal right next to the initial\nstate. An online variant of iterative deepening solves this problem; for an environment that is\na uniform tree, the competitive ratio of such an agent is a small constant.\nBecause of its method of backtracking, ONLINE-DFS-AGENT works only in state\nspaces where the actions are reversible. There are slightly more complex algorithms that\nwork in general state spaces, but no such algorithm has a bounded competitive ratio.\n4.5.3\nOnline local search\nLike depth-first search, hill-climbing search has the property of locality in its node expan-\nsions. In fact, because it keeps just one current state in memory, hill-climbing search is\nalready an online search algorithm! Unfortunately, it is not very useful in its simplest form\nbecause it leaves the agent sitting at local maxima with nowhere to go. Moreover, random\nrestarts cannot be used, because the agent cannot transport itself to a new state.\nInstead of random restarts, one might consider using a random walk to explore the\nRANDOM WALK\nenvironment. A random walk simply selects at random one of the available actions from the\nSection 4.5.\nOnline Search Agents and Unknown Environments\n151\nS\nG\nFigure 4.22\nAn environment in which a random walk will take exponentially many steps\nto find the goal.\ncurrent state; preference can be given to actions that have not yet been tried. It is easy to\nprove that a random walk will eventually find a goal or complete its exploration, provided\nthat the space is finite.14 On the other hand, the process can be very slow. Figure 4.22 shows\nan environment in which a random walk will take exponentially many steps to find the goal\nbecause, at each step, backward progress is twice as likely as forward progress. The example\nis contrived, of course, but there are many real-world state spaces whose topology causes\nthese kinds of “traps” for random walks.\nAugmenting hill climbing with memory rather than randomness turns out to be a more\neffective approach. The basic idea is to store a “current best estimate” H(s) of the cost to\nreach the goal from each state that has been visited. H(s) starts out being just the heuristic\nestimate h(s) and is updated as the agent gains experience in the state space. Figure 4.23\nshows a simple example in a one-dimensional state space. In (a), the agent seems to be\nstuck in a flat local minimum at the shaded state. Rather than staying where it is, the agent\nshould follow what seems to be the best path to the goal given the current cost estimates for\nits neighbors. The estimated cost to reach the goal through a neighbor s′ is the cost to get\nto s′ plus the estimated cost to get to a goal from there—that is, c(s, a, s′) + H(s′). In the\nexample, there are two actions, with estimated costs 1+9 and 1+2, so it seems best to move\nright. Now, it is clear that the cost estimate of 2 for the shaded state was overly optimistic.\nSince the best move cost 1 and led to a state that is at least 2 steps from a goal, the shaded\nstate must be at least 3 steps from a goal, so its H should be updated accordingly, as shown\nin Figure 4.23(b). Continuing this process, the agent will move back and forth twice more,\nupdating H each time and “flattening out” the local minimum until it escapes to the right.\nAn agent implementing this scheme, which is called learning real-time A∗(LRTA∗), is\nLRTA*\nshown in Figure 4.24. Like ONLINE-DFS-AGENT, it builds a map of the environment in\nthe result table. It updates the cost estimate for the state it has just left and then chooses the\n“apparently best” move according to its current cost estimates. One important detail is that\nactions that have not yet been tried in a state s are always assumed to lead immediately to the\ngoal with the least possible cost, namely h(s). This optimism under uncertainty encourages\nOPTIMISM UNDER\nUNCERTAINTY\nthe agent to explore new, possibly promising paths.\nAn LRTA∗agent is guaranteed to find a goal in any finite, safely explorable environment.\nUnlike A∗, however, it is not complete for infinite state spaces—there are cases where it can be\nled infinitely astray. It can explore an environment of n states in O(n2) steps in the worst case,\n14 Random walks are complete on infinite one-dimensional and two-dimensional grids. On a three-dimensional\ngrid, the probability that the walk ever returns to the starting point is only about 0.3405 (Hughes, 1995).\n152\nChapter\n4.\nBeyond Classical Search\n1\n2\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n1\n2\n2\n3\n4\n4\n4\n3\n3\n3\n1\n1\n1\n1\n1\n1\n1\n3\n1\n1\n1\n1\n1\n1\n1\n5\n3\n5\n5\n4\n(a)\n(b)\n(c)\n(d)\n(e)\n8\n9\n8\n9\n8\n9\n8\n9\n8\n9\n4\n4\n3\n4\nFigure 4.23\nFive iterations of LRTA∗on a one-dimensional state space. Each state is\nlabeled with H(s), the current cost estimate to reach a goal, and each link is labeled with its\nstep cost. The shaded state marks the location of the agent, and the updated cost estimates at\neach iteration are circled.\nfunction LRTA*-AGENT(s′) returns an action\ninputs: s′, a percept that identifies the current state\npersistent: result, a table, indexed by state and action, initially empty\nH , a table of cost estimates indexed by state, initially empty\ns, a, the previous state and action, initially null\nif GOAL-TEST(s′) then return stop\nif s′ is a new state (not in H ) then H [s′] ←h(s′)\nif s is not null\nresult[s,a] ←s′\nH [s] ←\nmin\nb ∈ACTIONS(s)LRTA*-COST(s,b,result[s,b],H )\na ←an action b in ACTIONS(s′) that minimizes LRTA*-COST(s′,b,result[s′,b],H )\ns ←s′\nreturn a\nfunction LRTA*-COST(s,a,s′,H ) returns a cost estimate\nif s′ is undefined then return h(s)\nelse return c(s, a, s′) + H[s′]\nFigure 4.24\nLRTA*-AGENT selects an action according to the values of neighboring\nstates, which are updated as the agent moves about the state space.\nSection 4.6.\nSummary\n153\nbut often does much better. The LRTA∗agent is just one of a large family of online agents that\none can define by specifying the action selection rule and the update rule in different ways.\nWe discuss this family, developed originally for stochastic environments, in Chapter 21.\n4.5.4\nLearning in online search\nThe initial ignorance of online search agents provides several opportunities for learning. First,\nthe agents learn a “map” of the environment—more precisely, the outcome of each action in\neach state—simply by recording each of their experiences. (Notice that the assumption of\ndeterministic environments means that one experience is enough for each action.) Second,\nthe local search agents acquire more accurate estimates of the cost of each state by using local\nupdating rules, as in LRTA∗. In Chapter 21, we show that these updates eventually converge\nto exact values for every state, provided that the agent explores the state space in the right\nway. Once exact values are known, optimal decisions can be taken simply by moving to the\nlowest-cost successor—that is, pure hill climbing is then an optimal strategy.\nIf you followed our suggestion to trace the behavior of ONLINE-DFS-AGENT in the\nenvironment of Figure 4.19, you will have noticed that the agent is not very bright. For\nexample, after it has seen that the Up action goes from (1,1) to (1,2), the agent still has no\nidea that the Down action goes back to (1,1) or that the Up action also goes from (2,1) to\n(2,2), from (2,2) to (2,3), and so on. In general, we would like the agent to learn that Up\nincreases the y-coordinate unless there is a wall in the way, that Down reduces it, and so on.\nFor this to happen, we need two things. First, we need a formal and explicitly manipulable\nrepresentation for these kinds of general rules; so far, we have hidden the information inside\nthe black box called the RESULT function. Part III is devoted to this issue. Second, we need\nalgorithms that can construct suitable general rules from the specific observations made by\nthe agent. These are covered in Chapter 18.\n4.6\nSUMMARY\nThis chapter has examined search algorithms for problems beyond the “classical” case of\nfinding the shortest path to a goal in an observable, deterministic, discrete environment.\n• Local search methods such as hill climbing operate on complete-state formulations,\nkeeping only a small number of nodes in memory. Several stochastic algorithms have\nbeen developed, including simulated annealing, which returns optimal solutions when\ngiven an appropriate cooling schedule.\n• Many local search methods apply also to problems in continuous spaces. Linear pro-\ngramming and convex optimization problems obey certain restrictions on the shape\nof the state space and the nature of the objective function, and admit polynomial-time\nalgorithms that are often extremely efficient in practice.\n• A genetic algorithm is a stochastic hill-climbing search in which a large population of\nstates is maintained. New states are generated by mutation and by crossover, which\ncombines pairs of states from the population.\n154\nChapter\n4.\nBeyond Classical Search\n• In nondeterministic environments, agents can apply AND–OR search to generate con-\ntingent plans that reach the goal regardless of which outcomes occur during execution.\n• When the environment is partially observable, the belief state represents the set of\npossible states that the agent might be in.\n• Standard search algorithms can be applied directly to belief-state space to solve sensor-\nless problems, and belief-state AND–OR search can solve general partially observable\nproblems. Incremental algorithms that construct solutions state-by-state within a belief\nstate are often more efficient.\n• Exploration problems arise when the agent has no idea about the states and actions of\nits environment. For safely explorable environments, online search agents can build a\nmap and find a goal if one exists. Updating heuristic estimates from experience provides\nan effective method to escape from local minima.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nLocal search techniques have a long history in mathematics and computer science. Indeed,\nthe Newton–Raphson method (Newton, 1671; Raphson, 1690) can be seen as a very effi-\ncient local search method for continuous spaces in which gradient information is available.\nBrent (1973) is a classic reference for optimization algorithms that do not require such in-\nformation. Beam search, which we have presented as a local search algorithm, originated\nas a bounded-width variant of dynamic programming for speech recognition in the HARPY\nsystem (Lowerre, 1976). A related algorithm is analyzed in depth by Pearl (1984, Ch. 5).\nThe topic of local search was reinvigorated in the early 1990s by surprisingly good re-\nsults for large constraint-satisfaction problems such as n-queens (Minton et al., 1992) and\nlogical reasoning (Selman et al., 1992) and by the incorporation of randomness, multiple\nsimultaneous searches, and other improvements. This renaissance of what Christos Papadim-\nitriou has called “New Age” algorithms also sparked increased interest among theoretical\ncomputer scientists (Koutsoupias and Papadimitriou, 1992; Aldous and Vazirani, 1994). In\nthe field of operations research, a variant of hill climbing called tabu search has gained popu-\nTABU SEARCH\nlarity (Glover and Laguna, 1997). This algorithm maintains a tabu list of k previously visited\nstates that cannot be revisited; as well as improving efficiency when searching graphs, this list\ncan allow the algorithm to escape from some local minima. Another useful improvement on\nhill climbing is the STAGE algorithm (Boyan and Moore, 1998). The idea is to use the local\nmaxima found by random-restart hill climbing to get an idea of the overall shape of the land-\nscape. The algorithm fits a smooth surface to the set of local maxima and then calculates the\nglobal maximum of that surface analytically. This becomes the new restart point. The algo-\nrithm has been shown to work in practice on hard problems. Gomes et al. (1998) showed that\nthe run times of systematic backtracking algorithms often have a heavy-tailed distribution,\nHEAVY-TAILED\nDISTRIBUTION\nwhich means that the probability of a very long run time is more than would be predicted if\nthe run times were exponentially distributed. When the run time distribution is heavy-tailed,\nrandom restarts find a solution faster, on average, than a single run to completion.\nBibliographical and Historical Notes\n155\nSimulated annealing was first described by Kirkpatrick et al. (1983), who borrowed\ndirectly from the Metropolis algorithm (which is used to simulate complex systems in\nphysics (Metropolis et al., 1953) and was supposedly invented at a Los Alamos dinner party).\nSimulated annealing is now a field in itself, with hundreds of papers published every year.\nFinding optimal solutions in continuous spaces is the subject matter of several fields,\nincluding optimization theory, optimal control theory, and the calculus of variations. The\nbasic techniques are explained well by Bishop (1995); Press et al. (2007) cover a wide range\nof algorithms and provide working software.\nAs Andrew Moore points out, researchers have taken inspiration for search and opti-\nmization algorithms from a wide variety of fields of study: metallurgy (simulated annealing),\nbiology (genetic algorithms), economics (market-based algorithms), entomology (ant colony\noptimization), neurology (neural networks), animal behavior (reinforcement learning), moun-\ntaineering (hill climbing), and others.\nLinear programming (LP) was first studied systematically by the Russian mathemati-\ncian Leonid Kantorovich (1939). It was one of the first applications of computers; the sim-\nplex algorithm (Dantzig, 1949) is still used despite worst-case exponential complexity. Kar-\nmarkar (1984) developed the far more efficient family of interior-point methods, which was\nshown to have polynomial complexity for the more general class of convex optimization prob-\nlems by Nesterov and Nemirovski (1994). Excellent introductions to convex optimization are\nprovided by Ben-Tal and Nemirovski (2001) and Boyd and Vandenberghe (2004).\nWork by Sewall Wright (1931) on the concept of a fitness landscape was an impor-\ntant precursor to the development of genetic algorithms. In the 1950s, several statisticians,\nincluding Box (1957) and Friedman (1959), used evolutionary techniques for optimization\nproblems, but it wasn’t until Rechenberg (1965) introduced evolution strategies to solve op-\nEVOLUTION\nSTRATEGY\ntimization problems for airfoils that the approach gained popularity. In the 1960s and 1970s,\nJohn Holland (1975) championed genetic algorithms, both as a useful tool and as a method\nto expand our understanding of adaptation, biological or otherwise (Holland, 1995). The ar-\ntificial life movement (Langton, 1995) takes this idea one step further, viewing the products\nARTIFICIAL LIFE\nof genetic algorithms as organisms rather than solutions to problems. Work in this field by\nHinton and Nowlan (1987) and Ackley and Littman (1991) has done much to clarify the im-\nplications of the Baldwin effect. For general background on evolution, we recommend Smith\nand Szathm´ary (1999), Ridley (2004), and Carroll (2007).\nMost comparisons of genetic algorithms to other approaches (especially stochastic hill\nclimbing) have found that the genetic algorithms are slower to converge (O’Reilly and Op-\npacher, 1994; Mitchell et al., 1996; Juels and Wattenberg, 1996; Baluja, 1997). Such findings\nare not universally popular within the GA community, but recent attempts within that com-\nmunity to understand population-based search as an approximate form of Bayesian learning\n(see Chapter 20) might help close the gap between the field and its critics (Pelikan et al.,\n1999). The theory of quadratic dynamical systems may also explain the performance of\nGAs (Rabani et al., 1998). See Lohn et al. (2001) for an example of GAs applied to antenna\ndesign, and Renner and Ekart (2003) for an application to computer-aided design.\nThe field of genetic programming is closely related to genetic algorithms. The princi-\nGENETIC\nPROGRAMMING\npal difference is that the representations that are mutated and combined are programs rather\n156\nChapter\n4.\nBeyond Classical Search\nthan bit strings. The programs are represented in the form of expression trees; the expressions\ncan be in a standard language such as Lisp or can be specially designed to represent circuits,\nrobot controllers, and so on. Crossover involves splicing together subtrees rather than sub-\nstrings. This form of mutation guarantees that the offspring are well-formed expressions,\nwhich would not be the case if programs were manipulated as strings.\nInterest in genetic programming was spurred by John Koza’s work (Koza, 1992, 1994),\nbut it goes back at least to early experiments with machine code by Friedberg (1958) and\nwith finite-state automata by Fogel et al. (1966). As with genetic algorithms, there is debate\nabout the effectiveness of the technique. Koza et al. (1999) describe experiments in the use\nof genetic programming to design circuit devices.\nThe journals Evolutionary Computation and IEEE Transactions on Evolutionary Com-\nputation cover genetic algorithms and genetic programming; articles are also found in Com-\nplex Systems, Adaptive Behavior, and Artificial Life. The main conference is the Genetic\nand Evolutionary Computation Conference (GECCO). Good overview texts on genetic algo-\nrithms are given by Mitchell (1996), Fogel (2000), and Langdon and Poli (2002), and by the\nfree online book by Poli et al. (2008).\nThe unpredictability and partial observability of real environments were recognized\nearly on in robotics projects that used planning techniques, including Shakey (Fikes et al.,\n1972) and FREDDY (Michie, 1974). The problems received more attention after the publica-\ntion of McDermott’s (1978a) influential article, Planning and Acting.\nThe first work to make explicit use of AND–OR trees seems to have been Slagle’s SAINT\nprogram for symbolic integration, mentioned in Chapter 1. Amarel (1967) applied the idea\nto propositional theorem proving, a topic discussed in Chapter 7, and introduced a search\nalgorithm similar to AND-OR-GRAPH-SEARCH. The algorithm was further developed and\nformalized by Nilsson (1971), who also described AO∗—which, as its name suggests, finds\noptimal solutions given an admissible heuristic. AO∗was analyzed and improved by Martelli\nand Montanari (1973). AO∗is a top-down algorithm; a bottom-up generalization of A∗is\nA∗LD, for A∗Lightest Derivation (Felzenszwalb and McAllester, 2007). Interest in AND–OR\nsearch has undergone a revival in recent years, with new algorithms for finding cyclic solu-\ntions (Jimenez and Torras, 2000; Hansen and Zilberstein, 2001) and new techniques inspired\nby dynamic programming (Bonet and Geffner, 2005).\nThe idea of transforming partially observable problems into belief-state problems orig-\ninated with Astrom (1965) for the much more complex case of probabilistic uncertainty (see\nChapter 17). Erdmann and Mason (1988) studied the problem of robotic manipulation with-\nout sensors, using a continuous form of belief-state search. They showed that it was possible\nto orient a part on a table from an arbitrary initial position by a well-designed sequence of tilt-\ning actions. More practical methods, based on a series of precisely oriented diagonal barriers\nacross a conveyor belt, use the same algorithmic insights (Wiegley et al., 1996).\nThe belief-state approach was reinvented in the context of sensorless and partially ob-\nservable search problems by Genesereth and Nourbakhsh (1993). Additional work was done\non sensorless problems in the logic-based planning community (Goldman and Boddy, 1996;\nSmith and Weld, 1998). This work has emphasized concise representations for belief states,\nas explained in Chapter 11. Bonet and Geffner (2000) introduced the first effective heuristics\nExercises\n157\nfor belief-state search; these were refined by Bryce et al. (2006). The incremental approach\nto belief-state search, in which solutions are constructed incrementally for subsets of states\nwithin each belief state, was studied in the planning literature by Kurien et al. (2002); several\nnew incremental algorithms were introduced for nondeterministic, partially observable prob-\nlems by Russell and Wolfe (2005). Additional references for planning in stochastic, partially\nobservable environments appear in Chapter 17.\nAlgorithms for exploring unknown state spaces have been of interest for many centuries.\nDepth-first search in a maze can be implemented by keeping one’s left hand on the wall; loops\ncan be avoided by marking each junction. Depth-first search fails with irreversible actions;\nthe more general problem of exploring Eulerian graphs (i.e., graphs in which each node has\nEULERIAN GRAPH\nequal numbers of incoming and outgoing edges) was solved by an algorithm due to Hierholzer\n(1873). The first thorough algorithmic study of the exploration problem for arbitrary graphs\nwas carried out by Deng and Papadimitriou (1990), who developed a completely general\nalgorithm but showed that no bounded competitive ratio is possible for exploring a general\ngraph. Papadimitriou and Yannakakis (1991) examined the question of finding paths to a goal\nin geometric path-planning environments (where all actions are reversible). They showed that\na small competitive ratio is achievable with square obstacles, but with general rectangular\nobstacles no bounded ratio can be achieved. (See Figure 4.20.)\nThe LRTA∗algorithm was developed by Korf (1990) as part of an investigation into\nreal-time search for environments in which the agent must act after searching for only a\nREAL-TIME SEARCH\nfixed amount of time (a common situation in two-player games). LRTA∗is in fact a special\ncase of reinforcement learning algorithms for stochastic environments (Barto et al., 1995). Its\npolicy of optimism under uncertainty—always head for the closest unvisited state—can result\nin an exploration pattern that is less efficient in the uninformed case than simple depth-first\nsearch (Koenig, 2000). Dasgupta et al. (1994) show that online iterative deepening search is\noptimally efficient for finding a goal in a uniform tree with no heuristic information. Sev-\neral informed variants on the LRTA∗theme have been developed with different methods for\nsearching and updating within the known portion of the graph (Pemberton and Korf, 1992).\nAs yet, there is no good understanding of how to find goals with optimal efficiency when\nusing heuristic information.\nEXERCISES\n4.1\nGive the name of the algorithm that results from each of the following special cases:\na. Local beam search with k = 1.\nb. Local beam search with one initial state and no limit on the number of states retained.\nc. Simulated annealing with T = 0 at all times (and omitting the termination test).\nd. Simulated annealing with T = ∞at all times.\ne. Genetic algorithm with population size N = 1.\n158\nChapter\n4.\nBeyond Classical Search\n4.2\nExercise 3.16 considers the problem of building railway tracks under the assumption\nthat pieces fit exactly with no slack. Now consider the real problem, in which pieces don’t\nfit exactly but allow for up to 10 degrees of rotation to either side of the “proper” alignment.\nExplain how to formulate the problem so it could be solved by simulated annealing.\n4.3\nIn this exercise, we explore the use of local search methods to solve TSPs of the type\ndefined in Exercise 3.30.\na. Implement and test a hill-climbing method to solve TSPs. Compare the results with op-\ntimal solutions obtained from the A∗algorithm with the MST heuristic (Exercise 3.30).\nb. Repeat part (a) using a genetic algorithm instead of hill climbing. You may want to\nconsult Larra˜naga et al. (1999) for some suggestions for representations.\n4.4\nGenerate a large number of 8-puzzle and 8-queens instances and solve them (where pos-\nsible) by hill climbing (steepest-ascent and first-choice variants), hill climbing with random\nrestart, and simulated annealing. Measure the search cost and percentage of solved problems\nand graph these against the optimal solution cost. Comment on your results.\n4.5\nThe AND-OR-GRAPH-SEARCH algorithm in Figure 4.11 checks for repeated states\nonly on the path from the root to the current state. Suppose that, in addition, the algorithm\nwere to store every visited state and check against that list. (See BREADTH-FIRST-SEARCH\nin Figure 3.11 for an example.) Determine the information that should be stored and how the\nalgorithm should use that information when a repeated state is found. (Hint: You will need to\ndistinguish at least between states for which a successful subplan was constructed previously\nand states for which no subplan could be found.) Explain how to use labels, as defined in\nSection 4.3.3, to avoid having multiple copies of subplans.\n4.6\nExplain precisely how to modify the AND-OR-GRAPH-SEARCH algorithm to generate\na cyclic plan if no acyclic plan exists. You will need to deal with three issues: labeling the plan\nsteps so that a cyclic plan can point back to an earlier part of the plan, modifying OR-SEARCH\nso that it continues to look for acyclic plans after finding a cyclic plan, and augmenting the\nplan representation to indicate whether a plan is cyclic. Show how your algorithm works on\n(a) the slippery vacuum world, and (b) the slippery, erratic vacuum world. You might wish to\nuse a computer implementation to check your results.\n4.7\nIn Section 4.4.1 we introduced belief states to solve sensorless search problems. A\nsequence of actions solves a sensorless problem if it maps every physical state in the initial\nbelief state b to a goal state. Suppose the agent knows h∗(s), the true optimal cost of solving\nthe physical state s in the fully observable problem, for every state s in b. Find an admissible\nheuristic h(b) for the sensorless problem in terms of these costs, and prove its admissibilty.\nComment on the accuracy of this heuristic on the sensorless vacuum problem of Figure 4.14.\nHow well does A∗perform?\n4.8\nThis exercise explores subset–superset relations between belief states in sensorless or\npartially observable environments.\na. Prove that if an action sequence is a solution for a belief state b, it is also a solution for\nany subset of b. Can anything be said about supersets of b?\nExercises\n159\nb. Explain in detail how to modify graph search for sensorless problems to take advantage\nof your answers in (a).\nc. Explain in detail how to modify AND–OR search for partially observable problems,\nbeyond the modifications you describe in (b).\n4.9\nOn page 139 it was assumed that a given action would have the same cost when ex-\necuted in any physical state within a given belief state. (This leads to a belief-state search\nproblem with well-defined step costs.) Now consider what happens when the assumption\ndoes not hold. Does the notion of optimality still make sense in this context, or does it require\nmodification? Consider also various possible definitions of the “cost” of executing an action\nin a belief state; for example, we could use the minimum of the physical costs; or the maxi-\nmum; or a cost interval with the lower bound being the minimum cost and the upper bound\nbeing the maximum; or just keep the set of all possible costs for that action. For each of these,\nexplore whether A∗(with modifications if necessary) can return optimal solutions.\n4.10\nConsider the sensorless version of the erratic vacuum world. Draw the belief-state\nspace reachable from the initial belief state {1, 2, 3, 4, 5, 6, 7, 8}, and explain why the problem\nis unsolvable.\n4.11\nWe can turn the navigation problem in Exercise 3.7 into an environment as follows:\n• The percept will be a list of the positions, relative to the agent, of the visible vertices.\nThe percept does not include the position of the robot! The robot must learn its own po-\nsition from the map; for now, you can assume that each location has a different “view.”\n• Each action will be a vector describing a straight-line path to follow. If the path is\nunobstructed, the action succeeds; otherwise, the robot stops at the point where its\npath first intersects an obstacle. If the agent returns a zero motion vector and is at the\ngoal (which is fixed and known), then the environment teleports the agent to a random\nlocation (not inside an obstacle).\n• The performance measure charges the agent 1 point for each unit of distance traversed\nand awards 1000 points each time the goal is reached.\na. Implement this environment and a problem-solving agent for it. After each teleporta-\ntion, the agent will need to formulate a new problem, which will involve discovering its\ncurrent location.\nb. Document your agent’s performance (by having the agent generate suitable commentary\nas it moves around) and report its performance over 100 episodes.\nc. Modify the environment so that 30% of the time the agent ends up at an unintended\ndestination (chosen randomly from the other visible vertices if any; otherwise, no move\nat all). This is a crude model of the motion errors of a real robot. Modify the agent\nso that when such an error is detected, it finds out where it is and then constructs a\nplan to get back to where it was and resume the old plan. Remember that sometimes\ngetting back to where it was might also fail! Show an example of the agent successfully\novercoming two successive motion errors and still reaching the goal.\n160\nChapter\n4.\nBeyond Classical Search\nd. Now try two different recovery schemes after an error: (1) head for the closest vertex on\nthe original route; and (2) replan a route to the goal from the new location. Compare the\nperformance of the three recovery schemes. Would the inclusion of search costs affect\nthe comparison?\ne. Now suppose that there are locations from which the view is identical. (For example,\nsuppose the world is a grid with square obstacles.) What kind of problem does the agent\nnow face? What do solutions look like?\n4.12\nSuppose that an agent is in a 3 × 3 maze environment like the one shown in Fig-\nure 4.19. The agent knows that its initial location is (1,1), that the goal is at (3,3), and that the\nactions Up, Down, Left, Right have their usual effects unless blocked by a wall. The agent\ndoes not know where the internal walls are. In any given state, the agent perceives the set of\nlegal actions; it can also tell whether the state is one it has visited before.\na. Explain how this online search problem can be viewed as an offline search in belief-state\nspace, where the initial belief state includes all possible environment configurations.\nHow large is the initial belief state? How large is the space of belief states?\nb. How many distinct percepts are possible in the initial state?\nc. Describe the first few branches of a contingency plan for this problem. How large\n(roughly) is the complete plan?\nNotice that this contingency plan is a solution for every possible environment fitting the given\ndescription. Therefore, interleaving of search and execution is not strictly necessary even in\nunknown environments.\n4.13\nIn this exercise, we examine hill climbing in the context of robot navigation, using the\nenvironment in Figure 3.31 as an example.\na. Repeat Exercise 4.11 using hill climbing. Does your agent ever get stuck in a local\nminimum? Is it possible for it to get stuck with convex obstacles?\nb. Construct a nonconvex polygonal environment in which the agent gets stuck.\nc. Modify the hill-climbing algorithm so that, instead of doing a depth-1 search to decide\nwhere to go next, it does a depth-k search. It should find the best k-step path and do\none step along it, and then repeat the process.\nd. Is there some k for which the new algorithm is guaranteed to escape from local minima?\ne. Explain how LRTA∗enables the agent to escape from local minima in this case.\n4.14\nLike DFS, online DFS is incomplete for reversible state spaces with infinite paths. For\nexample, suppose that states are points on the infinite two-dimensional grid and actions are\nunit vectors (1, 0), (0, 1), (−1, 0), (0, −1), tried in that order. Show that online DFS starting\nat (0, 0) will not reach (1, −1). Suppose the agent can observe, in addition to its current\nstate, all successor states and the actions that would lead to them. Write an algorithm that\nis complete even for bidirected state spaces with infinite paths. What states does it visit in\nreaching (1, −1)?\n" }, { "chapter": 5, "text": "5\nADVERSARIAL SEARCH\nIn which we examine the problems that arise when we try to plan ahead in a world\nwhere other agents are planning against us.\n5.1\nGAMES\nChapter 2 introduced multiagent environments, in which each agent needs to consider the\nactions of other agents and how they affect its own welfare. The unpredictability of these\nother agents can introduce contingencies into the agent’s problem-solving process, as dis-\ncussed in Chapter 4. In this chapter we cover competitive environments, in which the agents’\ngoals are in conflict, giving rise to adversarial search problems—often known as games.\nGAME\nMathematical game theory, a branch of economics, views any multiagent environment\nas a game, provided that the impact of each agent on the others is “significant,” regardless\nof whether the agents are cooperative or competitive.1 In AI, the most common games are\nof a rather specialized kind—what game theorists call deterministic, turn-taking, two-player,\nzero-sum games of perfect information (such as chess). In our terminology, this means\nZERO-SUM GAMES\nPERFECT\nINFORMATION\ndeterministic, fully observable environments in which two agents act alternately and in which\nthe utility values at the end of the game are always equal and opposite. For example, if one\nplayer wins a game of chess, the other player necessarily loses. It is this opposition between\nthe agents’ utility functions that makes the situation adversarial.\nGames have engaged the intellectual faculties of humans—sometimes to an alarming\ndegree—for as long as civilization has existed. For AI researchers, the abstract nature of\ngames makes them an appealing subject for study. The state of a game is easy to represent,\nand agents are usually restricted to a small number of actions whose outcomes are defined by\nprecise rules. Physical games, such as croquet and ice hockey, have much more complicated\ndescriptions, a much larger range of possible actions, and rather imprecise rules defining\nthe legality of actions. With the exception of robot soccer, these physical games have not\nattracted much interest in the AI community.\n1 Environments with very many agents are often viewed as economies rather than games.\n161\n162\nChapter\n5.\nAdversarial Search\nGames, unlike most of the toy problems studied in Chapter 3, are interesting because\nthey are too hard to solve. For example, chess has an average branching factor of about 35,\nand games often go to 50 moves by each player, so the search tree has about 35100 or 10154\nnodes (although the search graph has “only” about 1040 distinct nodes). Games, like the real\nworld, therefore require the ability to make some decision even when calculating the optimal\ndecision is infeasible. Games also penalize inefficiency severely. Whereas an implementation\nof A∗search that is half as efficient will simply take twice as long to run to completion, a chess\nprogram that is half as efficient in using its available time probably will be beaten into the\nground, other things being equal. Game-playing research has therefore spawned a number of\ninteresting ideas on how to make the best possible use of time.\nWe begin with a definition of the optimal move and an algorithm for finding it. We\nthen look at techniques for choosing a good move when time is limited. Pruning allows us\nPRUNING\nto ignore portions of the search tree that make no difference to the final choice, and heuristic\nevaluation functions allow us to approximate the true utility of a state without doing a com-\nplete search. Section 5.5 discusses games such as backgammon that include an element of\nchance; we also discuss bridge, which includes elements of imperfect information because\nIMPERFECT\nINFORMATION\nnot all cards are visible to each player. Finally, we look at how state-of-the-art game-playing\nprograms fare against human opposition and at directions for future developments.\nWe first consider games with two players, whom we call MAX and MIN for reasons that\nwill soon become obvious. MAX moves first, and then they take turns moving until the game\nis over. At the end of the game, points are awarded to the winning player and penalties are\ngiven to the loser. A game can be formally defined as a kind of search problem with the\nfollowing elements:\n• S0: The initial state, which specifies how the game is set up at the start.\n• PLAYER(s): Defines which player has the move in a state.\n• ACTIONS(s): Returns the set of legal moves in a state.\n• RESULT(s, a): The transition model, which defines the result of a move.\n• TERMINAL-TEST(s): A terminal test, which is true when the game is over and false\nTERMINAL TEST\notherwise. States where the game has ended are called terminal states.\nTERMINAL STATES\n• UTILITY(s, p): A utility function (also called an objective function or payoff function),\ndefines the final numeric value for a game that ends in terminal state s for a player p. In\nchess, the outcome is a win, loss, or draw, with values +1, 0, or 1\n2. Some games have a\nwider variety of possible outcomes; the payoffs in backgammon range from 0 to +192.\nA zero-sum game is (confusingly) defined as one where the total payoff to all players\nis the same for every instance of the game. Chess is zero-sum because every game has\npayoff of either 0 + 1, 1 + 0 or 1\n2 + 1\n2. “Constant-sum” would have been a better term,\nbut zero-sum is traditional and makes sense if you imagine each player is charged an\nentry fee of 1\n2.\nThe initial state, ACTIONS function, and RESULT function define the game tree for the\nGAME TREE\ngame—a tree where the nodes are game states and the edges are moves. Figure 5.1 shows\npart of the game tree for tic-tac-toe (noughts and crosses). From the initial state, MAX has\nnine possible moves. Play alternates between MAX’s placing an X and MIN’s placing an O\nSection 5.2.\nOptimal Decisions in Games\n163\nuntil we reach leaf nodes corresponding to terminal states such that one player has three in\na row or all the squares are filled. The number on each leaf node indicates the utility value\nof the terminal state from the point of view of MAX; high values are assumed to be good for\nMAX and bad for MIN (which is how the players get their names).\nFor tic-tac-toe the game tree is relatively small—fewer than 9! = 362, 880 terminal\nnodes. But for chess there are over 1040 nodes, so the game tree is best thought of as a\ntheoretical construct that we cannot realize in the physical world. But regardless of the size\nof the game tree, it is MAX’s job to search for a good move. We use the term search tree for a\nSEARCH TREE\ntree that is superimposed on the full game tree, and examines enough nodes to allow a player\nto determine what move to make.\nX\nX\nX\nX\nX\nX\nX\nX\nX\nX\nX\nO\nO\nX O\nO\nX O\nX O\nX\n. . .\n. . .\n. . .\n. . .\n. . .\n. . .\n. . .\nX\nX\n–1\n 0\n+1\nX\nX\nX\nX\nO\nX\nX\nO\nX\nX\nO\nO\nO\nX\nX\nX\nO\nO\nO\nO O X\nX\nMAX (X)\nMIN (O)\nMAX (X)\nMIN (O)\nTERMINAL\nUtility\nFigure 5.1\nA (partial) game tree for the game of tic-tac-toe. The top node is the initial\nstate, and MAX moves first, placing an X in an empty square. We show part of the tree, giving\nalternating moves by MIN (O) and MAX (X), until we eventually reach terminal states, which\ncan be assigned utilities according to the rules of the game.\n5.2\nOPTIMAL DECISIONS IN GAMES\nIn a normal search problem, the optimal solution would be a sequence of actions leading to\na goal state—a terminal state that is a win. In adversarial search, MIN has something to say\nabout it.\nMAX therefore must find a contingent strategy, which specifies MAX’s move in\nSTRATEGY\nthe initial state, then MAX’s moves in the states resulting from every possible response by\n164\nChapter\n5.\nAdversarial Search\nMAX\nA\nB\nC\nD\n3\n12\n8\n2\n4\n6\n14\n5\n2\n3\n2\n2\n3\na1\na2\na3\nb1\nb2\nb3\nc1\nc2\nc3\nd1\nd2\nd3\nMIN\nFigure 5.2\nA two-ply game tree. The △nodes are “MAX nodes,” in which it is MAX’s\nturn to move, and the ▽nodes are “MIN nodes.” The terminal nodes show the utility values\nfor MAX; the other nodes are labeled with their minimax values. MAX’s best move at the root\nis a1, because it leads to the state with the highest minimax value, and MIN’s best reply is b1,\nbecause it leads to the state with the lowest minimax value.\nMIN, then MAX’s moves in the states resulting from every possible response by MIN to those\nmoves, and so on. This is exactly analogous to the AND–OR search algorithm (Figure 4.11)\nwith MAX playing the role of OR and MIN equivalent to AND. Roughly speaking, an optimal\nstrategy leads to outcomes at least as good as any other strategy when one is playing an\ninfallible opponent. We begin by showing how to find this optimal strategy.\nEven a simple game like tic-tac-toe is too complex for us to draw the entire game tree\non one page, so we will switch to the trivial game in Figure 5.2. The possible moves for MAX\nat the root node are labeled a1, a2, and a3. The possible replies to a1 for MIN are b1, b2,\nb3, and so on. This particular game ends after one move each by MAX and MIN. (In game\nparlance, we say that this tree is one move deep, consisting of two half-moves, each of which\nis called a ply.) The utilities of the terminal states in this game range from 2 to 14.\nPLY\nGiven a game tree, the optimal strategy can be determined from the minimax value\nMINIMAX VALUE\nof each node, which we write as MINIMAX(n). The minimax value of a node is the utility\n(for MAX) of being in the corresponding state, assuming that both players play optimally\nfrom there to the end of the game. Obviously, the minimax value of a terminal state is just\nits utility. Furthermore, given a choice, MAX prefers to move to a state of maximum value,\nwhereas MIN prefers a state of minimum value. So we have the following:\nMINIMAX(s) =\n⎧\n⎨\n⎩\nUTILITY(s)\nif TERMINAL-TEST(s)\nmaxa∈Actions(s) MINIMAX(RESULT(s, a)) if PLAYER(s) = MAX\nmina∈Actions(s) MINIMAX(RESULT(s,a))\nif PLAYER(s) = MIN\nLet us apply these definitions to the game tree in Figure 5.2. The terminal nodes on the bottom\nlevel get their utility values from the game’s UTILITY function. The first MIN node, labeled\nB, has three successor states with values 3, 12, and 8, so its minimax value is 3. Similarly,\nthe other two MIN nodes have minimax value 2. The root node is a MAX node; its successor\nstates have minimax values 3, 2, and 2; so it has a minimax value of 3. We can also identify\nSection 5.2.\nOptimal Decisions in Games\n165\nthe minimax decision at the root: action a1 is the optimal choice for MAX because it leads to\nMINIMAX DECISION\nthe state with the highest minimax value.\nThis definition of optimal play for MAX assumes that MIN also plays optimally—it\nmaximizes the worst-case outcome for MAX. What if MIN does not play optimally? Then it is\neasy to show (Exercise 5.7) that MAX will do even better. Other strategies against suboptimal\nopponents may do better than the minimax strategy, but these strategies necessarily do worse\nagainst optimal opponents.\n5.2.1\nThe minimax algorithm\nThe minimax algorithm (Figure 5.3) computes the minimax decision from the current state.\nMINIMAX ALGORITHM\nIt uses a simple recursive computation of the minimax values of each successor state, directly\nimplementing the defining equations. The recursion proceeds all the way down to the leaves\nof the tree, and then the minimax values are backed up through the tree as the recursion\nunwinds. For example, in Figure 5.2, the algorithm first recurses down to the three bottom-\nleft nodes and uses the UTILITY function on them to discover that their values are 3, 12, and\n8, respectively. Then it takes the minimum of these values, 3, and returns it as the backed-\nup value of node B. A similar process gives the backed-up values of 2 for C and 2 for D.\nFinally, we take the maximum of 3, 2, and 2 to get the backed-up value of 3 for the root node.\nThe minimax algorithm performs a complete depth-first exploration of the game tree.\nIf the maximum depth of the tree is m and there are b legal moves at each point, then the\ntime complexity of the minimax algorithm is O(b m). The space complexity is O(bm) for an\nalgorithm that generates all actions at once, or O(m) for an algorithm that generates actions\none at a time (see page 87). For real games, of course, the time cost is totally impractical,\nbut this algorithm serves as the basis for the mathematical analysis of games and for more\npractical algorithms.\n5.2.2\nOptimal decisions in multiplayer games\nMany popular games allow more than two players. Let us examine how to extend the minimax\nidea to multiplayer games. This is straightforward from the technical viewpoint, but raises\nsome interesting new conceptual issues.\nFirst, we need to replace the single value for each node with a vector of values. For\nexample, in a three-player game with players A, B, and C, a vector ⟨vA, vB, vC⟩is associated\nwith each node. For terminal states, this vector gives the utility of the state from each player’s\nviewpoint. (In two-player, zero-sum games, the two-element vector can be reduced to a single\nvalue because the values are always opposite.) The simplest way to implement this is to have\nthe UTILITY function return a vector of utilities.\nNow we have to consider nonterminal states. Consider the node marked X in the game\ntree shown in Figure 5.4. In that state, player C chooses what to do. The two choices lead\nto terminal states with utility vectors ⟨vA = 1, vB = 2, vC = 6⟩and ⟨vA = 4, vB = 2, vC = 3⟩.\nSince 6 is bigger than 3, C should choose the first move. This means that if state X is reached,\nsubsequent play will lead to a terminal state with utilities ⟨vA = 1, vB = 2, vC = 6⟩. Hence,\nthe backed-up value of X is this vector. The backed-up value of a node n is always the utility\n166\nChapter\n5.\nAdversarial Search\nfunction MINIMAX-DECISION(state) returns an action\nreturn arg maxa ∈ACTIONS(s) MIN-VALUE(RESULT(state,a))\nfunction MAX-VALUE(state) returns a utility value\nif TERMINAL-TEST(state) then return UTILITY(state)\nv ←−∞\nfor each a in ACTIONS(state) do\nv ←MAX(v, MIN-VALUE(RESULT(s, a)))\nreturn v\nfunction MIN-VALUE(state) returns a utility value\nif TERMINAL-TEST(state) then return UTILITY(state)\nv ←∞\nfor each a in ACTIONS(state) do\nv ←MIN(v, MAX-VALUE(RESULT(s, a)))\nreturn v\nFigure 5.3\nAn algorithm for calculating minimax decisions. It returns the action corre-\nsponding to the best possible move, that is, the move that leads to the outcome with the\nbest utility, under the assumption that the opponent plays to minimize utility. The functions\nMAX-VALUE and MIN-VALUE go through the whole game tree, all the way to the leaves,\nto determine the backed-up value of a state. The notation argmaxa ∈S f(a) computes the\nelement a of set S that has the maximum value of f(a).\nto move\nA\nB\nC\nA\n(1, 2, 6)\n(4, 2, 3)\n(6, 1, 2)\n(7, 4,1)\n(5,1,1)\n(1, 5, 2)\n(7, 7,1)\n(5, 4, 5)\n(1, 2, 6)\n(6, 1, 2)\n(1, 5, 2)\n(5, 4, 5)\n(1, 2, 6)\n(1, 5, 2)\n(1, 2, 6)\nX\nFigure 5.4\nThe first three plies of a game tree with three players (A, B, C). Each node is\nlabeled with values from the viewpoint of each player. The best move is marked at the root.\nvector of the successor state with the highest value for the player choosing at n. Anyone\nwho plays multiplayer games, such as Diplomacy, quickly becomes aware that much more\nis going on than in two-player games. Multiplayer games usually involve alliances, whether\nALLIANCE\nformal or informal, among the players. Alliances are made and broken as the game proceeds.\nHow are we to understand such behavior? Are alliances a natural consequence of optimal\nstrategies for each player in a multiplayer game? It turns out that they can be. For example,\nSection 5.3.\nAlpha–Beta Pruning\n167\nsuppose A and B are in weak positions and C is in a stronger position. Then it is often\noptimal for both A and B to attack C rather than each other, lest C destroy each of them\nindividually. In this way, collaboration emerges from purely selfish behavior. Of course,\nas soon as C weakens under the joint onslaught, the alliance loses its value, and either A\nor B could violate the agreement. In some cases, explicit alliances merely make concrete\nwhat would have happened anyway. In other cases, a social stigma attaches to breaking an\nalliance, so players must balance the immediate advantage of breaking an alliance against the\nlong-term disadvantage of being perceived as untrustworthy. See Section 17.5 for more on\nthese complications.\nIf the game is not zero-sum, then collaboration can also occur with just two players.\nSuppose, for example, that there is a terminal state with utilities ⟨vA = 1000, vB = 1000⟩and\nthat 1000 is the highest possible utility for each player. Then the optimal strategy is for both\nplayers to do everything possible to reach this state—that is, the players will automatically\ncooperate to achieve a mutually desirable goal.\n5.3\nALPHA–BETA PRUNING\nThe problem with minimax search is that the number of game states it has to examine is\nexponential in the depth of the tree. Unfortunately, we can’t eliminate the exponent, but it\nturns out we can effectively cut it in half. The trick is that it is possible to compute the correct\nminimax decision without looking at every node in the game tree. That is, we can borrow the\nidea of pruning from Chapter 3 to eliminate large parts of the tree from consideration. The\nparticular technique we examine is called alpha–beta pruning. When applied to a standard\nALPHA–BETA\nPRUNING\nminimax tree, it returns the same move as minimax would, but prunes away branches that\ncannot possibly influence the final decision.\nConsider again the two-ply game tree from Figure 5.2. Let’s go through the calculation\nof the optimal decision once more, this time paying careful attention to what we know at\neach point in the process. The steps are explained in Figure 5.5. The outcome is that we can\nidentify the minimax decision without ever evaluating two of the leaf nodes.\nAnother way to look at this is as a simplification of the formula for MINIMAX. Let the\ntwo unevaluated successors of node C in Figure 5.5 have values x and y. Then the value of\nthe root node is given by\nMINIMAX(root) = max(min(3, 12, 8), min(2, x, y), min(14, 5, 2))\n= max(3, min(2, x, y), 2)\n= max(3, z, 2)\nwhere z = min(2, x, y) ≤2\n= 3.\nIn other words, the value of the root and hence the minimax decision are independent of the\nvalues of the pruned leaves x and y.\nAlpha–beta pruning can be applied to trees of any depth, and it is often possible to\nprune entire subtrees rather than just leaves. The general principle is this: consider a node n\n168\nChapter\n5.\nAdversarial Search\n(a)\n(b)\n(c)\n(d)\n(e)\n(f)\n3\n3\n12\n3\n12\n8\n3\n12\n8\n2\n3\n12\n8\n2\n14\n3\n12\n8\n2\n14\n5\n2\nA\nB\nA\nB\nA\nB\nC\nD\nA\nB\nC\nD\nA\nB\nA\nB\nC\n[−∞, +∞]\n[−∞, +∞]\n[3, +∞]\n[3, +∞]\n[3, 3]\n[3, 14]\n[−∞, 2]\n[−∞, 2]\n[2, 2]\n[3, 3]\n[3, 3]\n[3, 3]\n[3, 3]\n[−∞, 3]\n[−∞, 3]\n[−∞, 2]\n[−∞, 14]\nFigure 5.5\nStages in the calculation of the optimal decision for the game tree in Figure 5.2.\nAt each point, we show the range of possible values for each node. (a) The first leaf below B\nhas the value 3. Hence, B, which is a MIN node, has a value of at most 3. (b) The second leaf\nbelow B has a value of 12; MIN would avoid this move, so the value of B is still at most 3.\n(c) The third leaf below B has a value of 8; we have seen all B’s successor states, so the\nvalue of B is exactly 3. Now, we can infer that the value of the root is at least 3, because\nMAX has a choice worth 3 at the root. (d) The first leaf below C has the value 2. Hence,\nC, which is a MIN node, has a value of at most 2. But we know that B is worth 3, so MAX\nwould never choose C. Therefore, there is no point in looking at the other successor states\nof C. This is an example of alpha–beta pruning. (e) The first leaf below D has the value 14,\nso D is worth at most 14. This is still higher than MAX’s best alternative (i.e., 3), so we need\nto keep exploring D’s successor states. Notice also that we now have bounds on all of the\nsuccessors of the root, so the root’s value is also at most 14. (f) The second successor of D\nis worth 5, so again we need to keep exploring. The third successor is worth 2, so now D is\nworth exactly 2. MAX’s decision at the root is to move to B, giving a value of 3.\nsomewhere in the tree (see Figure 5.6), such that Player has a choice of moving to that node.\nIf Player has a better choice m either at the parent node of n or at any choice point further up,\nthen n will never be reached in actual play. So once we have found out enough about n (by\nexamining some of its descendants) to reach this conclusion, we can prune it.\nRemember that minimax search is depth-first, so at any one time we just have to con-\nsider the nodes along a single path in the tree. Alpha–beta pruning gets its name from the\nfollowing two parameters that describe bounds on the backed-up values that appear anywhere\nalong the path:\nSection 5.3.\nAlpha–Beta Pruning\n169\nPlayer\nOpponent\nPlayer\nOpponent\nm\nn\n•\n•\n•\nFigure 5.6\nThe general case for alpha–beta pruning. If m is better than n for Player, we\nwill never get to n in play.\nα = the value of the best (i.e., highest-value) choice we have found so far at any choice point\nalong the path for MAX.\nβ = the value of the best (i.e., lowest-value) choice we have found so far at any choice point\nalong the path for MIN.\nAlpha–beta search updates the values of α and β as it goes along and prunes the remaining\nbranches at a node (i.e., terminates the recursive call) as soon as the value of the current\nnode is known to be worse than the current α or β value for MAX or MIN, respectively. The\ncomplete algorithm is given in Figure 5.7. We encourage you to trace its behavior when\napplied to the tree in Figure 5.5.\n5.3.1\nMove ordering\nThe effectiveness of alpha–beta pruning is highly dependent on the order in which the states\nare examined. For example, in Figure 5.5(e) and (f), we could not prune any successors of D\nat all because the worst successors (from the point of view of MIN) were generated first. If\nthe third successor of D had been generated first, we would have been able to prune the other\ntwo. This suggests that it might be worthwhile to try to examine first the successors that are\nlikely to be best.\nIf this can be done,2 then it turns out that alpha–beta needs to examine only O(bm/2)\nnodes to pick the best move, instead of O(bm) for minimax. This means that the effective\nbranching factor becomes\n√\nb instead of b—for chess, about 6 instead of 35. Put another\nway, alpha–beta can solve a tree roughly twice as deep as minimax in the same amount of\ntime. If successors are examined in random order rather than best-first, the total number of\nnodes examined will be roughly O(b3m/4) for moderate b. For chess, a fairly simple ordering\nfunction (such as trying captures first, then threats, then forward moves, and then backward\nmoves) gets you to within about a factor of 2 of the best-case O(bm/2) result.\n2 Obviously, it cannot be done perfectly; otherwise, the ordering function could be used to play a perfect game!\n170\nChapter\n5.\nAdversarial Search\nfunction ALPHA-BETA-SEARCH(state) returns an action\nv ←MAX-VALUE(state,−∞,+∞)\nreturn the action in ACTIONS(state) with value v\nfunction MAX-VALUE(state,α,β) returns a utility value\nif TERMINAL-TEST(state) then return UTILITY(state)\nv ←−∞\nfor each a in ACTIONS(state) do\nv ←MAX(v, MIN-VALUE(RESULT(s,a),α,β))\nif v ≥β then return v\nα ←MAX(α, v)\nreturn v\nfunction MIN-VALUE(state,α,β) returns a utility value\nif TERMINAL-TEST(state) then return UTILITY(state)\nv ←+∞\nfor each a in ACTIONS(state) do\nv ←MIN(v, MAX-VALUE(RESULT(s,a) ,α,β))\nif v ≤α then return v\nβ ←MIN(β, v)\nreturn v\nFigure 5.7\nThe alpha–beta search algorithm. Notice that these routines are the same as\nthe MINIMAX functions in Figure 5.3, except for the two lines in each of MIN-VALUE and\nMAX-VALUE that maintain α and β (and the bookkeeping to pass these parameters along).\nAdding dynamic move-ordering schemes, such as trying first the moves that were found\nto be best in the past, brings us quite close to the theoretical limit. The past could be the\nprevious move—often the same threats remain—or it could come from previous exploration\nof the current move. One way to gain information from the current move is with iterative\ndeepening search. First, search 1 ply deep and record the best path of moves. Then search\n1 ply deeper, but use the recorded path to inform move ordering. As we saw in Chapter 3,\niterative deepening on an exponential game tree adds only a constant fraction to the total\nsearch time, which can be more than made up from better move ordering. The best moves are\noften called killer moves and to try them first is called the killer move heuristic.\nKILLER MOVES\nIn Chapter 3, we noted that repeated states in the search tree can cause an exponential\nincrease in search cost. In many games, repeated states occur frequently because of transpo-\nsitions—different permutations of the move sequence that end up in the same position. For\nTRANSPOSITION\nexample, if White has one move, a1, that can be answered by Black with b1 and an unre-\nlated move a2 on the other side of the board that can be answered by b2, then the sequences\n[a1, b1, a2, b2] and [a2, b2, a1, b1] both end up in the same position. It is worthwhile to store\nthe evaluation of the resulting position in a hash table the first time it is encountered so that\nwe don’t have to recompute it on subsequent occurrences. The hash table of previously seen\npositions is traditionally called a transposition table; it is essentially identical to the explored\nTRANSPOSITION\nTABLE\nSection 5.4.\nImperfect Real-Time Decisions\n171\nlist in GRAPH-SEARCH (Section 3.3). Using a transposition table can have a dramatic effect,\nsometimes as much as doubling the reachable search depth in chess. On the other hand, if we\nare evaluating a million nodes per second, at some point it is not practical to keep all of them\nin the transposition table. Various strategies have been used to choose which nodes to keep\nand which to discard.\n5.4\nIMPERFECT REAL-TIME DECISIONS\nThe minimax algorithm generates the entire game search space, whereas the alpha–beta algo-\nrithm allows us to prune large parts of it. However, alpha–beta still has to search all the way\nto terminal states for at least a portion of the search space. This depth is usually not practical,\nbecause moves must be made in a reasonable amount of time—typically a few minutes at\nmost. Claude Shannon’s paper Programming a Computer for Playing Chess (1950) proposed\ninstead that programs should cut off the search earlier and apply a heuristic evaluation func-\ntion to states in the search, effectively turning nonterminal nodes into terminal leaves. In\nEVALUATION\nFUNCTION\nother words, the suggestion is to alter minimax or alpha–beta in two ways: replace the utility\nfunction by a heuristic evaluation function EVAL, which estimates the position’s utility, and\nreplace the terminal test by a cutoff test that decides when to apply EVAL. That gives us the\nCUTOFF TEST\nfollowing for heuristic minimax for state s and maximum depth d:\nH-MINIMAX(s, d) =\n⎧\n⎨\n⎩\nEVAL(s)\nif CUTOFF-TEST(s, d)\nmaxa∈Actions(s) H-MINIMAX(RESULT(s, a), d + 1)\nif PLAYER(s) = MAX\nmina∈Actions(s) H-MINIMAX(RESULT(s,a), d + 1)\nif PLAYER(s) = MIN.\n5.4.1\nEvaluation functions\nAn evaluation function returns an estimate of the expected utility of the game from a given\nposition, just as the heuristic functions of Chapter 3 return an estimate of the distance to\nthe goal. The idea of an estimator was not new when Shannon proposed it. For centuries,\nchess players (and aficionados of other games) have developed ways of judging the value of\na position because humans are even more limited in the amount of search they can do than\nare computer programs. It should be clear that the performance of a game-playing program\ndepends strongly on the quality of its evaluation function. An inaccurate evaluation function\nwill guide an agent toward positions that turn out to be lost. How exactly do we design good\nevaluation functions?\nFirst, the evaluation function should order the terminal states in the same way as the\ntrue utility function: states that are wins must evaluate better than draws, which in turn must\nbe better than losses. Otherwise, an agent using the evaluation function might err even if it\ncan see ahead all the way to the end of the game. Second, the computation must not take\ntoo long! (The whole point is to search faster.) Third, for nonterminal states, the evaluation\nfunction should be strongly correlated with the actual chances of winning.\n172\nChapter\n5.\nAdversarial Search\nOne might well wonder about the phrase “chances of winning.” After all, chess is not a\ngame of chance: we know the current state with certainty, and no dice are involved. But if the\nsearch must be cut off at nonterminal states, then the algorithm will necessarily be uncertain\nabout the final outcomes of those states. This type of uncertainty is induced by computational,\nrather than informational, limitations. Given the limited amount of computation that the\nevaluation function is allowed to do for a given state, the best it can do is make a guess about\nthe final outcome.\nLet us make this idea more concrete. Most evaluation functions work by calculating\nvarious features of the state—for example, in chess, we would have features for the number\nof white pawns, black pawns, white queens, black queens, and so on. The features, taken\ntogether, define various categories or equivalence classes of states: the states in each category\nhave the same values for all the features. For example, one category contains all two-pawn\nvs. one-pawn endgames. Any given category, generally speaking, will contain some states\nthat lead to wins, some that lead to draws, and some that lead to losses. The evaluation\nfunction cannot know which states are which, but it can return a single value that reflects the\nproportion of states with each outcome. For example, suppose our experience suggests that\n72% of the states encountered in the two-pawns vs. one-pawn category lead to a win (utility\n+1); 20% to a loss (0), and 8% to a draw (1/2). Then a reasonable evaluation for states in\nthe category is the expected value: (0.72 × +1) + (0.20 × 0) + (0.08 × 1/2) = 0.76. In\nEXPECTED VALUE\nprinciple, the expected value can be determined for each category, resulting in an evaluation\nfunction that works for any state. As with terminal states, the evaluation function need not\nreturn actual expected values as long as the ordering of the states is the same.\nIn practice, this kind of analysis requires too many categories and hence too much\nexperience to estimate all the probabilities of winning. Instead, most evaluation functions\ncompute separate numerical contributions from each feature and then combine them to find\nthe total value. For example, introductory chess books give an approximate material value\nMATERIAL VALUE\nfor each piece: each pawn is worth 1, a knight or bishop is worth 3, a rook 5, and the queen 9.\nOther features such as “good pawn structure” and “king safety” might be worth half a pawn,\nsay. These feature values are then simply added up to obtain the evaluation of the position.\nA secure advantage equivalent to a pawn gives a substantial likelihood of winning, and\na secure advantage equivalent to three pawns should give almost certain victory, as illustrated\nin Figure 5.8(a). Mathematically, this kind of evaluation function is called a weighted linear\nfunction because it can be expressed as\nWEIGHTED LINEAR\nFUNCTION\nEVAL(s) = w1f1(s) + w2f2(s) + · · · + wnfn(s) =\nn\n\f\ni=1\nwifi(s) ,\nwhere each wi is a weight and each fi is a feature of the position. For chess, the fi could be\nthe numbers of each kind of piece on the board, and the wi could be the values of the pieces\n(1 for pawn, 3 for bishop, etc.).\nAdding up the values of features seems like a reasonable thing to do, but in fact it\ninvolves a strong assumption: that the contribution of each feature is independent of the\nvalues of the other features. For example, assigning the value 3 to a bishop ignores the fact\nthat bishops are more powerful in the endgame, when they have a lot of space to maneuver.\nSection 5.4.\nImperfect Real-Time Decisions\n173\n(b) White to move\n(a) White to move\nFigure 5.8\nTwo chess positions that differ only in the position of the rook at lower right.\nIn (a), Black has an advantage of a knight and two pawns, which should be enough to win\nthe game. In (b), White will capture the queen, giving it an advantage that should be strong\nenough to win.\nFor this reason, current programs for chess and other games also use nonlinear combinations\nof features. For example, a pair of bishops might be worth slightly more than twice the value\nof a single bishop, and a bishop is worth more in the endgame (that is, when the move number\nfeature is high or the number of remaining pieces feature is low).\nThe astute reader will have noticed that the features and weights are not part of the rules\nof chess! They come from centuries of human chess-playing experience. In games where this\nkind of experience is not available, the weights of the evaluation function can be estimated\nby the machine learning techniques of Chapter 18. Reassuringly, applying these techniques\nto chess has confirmed that a bishop is indeed worth about three pawns.\n5.4.2\nCutting off search\nThe next step is to modify ALPHA-BETA-SEARCH so that it will call the heuristic EVAL\nfunction when it is appropriate to cut off the search. We replace the two lines in Figure 5.7\nthat mention TERMINAL-TEST with the following line:\nif CUTOFF-TEST(state, depth) then return EVAL(state)\nWe also must arrange for some bookkeeping so that the current depth is incremented on each\nrecursive call. The most straightforward approach to controlling the amount of search is to set\na fixed depth limit so that CUTOFF-TEST(state, depth) returns true for all depth greater than\nsome fixed depth d. (It must also return true for all terminal states, just as TERMINAL-TEST\ndid.) The depth d is chosen so that a move is selected within the allocated time. A more\nrobust approach is to apply iterative deepening. (See Chapter 3.) When time runs out, the\nprogram returns the move selected by the deepest completed search. As a bonus, iterative\ndeepening also helps with move ordering.\n174\nChapter\n5.\nAdversarial Search\nThese simple approaches can lead to errors due to the approximate nature of the eval-\nuation function. Consider again the simple evaluation function for chess based on material\nadvantage. Suppose the program searches to the depth limit, reaching the position in Fig-\nure 5.8(b), where Black is ahead by a knight and two pawns. It would report this as the\nheuristic value of the state, thereby declaring that the state is a probable win by Black. But\nWhite’s next move captures Black’s queen with no compensation. Hence, the position is\nreally won for White, but this can be seen only by looking ahead one more ply.\nObviously, a more sophisticated cutoff test is needed. The evaluation function should be\napplied only to positions that are quiescent—that is, unlikely to exhibit wild swings in value\nQUIESCENCE\nin the near future. In chess, for example, positions in which favorable captures can be made\nare not quiescent for an evaluation function that just counts material. Nonquiescent positions\ncan be expanded further until quiescent positions are reached. This extra search is called a\nquiescence search; sometimes it is restricted to consider only certain types of moves, such\nQUIESCENCE\nSEARCH\nas capture moves, that will quickly resolve the uncertainties in the position.\nThe horizon effect is more difficult to eliminate. It arises when the program is facing\nHORIZON EFFECT\nan opponent’s move that causes serious damage and is ultimately unavoidable, but can be\ntemporarily avoided by delaying tactics. Consider the chess game in Figure 5.9. It is clear\nthat there is no way for the black bishop to escape. For example, the white rook can capture\nit by moving to h1, then a1, then a2; a capture at depth 6 ply. But Black does have a sequence\nof moves that pushes the capture of the bishop “over the horizon.” Suppose Black searches\nto depth 8 ply. Most moves by Black will lead to the eventual capture of the bishop, and thus\nwill be marked as “bad” moves. But Black will consider checking the white king with the\npawn at e4. This will lead to the king capturing the pawn. Now Black will consider checking\nagain, with the pawn at f5, leading to another pawn capture. That takes up 4 ply, and from\nthere the remaining 4 ply is not enough to capture the bishop. Black thinks that the line of\nplay has saved the bishop at the price of two pawns, when actually all it has done is push the\ninevitable capture of the bishop beyond the horizon that Black can see.\nOne strategy to mitigate the horizon effect is the singular extension, a move that is\nSINGULAR\nEXTENSION\n“clearly better” than all other moves in a given position. Once discovered anywhere in the\ntree in the course of a search, this singular move is remembered. When the search reaches the\nnormal depth limit, the algorithm checks to see if the singular extension is a legal move; if it\nis, the algorithm allows the move to be considered. This makes the tree deeper, but because\nthere will be few singular extensions, it does not add many total nodes to the tree.\n5.4.3\nForward pruning\nSo far, we have talked about cutting off search at a certain level and about doing alpha–\nbeta pruning that provably has no effect on the result (at least with respect to the heuristic\nevaluation values). It is also possible to do forward pruning, meaning that some moves at\nFORWARD PRUNING\na given node are pruned immediately without further consideration. Clearly, most humans\nplaying chess consider only a few moves from each position (at least consciously). One\napproach to forward pruning is beam search: on each ply, consider only a “beam” of the n\nBEAM SEARCH\nbest moves (according to the evaluation function) rather than considering all possible moves.\nSection 5.4.\nImperfect Real-Time Decisions\n175\na b c d e f g h\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8\nFigure 5.9\nThe horizon effect. With Black to move, the black bishop is surely doomed.\nBut Black can forestall that event by checking the white king with its pawns, forcing the king\nto capture the pawns. This pushes the inevitable loss of the bishop over the horizon, and thus\nthe pawn sacrifices are seen by the search algorithm as good moves rather than bad ones.\nUnfortunately, this approach is rather dangerous because there is no guarantee that the best\nmove will not be pruned away.\nThe PROBCUT, or probabilistic cut, algorithm (Buro, 1995) is a forward-pruning ver-\nsion of alpha–beta search that uses statistics gained from prior experience to lessen the chance\nthat the best move will be pruned. Alpha–beta search prunes any node that is provably out-\nside the current (α, β) window. PROBCUT also prunes nodes that are probably outside the\nwindow. It computes this probability by doing a shallow search to compute the backed-up\nvalue v of a node and then using past experience to estimate how likely it is that a score of v\nat depth d in the tree would be outside (α, β). Buro applied this technique to his Othello pro-\ngram, LOGISTELLO, and found that a version of his program with PROBCUT beat the regular\nversion 64% of the time, even when the regular version was given twice as much time.\nCombining all the techniques described here results in a program that can play cred-\nitable chess (or other games). Let us assume we have implemented an evaluation function for\nchess, a reasonable cutoff test with a quiescence search, and a large transposition table. Let\nus also assume that, after months of tedious bit-bashing, we can generate and evaluate around\na million nodes per second on the latest PC, allowing us to search roughly 200 million nodes\nper move under standard time controls (three minutes per move). The branching factor for\nchess is about 35, on average, and 355 is about 50 million, so if we used minimax search,\nwe could look ahead only about five plies. Though not incompetent, such a program can be\nfooled easily by an average human chess player, who can occasionally plan six or eight plies\nahead. With alpha–beta search we get to about 10 plies, which results in an expert level of\nplay. Section 5.8 describes additional pruning techniques that can extend the effective search\ndepth to roughly 14 plies. To reach grandmaster status we would need an extensively tuned\nevaluation function and a large database of optimal opening and endgame moves.\n176\nChapter\n5.\nAdversarial Search\n5.4.4\nSearch versus lookup\nSomehow it seems like overkill for a chess program to start a game by considering a tree of a\nbillion game states, only to conclude that it will move its pawn to e4. Books describing good\nplay in the opening and endgame in chess have been available for about a century (Tattersall,\n1911). It is not surprising, therefore, that many game-playing programs use table lookup\nrather than search for the opening and ending of games.\nFor the openings, the computer is mostly relying on the expertise of humans. The best\nadvice of human experts on how to play each opening is copied from books and entered into\ntables for the computer’s use. However, computers can also gather statistics from a database\nof previously played games to see which opening sequences most often lead to a win. In\nthe early moves there are few choices, and thus much expert commentary and past games on\nwhich to draw. Usually after ten moves we end up in a rarely seen position, and the program\nmust switch from table lookup to search.\nNear the end of the game there are again fewer possible positions, and thus more chance\nto do lookup.\nBut here it is the computer that has the expertise: computer analysis of\nendgames goes far beyond anything achieved by humans. A human can tell you the gen-\neral strategy for playing a king-and-rook-versus-king (KRK) endgame: reduce the opposing\nking’s mobility by squeezing it toward one edge of the board, using your king to prevent the\nopponent from escaping the squeeze. Other endings, such as king, bishop, and knight versus\nking (KBNK), are difficult to master and have no succinct strategy description. A computer,\non the other hand, can completely solve the endgame by producing a policy, which is a map-\nPOLICY\nping from every possible state to the best move in that state. Then we can just look up the best\nmove rather than recompute it anew. How big will the KBNK lookup table be? It turns out\nthere are 462 ways that two kings can be placed on the board without being adjacent. After\nthe kings are placed, there are 62 empty squares for the bishop, 61 for the knight, and two\npossible players to move next, so there are just 462 × 62 × 61 × 2 = 3, 494, 568 possible\npositions. Some of these are checkmates; mark them as such in a table. Then do a retrograde\nRETROGRADE\nminimax search: reverse the rules of chess to do unmoves rather than moves. Any move by\nWhite that, no matter what move Black responds with, ends up in a position marked as a win,\nmust also be a win. Continue this search until all 3,494,568 positions are resolved as win,\nloss, or draw, and you have an infallible lookup table for all KBNK endgames.\nUsing this technique and a tour de force of optimization tricks, Ken Thompson (1986,\n1996) and Lewis Stiller (1992, 1996) solved all chess endgames with up to five pieces and\nsome with six pieces, making them available on the Internet. Stiller discovered one case\nwhere a forced mate existed but required 262 moves; this caused some consternation because\nthe rules of chess require a capture or pawn move to occur within 50 moves. Later work by\nMarc Bourzutschky and Yakov Konoval (Bourzutschky, 2006) solved all pawnless six-piece\nand some seven-piece endgames; there is a KQNKRBN endgame that with best play requires\n517 moves until a capture, which then leads to a mate.\nIf we could extend the chess endgame tables from 6 pieces to 32, then White would\nknow on the opening move whether it would be a win, loss, or draw. This has not happened\nso far for chess, but it has happened for checkers, as explained in the historical notes section.\nSection 5.5.\nStochastic Games\n177\n5.5\nSTOCHASTIC GAMES\nIn real life, many unpredictable external events can put us into unforeseen situations. Many\ngames mirror this unpredictability by including a random element, such as the throwing of\ndice. We call these stochastic games. Backgammon is a typical game that combines luck\nSTOCHASTIC GAMES\nand skill. Dice are rolled at the beginning of a player’s turn to determine the legal moves. In\nthe backgammon position of Figure 5.10, for example, White has rolled a 6–5 and has four\npossible moves.\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n24\n23\n22\n21\n20\n19\n18\n17\n16\n15\n14\n13\n0\n25\nFigure 5.10\nA typical backgammon position. The goal of the game is to move all one’s\npieces off the board. White moves clockwise toward 25, and Black moves counterclockwise\ntoward 0. A piece can move to any position unless multiple opponent pieces are there; if there\nis one opponent, it is captured and must start over. In the position shown, White has rolled\n6–5 and must choose among four legal moves: (5–10,5–11), (5–11,19–24), (5–10,10–16),\nand (5–11,11–16), where the notation (5–11,11–16) means move one piece from position 5\nto 11, and then move a piece from 11 to 16.\nAlthough White knows what his or her own legal moves are, White does not know what\nBlack is going to roll and thus does not know what Black’s legal moves will be. That means\nWhite cannot construct a standard game tree of the sort we saw in chess and tic-tac-toe. A\ngame tree in backgammon must include chance nodes in addition to MAX and MIN nodes.\nCHANCE NODES\nChance nodes are shown as circles in Figure 5.11. The branches leading from each chance\nnode denote the possible dice rolls; each branch is labeled with the roll and its probability.\nThere are 36 ways to roll two dice, each equally likely; but because a 6–5 is the same as a 5–6,\nthere are only 21 distinct rolls. The six doubles (1–1 through 6–6) each have a probability of\n1/36, so we say P(1–1) = 1/36. The other 15 distinct rolls each have a 1/18 probability.\n178\nChapter\n5.\nAdversarial Search\nCHANCE\nMIN\nMAX\nCHANCE\nMAX\n. . .\n. . .\nB\n1\n. . .\n1,1\n1/36\n1,2\n1/18\nTERMINAL\n1,2\n1/18\n...\n...\n...\n...\n...\n...\n...\n1,1\n1/36\n...\n...\n...\n...\n...\n...\nC\n. . .\n1/18\n6,5\n6,6\n1/36\n1/18\n6,5\n6,6\n1/36\n2\n–1\n1\n–1\nFigure 5.11\nSchematic game tree for a backgammon position.\nThe next step is to understand how to make correct decisions. Obviously, we still want\nto pick the move that leads to the best position. However, positions do not have definite\nminimax values. Instead, we can only calculate the expected value of a position: the average\nEXPECTED VALUE\nover all possible outcomes of the chance nodes.\nThis leads us to generalize the minimax value for deterministic games to an expecti-\nminimax value for games with chance nodes. Terminal nodes and MAX and MIN nodes (for\nEXPECTIMINIMAX\nVALUE\nwhich the dice roll is known) work exactly the same way as before. For chance nodes we\ncompute the expected value, which is the sum of the value over all outcomes, weighted by\nthe probability of each chance action:\nEXPECTIMINIMAX(s) =\n⎧\n⎪\n⎪\n⎨\n⎪\n⎪\n⎩\nUTILITY(s)\nif TERMINAL-TEST(s)\nmaxa EXPECTIMINIMAX(RESULT(s, a))\nif PLAYER(s) = MAX\nmina EXPECTIMINIMAX(RESULT(s, a))\nif PLAYER(s) = MIN\n\u0002\nr P(r)EXPECTIMINIMAX(RESULT(s, r)) if PLAYER(s) = CHANCE\nwhere r represents a possible dice roll (or other chance event) and RESULT(s, r) is the same\nstate as s, with the additional fact that the result of the dice roll is r.\n5.5.1\nEvaluation functions for games of chance\nAs with minimax, the obvious approximation to make with expectiminimax is to cut the\nsearch off at some point and apply an evaluation function to each leaf. One might think that\nevaluation functions for games such as backgammon should be just like evaluation functions\nSection 5.5.\nStochastic Games\n179\nfor chess—they just need to give higher scores to better positions. But in fact, the presence of\nchance nodes means that one has to be more careful about what the evaluation values mean.\nFigure 5.12 shows what happens: with an evaluation function that assigns the values [1, 2,\n3, 4] to the leaves, move a1 is best; with values [1, 20, 30, 400], move a2 is best. Hence,\nthe program behaves totally differently if we make a change in the scale of some evaluation\nvalues! It turns out that to avoid this sensitivity, the evaluation function must be a positive\nlinear transformation of the probability of winning from a position (or, more generally, of the\nexpected utility of the position). This is an important and general property of situations in\nwhich uncertainty is involved, and we discuss it further in Chapter 16.\nCHANCE\nMIN\nMAX\n2\n2\n3\n3\n1\n1\n4\n4\n2\n3\n1\n4\n.9\n.1\n.9\n.1\n2.1\n1.3\n20\n20\n30\n30\n1\n1\n400\n400\n20\n30\n1\n400\n.9\n.1\n.9\n.1\n21\n40.9\na1\na2\na1\na2\nFigure 5.12\nAn order-preserving transformation on leaf values changes the best move.\nIf the program knew in advance all the dice rolls that would occur for the rest of the\ngame, solving a game with dice would be just like solving a game without dice, which mini-\nmax does in O(bm) time, where b is the branching factor and m is the maximum depth of the\ngame tree. Because expectiminimax is also considering all the possible dice-roll sequences,\nit will take O(bmnm), where n is the number of distinct rolls.\nEven if the search depth is limited to some small depth d, the extra cost compared with\nthat of minimax makes it unrealistic to consider looking ahead very far in most games of\nchance. In backgammon n is 21 and b is usually around 20, but in some situations can be as\nhigh as 4000 for dice rolls that are doubles. Three plies is probably all we could manage.\nAnother way to think about the problem is this: the advantage of alpha–beta is that\nit ignores future developments that just are not going to happen, given best play. Thus, it\nconcentrates on likely occurrences. In games with dice, there are no likely sequences of\nmoves, because for those moves to take place, the dice would first have to come out the right\nway to make them legal. This is a general problem whenever uncertainty enters the picture:\nthe possibilities are multiplied enormously, and forming detailed plans of action becomes\npointless because the world probably will not play along.\nIt may have occurred to you that something like alpha–beta pruning could be applied\n180\nChapter\n5.\nAdversarial Search\nto game trees with chance nodes. It turns out that it can. The analysis for MIN and MAX\nnodes is unchanged, but we can also prune chance nodes, using a bit of ingenuity. Consider\nthe chance node C in Figure 5.11 and what happens to its value as we examine and evaluate\nits children. Is it possible to find an upper bound on the value of C before we have looked\nat all its children? (Recall that this is what alpha–beta needs in order to prune a node and its\nsubtree.) At first sight, it might seem impossible because the value of C is the average of its\nchildren’s values, and in order to compute the average of a set of numbers, we must look at\nall the numbers. But if we put bounds on the possible values of the utility function, then we\ncan arrive at bounds for the average without looking at every number. For example, say that\nall utility values are between −2 and +2; then the value of leaf nodes is bounded, and in turn\nwe can place an upper bound on the value of a chance node without looking at all its children.\nAn alternative is to do Monte Carlo simulation to evaluate a position. Start with\nMONTE CARLO\nSIMULATION\nan alpha–beta (or other) search algorithm. From a start position, have the algorithm play\nthousands of games against itself, using random dice rolls. In the case of backgammon, the\nresulting win percentage has been shown to be a good approximation of the value of the\nposition, even if the algorithm has an imperfect heuristic and is searching only a few plies\n(Tesauro, 1995). For games with dice, this type of simulation is called a rollout.\nROLLOUT\n5.6\nPARTIALLY OBSERVABLE GAMES\nChess has often been described as war in miniature, but it lacks at least one major charac-\nteristic of real wars, namely, partial observability. In the “fog of war,” the existence and\ndisposition of enemy units is often unknown until revealed by direct contact. As a result,\nwarfare includes the use of scouts and spies to gather information and the use of concealment\nand bluff to confuse the enemy. Partially observable games share these characteristics and\nare thus qualitatively different from the games described in the preceding sections.\n5.6.1\nKriegspiel: Partially observable chess\nIn deterministic partially observable games, uncertainty about the state of the board arises en-\ntirely from lack of access to the choices made by the opponent. This class includes children’s\ngames such as Battleships (where each player’s ships are placed in locations hidden from the\nopponent but do not move) and Stratego (where piece locations are known but piece types are\nhidden). We will examine the game of Kriegspiel, a partially observable variant of chess in\nKRIEGSPIEL\nwhich pieces can move but are completely invisible to the opponent.\nThe rules of Kriegspiel are as follows: White and Black each see a board containing\nonly their own pieces. A referee, who can see all the pieces, adjudicates the game and period-\nically makes announcements that are heard by both players. On his turn, White proposes to\nthe referee any move that would be legal if there were no black pieces. If the move is in fact\nnot legal (because of the black pieces), the referee announces “illegal.” In this case, White\nmay keep proposing moves until a legal one is found—and learns more about the location of\nBlack’s pieces in the process. Once a legal move is proposed, the referee announces one or\nSection 5.6.\nPartially Observable Games\n181\nmore of the following: “Capture on square X” if there is a capture, and “Check by D” if the\nblack king is in check, where D is the direction of the check, and can be one of “Knight,”\n“Rank,” “File,” “Long diagonal,” or “Short diagonal.” (In case of discovered check, the ref-\neree may make two “Check” announcements.) If Black is checkmated or stalemated, the\nreferee says so; otherwise, it is Black’s turn to move.\nKriegspiel may seem terrifyingly impossible, but humans manage it quite well and com-\nputer programs are beginning to catch up. It helps to recall the notion of a belief state as\ndefined in Section 4.4 and illustrated in Figure 4.14—the set of all logically possible board\nstates given the complete history of percepts to date. Initially, White’s belief state is a sin-\ngleton because Black’s pieces haven’t moved yet. After White makes a move and Black re-\nsponds, White’s belief state contains 20 positions because Black has 20 replies to any White\nmove. Keeping track of the belief state as the game progresses is exactly the problem of state\nestimation, for which the update step is given in Equation (4.6). We can map Kriegspiel\nstate estimation directly onto the partially observable, nondeterministic framework of Sec-\ntion 4.4 if we consider the opponent as the source of nondeterminism; that is, the RESULTS\nof White’s move are composed from the (predictable) outcome of White’s own move and the\nunpredictable outcome given by Black’s reply.3\nGiven a current belief state, White may ask, “Can I win the game?” For a partially\nobservable game, the notion of a strategy is altered; instead of specifying a move to make\nfor each possible move the opponent might make, we need a move for every possible percept\nsequence that might be received. For Kriegspiel, a winning strategy, or guaranteed check-\nmate, is one that, for each possible percept sequence, leads to an actual checkmate for every\nGUARANTEED\nCHECKMATE\npossible board state in the current belief state, regardless of how the opponent moves. With\nthis definition, the opponent’s belief state is irrelevant—the strategy has to work even if the\nopponent can see all the pieces. This greatly simplifies the computation. Figure 5.13 shows\npart of a guaranteed checkmate for the KRK (king and rook against king) endgame. In this\ncase, Black has just one piece (the king), so a belief state for White can be shown in a single\nboard by marking each possible position of the Black king.\nThe general AND-OR search algorithm can be applied to the belief-state space to find\nguaranteed checkmates, just as in Section 4.4. The incremental belief-state algorithm men-\ntioned in that section often finds midgame checkmates up to depth 9—probably well beyond\nthe abilities of human players.\nIn addition to guaranteed checkmates, Kriegspiel admits an entirely new concept that\nmakes no sense in fully observable games: probabilistic checkmate. Such checkmates are\nPROBABILISTIC\nCHECKMATE\nstill required to work in every board state in the belief state; they are probabilistic with respect\nto randomization of the winning player’s moves. To get the basic idea, consider the problem\nof finding a lone black king using just the white king. Simply by moving randomly, the\nwhite king will eventually bump into the black king even if the latter tries to avoid this fate,\nsince Black cannot keep guessing the right evasive moves indefinitely. In the terminology of\nprobability theory, detection occurs with probability 1. The KBNK endgame—king, bishop\n3 Sometimes, the belief state will become too large to represent just as a list of board states, but we will ignore\nthis issue for now; Chapters 7 and 8 suggest methods for compactly representing very large belief states.\n182\nChapter\n5.\nAdversarial Search\na\n1\n2\n3\n4\nd\nb\nc\nKc3 ?\n“Illegal”\n“OK”\nRc3 ?\n“OK”\n“Check”\nFigure 5.13\nPart of a guaranteed checkmate in the KRK endgame, shown on a reduced\nboard. In the initial belief state, Black’s king is in one of three possible locations. By a\ncombination of probing moves, the strategy narrows this down to one. Completion of the\ncheckmate is left as an exercise.\nand knight against king—is won in this sense; White presents Black with an infinite random\nsequence of choices, for one of which Black will guess incorrectly and reveal his position,\nleading to checkmate. The KBBK endgame, on the other hand, is won with probability 1−ϵ.\nWhite can force a win only by leaving one of his bishops unprotected for one move. If\nBlack happens to be in the right place and captures the bishop (a move that would lose if the\nbishops are protected), the game is drawn. White can choose to make the risky move at some\nrandomly chosen point in the middle of a very long sequence, thus reducing ϵ to an arbitrarily\nsmall constant, but cannot reduce ϵ to zero.\nIt is quite rare that a guaranteed or probabilistic checkmate can be found within any\nreasonable depth, except in the endgame. Sometimes a checkmate strategy works for some of\nthe board states in the current belief state but not others. Trying such a strategy may succeed,\nleading to an accidental checkmate—accidental in the sense that White could not know that\nACCIDENTAL\nCHECKMATE\nit would be checkmate—if Black’s pieces happen to be in the right places. (Most checkmates\nin games between humans are of this accidental nature.) This idea leads naturally to the\nquestion of how likely it is that a given strategy will win, which leads in turn to the question\nof how likely it is that each board state in the current belief state is the true board state.\nSection 5.6.\nPartially Observable Games\n183\nOne’s first inclination might be to propose that all board states in the current belief state\nare equally likely—but this can’t be right. Consider, for example, White’s belief state after\nBlack’s first move of the game. By definition (assuming that Black plays optimally), Black\nmust have played an optimal move, so all board states resulting from suboptimal moves ought\nto be assigned zero probability. This argument is not quite right either, because each player’s\ngoal is not just to move pieces to the right squares but also to minimize the information that\nthe opponent has about their location. Playing any predictable “optimal” strategy provides\nthe opponent with information. Hence, optimal play in partially observable games requires\na willingness to play somewhat randomly. (This is why restaurant hygiene inspectors do\nrandom inspection visits.) This means occasionally selecting moves that may seem “intrinsi-\ncally” weak—but they gain strength from their very unpredictability, because the opponent is\nunlikely to have prepared any defense against them.\nFrom these considerations, it seems that the probabilities associated with the board\nstates in the current belief state can only be calculated given an optimal randomized strat-\negy; in turn, computing that strategy seems to require knowing the probabilities of the var-\nious states the board might be in. This conundrum can be resolved by adopting the game-\ntheoretic notion of an equilibrium solution, which we pursue further in Chapter 17. An\nequilibrium specifies an optimal randomized strategy for each player. Computing equilib-\nria is prohibitively expensive, however, even for small games, and is out of the question for\nKriegspiel. At present, the design of effective algorithms for general Kriegspiel play is an\nopen research topic. Most systems perform bounded-depth lookahead in their own belief-\nstate space, ignoring the opponent’s belief state. Evaluation functions resemble those for the\nobservable game but include a component for the size of the belief state—smaller is better!\n5.6.2\nCard games\nCard games provide many examples of stochastic partial observability, where the missing\ninformation is generated randomly. For example, in many games, cards are dealt randomly at\nthe beginning of the game, with each player receiving a hand that is not visible to the other\nplayers. Such games include bridge, whist, hearts, and some forms of poker.\nAt first sight, it might seem that these card games are just like dice games: the cards are\ndealt randomly and determine the moves available to each player, but all the “dice” are rolled\nat the beginning! Even though this analogy turns out to be incorrect, it suggests an effective\nalgorithm: consider all possible deals of the invisible cards; solve each one as if it were a\nfully observable game; and then choose the move that has the best outcome averaged over all\nthe deals. Suppose that each deal s occurs with probability P(s); then the move we want is\nargmax\na\n\f\ns\nP(s) MINIMAX(RESULT(s,a)) .\n(5.1)\nHere, we run exact MINIMAX if computationally feasible; otherwise, we run H-MINIMAX.\nNow, in most card games, the number of possible deals is rather large. For example,\nin bridge play, each player sees just two of the four hands; there are two unseen hands of 13\ncards each, so the number of deals is\n\u001426\n13\n\u0015\n= 10, 400, 600. Solving even one deal is quite\ndifficult, so solving ten million is out of the question. Instead, we resort to a Monte Carlo\n184\nChapter\n5.\nAdversarial Search\napproximation: instead of adding up all the deals, we take a random sample of N deals,\nwhere the probability of deal s appearing in the sample is proportional to P(s):\nargmax\na\n1\nN\nN\n\f\ni = 1\nMINIMAX(RESULT(si, a)) .\n(5.2)\n(Notice that P(s) does not appear explicitly in the summation, because the samples are al-\nready drawn according to P(s).) As N grows large, the sum over the random sample tends\nto the exact value, but even for fairly small N—say, 100 to 1,000—the method gives a good\napproximation. It can also be applied to deterministic games such as Kriegspiel, given some\nreasonable estimate of P(s).\nFor games like whist and hearts, where there is no bidding or betting phase before play\ncommences, each deal will be equally likely and so the values of P(s) are all equal. For\nbridge, play is preceded by a bidding phase in which each team indicates how many tricks it\nexpects to win. Since players bid based on the cards they hold, the other players learn more\nabout the probability of each deal. Taking this into account in deciding how to play the hand\nis tricky, for the reasons mentioned in our description of Kriegspiel: players may bid in such\na way as to minimize the information conveyed to their opponents. Even so, the approach is\nquite effective for bridge, as we show in Section 5.7.\nThe strategy described in Equations 5.1 and 5.2 is sometimes called averaging over\nclairvoyance because it assumes that the game will become observable to both players im-\nmediately after the first move. Despite its intuitive appeal, the strategy can lead one astray.\nConsider the following story:\nDay 1: Road A leads to a heap of gold; Road B leads to a fork. Take the left fork and\nyou’ll find a bigger heap of gold, but take the right fork and you’ll be run over by a bus.\nDay 2: Road A leads to a heap of gold; Road B leads to a fork. Take the right fork and\nyou’ll find a bigger heap of gold, but take the left fork and you’ll be run over by a bus.\nDay 3: Road A leads to a heap of gold; Road B leads to a fork. One branch of the\nfork leads to a bigger heap of gold, but take the wrong fork and you’ll be hit by a bus.\nUnfortunately you don’t know which fork is which.\nAveraging over clairvoyance leads to the following reasoning: on Day 1, B is the right choice;\non Day 2, B is the right choice; on Day 3, the situation is the same as either Day 1 or Day 2,\nso B must still be the right choice.\nNow we can see how averaging over clairvoyance fails: it does not consider the belief\nstate that the agent will be in after acting. A belief state of total ignorance is not desirable, es-\npecially when one possibility is certain death. Because it assumes that every future state will\nautomatically be one of perfect knowledge, the approach never selects actions that gather in-\nformation (like the first move in Figure 5.13); nor will it choose actions that hide information\nfrom the opponent or provide information to a partner because it assumes that they already\nknow the information; and it will never bluff in poker,4 because it assumes the opponent can\nBLUFF\nsee its cards. In Chapter 17, we show how to construct algorithms that do all these things by\nvirtue of solving the true partially observable decision problem.\n4 Bluffing—betting as if one’s hand is good, even when it’s not—is a core part of poker strategy.\nSection 5.7.\nState-of-the-Art Game Programs\n185\n5.7\nSTATE-OF-THE-ART GAME PROGRAMS\nIn 1965, the Russian mathematician Alexander Kronrod called chess “the Drosophila of ar-\ntificial intelligence.” John McCarthy disagrees: whereas geneticists use fruit flies to make\ndiscoveries that apply to biology more broadly, AI has used chess to do the equivalent of\nbreeding very fast fruit flies. Perhaps a better analogy is that chess is to AI as Grand Prix\nmotor racing is to the car industry: state-of-the-art game programs are blindingly fast, highly\noptimized machines that incorporate the latest engineering advances, but they aren’t much\nuse for doing the shopping or driving off-road. Nonetheless, racing and game-playing gen-\nerate excitement and a steady stream of innovations that have been adopted by the wider\ncommunity. In this section we look at what it takes to come out on top in various games.\nChess: IBM’s DEEP BLUE chess program, now retired, is well known for defeating world\nCHESS\nchampion Garry Kasparov in a widely publicized exhibition match. Deep Blue ran on a par-\nallel computer with 30 IBM RS/6000 processors doing alpha–beta search. The unique part\nwas a configuration of 480 custom VLSI chess processors that performed move generation\nand move ordering for the last few levels of the tree, and evaluated the leaf nodes. Deep Blue\nsearched up to 30 billion positions per move, reaching depth 14 routinely. The key to its\nsuccess seems to have been its ability to generate singular extensions beyond the depth limit\nfor sufficiently interesting lines of forcing/forced moves. In some cases the search reached a\ndepth of 40 plies. The evaluation function had over 8000 features, many of them describing\nhighly specific patterns of pieces. An “opening book” of about 4000 positions was used, as\nwell as a database of 700,000 grandmaster games from which consensus recommendations\ncould be extracted. The system also used a large endgame database of solved positions con-\ntaining all positions with five pieces and many with six pieces. This database had the effect\nof substantially extending the effective search depth, allowing Deep Blue to play perfectly in\nsome cases even when it was many moves away from checkmate.\nThe success of DEEP BLUE reinforced the widely held belief that progress in computer\ngame-playing has come primarily from ever-more-powerful hardware—a view encouraged\nby IBM. But algorithmic improvements have allowed programs running on standard PCs\nto win World Computer Chess Championships. A variety of pruning heuristics are used to\nreduce the effective branching factor to less than 3 (compared with the actual branching factor\nof about 35). The most important of these is the null move heuristic, which generates a good\nNULL MOVE\nlower bound on the value of a position, using a shallow search in which the opponent gets\nto move twice at the beginning. This lower bound often allows alpha–beta pruning without\nthe expense of a full-depth search. Also important is futility pruning, which helps decide in\nFUTILITY PRUNING\nadvance which moves will cause a beta cutoff in the successor nodes.\nHYDRA can be seen as the successor to DEEP BLUE. HYDRA runs on a 64-processor\ncluster with 1 gigabyte per processor and with custom hardware in the form of FPGA (Field\nProgrammable Gate Array) chips. HYDRA reaches 200 million evaluations per second, about\nthe same as Deep Blue, but HYDRA reaches 18 plies deep rather than just 14 because of\naggressive use of the null move heuristic and forward pruning.\n186\nChapter\n5.\nAdversarial Search\nRYBKA, winner of the 2008 and 2009 World Computer Chess Championships, is con-\nsidered the strongest current computer player. It uses an off-the-shelf 8-core 3.2 GHz Intel\nXeon processor, but little is known about the design of the program. RYBKA’s main ad-\nvantage appears to be its evaluation function, which has been tuned by its main developer,\nInternational Master Vasik Rajlich, and at least three other grandmasters.\nThe most recent matches suggest that the top computer chess programs have pulled\nahead of all human contenders. (See the historical notes for details.)\nCheckers: Jonathan Schaeffer and colleagues developed CHINOOK, which runs on regular\nCHECKERS\nPCs and uses alpha–beta search. Chinook defeated the long-running human champion in an\nabbreviated match in 1990, and since 2007 CHINOOK has been able to play perfectly by using\nalpha–beta search combined with a database of 39 trillion endgame positions.\nOthello, also called Reversi, is probably more popular as a computer game than as a board\nOTHELLO\ngame. It has a smaller search space than chess, usually 5 to 15 legal moves, but evaluation\nexpertise had to be developed from scratch. In 1997, the LOGISTELLO program (Buro, 2002)\ndefeated the human world champion, Takeshi Murakami, by six games to none. It is generally\nacknowledged that humans are no match for computers at Othello.\nBackgammon: Section 5.5 explained why the inclusion of uncertainty from dice rolls makes\nBACKGAMMON\ndeep search an expensive luxury. Most work on backgammon has gone into improving the\nevaluation function.\nGerry Tesauro (1992) combined reinforcement learning with neural\nnetworks to develop a remarkably accurate evaluator that is used with a search to depth 2\nor 3. After playing more than a million training games against itself, Tesauro’s program,\nTD-GAMMON, is competitive with top human players. The program’s opinions on the open-\ning moves of the game have in some cases radically altered the received wisdom.\nGo is the most popular board game in Asia. Because the board is 19 × 19 and moves are\nGO\nallowed into (almost) every empty square, the branching factor starts at 361, which is too\ndaunting for regular alpha–beta search methods. In addition, it is difficult to write an eval-\nuation function because control of territory is often very unpredictable until the endgame.\nTherefore the top programs, such as MOGO, avoid alpha–beta search and instead use Monte\nCarlo rollouts. The trick is to decide what moves to make in the course of the rollout. There is\nno aggressive pruning; all moves are possible. The UCT (upper confidence bounds on trees)\nmethod works by making random moves in the first few iterations, and over time guiding\nthe sampling process to prefer moves that have led to wins in previous samples. Some tricks\nare added, including knowledge-based rules that suggest particular moves whenever a given\npattern is detected and limited local search to decide tactical questions. Some programs also\ninclude special techniques from combinatorial game theory to analyze endgames. These\nCOMBINATORIAL\nGAME THEORY\ntechniques decompose a position into sub-positions that can be analyzed separately and then\ncombined (Berlekamp and Wolfe, 1994; M¨uller, 2003). The optimal solutions obtained in\nthis way have surprised many professional Go players, who thought they had been playing\noptimally all along. Current Go programs play at the master level on a reduced 9 × 9 board,\nbut are still at advanced amateur level on a full board.\nBridge is a card game of imperfect information: a player’s cards are hidden from the other\nBRIDGE\nplayers. Bridge is also a multiplayer game with four players instead of two, although the\nSection 5.8.\nAlternative Approaches\n187\nplayers are paired into two teams. As in Section 5.6, optimal play in partially observable\ngames like bridge can include elements of information gathering, communication, and careful\nweighing of probabilities. Many of these techniques are used in the Bridge Baron program\n(Smith et al., 1998), which won the 1997 computer bridge championship. While it does\nnot play optimally, Bridge Baron is one of the few successful game-playing systems to use\ncomplex, hierarchical plans (see Chapter 11) involving high-level ideas, such as finessing and\nsqueezing, that are familiar to bridge players.\nThe GIB program (Ginsberg, 1999) won the 2000 computer bridge championship quite\ndecisively using the Monte Carlo method. Since then, other winning programs have followed\nGIB’s lead. GIB’s major innovation is using explanation-based generalization to compute\nEXPLANATION-\nBASED\nGENERALIZATION\nand cache general rules for optimal play in various standard classes of situations rather than\nevaluating each situation individually. For example, in a situation where one player has the\ncards A-K-Q-J-4-3-2 of one suit and another player has 10-9-8-7-6-5, there are 7 × 6 = 42\nways that the first player can lead from that suit and the second player can follow. But GIB\ntreats these situations as just two: the first player can lead either a high card or a low card;\nthe exact cards played don’t matter. With this optimization (and a few others), GIB can solve\na 52-card, fully observable deal exactly in about a second. GIB’s tactical accuracy makes up\nfor its inability to reason about information. It finished 12th in a field of 35 in the par contest\n(involving just play of the hand, not bidding) at the 1998 human world championship, far\nexceeding the expectations of many human experts.\nThere are several reasons why GIB plays at expert level with Monte Carlo simulation,\nwhereas Kriegspiel programs do not. First, GIB’s evaluation of the fully observable version\nof the game is exact, searching the full game tree, while Kriegspiel programs rely on inexact\nheuristics. But far more important is the fact that in bridge, most of the uncertainty in the\npartially observable information comes from the randomness of the deal, not from the adver-\nsarial play of the opponent. Monte Carlo simulation handles randomness well, but does not\nalways handle strategy well, especially when the strategy involves the value of information.\nScrabble: Most people think the hard part about Scrabble is coming up with good words, but\nSCRABBLE\ngiven the official dictionary, it turns out to be rather easy to program a move generator to find\nthe highest-scoring move (Gordon, 1994). That doesn’t mean the game is solved, however:\nmerely taking the top-scoring move each turn results in a good but not expert player. The\nproblem is that Scrabble is both partially observable and stochastic: you don’t know what\nletters the other player has or what letters you will draw next. So playing Scrabble well\ncombines the difficulties of backgammon and bridge. Nevertheless, in 2006, the QUACKLE\nprogram defeated the former world champion, David Boys, 3–2.\n5.8\nALTERNATIVE APPROACHES\nBecause calculating optimal decisions in games is intractable in most cases, all algorithms\nmust make some assumptions and approximations. The standard approach, based on mini-\nmax, evaluation functions, and alpha–beta, is just one way to do this. Probably because it has\n188\nChapter\n5.\nAdversarial Search\nMAX\n99\n1000\n1000\n1000\n100\n101\n102\n100\n100\n99\nMIN\nFigure 5.14\nA two-ply game tree for which heuristic minimax may make an error.\nbeen worked on for so long, the standard approach dominates other methods in tournament\nplay. Some believe that this has caused game playing to become divorced from the main-\nstream of AI research: the standard approach no longer provides much room for new insight\ninto general questions of decision making. In this section, we look at the alternatives.\nFirst, let us consider heuristic minimax. It selects an optimal move in a given search\ntree provided that the leaf node evaluations are exactly correct. In reality, evaluations are\nusually crude estimates of the value of a position and can be considered to have large errors\nassociated with them. Figure 5.14 shows a two-ply game tree for which minimax suggests\ntaking the right-hand branch because 100 > 99. That is the correct move if the evaluations\nare all correct. But of course the evaluation function is only approximate. Suppose that\nthe evaluation of each node has an error that is independent of other nodes and is randomly\ndistributed with mean zero and standard deviation of σ. Then when σ = 5, the left-hand\nbranch is actually better 71% of the time, and 58% of the time when σ = 2. The intuition\nbehind this is that the right-hand branch has four nodes that are close to 99; if an error in\nthe evaluation of any one of the four makes the right-hand branch slip below 99, then the\nleft-hand branch is better.\nIn reality, circumstances are actually worse than this because the error in the evaluation\nfunction is not independent. If we get one node wrong, the chances are high that nearby nodes\nin the tree will also be wrong. The fact that the node labeled 99 has siblings labeled 1000\nsuggests that in fact it might have a higher true value. We can use an evaluation function\nthat returns a probability distribution over possible values, but it is difficult to combine these\ndistributions properly, because we won’t have a good model of the very strong dependencies\nthat exist between the values of sibling nodes\nNext, we consider the search algorithm that generates the tree. The aim of an algorithm\ndesigner is to specify a computation that runs quickly and yields a good move. The alpha–beta\nalgorithm is designed not just to select a good move but also to calculate bounds on the values\nof all the legal moves. To see why this extra information is unnecessary, consider a position\nin which there is only one legal move. Alpha–beta search still will generate and evaluate a\nlarge search tree, telling us that the only move is the best move and assigning it a value. But\nsince we have to make the move anyway, knowing the move’s value is useless. Similarly, if\nthere is one obviously good move and several moves that are legal but lead to a quick loss, we\nSection 5.9.\nSummary\n189\nwould not want alpha–beta to waste time determining a precise value for the lone good move.\nBetter to just make the move quickly and save the time for later. This leads to the idea of the\nutility of a node expansion. A good search algorithm should select node expansions of high\nutility—that is, ones that are likely to lead to the discovery of a significantly better move. If\nthere are no node expansions whose utility is higher than their cost (in terms of time), then\nthe algorithm should stop searching and make a move. Notice that this works not only for\nclear-favorite situations but also for the case of symmetrical moves, for which no amount of\nsearch will show that one move is better than another.\nThis kind of reasoning about what computations to do is called metareasoning (rea-\nMETAREASONING\nsoning about reasoning). It applies not just to game playing but to any kind of reasoning\nat all. All computations are done in the service of trying to reach better decisions, all have\ncosts, and all have some likelihood of resulting in a certain improvement in decision quality.\nAlpha–beta incorporates the simplest kind of metareasoning, namely, a theorem to the effect\nthat certain branches of the tree can be ignored without loss. It is possible to do much better.\nIn Chapter 16, we see how these ideas can be made precise and implementable.\nFinally, let us reexamine the nature of search itself. Algorithms for heuristic search\nand for game playing generate sequences of concrete states, starting from the initial state\nand then applying an evaluation function. Clearly, this is not how humans play games. In\nchess, one often has a particular goal in mind—for example, trapping the opponent’s queen—\nand can use this goal to selectively generate plausible plans for achieving it. This kind of\ngoal-directed reasoning or planning sometimes eliminates combinatorial search altogether.\nDavid Wilkins’ (1980) PARADISE is the only program to have used goal-directed reasoning\nsuccessfully in chess: it was capable of solving some chess problems requiring an 18-move\ncombination. As yet there is no good understanding of how to combine the two kinds of\nalgorithms into a robust and efficient system, although Bridge Baron might be a step in the\nright direction. A fully integrated system would be a significant achievement not just for\ngame-playing research but also for AI research in general, because it would be a good basis\nfor a general intelligent agent.\n5.9\nSUMMARY\nWe have looked at a variety of games to understand what optimal play means and to under-\nstand how to play well in practice. The most important ideas are as follows:\n• A game can be defined by the initial state (how the board is set up), the legal actions\nin each state, the result of each action, a terminal test (which says when the game is\nover), and a utility function that applies to terminal states.\n• In two-player zero-sum games with perfect information, the minimax algorithm can\nselect optimal moves by a depth-first enumeration of the game tree.\n• The alpha–beta search algorithm computes the same optimal move as minimax, but\nachieves much greater efficiency by eliminating subtrees that are provably irrelevant.\n• Usually, it is not feasible to consider the whole game tree (even with alpha–beta), so we\n190\nChapter\n5.\nAdversarial Search\nneed to cut the search off at some point and apply a heuristic evaluation function that\nestimates the utility of a state.\n• Many game programs precompute tables of best moves in the opening and endgame so\nthat they can look up a move rather than search.\n• Games of chance can be handled by an extension to the minimax algorithm that eval-\nuates a chance node by taking the average utility of all its children, weighted by the\nprobability of each child.\n• Optimal play in games of imperfect information, such as Kriegspiel and bridge, re-\nquires reasoning about the current and future belief states of each player. A simple\napproximation can be obtained by averaging the value of an action over each possible\nconfiguration of missing information.\n• Programs have bested even champion human players at games such as chess, checkers,\nand Othello. Humans retain the edge in several games of imperfect information, such\nas poker, bridge, and Kriegspiel, and in games with very large branching factors and\nlittle good heuristic knowledge, such as Go.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nThe early history of mechanical game playing was marred by numerous frauds. The most\nnotorious of these was Baron Wolfgang von Kempelen’s (1734–1804) “The Turk,” a supposed\nchess-playing automaton that defeated Napoleon before being exposed as a magician’s trick\ncabinet housing a human chess expert (see Levitt, 2000). It played from 1769 to 1854. In\n1846, Charles Babbage (who had been fascinated by the Turk) appears to have contributed\nthe first serious discussion of the feasibility of computer chess and checkers (Morrison and\nMorrison, 1961). He did not understand the exponential complexity of search trees, claiming\n“the combinations involved in the Analytical Engine enormously surpassed any required,\neven by the game of chess.” Babbage also designed, but did not build, a special-purpose\nmachine for playing tic-tac-toe. The first true game-playing machine was built around 1890\nby the Spanish engineer Leonardo Torres y Quevedo. It specialized in the “KRK” (king and\nrook vs. king) chess endgame, guaranteeing a win with king and rook from any position.\nThe minimax algorithm is traced to a 1912 paper by Ernst Zermelo, the developer of\nmodern set theory. The paper unfortunately contained several errors and did not describe min-\nimax correctly. On the other hand, it did lay out the ideas of retrograde analysis and proposed\n(but did not prove) what became known as Zermelo’s theorem: that chess is determined—\nWhite can force a win or Black can or it is a draw; we just don’t know which. Zermelo says\nthat should we eventually know, “Chess would of course lose the character of a game at all.”\nA solid foundation for game theory was developed in the seminal work Theory of Games\nand Economic Behavior (von Neumann and Morgenstern, 1944), which included an analysis\nshowing that some games require strategies that are randomized (or otherwise unpredictable).\nSee Chapter 17 for more information.\nBibliographical and Historical Notes\n191\nJohn McCarthy conceived the idea of alpha–beta search in 1956, although he did not\npublish it. The NSS chess program (Newell et al., 1958) used a simplified version of alpha–\nbeta; it was the first chess program to do so. Alpha–beta pruning was described by Hart and\nEdwards (1961) and Hart et al. (1972). Alpha–beta was used by the “Kotok–McCarthy” chess\nprogram written by a student of John McCarthy (Kotok, 1962). Knuth and Moore (1975)\nproved the correctness of alpha–beta and analysed its time complexity. Pearl (1982b) shows\nalpha–beta to be asymptotically optimal among all fixed-depth game-tree search algorithms.\nSeveral attempts have been made to overcome the problems with the “standard ap-\nproach” that were outlined in Section 5.8. The first nonexhaustive heuristic search algorithm\nwith some theoretical grounding was probably B∗(Berliner, 1979), which attempts to main-\ntain interval bounds on the possible value of a node in the game tree rather than giving it\na single point-valued estimate. Leaf nodes are selected for expansion in an attempt to re-\nfine the top-level bounds until one move is “clearly best.” Palay (1985) extends the B∗idea\nusing probability distributions on values in place of intervals. David McAllester’s (1988)\nconspiracy number search expands leaf nodes that, by changing their values, could cause\nthe program to prefer a new move at the root. MGSS∗(Russell and Wefald, 1989) uses the\ndecision-theoretic techniques of Chapter 16 to estimate the value of expanding each leaf in\nterms of the expected improvement in decision quality at the root. It outplayed an alpha–\nbeta algorithm at Othello despite searching an order of magnitude fewer nodes. The MGSS∗\napproach is, in principle, applicable to the control of any form of deliberation.\nAlpha–beta search is in many ways the two-player analog of depth-first branch-and-\nbound, which is dominated by A∗in the single-agent case. The SSS∗algorithm (Stockman,\n1979) can be viewed as a two-player A∗and never expands more nodes than alpha–beta to\nreach the same decision. The memory requirements and computational overhead of the queue\nmake SSS∗in its original form impractical, but a linear-space version has been developed\nfrom the RBFS algorithm (Korf and Chickering, 1996). Plaat et al. (1996) developed a new\nview of SSS∗as a combination of alpha–beta and transposition tables, showing how to over-\ncome the drawbacks of the original algorithm and developing a new variant called MTD(f)\nthat has been adopted by a number of top programs.\nD. F. Beal (1980) and Dana Nau (1980, 1983) studied the weaknesses of minimax ap-\nplied to approximate evaluations. They showed that under certain assumptions about the dis-\ntribution of leaf values in the tree, minimaxing can yield values at the root that are actually less\nreliable than the direct use of the evaluation function itself. Pearl’s book Heuristics (1984)\npartially explains this apparent paradox and analyzes many game-playing algorithms. Baum\nand Smith (1997) propose a probability-based replacement for minimax, showing that it re-\nsults in better choices in certain games. The expectiminimax algorithm was proposed by\nDonald Michie (1966). Bruce Ballard (1983) extended alpha–beta pruning to cover trees\nwith chance nodes and Hauk (2004) reexamines this work and provides empirical results.\nKoller and Pfeffer (1997) describe a system for completely solving partially observ-\nable games. The system is quite general, handling games whose optimal strategy requires\nrandomized moves and games that are more complex than those handled by any previous\nsystem. Still, it can’t handle games as complex as poker, bridge, and Kriegspiel. Frank\net al. (1998) describe several variants of Monte Carlo search, including one where MIN has\n192\nChapter\n5.\nAdversarial Search\ncomplete information but MAX does not. Among deterministic, partially observable games,\nKriegspiel has received the most attention. Ferguson demonstrated hand-derived random-\nized strategies for winning Kriegspiel with a bishop and knight (1992) or two bishops (1995)\nagainst a king. The first Kriegspiel programs concentrated on finding endgame checkmates\nand performed AND–OR search in belief-state space (Sakuta and Iida, 2002; Bolognesi and\nCiancarini, 2003). Incremental belief-state algorithms enabled much more complex midgame\ncheckmates to be found (Russell and Wolfe, 2005; Wolfe and Russell, 2007), but efficient\nstate estimation remains the primary obstacle to effective general play (Parker et al., 2005).\nChess was one of the first tasks undertaken in AI, with early efforts by many of the pio-\nneers of computing, including Konrad Zuse in 1945, Norbert Wiener in his book Cybernetics\n(1948), and Alan Turing in 1950 (see Turing et al., 1953). But it was Claude Shannon’s\narticle Programming a Computer for Playing Chess (1950) that had the most complete set\nof ideas, describing a representation for board positions, an evaluation function, quiescence\nsearch, and some ideas for selective (nonexhaustive) game-tree search. Slater (1950) and the\ncommentators on his article also explored the possibilities for computer chess play.\nD. G. Prinz (1952) completed a program that solved chess endgame problems but did\nnot play a full game. Stan Ulam and a group at the Los Alamos National Lab produced a\nprogram that played chess on a 6 × 6 board with no bishops (Kister et al., 1957). It could\nsearch 4 plies deep in about 12 minutes. Alex Bernstein wrote the first documented program\nto play a full game of standard chess (Bernstein and Roberts, 1958).5\nThe first computer chess match featured the Kotok–McCarthy program from MIT (Ko-\ntok, 1962) and the ITEP program written in the mid-1960s at Moscow’s Institute of Theo-\nretical and Experimental Physics (Adelson-Velsky et al., 1970). This intercontinental match\nwas played by telegraph. It ended with a 3–1 victory for the ITEP program in 1967. The first\nchess program to compete successfully with humans was MIT’s MACHACK-6 (Greenblatt\net al., 1967). Its Elo rating of approximately 1400 was well above the novice level of 1000.\nThe Fredkin Prize, established in 1980, offered awards for progressive milestones in\nchess play. The $5,000 prize for the first program to achieve a master rating went to BELLE\n(Condon and Thompson, 1982), which achieved a rating of 2250. The $10,000 prize for the\nfirst program to achieve a USCF (United States Chess Federation) rating of 2500 (near the\ngrandmaster level) was awarded to DEEP THOUGHT (Hsu et al., 1990) in 1989. The grand\nprize, $100,000, went to DEEP BLUE (Campbell et al., 2002; Hsu, 2004) for its landmark\nvictory over world champion Garry Kasparov in a 1997 exhibition match. Kasparov wrote:\nThe decisive game of the match was Game 2, which left a scar in my memory . . . we saw\nsomething that went well beyond our wildest expectations of how well a computer would\nbe able to foresee the long-term positional consequences of its decisions. The machine\nrefused to move to a position that had a decisive short-term advantage—showing a very\nhuman sense of danger. (Kasparov, 1997)\nProbably the most complete description of a modern chess program is provided by Ernst\nHeinz (2000), whose DARKTHOUGHT program was the highest-ranked noncommercial PC\nprogram at the 1999 world championships.\n5 A Russian program, BESM may have predated Bernstein’s program.\nBibliographical and Historical Notes\n193\n(a)\n(b)\nFigure 5.15\nPioneers in computer chess: (a) Herbert Simon and Allen Newell, developers\nof the NSS program (1958); (b) John McCarthy and the Kotok–McCarthy program on an\nIBM 7090 (1967).\nIn recent years, chess programs are pulling ahead of even the world’s best humans.\nIn 2004–2005 HYDRA defeated grand master Evgeny Vladimirov 3.5–0.5, world champion\nRuslan Ponomariov 2–0, and seventh-ranked Michael Adams 5.5–0.5. In 2006, DEEP FRITZ\nbeat world champion Vladimir Kramnik 4–2, and in 2007 RYBKA defeated several grand\nmasters in games in which it gave odds (such as a pawn) to the human players. As of 2009,\nthe highest Elo rating ever recorded was Kasparov’s 2851. HYDRA (Donninger and Lorenz,\n2004) is rated somewhere between 2850 and 3000, based mostly on its trouncing of Michael\nAdams. The RYBKA program is rated between 2900 and 3100, but this is based on a small\nnumber of games and is not considered reliable. Ross (2004) shows how human players have\nlearned to exploit some of the weaknesses of the computer programs.\nCheckers was the first of the classic games fully played by a computer. Christopher\nStrachey (1952) wrote the first working program for checkers. Beginning in 1952, Arthur\nSamuel of IBM, working in his spare time, developed a checkers program that learned its\nown evaluation function by playing itself thousands of times (Samuel, 1959, 1967). We\ndescribe this idea in more detail in Chapter 21. Samuel’s program began as a novice but\nafter only a few days’ self-play had improved itself beyond Samuel’s own level. In 1962 it\ndefeated Robert Nealy, a champion at “blind checkers,” through an error on his part. When\none considers that Samuel’s computing equipment (an IBM 704) had 10,000 words of main\nmemory, magnetic tape for long-term storage, and a .000001 GHz processor, the win remains\na great accomplishment.\nThe challenge started by Samuel was taken up by Jonathan Schaeffer of the University\nof Alberta. His CHINOOK program came in second in the 1990 U.S. Open and earned the\nright to challenge for the world championship. It then ran up against a problem, in the form\nof Marion Tinsley. Dr. Tinsley had been world champion for over 40 years, losing only\nthree games in all that time. In the first match against CHINOOK, Tinsley suffered his fourth\n194\nChapter\n5.\nAdversarial Search\nand fifth losses, but won the match 20.5–18.5. A rematch at the 1994 world championship\nended prematurely when Tinsley had to withdraw for health reasons. CHINOOK became the\nofficial world champion. Schaeffer kept on building on his database of endgames, and in\n2007 “solved” checkers (Schaeffer et al., 2007; Schaeffer, 2008). This had been predicted by\nRichard Bellman (1965). In the paper that introduced the dynamic programming approach\nto retrograde analysis, he wrote, “In checkers, the number of possible moves in any given\nsituation is so small that we can confidently expect a complete digital computer solution to\nthe problem of optimal play in this game.” Bellman did not, however, fully appreciate the\nsize of the checkers game tree. There are about 500 quadrillion positions. After 18 years\nof computation on a cluster of 50 or more machines, Jonathan Schaeffer’s team completed\nan endgame table for all checkers positions with 10 or fewer pieces: over 39 trillion entries.\nFrom there, they were able to do forward alpha–beta search to derive a policy that proves\nthat checkers is in fact a draw with best play by both sides. Note that this is an application\nof bidirectional search (Section 3.4.6). Building an endgame table for all of checkers would\nbe impractical: it would require a billion gigabytes of storage. Searching without any table\nwould also be impractical: the search tree has about 847 positions, and would take thousands\nof years to search with today’s technology. Only a combination of clever search, endgame\ndata, and a drop in the price of processors and memory could solve checkers. Thus, checkers\njoins Qubic (Patashnik, 1980), Connect Four (Allis, 1988), and Nine-Men’s Morris (Gasser,\n1998) as games that have been solved by computer analysis.\nBackgammon, a game of chance, was analyzed mathematically by Gerolamo Cardano\n(1663), but only taken up for computer play in the late 1970s, first with the BKG pro-\ngram (Berliner, 1980b); it used a complex, manually constructed evaluation function and\nsearched only to depth 1. It was the first program to defeat a human world champion at a ma-\njor classic game (Berliner, 1980a). Berliner readily acknowledged that BKG was very lucky\nwith the dice. Gerry Tesauro’s (1995) TD-GAMMON played consistently at world champion\nlevel. The BGBLITZ program was the winner of the 2008 Computer Olympiad.\nGo is a deterministic game, but the large branching factor makes it challeging. The key\nissues and early literature in computer Go are summarized by Bouzy and Cazenave (2001) and\nM¨uller (2002). Up to 1997 there were no competent Go programs. Now the best programs\nplay most of their moves at the master level; the only problem is that over the course of a\ngame they usually make at least one serious blunder that allows a strong opponent to win.\nWhereas alpha–beta search reigns in most games, many recent Go programs have adopted\nMonte Carlo methods based on the UCT (upper confidence bounds on trees) scheme (Kocsis\nand Szepesvari, 2006). The strongest Go program as of 2009 is Gelly and Silver’s MOGO\n(Wang and Gelly, 2007; Gelly and Silver, 2008). In August 2008, MOGO scored a surprising\nwin against top professional Myungwan Kim, albeit with MOGO receiving a handicap of\nnine stones (about the equivalent of a queen handicap in chess). Kim estimated MOGO’s\nstrength at 2–3 dan, the low end of advanced amateur. For this match, MOGO was run on\nan 800-processor 15 teraflop supercomputer (1000 times Deep Blue). A few weeks later,\nMOGO, with only a five-stone handicap, won against a 6-dan professional. In the 9 × 9 form\nof Go, MOGO is at approximately the 1-dan professional level. Rapid advances are likely\nas experimentation continues with new forms of Monte Carlo search. The Computer Go\nExercises\n195\nNewsletter, published by the Computer Go Association, describes current developments.\nBridge: Smith et al. (1998) report on how their planning-based program won the 1998\ncomputer bridge championship, and (Ginsberg, 2001) describes how his GIB program, based\non Monte Carlo simulation, won the following computer championship and did surprisingly\nwell against human players and standard book problem sets. From 2001–2007, the computer\nbridge championship was won five times by JACK and twice by WBRIDGE5. Neither has\nhad academic articles explaining their structure, but both are rumored to use the Monte Carlo\ntechnique, which was first proposed for bridge by Levy (1989).\nScrabble: A good description of a top program, MAVEN, is given by its creator, Brian\nSheppard (2002). Generating the highest-scoring move is described by Gordon (1994), and\nmodeling opponents is covered by Richards and Amir (2007).\nSoccer (Kitano et al., 1997b; Visser et al., 2008) and billiards (Lam and Greenspan,\n2008; Archibald et al., 2009) and other stochastic games with a continuous space of actions\nare beginning to attract attention in AI, both in simulation and with physical robot players.\nComputer game competitions occur annually, and papers appear in a variety of venues.\nThe rather misleadingly named conference proceedings Heuristic Programming in Artificial\nIntelligence report on the Computer Olympiads, which include a wide variety of games. The\nGeneral Game Competition (Love et al., 2006) tests programs that must learn to play an un-\nknown game given only a logical description of the rules of the game. There are also several\nedited collections of important papers on game-playing research (Levy, 1988a, 1988b; Mars-\nland and Schaeffer, 1990). The International Computer Chess Association (ICCA), founded\nin 1977, publishes the ICGA Journal (formerly the ICCA Journal). Important papers have\nbeen published in the serial anthology Advances in Computer Chess, starting with Clarke\n(1977). Volume 134 of the journal Artificial Intelligence (2002) contains descriptions of\nstate-of-the-art programs for chess, Othello, Hex, shogi, Go, backgammon, poker, Scrabble,\nand other games. Since 1998, a biennial Computers and Games conference has been held.\nEXERCISES\n5.1\nSuppose you have an oracle, OM(s), that correctly predicts the opponent’s move in\nany state. Using this, formulate the definition of a game as a (single-agent) search problem.\nDescribe an algorithm for finding the optimal move.\n5.2\nConsider the problem of solving two 8-puzzles.\na. Give a complete problem formulation in the style of Chapter 3.\nb. How large is the reachable state space? Give an exact numerical expression.\nc. Suppose we make the problem adversarial as follows: the two players take turns mov-\ning; a coin is flipped to determine the puzzle on which to make a move in that turn; and\nthe winner is the first to solve one puzzle. Which algorithm can be used to choose a\nmove in this setting?\nd. Give an informal proof that someone will eventually win if both play perfectly.\n196\nChapter\n5.\nAdversarial Search\n(b)\n(a)\na\nf\ne\nd\nc\nb\nbd\ncd\nad\nce\ncf\ncc\nae\naf\nac\nde\ndf\ndd\ndd\n?\n?\n?\n?\n?\nP\nE\nFigure 5.16\n(a) A map where the cost of every edge is 1. Initially the pursuer P is at node\nb and the evader E is at node d. (b) A partial game tree for this map. Each node is labeled\nwith the P, E positions. P moves first. Branches marked “?” have yet to be explored.\n5.3\nImagine that, in Exercise 3.3, one of the friends wants to avoid the other. The problem\nthen becomes a two-player pursuit–evasion game. We assume now that the players take\nPURSUIT–EVASION\nturns moving. The game ends only when the players are on the same node; the terminal\npayoff to the pursuer is minus the total time taken. (The evader “wins” by never losing.) An\nexample is shown in Figure 5.16.\na. Copy the game tree and mark the values of the terminal nodes.\nb. Next to each internal node, write the strongest fact you can infer about its value (a\nnumber, one or more inequalities such as “≥14”, or a “?”).\nc. Beneath each question mark, write the name of the node reached by that branch.\nd. Explain how a bound on the value of the nodes in (c) can be derived from consideration\nof shortest-path lengths on the map, and derive such bounds for these nodes. Remember\nthe cost to get to each leaf as well as the cost to solve it.\ne. Now suppose that the tree as given, with the leaf bounds from (d), is evaluated from left\nto right. Circle those “?” nodes that would not need to be expanded further, given the\nbounds from part (d), and cross out those that need not be considered at all.\nf. Can you prove anything in general about who wins the game on a map that is a tree?\nExercises\n197\n5.4\nDescribe and implement state descriptions, move generators, terminal tests, utility func-\ntions, and evaluation functions for one or more of the following stochastic games: Monopoly,\nScrabble, bridge play with a given contract, or Texas hold’em poker.\n5.5\nDescribe and implement a real-time, multiplayer game-playing environment, where\ntime is part of the environment state and players are given fixed time allocations.\n5.6\nDiscuss how well the standard approach to game playing would apply to games such as\ntennis, pool, and croquet, which take place in a continuous physical state space.\n5.7\nProve the following assertion: For every game tree, the utility obtained by MAX using\nminimax decisions against a suboptimal MIN will be never be lower than the utility obtained\nplaying against an optimal MIN. Can you come up with a game tree in which MAX can do\nstill better using a suboptimal strategy against a suboptimal MIN?\nA\nB\n1\n4\n3\n2\nFigure 5.17\nThe starting position of a simple game. Player A moves first. The two players\ntake turns moving, and each player must move his token to an open adjacent space in either\ndirection. If the opponent occupies an adjacent space, then a player may jump over the\nopponent to the next open space if any. (For example, if A is on 3 and B is on 2, then A may\nmove back to 1.) The game ends when one player reaches the opposite end of the board. If\nplayer A reaches space 4 first, then the value of the game to A is +1; if player B reaches\nspace 1 first, then the value of the game to A is −1.\n5.8\nConsider the two-player game described in Figure 5.17.\na. Draw the complete game tree, using the following conventions:\n• Write each state as (sA, sB), where sA and sB denote the token locations.\n• Put each terminal state in a square box and write its game value in a circle.\n• Put loop states (states that already appear on the path to the root) in double square\nboxes. Since their value is unclear, annotate each with a “?” in a circle.\nb. Now mark each node with its backed-up minimax value (also in a circle). Explain how\nyou handled the “?” values and why.\nc. Explain why the standard minimax algorithm would fail on this game tree and briefly\nsketch how you might fix it, drawing on your answer to (b). Does your modified algo-\nrithm give optimal decisions for all games with loops?\nd. This 4-square game can be generalized to n squares for any n > 2. Prove that A wins\nif n is even and loses if n is odd.\n5.9\nThis problem exercises the basic concepts of game playing, using tic-tac-toe (noughts\nand crosses) as an example. We define Xn as the number of rows, columns, or diagonals\n198\nChapter\n5.\nAdversarial Search\nwith exactly n X’s and no O’s. Similarly, On is the number of rows, columns, or diagonals\nwith just n O’s. The utility function assigns +1 to any position with X3 = 1 and −1 to any\nposition with O3 = 1. All other terminal positions have utility 0. For nonterminal positions,\nwe use a linear evaluation function defined as Eval(s) = 3X2(s)+X1(s)−(3O2(s)+O1(s)).\na. Approximately how many possible games of tic-tac-toe are there?\nb. Show the whole game tree starting from an empty board down to depth 2 (i.e., one X\nand one O on the board), taking symmetry into account.\nc. Mark on your tree the evaluations of all the positions at depth 2.\nd. Using the minimax algorithm, mark on your tree the backed-up values for the positions\nat depths 1 and 0, and use those values to choose the best starting move.\ne. Circle the nodes at depth 2 that would not be evaluated if alpha–beta pruning were\napplied, assuming the nodes are generated in the optimal order for alpha–beta pruning.\n5.10\nConsider the family of generalized tic-tac-toe games, defined as follows. Each partic-\nular game is specified by a set S of squares and a collection W of winning positions. Each\nwinning position is a subset of S. For example, in standard tic-tac-toe, S is a set of 9 squares\nand W is a collection of 8 subsets of W: the three rows, the three columns, and the two diag-\nonals. In other respects, the game is identical to standard tic-tac-toe. Starting from an empty\nboard, players alternate placing their marks on an empty square. A player who marks every\nsquare in a winning position wins the game. It is a tie if all squares are marked and neither\nplayer has won.\na. Let N = |S|, the number of squares. Give an upper bound on the number of nodes in\nthe complete game tree for generalized tic-tac-toe as a function of N.\nb. Give a lower bound on the size of the game tree for the worst case, where W = { }.\nc. Propose a plausible evaluation function that can be used for any instance of generalized\ntic-tac-toe. The function may depend on S and W.\nd. Assume that it is possible to generate a new board and check whether it is a winning\nposition in 100N machine instructions and assume a 2 gigahertz processor. Ignore\nmemory limitations. Using your estimate in (a), roughly how large a game tree can be\ncompletely solved by alpha–beta in a second of CPU time? a minute? an hour?\n5.11\nDevelop a general game-playing program, capable of playing a variety of games.\na. Implement move generators and evaluation functions for one or more of the following\ngames: Kalah, Othello, checkers, and chess.\nb. Construct a general alpha–beta game-playing agent.\nc. Compare the effect of increasing search depth, improving move ordering, and improv-\ning the evaluation function. How close does your effective branching factor come to the\nideal case of perfect move ordering?\nd. Implement a selective search algorithm, such as B* (Berliner, 1979), conspiracy number\nsearch (McAllester, 1988), or MGSS* (Russell and Wefald, 1989) and compare its\nperformance to A*.\nExercises\n199\nn1\nn2\nnj\nFigure 5.18\nSituation when considering whether to prune node nj.\n5.12\nDescribe how the minimax and alpha–beta algorithms change for two-player, non-\nzero-sum games in which each player has a distinct utility function and both utility functions\nare known to both players. If there are no constraints on the two terminal utilities, is it possible\nfor any node to be pruned by alpha–beta? What if the player’s utility functions on any state\ndiffer by at most a constant k, making the game almost cooperative?\n5.13\nDevelop a formal proof of correctness for alpha–beta pruning. To do this, consider the\nsituation shown in Figure 5.18. The question is whether to prune node nj, which is a max-\nnode and a descendant of node n1. The basic idea is to prune it if and only if the minimax\nvalue of n1 can be shown to be independent of the value of nj.\na. Mode n1 takes on the minimum value among its children: n1 = min(n2, n21, . . . , n2b2).\nFind a similar expression for n2 and hence an expression for n1 in terms of nj.\nb. Let li be the minimum (or maximum) value of the nodes to the left of node ni at depth i,\nwhose minimax value is already known. Similarly, let ri be the minimum (or maximum)\nvalue of the unexplored nodes to the right of ni at depth i. Rewrite your expression for\nn1 in terms of the li and ri values.\nc. Now reformulate the expression to show that in order to affect n1, nj must not exceed\na certain bound derived from the li values.\nd. Repeat the process for the case where nj is a min-node.\n5.14\nProve that alpha–beta pruning takes time O(2m/2) with optimal move ordering, where\nm is the maximum depth of the game tree.\n5.15\nSuppose you have a chess program that can evaluate 10 million nodes per second.\nDecide on a compact representation of a game state for storage in a transposition table. About\nhow many entries can you fit in a 2-gigabyte in-memory table? Will that be enough for the\n200\nChapter\n5.\nAdversarial Search\n0.5\n0.5\n0.5\n0.5\n2\n2\n1\n2\n0\n2\n-1\n0\nFigure 5.19\nThe complete game tree for a trivial game with chance nodes.\nthree minutes of search allocated for one move? How many table lookups can you do in the\ntime it would take to do one evaluation? Now suppose the transposition table is stored on\ndisk. About how many evaluations could you do in the time it takes to do one disk seek with\nstandard disk hardware?\n5.16\nThis question considers pruning in games with chance nodes. Figure 5.19 shows the\ncomplete game tree for a trivial game. Assume that the leaf nodes are to be evaluated in left-\nto-right order, and that before a leaf node is evaluated, we know nothing about its value—the\nrange of possible values is −∞to ∞.\na. Copy the figure, mark the value of all the internal nodes, and indicate the best move at\nthe root with an arrow.\nb. Given the values of the first six leaves, do we need to evaluate the seventh and eighth\nleaves? Given the values of the first seven leaves, do we need to evaluate the eighth\nleaf? Explain your answers.\nc. Suppose the leaf node values are known to lie between –2 and 2 inclusive. After the\nfirst two leaves are evaluated, what is the value range for the left-hand chance node?\nd. Circle all the leaves that need not be evaluated under the assumption in (c).\n5.17\nImplement the expectiminimax algorithm and the *-alpha–beta algorithm, which is\ndescribed by Ballard (1983), for pruning game trees with chance nodes. Try them on a game\nsuch as backgammon and measure the pruning effectiveness of *-alpha–beta.\n5.18\nProve that with a positive linear transformation of leaf values (i.e., transforming a\nvalue x to ax + b where a > 0), the choice of move remains unchanged in a game tree, even\nwhen there are chance nodes.\n5.19\nConsider the following procedure for choosing moves in games with chance nodes:\n• Generate some dice-roll sequences (say, 50) down to a suitable depth (say, 8).\n• With known dice rolls, the game tree becomes deterministic. For each dice-roll se-\nquence, solve the resulting deterministic game tree using alpha–beta.\nExercises\n201\n• Use the results to estimate the value of each move and to choose the best.\nWill this procedure work well? Why (or why not)?\n5.20\nIn the following, a “max” tree consists only of max nodes, whereas an “expectimax”\ntree consists of a max node at the root with alternating layers of chance and max nodes. At\nchance nodes, all outcome probabilities are nonzero. The goal is to find the value of the root\nwith a bounded-depth search. For each of (a)–(f), either give an example or explain why this\nis impossible.\na. Assuming that leaf values are finite but unbounded, is pruning (as in alpha–beta) ever\npossible in a max tree?\nb. Is pruning ever possible in an expectimax tree under the same conditions?\nc. If leaf values are all nonnegative, is pruning ever possible in a max tree? Give an\nexample, or explain why not.\nd. If leaf values are all nonnegative, is pruning ever possible in an expectimax tree? Give\nan example, or explain why not.\ne. If leaf values are all in the range [0, 1], is pruning ever possible in a max tree? Give an\nexample, or explain why not.\nf. If leaf values are all in the range [0, 1], is pruning ever possible in an expectimax tree?\ng. Consider the outcomes of a chance node in an expectimax tree. Which of the following\nevaluation orders is most likely to yield pruning opportunities?\n(i) Lowest probability first\n(ii) Highest probability first\n(iii) Doesn’t make any difference\n5.21\nWhich of the following are true and which are false? Give brief explanations.\na. In a fully observable, turn-taking, zero-sum game between two perfectly rational play-\ners, it does not help the first player to know what strategy the second player is using—\nthat is, what move the second player will make, given the first player’s move.\nb. In a partially observable, turn-taking, zero-sum game between two perfectly rational\nplayers, it does not help the first player to know what move the second player will\nmake, given the first player’s move.\nc. A perfectly rational backgammon agent never loses.\n5.22\nConsider carefully the interplay of chance events and partial information in each of the\ngames in Exercise 5.4.\na. For which is the standard expectiminimax model appropriate? Implement the algorithm\nand run it in your game-playing agent, with appropriate modifications to the game-\nplaying environment.\nb. For which would the scheme described in Exercise 5.19 be appropriate?\nc. Discuss how you might deal with the fact that in some of the games, the players do not\nhave the same knowledge of the current state.\n" }, { "chapter": 6, "text": "6\nCONSTRAINT\nSATISFACTION PROBLEMS\nIn which we see how treating states as more than just little black boxes leads to the\ninvention of a range of powerful new search methods and a deeper understanding\nof problem structure and complexity.\nChapters 3 and 4 explored the idea that problems can be solved by searching in a space of\nstates. These states can be evaluated by domain-specific heuristics and tested to see whether\nthey are goal states. From the point of view of the search algorithm, however, each state is\natomic, or indivisible—a black box with no internal structure.\nThis chapter describes a way to solve a wide variety of problems more efficiently. We\nuse a factored representation for each state: a set of variables, each of which has a value.\nA problem is solved when each variable has a value that satisfies all the constraints on the\nvariable. A problem described this way is called a constraint satisfaction problem, or CSP.\nCONSTRAINT\nSATISFACTION\nPROBLEM\nCSP search algorithms take advantage of the structure of states and use general-purpose\nrather than problem-specific heuristics to enable the solution of complex problems. The main\nidea is to eliminate large portions of the search space all at once by identifying variable/value\ncombinations that violate the constraints.\n6.1\nDEFINING CONSTRAINT SATISFACTION PROBLEMS\nA constraint satisfaction problem consists of three components, X, D, and C:\nX is a set of variables, {X1, . . . , Xn}.\nD is a set of domains, {D1, . . . , Dn}, one for each variable.\nC is a set of constraints that specify allowable combinations of values.\nEach domain Di consists of a set of allowable values, {v1, . . . , vk} for variable Xi. Each\nconstraint Ci consists of a pair ⟨scope, rel⟩, where scope is a tuple of variables that participate\nin the constraint and rel is a relation that defines the values that those variables can take on. A\nrelation can be represented as an explicit list of all tuples of values that satisfy the constraint,\nor as an abstract relation that supports two operations: testing if a tuple is a member of the\nrelation and enumerating the members of the relation. For example, if X1 and X2 both have\n202\nSection 6.1.\nDefining Constraint Satisfaction Problems\n203\nthe domain {A,B}, then the constraint saying the two variables must have different values\ncan be written as ⟨(X1, X2), [(A, B), (B, A)]⟩or as ⟨(X1, X2), X1 ̸= X2⟩.\nTo solve a CSP, we need to define a state space and the notion of a solution. Each\nstate in a CSP is defined by an assignment of values to some or all of the variables, {Xi =\nASSIGNMENT\nvi, Xj = vj, . . .}. An assignment that does not violate any constraints is called a consistent\nCONSISTENT\nor legal assignment. A complete assignment is one in which every variable is assigned, and\nCOMPLETE\nASSIGNMENT\na solution to a CSP is a consistent, complete assignment. A partial assignment is one that\nSOLUTION\nPARTIAL\nASSIGNMENT\nassigns values to only some of the variables.\n6.1.1\nExample problem: Map coloring\nSuppose that, having tired of Romania, we are looking at a map of Australia showing each\nof its states and territories (Figure 6.1(a)). We are given the task of coloring each region\neither red, green, or blue in such a way that no neighboring regions have the same color. To\nformulate this as a CSP, we define the variables to be the regions\nX = {WA, NT, Q, NSW , V, SA, T} .\nThe domain of each variable is the set Di = {red, green, blue}. The constraints require\nneighboring regions to have distinct colors. Since there are nine places where regions border,\nthere are nine constraints:\nC = {SA ̸= WA, SA ̸= NT, SA ̸= Q, SA ̸= NSW , SA ̸= V,\nWA ̸= NT, NT ̸= Q, Q ̸= NSW , NSW ̸= V } .\nHere we are using abbreviations; SA ̸= WA is a shortcut for ⟨(SA, WA), SA ̸= WA⟩, where\nSA ̸= WA can be fully enumerated in turn as\n{(red, green), (red, blue), (green, red), (green, blue), (blue, red), (blue, green)} .\nThere are many possible solutions to this problem, such as\n{WA = red, NT = green, Q = red, NSW = green, V = red, SA = blue, T = red }.\nIt can be helpful to visualize a CSP as a constraint graph, as shown in Figure 6.1(b). The\nCONSTRAINT GRAPH\nnodes of the graph correspond to variables of the problem, and a link connects any two vari-\nables that participate in a constraint.\nWhy formulate a problem as a CSP? One reason is that the CSPs yield a natural rep-\nresentation for a wide variety of problems; if you already have a CSP-solving system, it is\noften easier to solve a problem using it than to design a custom solution using another search\ntechnique. In addition, CSP solvers can be faster than state-space searchers because the CSP\nsolver can quickly eliminate large swatches of the search space. For example, once we have\nchosen {SA = blue} in the Australia problem, we can conclude that none of the five neighbor-\ning variables can take on the value blue. Without taking advantage of constraint propagation,\na search procedure would have to consider 35 = 243 assignments for the five neighboring\nvariables; with constraint propagation we never have to consider blue as a value, so we have\nonly 25 = 32 assignments to look at, a reduction of 87%.\nIn regular state-space search we can only ask: is this specific state a goal? No? What\nabout this one? With CSPs, once we find out that a partial assignment is not a solution, we can\n204\nChapter\n6.\nConstraint Satisfaction Problems\nWestern\nAustralia\nNorthern\nTerritory\nSouth\nAustralia\nQueensland\nNew\nSouth\nWales\nVictoria\nTasmania\nWA\nNT\nSA\nQ\nNSW\nV\nT\n(a)\n(b)\nFigure 6.1\n(a) The principal states and territories of Australia. Coloring this map can\nbe viewed as a constraint satisfaction problem (CSP). The goal is to assign colors to each\nregion so that no neighboring regions have the same color. (b) The map-coloring problem\nrepresented as a constraint graph.\nimmediately discard further refinements of the partial assignment. Furthermore, we can see\nwhy the assignment is not a solution—we see which variables violate a constraint—so we can\nfocus attention on the variables that matter. As a result, many problems that are intractable\nfor regular state-space search can be solved quickly when formulated as a CSP.\n6.1.2\nExample problem: Job-shop scheduling\nFactories have the problem of scheduling a day’s worth of jobs, subject to various constraints.\nIn practice, many of these problems are solved with CSP techniques. Consider the problem of\nscheduling the assembly of a car. The whole job is composed of tasks, and we can model each\ntask as a variable, where the value of each variable is the time that the task starts, expressed\nas an integer number of minutes. Constraints can assert that one task must occur before\nanother—for example, a wheel must be installed before the hubcap is put on—and that only\nso many tasks can go on at once. Constraints can also specify that a task takes a certain\namount of time to complete.\nWe consider a small part of the car assembly, consisting of 15 tasks: install axles (front\nand back), affix all four wheels (right and left, front and back), tighten nuts for each wheel,\naffix hubcaps, and inspect the final assembly. We can represent the tasks with 15 variables:\nX = {AxleF , AxleB, Wheel RF , WheelLF , WheelRB, WheelLB, NutsRF ,\nNutsLF , NutsRB, NutsLB, CapRF , CapLF , CapRB, CapLB, Inspect} .\nThe value of each variable is the time that the task starts. Next we represent precedence\nconstraints between individual tasks. Whenever a task T1 must occur before task T2, and\nPRECEDENCE\nCONSTRAINTS\ntask T1 takes duration d1 to complete, we add an arithmetic constraint of the form\nT1 + d1 ≤T2 .\nSection 6.1.\nDefining Constraint Satisfaction Problems\n205\nIn our example, the axles have to be in place before the wheels are put on, and it takes 10\nminutes to install an axle, so we write\nAxleF + 10 ≤WheelRF ;\nAxleF + 10 ≤WheelLF ;\nAxleB + 10 ≤WheelRB;\nAxleB + 10 ≤WheelLB .\nNext we say that, for each wheel, we must affix the wheel (which takes 1 minute), then tighten\nthe nuts (2 minutes), and finally attach the hubcap (1 minute, but not represented yet):\nWheelRF + 1 ≤NutsRF ;\nNutsRF + 2 ≤CapRF ;\nWheelLF + 1 ≤NutsLF;\nNutsLF + 2 ≤CapLF;\nWheelRB + 1 ≤NutsRB;\nNutsRB + 2 ≤CapRB;\nWheelLB + 1 ≤NutsLB;\nNutsLB + 2 ≤CapLB .\nSuppose we have four workers to install wheels, but they have to share one tool that helps put\nthe axle in place. We need a disjunctive constraint to say that AxleF and AxleB must not\nDISJUNCTIVE\nCONSTRAINT\noverlap in time; either one comes first or the other does:\n(AxleF + 10 ≤AxleB)\nor\n(AxleB + 10 ≤AxleF) .\nThis looks like a more complicated constraint, combining arithmetic and logic. But it still\nreduces to a set of pairs of values that AxleF and AxleF can take on.\nWe also need to assert that the inspection comes last and takes 3 minutes. For every\nvariable except Inspect we add a constraint of the form X + dX ≤Inspect. Finally, suppose\nthere is a requirement to get the whole assembly done in 30 minutes. We can achieve that by\nlimiting the domain of all variables:\nDi = {1, 2, 3, . . . , 27} .\nThis particular problem is trivial to solve, but CSPs have been applied to job-shop schedul-\ning problems like this with thousands of variables. In some cases, there are complicated\nconstraints that are difficult to specify in the CSP formalism, and more advanced planning\ntechniques are used, as discussed in Chapter 11.\n6.1.3\nVariations on the CSP formalism\nThe simplest kind of CSP involves variables that have discrete, finite domains.\nMap-\nDISCRETE DOMAIN\nFINITE DOMAIN\ncoloring problems and scheduling with time limits are both of this kind. The 8-queens prob-\nlem described in Chapter 3 can also be viewed as a finite-domain CSP, where the variables\nQ1, . . . , Q8 are the positions of each queen in columns 1, . . . , 8 and each variable has the\ndomain Di = {1, 2, 3, 4, 5, 6, 7, 8}.\nA discrete domain can be infinite, such as the set of integers or strings. (If we didn’t put\nINFINITE\na deadline on the job-scheduling problem, there would be an infinite number of start times\nfor each variable.) With infinite domains, it is no longer possible to describe constraints by\nenumerating all allowed combinations of values. Instead, a constraint language must be\nCONSTRAINT\nLANGUAGE\nused that understands constraints such as T1 + d1 ≤T2 directly, without enumerating the\nset of pairs of allowable values for (T1, T2). Special solution algorithms (which we do not\ndiscuss here) exist for linear constraints on integer variables—that is, constraints, such as\nLINEAR\nCONSTRAINTS\nthe one just given, in which each variable appears only in linear form. It can be shown that\nno algorithm exists for solving general nonlinear constraints on integer variables.\nNONLINEAR\nCONSTRAINTS\n206\nChapter\n6.\nConstraint Satisfaction Problems\nConstraint satisfaction problems with continuous domains are common in the real\nCONTINUOUS\nDOMAINS\nworld and are widely studied in the field of operations research. For example, the scheduling\nof experiments on the Hubble Space Telescope requires very precise timing of observations;\nthe start and finish of each observation and maneuver are continuous-valued variables that\nmust obey a variety of astronomical, precedence, and power constraints. The best-known\ncategory of continuous-domain CSPs is that of linear programming problems, where con-\nstraints must be linear equalities or inequalities. Linear programming problems can be solved\nin time polynomial in the number of variables. Problems with different types of constraints\nand objective functions have also been studied—quadratic programming, second-order conic\nprogramming, and so on.\nIn addition to examining the types of variables that can appear in CSPs, it is useful to\nlook at the types of constraints. The simplest type is the unary constraint, which restricts\nUNARY CONSTRAINT\nthe value of a single variable. For example, in the map-coloring problem it could be the case\nthat South Australians won’t tolerate the color green; we can express that with the unary\nconstraint ⟨(SA), SA ̸= green⟩\nA binary constraint relates two variables. For example, SA ̸= NSW is a binary\nBINARY CONSTRAINT\nconstraint. A binary CSP is one with only binary constraints; it can be represented as a\nconstraint graph, as in Figure 6.1(b).\nWe can also describe higher-order constraints, such as asserting that the value of Y is\nbetween X and Z, with the ternary constraint Between(X, Y, Z).\nA constraint involving an arbitrary number of variables is called a global constraint.\nGLOBAL\nCONSTRAINT\n(The name is traditional but confusing because it need not involve all the variables in a prob-\nlem). One of the most common global constraints is Alldiff, which says that all of the\nvariables involved in the constraint must have different values. In Sudoku problems (see\nSection 6.2.6), all variables in a row or column must satisfy an Alldiffconstraint.\nAn-\nother example is provided by cryptarithmetic puzzles. (See Figure 6.2(a).) Each letter in a\nCRYPTARITHMETIC\ncryptarithmetic puzzle represents a different digit. For the case in Figure 6.2(a), this would\nbe represented as the global constraint Alldiff(F, T, U, W, R, O). The addition constraints\non the four columns of the puzzle can be written as the following n-ary constraints:\nO + O = R + 10 · C10\nC10 + W + W = U + 10 · C100\nC100 + T + T = O + 10 · C1000\nC1000 = F ,\nwhere C10, C100, and C1000 are auxiliary variables representing the digit carried over into the\ntens, hundreds, or thousands column. These constraints can be represented in a constraint\nhypergraph, such as the one shown in Figure 6.2(b). A hypergraph consists of ordinary nodes\nCONSTRAINT\nHYPERGRAPH\n(the circles in the figure) and hypernodes (the squares), which represent n-ary constraints.\nAlternatively, as Exercise 6.6 asks you to prove, every finite-domain constraint can be\nreduced to a set of binary constraints if enough auxiliary variables are introduced, so we could\ntransform any CSP into one with only binary constraints; this makes the algorithms simpler.\nAnother way to convert an n-ary CSP to a binary one is the dual graph transformation: create\nDUAL GRAPH\na new graph in which there will be one variable for each constraint in the original graph, and\nSection 6.1.\nDefining Constraint Satisfaction Problems\n207\n(a)\nO\nW\nT\nF\nU\nR\n(b)\n+\nF\nT\nT\nO\nW\nW\nU\nO\nO\nR\nC3\nC1\nC2\nFigure 6.2\n(a) A cryptarithmetic problem. Each letter stands for a distinct digit; the aim is\nto find a substitution of digits for letters such that the resulting sum is arithmetically correct,\nwith the added restriction that no leading zeroes are allowed. (b) The constraint hypergraph\nfor the cryptarithmetic problem, showing the Alldiffconstraint (square box at the top) as\nwell as the column addition constraints (four square boxes in the middle). The variables C1,\nC2, and C3 represent the carry digits for the three columns.\none binary constraint for each pair of constraints in the original graph that share variables. For\nexample, if the original graph has variables {X, Y, Z} and constraints ⟨(X, Y, Z), C1⟩and\n⟨(X, Y ), C2⟩then the dual graph would have variables {C1, C2} with the binary constraint\n⟨(X, Y ), R1⟩, where (X, Y ) are the shared variables and R1 is a new relation that defines the\nconstraint between the shared variables, as specified by the original C1 and C2.\nThere are however two reasons why we might prefer a global constraint such as Alldiff\nrather than a set of binary constraints. First, it is easier and less error-prone to write the\nproblem description using Alldiff. Second, it is possible to design special-purpose inference\nalgorithms for global constraints that are not available for a set of more primitive constraints.\nWe describe these inference algorithms in Section 6.2.5.\nThe constraints we have described so far have all been absolute constraints, violation of\nwhich rules out a potential solution. Many real-world CSPs include preference constraints\nPREFERENCE\nCONSTRAINTS\nindicating which solutions are preferred. For example, in a university class-scheduling prob-\nlem there are absolute constraints that no professor can teach two classes at the same time.\nBut we also may allow preference constraints: Prof. R might prefer teaching in the morning,\nwhereas Prof. N prefers teaching in the afternoon. A schedule that has Prof. R teaching at\n2 p.m. would still be an allowable solution (unless Prof. R happens to be the department chair)\nbut would not be an optimal one. Preference constraints can often be encoded as costs on in-\ndividual variable assignments—for example, assigning an afternoon slot for Prof. R costs\n2 points against the overall objective function, whereas a morning slot costs 1. With this\nformulation, CSPs with preferences can be solved with optimization search methods, either\npath-based or local. We call such a problem a constraint optimization problem, or COP.\nCONSTRAINT\nOPTIMIZATION\nPROBLEM\nLinear programming problems do this kind of optimization.\n208\nChapter\n6.\nConstraint Satisfaction Problems\n6.2\nCONSTRAINT PROPAGATION: INFERENCE IN CSPS\nIn regular state-space search, an algorithm can do only one thing: search. In CSPs there is a\nchoice: an algorithm can search (choose a new variable assignment from several possibilities)\nor do a specific type of inference called constraint propagation: using the constraints to\nINFERENCE\nCONSTRAINT\nPROPAGATION\nreduce the number of legal values for a variable, which in turn can reduce the legal values\nfor another variable, and so on. Constraint propagation may be intertwined with search, or it\nmay be done as a preprocessing step, before search starts. Sometimes this preprocessing can\nsolve the whole problem, so no search is required at all.\nThe key idea is local consistency. If we treat each variable as a node in a graph (see\nLOCAL\nCONSISTENCY\nFigure 6.1(b)) and each binary constraint as an arc, then the process of enforcing local con-\nsistency in each part of the graph causes inconsistent values to be eliminated throughout the\ngraph. There are different types of local consistency, which we now cover in turn.\n6.2.1\nNode consistency\nA single variable (corresponding to a node in the CSP network) is node-consistent if all\nNODE CONSISTENCY\nthe values in the variable’s domain satisfy the variable’s unary constraints. For example,\nin the variant of the Australia map-coloring problem (Figure 6.1) where South Australians\ndislike green, the variable SA starts with domain {red, green, blue}, and we can make it\nnode consistent by eliminating green, leaving SA with the reduced domain {red, blue}. We\nsay that a network is node-consistent if every variable in the network is node-consistent.\nIt is always possible to eliminate all the unary constraints in a CSP by running node\nconsistency. It is also possible to transform all n-ary constraints into binary ones (see Ex-\nercise 6.6). Because of this, it is common to define CSP solvers that work with only binary\nconstraints; we make that assumption for the rest of this chapter, except where noted.\n6.2.2\nArc consistency\nA variable in a CSP is arc-consistent if every value in its domain satisfies the variable’s\nARC CONSISTENCY\nbinary constraints. More formally, Xi is arc-consistent with respect to another variable Xj if\nfor every value in the current domain Di there is some value in the domain Dj that satisfies\nthe binary constraint on the arc (Xi, Xj). A network is arc-consistent if every variable is arc\nconsistent with every other variable. For example, consider the constraint Y = X2 where the\ndomain of both X and Y is the set of digits. We can write this constraint explicitly as\n⟨(X, Y ), {(0, 0), (1, 1), (2, 4), (3, 9))}⟩.\nTo make X arc-consistent with respect to Y , we reduce X’s domain to {0, 1, 2, 3}. If we\nalso make Y arc-consistent with respect to X, then Y ’s domain becomes {0, 1, 4, 9} and the\nwhole CSP is arc-consistent.\nOn the other hand, arc consistency can do nothing for the Australia map-coloring prob-\nlem. Consider the following inequality constraint on (SA, WA):\n{(red, green), (red, blue), (green, red), (green, blue), (blue, red), (blue, green)} .\nSection 6.2.\nConstraint Propagation: Inference in CSPs\n209\nfunction AC-3(csp) returns false if an inconsistency is found and true otherwise\ninputs: csp, a binary CSP with components (X, D, C)\nlocal variables: queue, a queue of arcs, initially all the arcs in csp\nwhile queue is not empty do\n(Xi, Xj) ←REMOVE-FIRST(queue)\nif REVISE(csp, Xi, Xj) then\nif size of Di = 0 then return false\nfor each Xk in Xi.NEIGHBORS - {Xj} do\nadd (Xk, Xi) to queue\nreturn true\nfunction REVISE(csp, Xi, Xj) returns true iff we revise the domain of Xi\nrevised ←false\nfor each x in Di do\nif no value y in Dj allows (x,y) to satisfy the constraint between Xi and Xj then\ndelete x from Di\nrevised ←true\nreturn revised\nFigure 6.3\nThe arc-consistency algorithm AC-3. After applying AC-3, either every arc\nis arc-consistent, or some variable has an empty domain, indicating that the CSP cannot be\nsolved. The name “AC-3” was used by the algorithm’s inventor (Mackworth, 1977) because\nit’s the third version developed in the paper.\nNo matter what value you choose for SA (or for WA), there is a valid value for the other\nvariable. So applying arc consistency has no effect on the domains of either variable.\nThe most popular algorithm for arc consistency is called AC-3 (see Figure 6.3). To\nmake every variable arc-consistent, the AC-3 algorithm maintains a queue of arcs to consider.\n(Actually, the order of consideration is not important, so the data structure is really a set, but\ntradition calls it a queue.) Initially, the queue contains all the arcs in the CSP. AC-3 then pops\noff an arbitrary arc (Xi, Xj) from the queue and makes Xi arc-consistent with respect to Xj.\nIf this leaves Di unchanged, the algorithm just moves on to the next arc. But if this revises\nDi (makes the domain smaller), then we add to the queue all arcs (Xk, Xi) where Xk is a\nneighbor of Xi. We need to do that because the change in Di might enable further reductions\nin the domains of Dk, even if we have previously considered Xk. If Di is revised down to\nnothing, then we know the whole CSP has no consistent solution, and AC-3 can immediately\nreturn failure. Otherwise, we keep checking, trying to remove values from the domains of\nvariables until no more arcs are in the queue. At that point, we are left with a CSP that is\nequivalent to the original CSP—they both have the same solutions—but the arc-consistent\nCSP will in most cases be faster to search because its variables have smaller domains.\nThe complexity of AC-3 can be analyzed as follows. Assume a CSP with n variables,\neach with domain size at most d, and with c binary constraints (arcs). Each arc (Xk, Xi) can\nbe inserted in the queue only d times because Xi has at most d values to delete. Checking\n210\nChapter\n6.\nConstraint Satisfaction Problems\nconsistency of an arc can be done in O(d2) time, so we get O(cd3) total worst-case time.1\nIt is possible to extend the notion of arc consistency to handle n-ary rather than just\nbinary constraints; this is called generalized arc consistency or sometimes hyperarc consis-\ntency, depending on the author. A variable Xi is generalized arc consistent with respect to\nGENERALIZED ARC\nCONSISTENT\nan n-ary constraint if for every value v in the domain of Xi there exists a tuple of values that\nis a member of the constraint, has all its values taken from the domains of the corresponding\nvariables, and has its Xi component equal to v. For example, if all variables have the do-\nmain {0, 1, 2, 3}, then to make the variable X consistent with the constraint X < Y < Z,\nwe would have to eliminate 2 and 3 from the domain of X because the constraint cannot be\nsatisfied when X is 2 or 3.\n6.2.3\nPath consistency\nArc consistency can go a long way toward reducing the domains of variables, sometimes\nfinding a solution (by reducing every domain to size 1) and sometimes finding that the CSP\ncannot be solved (by reducing some domain to size 0). But for other networks, arc consistency\nfails to make enough inferences. Consider the map-coloring problem on Australia, but with\nonly two colors allowed, red and blue. Arc consistency can do nothing because every variable\nis already arc consistent: each can be red with blue at the other end of the arc (or vice versa).\nBut clearly there is no solution to the problem: because Western Australia, Northern Territory\nand South Australia all touch each other, we need at least three colors for them alone.\nArc consistency tightens down the domains (unary constraints) using the arcs (binary\nconstraints). To make progress on problems like map coloring, we need a stronger notion of\nconsistency. Path consistency tightens the binary constraints by using implicit constraints\nPATH CONSISTENCY\nthat are inferred by looking at triples of variables.\nA two-variable set {Xi, Xj} is path-consistent with respect to a third variable Xm if,\nfor every assignment {Xi = a, Xj = b} consistent with the constraints on {Xi, Xj}, there is\nan assignment to Xm that satisfies the constraints on {Xi, Xm} and {Xm, Xj}. This is called\npath consistency because one can think of it as looking at a path from Xi to Xj with Xm in\nthe middle.\nLet’s see how path consistency fares in coloring the Australia map with two colors. We\nwill make the set {WA, SA} path consistent with respect to NT. We start by enumerating the\nconsistent assignments to the set. In this case, there are only two: {WA = red, SA = blue}\nand {WA = blue, SA = red}. We can see that with both of these assignments NT can be\nneither red nor blue (because it would conflict with either WA or SA). Because there is no\nvalid choice for NT, we eliminate both assignments, and we end up with no valid assignments\nfor {WA, SA}. Therefore, we know that there can be no solution to this problem. The PC-2\nalgorithm (Mackworth, 1977) achieves path consistency in much the same way that AC-3\nachieves arc consistency. Because it is so similar, we do not show it here.\n1 The AC-4 algorithm (Mohr and Henderson, 1986) runs in O(cd2) worst-case time but can be slower than AC-3\non average cases. See Exercise 6.13.\nSection 6.2.\nConstraint Propagation: Inference in CSPs\n211\n6.2.4\nK-consistency\nStronger forms of propagation can be defined with the notion of k-consistency. A CSP is\nK-CONSISTENCY\nk-consistent if, for any set of k −1 variables and for any consistent assignment to those\nvariables, a consistent value can always be assigned to any kth variable. 1-consistency says\nthat, given the empty set, we can make any set of one variable consistent: this is what we\ncalled node consistency. 2-consistency is the same as arc consistency. For binary constraint\nnetworks, 3-consistency is the same as path consistency.\nA CSP is strongly k-consistent if it is k-consistent and is also (k −1)-consistent,\nSTRONGLY\nK-CONSISTENT\n(k −2)-consistent, . . . all the way down to 1-consistent. Now suppose we have a CSP with\nn nodes and make it strongly n-consistent (i.e., strongly k-consistent for k = n). We can\nthen solve the problem as follows: First, we choose a consistent value for X1. We are then\nguaranteed to be able to choose a value for X2 because the graph is 2-consistent, for X3\nbecause it is 3-consistent, and so on. For each variable Xi, we need only search through the d\nvalues in the domain to find a value consistent with X1, . . . , Xi−1. We are guaranteed to find\na solution in time O(n2d). Of course, there is no free lunch: any algorithm for establishing\nn-consistency must take time exponential in n in the worst case. Worse, n-consistency also\nrequires space that is exponential in n. The memory issue is even more severe than the time.\nIn practice, determining the appropriate level of consistency checking is mostly an empirical\nscience. It can be said practitioners commonly compute 2-consistency and less commonly\n3-consistency.\n6.2.5\nGlobal constraints\nRemember that a global constraint is one involving an arbitrary number of variables (but not\nnecessarily all variables). Global constraints occur frequently in real problems and can be\nhandled by special-purpose algorithms that are more efficient than the general-purpose meth-\nods described so far. For example, the Alldiffconstraint says that all the variables involved\nmust have distinct values (as in the cryptarithmetic problem above and Sudoku puzzles be-\nlow). One simple form of inconsistency detection for Alldiffconstraints works as follows:\nif m variables are involved in the constraint, and if they have n possible distinct values alto-\ngether, and m > n, then the constraint cannot be satisfied.\nThis leads to the following simple algorithm: First, remove any variable in the con-\nstraint that has a singleton domain, and delete that variable’s value from the domains of the\nremaining variables. Repeat as long as there are singleton variables. If at any point an empty\ndomain is produced or there are more variables than domain values left, then an inconsistency\nhas been detected.\nThis method can detect the inconsistency in the assignment {WA = red, NSW = red}\nfor Figure 6.1. Notice that the variables SA, NT, and Q are effectively connected by an\nAlldiffconstraint because each pair must have two different colors. After applying AC-3\nwith the partial assignment, the domain of each variable is reduced to {green, blue}. That\nis, we have three variables and only two colors, so the Alldiffconstraint is violated. Thus,\na simple consistency procedure for a higher-order constraint is sometimes more effective\nthan applying arc consistency to an equivalent set of binary constraints. There are more\n212\nChapter\n6.\nConstraint Satisfaction Problems\ncomplex inference algorithms for Alldiff(see van Hoeve and Katriel, 2006) that propagate\nmore constraints but are more computationally expensive to run.\nAnother important higher-order constraint is the resource constraint, sometimes called\nRESOURCE\nCONSTRAINT\nthe atmost constraint. For example, in a scheduling problem, let P1, . . . , P4 denote the\nnumbers of personnel assigned to each of four tasks. The constraint that no more than 10\npersonnel are assigned in total is written as Atmost(10, P1, P2, P3, P4). We can detect an\ninconsistency simply by checking the sum of the minimum values of the current domains;\nfor example, if each variable has the domain {3, 4, 5, 6}, the Atmost constraint cannot be\nsatisfied. We can also enforce consistency by deleting the maximum value of any domain if it\nis not consistent with the minimum values of the other domains. Thus, if each variable in our\nexample has the domain {2, 3, 4, 5, 6}, the values 5 and 6 can be deleted from each domain.\nFor large resource-limited problems with integer values—such as logistical problems\ninvolving moving thousands of people in hundreds of vehicles—it is usually not possible to\nrepresent the domain of each variable as a large set of integers and gradually reduce that set by\nconsistency-checking methods. Instead, domains are represented by upper and lower bounds\nand are managed by bounds propagation. For example, in an airline-scheduling problem,\nBOUNDS\nPROPAGATION\nlet’s suppose there are two flights, F1 and F2, for which the planes have capacities 165 and\n385, respectively. The initial domains for the numbers of passengers on each flight are then\nD1 = [0, 165]\nand\nD2 = [0, 385] .\nNow suppose we have the additional constraint that the two flights together must carry 420\npeople: F1 + F2 = 420. Propagating bounds constraints, we reduce the domains to\nD1 = [35, 165]\nand\nD2 = [255, 385] .\nWe say that a CSP is bounds consistent if for every variable X, and for both the lower-\nBOUNDS\nCONSISTENT\nbound and upper-bound values of X, there exists some value of Y that satisfies the constraint\nbetween X and Y for every variable Y . This kind of bounds propagation is widely used in\npractical constraint problems.\n6.2.6\nSudoku example\nThe popular Sudoku puzzle has introduced millions of people to constraint satisfaction prob-\nSUDOKU\nlems, although they may not recognize it. A Sudoku board consists of 81 squares, some of\nwhich are initially filled with digits from 1 to 9. The puzzle is to fill in all the remaining\nsquares such that no digit appears twice in any row, column, or 3 × 3 box (see Figure 6.4). A\nrow, column, or box is called a unit.\nThe Sudoku puzzles that are printed in newspapers and puzzle books have the property\nthat there is exactly one solution. Although some can be tricky to solve by hand, taking tens\nof minutes, even the hardest Sudoku problems yield to a CSP solver in less than 0.1 second.\nA Sudoku puzzle can be considered a CSP with 81 variables, one for each square. We\nuse the variable names A1 through A9 for the top row (left to right), down to I1 through I9\nfor the bottom row. The empty squares have the domain {1, 2, 3, 4, 5, 6, 7, 8, 9} and the pre-\nfilled squares have a domain consisting of a single value. In addition, there are 27 different\nSection 6.2.\nConstraint Propagation: Inference in CSPs\n213\n3\n2\n6\n9\n3\n5\n1\n1 8\n6 4\n8 1\n2 9\n7\n8\n6 7\n8 2\n2 6\n9 5\n8\n2\n3\n9\n5\n1\n3\n3\n2\n6\n9\n3\n5\n1\n1 8\n6 4\n8 1\n2 9\n7\n8\n6 7\n8 2\n2 6\n9 5\n8\n2\n3\n9\n5\n1\n3\n4 8\n9\n1\n5 7\n6 7\n4\n8 2\n2 5\n7\n9 3\n5 4\n3\n7 6\n2 9 5 6 4 1 3\n1 3\n9\n4 5\n3 7\n8\n1 4\n1 4\n5\n7 6\n6 9\n4\n7\n8 2\n1\n2\n3\n4\n5\n6\n7\n8\n9\nA\nB\nC\nD\nE\nF\nG\nH\n I\nA\nB\nC\nD\nE\nF\nG\nH\n I\n1\n2\n3\n4\n5\n6\n7\n8\n9\n(a)\n(b)\nFigure 6.4\n(a) A Sudoku puzzle and (b) its solution.\nAlldiffconstraints: one for each row, column, and box of 9 squares.\nAlldiff(A1, A2, A3, A4, A5, A6, A7, A8, A9)\nAlldiff(B1, B2, B3, B4, B5, B6, B7, B8, B9)\n· · ·\nAlldiff(A1, B1, C1, D1, E1, F1, G1, H1, I1)\nAlldiff(A2, B2, C2, D2, E2, F2, G2, H2, I2)\n· · ·\nAlldiff(A1, A2, A3, B1, B2, B3, C1, C2, C3)\nAlldiff(A4, A5, A6, B4, B5, B6, C4, C5, C6)\n· · ·\nLet us see how far arc consistency can take us. Assume that the Alldiffconstraints have been\nexpanded into binary constraints (such as A1 ̸= A2) so that we can apply the AC-3 algorithm\ndirectly. Consider variable E6 from Figure 6.4(a)—the empty square between the 2 and the\n8 in the middle box. From the constraints in the box, we can remove not only 2 and 8 but also\n1 and 7 from E6’s domain. From the constraints in its column, we can eliminate 5, 6, 2, 8,\n9, and 3. That leaves E6 with a domain of {4}; in other words, we know the answer for E6.\nNow consider variable I6—the square in the bottom middle box surrounded by 1, 3, and 3.\nApplying arc consistency in its column, we eliminate 5, 6, 2, 4 (since we now know E6 must\nbe 4), 8, 9, and 3. We eliminate 1 by arc consistency with I5, and we are left with only the\nvalue 7 in the domain of I6. Now there are 8 known values in column 6, so arc consistency\ncan infer that A6 must be 1. Inference continues along these lines, and eventually, AC-3 can\nsolve the entire puzzle—all the variables have their domains reduced to a single value, as\nshown in Figure 6.4(b).\nOf course, Sudoku would soon lose its appeal if every puzzle could be solved by a\n214\nChapter\n6.\nConstraint Satisfaction Problems\nmechanical application of AC-3, and indeed AC-3 works only for the easiest Sudoku puzzles.\nSlightly harder ones can be solved by PC-2, but at a greater computational cost: there are\n255,960 different path constraints to consider in a Sudoku puzzle. To solve the hardest puzzles\nand to make efficient progress, we will have to be more clever.\nIndeed, the appeal of Sudoku puzzles for the human solver is the need to be resourceful\nin applying more complex inference strategies. Aficionados give them colorful names, such\nas “naked triples.” That strategy works as follows: in any unit (row, column or box), find\nthree squares that each have a domain that contains the same three numbers or a subset of\nthose numbers. For example, the three domains might be {1, 8}, {3, 8}, and {1, 3, 8}. From\nthat we don’t know which square contains 1, 3, or 8, but we do know that the three numbers\nmust be distributed among the three squares. Therefore we can remove 1, 3, and 8 from the\ndomains of every other square in the unit.\nIt is interesting to note how far we can go without saying much that is specific to Su-\ndoku. We do of course have to say that there are 81 variables, that their domains are the digits\n1 to 9, and that there are 27 Alldiffconstraints. But beyond that, all the strategies—arc con-\nsistency, path consistency, etc.—apply generally to all CSPs, not just to Sudoku problems.\nEven naked triples is really a strategy for enforcing consistency of Alldiffconstraints and\nhas nothing to do with Sudoku per se. This is the power of the CSP formalism: for each new\nproblem area, we only need to define the problem in terms of constraints; then the general\nconstraint-solving mechanisms can take over.\n6.3\nBACKTRACKING SEARCH FOR CSPS\nSudoku problems are designed to be solved by inference over constraints. But many other\nCSPs cannot be solved by inference alone; there comes a time when we must search for a\nsolution. In this section we look at backtracking search algorithms that work on partial as-\nsignments; in the next section we look at local search algorithms over complete assignments.\nWe could apply a standard depth-limited search (from Chapter 3). A state would be a\npartial assignment, and an action would be adding var = value to the assignment. But for a\nCSP with n variables of domain size d, we quickly notice something terrible: the branching\nfactor at the top level is nd because any of d values can be assigned to any of n variables. At\nthe next level, the branching factor is (n −1)d, and so on for n levels. We generate a tree\nwith n! · dn leaves, even though there are only dn possible complete assignments!\nOur seemingly reasonable but naive formulation ignores crucial property common to\nall CSPs: commutativity. A problem is commutative if the order of application of any given\nCOMMUTATIVITY\nset of actions has no effect on the outcome. CSPs are commutative because when assigning\nvalues to variables, we reach the same partial assignment regardless of order. Therefore, we\nneed only consider a single variable at each node in the search tree. For example, at the root\nnode of a search tree for coloring the map of Australia, we might make a choice between\nSA = red, SA = green, and SA = blue, but we would never choose between SA = red and\nWA = blue. With this restriction, the number of leaves is dn, as we would hope.\nSection 6.3.\nBacktracking Search for CSPs\n215\nfunction BACKTRACKING-SEARCH(csp) returns a solution, or failure\nreturn BACKTRACK({ },csp)\nfunction BACKTRACK(assignment,csp) returns a solution, or failure\nif assignment is complete then return assignment\nvar ←SELECT-UNASSIGNED-VARIABLE(csp)\nfor each value in ORDER-DOMAIN-VALUES(var,assignment,csp) do\nif value is consistent with assignment then\nadd {var = value} to assignment\ninferences ←INFERENCE(csp,var,value)\nif inferences ̸= failure then\nadd inferences to assignment\nresult ←BACKTRACK(assignment,csp)\nif result ̸= failure then\nreturn result\nremove {var = value} and inferences from assignment\nreturn failure\nFigure 6.5\nA simple backtracking algorithm for constraint satisfaction problems. The al-\ngorithm is modeled on the recursive depth-first search of Chapter 3. By varying the functions\nSELECT-UNASSIGNED-VARIABLE and ORDER-DOMAIN-VALUES, we can implement the\ngeneral-purpose heuristics discussed in the text. The function INFERENCE can optionally be\nused to impose arc-, path-, or k-consistency, as desired. If a value choice leads to failure\n(noticed either by INFERENCE or by BACKTRACK), then value assignments (including those\nmade by INFERENCE) are removed from the current assignment and a new value is tried.\nThe term backtracking search is used for a depth-first search that chooses values for\nBACKTRACKING\nSEARCH\none variable at a time and backtracks when a variable has no legal values left to assign. The\nalgorithm is shown in Figure 6.5. It repeatedly chooses an unassigned variable, and then tries\nall values in the domain of that variable in turn, trying to find a solution. If an inconsistency is\ndetected, then BACKTRACK returns failure, causing the previous call to try another value. Part\nof the search tree for the Australia problem is shown in Figure 6.6, where we have assigned\nvariables in the order WA, NT, Q, . . .. Because the representation of CSPs is standardized,\nthere is no need to supply BACKTRACKING-SEARCH with a domain-specific initial state,\naction function, transition model, or goal test.\nNotice that BACKTRACKING-SEARCH keeps only a single representation of a state and\nalters that representation rather than creating new ones, as described on page 87.\nIn Chapter 3 we improved the poor performance of uninformed search algorithms by\nsupplying them with domain-specific heuristic functions derived from our knowledge of the\nproblem. It turns out that we can solve CSPs efficiently without such domain-specific knowl-\nedge. Instead, we can add some sophistication to the unspecified functions in Figure 6.5,\nusing them to address the following questions:\n1. Which variable should be assigned next (SELECT-UNASSIGNED-VARIABLE), and in\nwhat order should its values be tried (ORDER-DOMAIN-VALUES)?\n216\nChapter\n6.\nConstraint Satisfaction Problems\nWA=red\nWA=blue\nWA=green\nWA=red\nNT=blue\nWA=red\nNT=green\nWA=red\nNT=green\nQ=red\nWA=red\nNT=green\nQ=blue\nFigure 6.6\nPart of the search tree for the map-coloring problem in Figure 6.1.\n2. What inferences should be performed at each step in the search (INFERENCE)?\n3. When the search arrives at an assignment that violates a constraint, can the search avoid\nrepeating this failure?\nThe subsections that follow answer each of these questions in turn.\n6.3.1\nVariable and value ordering\nThe backtracking algorithm contains the line\nvar ←SELECT-UNASSIGNED-VARIABLE(csp) .\nThe simplest strategy for SELECT-UNASSIGNED-VARIABLE is to choose the next unassigned\nvariable in order, {X1, X2, . . .}. This static variable ordering seldom results in the most effi-\ncient search. For example, after the assignments for WA = red and NT = green in Figure 6.6,\nthere is only one possible value for SA, so it makes sense to assign SA = blue next rather than\nassigning Q. In fact, after SA is assigned, the choices for Q, NSW , and V are all forced. This\nintuitive idea—choosing the variable with the fewest “legal” values—is called the minimum-\nremaining-values (MRV) heuristic. It also has been called the “most constrained variable” or\nMINIMUM-\nREMAINING-VALUES\n“fail-first” heuristic, the latter because it picks a variable that is most likely to cause a failure\nsoon, thereby pruning the search tree. If some variable X has no legal values left, the MRV\nheuristic will select X and failure will be detected immediately—avoiding pointless searches\nthrough other variables. The MRV heuristic usually performs better than a random or static\nordering, sometimes by a factor of 1,000 or more, although the results vary widely depending\non the problem.\nThe MRV heuristic doesn’t help at all in choosing the first region to color in Australia,\nbecause initially every region has three legal colors. In this case, the degree heuristic comes\nDEGREE HEURISTIC\nin handy. It attempts to reduce the branching factor on future choices by selecting the vari-\nable that is involved in the largest number of constraints on other unassigned variables. In\nFigure 6.1, SA is the variable with highest degree, 5; the other variables have degree 2 or 3,\nexcept for T, which has degree 0. In fact, once SA is chosen, applying the degree heuris-\ntic solves the problem without any false steps—you can choose any consistent color at each\nchoice point and still arrive at a solution with no backtracking. The minimum-remaining-\nSection 6.3.\nBacktracking Search for CSPs\n217\nvalues heuristic is usually a more powerful guide, but the degree heuristic can be useful as a\ntie-breaker.\nOnce a variable has been selected, the algorithm must decide on the order in which to\nexamine its values. For this, the least-constraining-value heuristic can be effective in some\nLEAST-\nCONSTRAINING-\nVALUE\ncases. It prefers the value that rules out the fewest choices for the neighboring variables in\nthe constraint graph. For example, suppose that in Figure 6.1 we have generated the partial\nassignment with WA = red and NT = green and that our next choice is for Q. Blue would\nbe a bad choice because it eliminates the last legal value left for Q’s neighbor, SA. The\nleast-constraining-value heuristic therefore prefers red to blue. In general, the heuristic is\ntrying to leave the maximum flexibility for subsequent variable assignments. Of course, if we\nare trying to find all the solutions to a problem, not just the first one, then the ordering does\nnot matter because we have to consider every value anyway. The same holds if there are no\nsolutions to the problem.\nWhy should variable selection be fail-first, but value selection be fail-last? It turns out\nthat, for a wide variety of problems, a variable ordering that chooses a variable with the\nminimum number of remaining values helps minimize the number of nodes in the search tree\nby pruning larger parts of the tree earlier. For value ordering, the trick is that we only need\none solution; therefore it makes sense to look for the most likely values first. If we wanted to\nenumerate all solutions rather than just find one, then value ordering would be irrelevant.\n6.3.2\nInterleaving search and inference\nSo far we have seen how AC-3 and other algorithms can infer reductions in the domain of\nvariables before we begin the search. But inference can be even more powerful in the course\nof a search: every time we make a choice of a value for a variable, we have a brand-new\nopportunity to infer new domain reductions on the neighboring variables.\nOne of the simplest forms of inference is called forward checking. Whenever a vari-\nFORWARD\nCHECKING\nable X is assigned, the forward-checking process establishes arc consistency for it: for each\nunassigned variable Y that is connected to X by a constraint, delete from Y ’s domain any\nvalue that is inconsistent with the value chosen for X. Because forward checking only does\narc consistency inferences, there is no reason to do forward checking if we have already done\narc consistency as a preprocessing step.\nFigure 6.7 shows the progress of backtracking search on the Australia CSP with for-\nward checking. There are two important points to notice about this example. First, notice\nthat after WA = red and Q = green are assigned, the domains of NT and SA are reduced\nto a single value; we have eliminated branching on these variables altogether by propagat-\ning information from WA and Q. A second point to notice is that after V = blue, the do-\nmain of SA is empty. Hence, forward checking has detected that the partial assignment\n{WA = red, Q = green, V = blue} is inconsistent with the constraints of the problem, and\nthe algorithm will therefore backtrack immediately.\nFor many problems the search will be more effective if we combine the MRV heuris-\ntic with forward checking. Consider Figure 6.7 after assigning {WA = red}. Intuitively, it\nseems that that assignment constrains its neighbors, NT and SA, so we should handle those\n218\nChapter\n6.\nConstraint Satisfaction Problems\nInitial domains\nAfter WA=red\nAfter Q=green\nAfter V=blue\nR G B\nR\nR\nB\nR G B\nR G B\nB\nR G B\nR G B\nR G B\nR\nR\nR\nR G B\nB\nB\nG B\nR G B\nG\nG\nR G B\nR G B\nB\nG B\nR G B\nR G B\nR G B\nR G B\nWA\nT\nSA\nV\nNSW\nQ\nNT\nFigure 6.7\nThe progress of a map-coloring search with forward checking. WA = red\nis assigned first; then forward checking deletes red from the domains of the neighboring\nvariables NT and SA. After Q = green is assigned, green is deleted from the domains of\nNT, SA, and NSW . After V = blue is assigned, blue is deleted from the domains of NSW\nand SA, leaving SA with no legal values.\nvariables next, and then all the other variables will fall into place. That’s exactly what hap-\npens with MRV: NT and SA have two values, so one of them is chosen first, then the other,\nthen Q, NSW , and V in order. Finally T still has three values, and any one of them works.\nWe can view forward checking as an efficient way to incrementally compute the information\nthat the MRV heuristic needs to do its job.\nAlthough forward checking detects many inconsistencies, it does not detect all of them.\nThe problem is that it makes the current variable arc-consistent, but doesn’t look ahead and\nmake all the other variables arc-consistent. For example, consider the third row of Figure 6.7.\nIt shows that when WA is red and Q is green, both NT and SA are forced to be blue. Forward\nchecking does not look far enough ahead to notice that this is an inconsistency: NT and SA\nare adjacent and so cannot have the same value.\nThe algorithm called MAC (for Maintaining Arc Consistency (MAC)) detects this\nMAINTAINING ARC\nCONSISTENCY (MAC)\ninconsistency. After a variable Xi is assigned a value, the INFERENCE procedure calls AC-3,\nbut instead of a queue of all arcs in the CSP, we start with only the arcs (Xj, Xi) for all\nXj that are unassigned variables that are neighbors of Xi. From there, AC-3 does constraint\npropagation in the usual way, and if any variable has its domain reduced to the empty set, the\ncall to AC-3 fails and we know to backtrack immediately. We can see that MAC is strictly\nmore powerful than forward checking because forward checking does the same thing as MAC\non the initial arcs in MAC’s queue; but unlike MAC, forward checking does not recursively\npropagate constraints when changes are made to the domains of variables.\n6.3.3\nIntelligent backtracking: Looking backward\nThe BACKTRACKING-SEARCH algorithm in Figure 6.5 has a very simple policy for what to\ndo when a branch of the search fails: back up to the preceding variable and try a different\nvalue for it. This is called chronological backtracking because the most recent decision\nCHRONOLOGICAL\nBACKTRACKING\npoint is revisited. In this subsection, we consider better possibilities.\nConsider what happens when we apply simple backtracking in Figure 6.1 with a fixed\nvariable ordering Q, NSW , V , T, SA, WA, NT. Suppose we have generated the partial\nassignment {Q = red, NSW = green, V = blue, T = red}. When we try the next variable,\nSA, we see that every value violates a constraint. We back up to T and try a new color for\nSection 6.3.\nBacktracking Search for CSPs\n219\nTasmania! Obviously this is silly—recoloring Tasmania cannot possibly resolve the problem\nwith South Australia.\nA more intelligent approach to backtracking is to backtrack to a variable that might fix\nthe problem—a variable that was responsible for making one of the possible values of SA\nimpossible. To do this, we will keep track of a set of assignments that are in conflict with\nsome value for SA. The set (in this case {Q = red, NSW = green, V = blue, }), is called the\nconflict set for SA. The backjumping method backtracks to the most recent assignment in\nCONFLICT SET\nBACKJUMPING\nthe conflict set; in this case, backjumping would jump over Tasmania and try a new value\nfor V . This method is easily implemented by a modification to BACKTRACK such that it\naccumulates the conflict set while checking for a legal value to assign. If no legal value is\nfound, the algorithm should return the most recent element of the conflict set along with the\nfailure indicator.\nThe sharp-eyed reader will have noticed that forward checking can supply the conflict\nset with no extra work: whenever forward checking based on an assignment X = x deletes a\nvalue from Y ’s domain, it should add X = x to Y ’s conflict set. If the last value is deleted\nfrom Y ’s domain, then the assignments in the conflict set of Y are added to the conflict set\nof X. Then, when we get to Y , we know immediately where to backtrack if needed.\nThe eagle-eyed reader will have noticed something odd: backjumping occurs when\nevery value in a domain is in conflict with the current assignment; but forward checking\ndetects this event and prevents the search from ever reaching such a node! In fact, it can be\nshown that every branch pruned by backjumping is also pruned by forward checking. Hence,\nsimple backjumping is redundant in a forward-checking search or, indeed, in a search that\nuses stronger consistency checking, such as MAC.\nDespite the observations of the preceding paragraph, the idea behind backjumping re-\nmains a good one: to backtrack based on the reasons for failure. Backjumping notices failure\nwhen a variable’s domain becomes empty, but in many cases a branch is doomed long before\nthis occurs. Consider again the partial assignment {WA = red, NSW = red} (which, from\nour earlier discussion, is inconsistent). Suppose we try T = red next and then assign NT, Q,\nV , SA. We know that no assignment can work for these last four variables, so eventually we\nrun out of values to try at NT. Now, the question is, where to backtrack? Backjumping cannot\nwork, because NT does have values consistent with the preceding assigned variables—NT\ndoesn’t have a complete conflict set of preceding variables that caused it to fail. We know,\nhowever, that the four variables NT, Q, V , and SA, taken together, failed because of a set of\npreceding variables, which must be those variables that directly conflict with the four. This\nleads to a deeper notion of the conflict set for a variable such as NT: it is that set of preced-\ning variables that caused NT, together with any subsequent variables, to have no consistent\nsolution. In this case, the set is WA and NSW , so the algorithm should backtrack to NSW\nand skip over Tasmania. A backjumping algorithm that uses conflict sets defined in this way\nis called conflict-directed backjumping.\nCONFLICT-DIRECTED\nBACKJUMPING\nWe must now explain how these new conflict sets are computed. The method is in\nfact quite simple. The “terminal” failure of a branch of the search always occurs because a\nvariable’s domain becomes empty; that variable has a standard conflict set. In our example,\nSA fails, and its conflict set is (say) {WA, NT, Q}. We backjump to Q, and Q absorbs\n220\nChapter\n6.\nConstraint Satisfaction Problems\nthe conflict set from SA (minus Q itself, of course) into its own direct conflict set, which is\n{NT, NSW }; the new conflict set is {WA, NT, NSW }. That is, there is no solution from\nQ onward, given the preceding assignment to {WA, NT, NSW }. Therefore, we backtrack\nto NT, the most recent of these. NT absorbs {WA, NT, NSW } −{NT} into its own\ndirect conflict set {WA}, giving {WA, NSW } (as stated in the previous paragraph). Now\nthe algorithm backjumps to NSW , as we would hope. To summarize: let Xj be the current\nvariable, and let conf (Xj) be its conflict set. If every possible value for Xj fails, backjump\nto the most recent variable Xi in conf (Xj), and set\nconf (Xi) ←conf (Xi) ∪conf (Xj) −{Xi} .\nWhen we reach a contradiction, backjumping can tell us how far to back up, so we don’t\nwaste time changing variables that won’t fix the problem. But we would also like to avoid\nrunning into the same problem again. When the search arrives at a contradiction, we know\nthat some subset of the conflict set is responsible for the problem. Constraint learning is the\nCONSTRAINT\nLEARNING\nidea of finding a minimum set of variables from the conflict set that causes the problem. This\nset of variables, along with their corresponding values, is called a no-good. We then record\nNO-GOOD\nthe no-good, either by adding a new constraint to the CSP or by keeping a separate cache of\nno-goods.\nFor example, consider the state {WA = red, NT = green, Q = blue} in the bottom\nrow of Figure 6.6. Forward checking can tell us this state is a no-good because there is no\nvalid assignment to SA. In this particular case, recording the no-good would not help, because\nonce we prune this branch from the search tree, we will never encounter this combination\nagain. But suppose that the search tree in Figure 6.6 were actually part of a larger search tree\nthat started by first assigning values for V and T. Then it would be worthwhile to record\n{WA = red, NT = green, Q = blue} as a no-good because we are going to run into the\nsame problem again for each possible set of assignments to V and T.\nNo-goods can be effectively used by forward checking or by backjumping. Constraint\nlearning is one of the most important techniques used by modern CSP solvers to achieve\nefficiency on complex problems.\n6.4\nLOCAL SEARCH FOR CSPS\nLocal search algorithms (see Section 4.1) turn out to be effective in solving many CSPs. They\nuse a complete-state formulation: the initial state assigns a value to every variable, and the\nsearch changes the value of one variable at a time. For example, in the 8-queens problem (see\nFigure 4.3), the initial state might be a random configuration of 8 queens in 8 columns, and\neach step moves a single queen to a new position in its column. Typically, the initial guess\nviolates several constraints. The point of local search is to eliminate the violated constraints.2\nIn choosing a new value for a variable, the most obvious heuristic is to select the value\nthat results in the minimum number of conflicts with other variables—the min-conflicts\nMIN-CONFLICTS\n2 Local search can easily be extended to constraint optimization problems (COPs). In that case, all the techniques\nfor hill climbing and simulated annealing can be applied to optimize the objective function.\nSection 6.4.\nLocal Search for CSPs\n221\nfunction MIN-CONFLICTS(csp,max steps) returns a solution or failure\ninputs: csp, a constraint satisfaction problem\nmax steps, the number of steps allowed before giving up\ncurrent ←an initial complete assignment for csp\nfor i = 1 to max steps do\nif current is a solution for csp then return current\nvar ←a randomly chosen conflicted variable from csp.VARIABLES\nvalue ←the value v for var that minimizes CONFLICTS(var,v,current,csp)\nset var = value in current\nreturn failure\nFigure 6.8\nThe MIN-CONFLICTS algorithm for solving CSPs by local search. The initial\nstate may be chosen randomly or by a greedy assignment process that chooses a minimal-\nconflict value for each variable in turn. The CONFLICTS function counts the number of\nconstraints violated by a particular value, given the rest of the current assignment.\n2\n2\n1\n2\n3\n1\n2\n3\n3\n2\n3\n2\n3\n0\nFigure 6.9\nA two-step solution using min-conflicts for an 8-queens problem. At each\nstage, a queen is chosen for reassignment in its column. The number of conflicts (in this\ncase, the number of attacking queens) is shown in each square. The algorithm moves the\nqueen to the min-conflicts square, breaking ties randomly.\nheuristic. The algorithm is shown in Figure 6.8 and its application to an 8-queens problem is\ndiagrammed in Figure 6.9.\nMin-conflicts is surprisingly effective for many CSPs. Amazingly, on the n-queens\nproblem, if you don’t count the initial placement of queens, the run time of min-conflicts is\nroughly independent of problem size. It solves even the million-queens problem in an aver-\nage of 50 steps (after the initial assignment). This remarkable observation was the stimulus\nleading to a great deal of research in the 1990s on local search and the distinction between\neasy and hard problems, which we take up in Chapter 7. Roughly speaking, n-queens is\neasy for local search because solutions are densely distributed throughout the state space.\nMin-conflicts also works well for hard problems. For example, it has been used to schedule\nobservations for the Hubble Space Telescope, reducing the time taken to schedule a week of\nobservations from three weeks (!) to around 10 minutes.\n222\nChapter\n6.\nConstraint Satisfaction Problems\nAll the local search techniques from Section 4.1 are candidates for application to CSPs,\nand some of those have proved especially effective. The landscape of a CSP under the min-\nconflicts heuristic usually has a series of plateaux. There may be millions of variable as-\nsignments that are only one conflict away from a solution. Plateau search—allowing side-\nways moves to another state with the same score—can help local search find its way off this\nplateau. This wandering on the plateau can be directed with tabu search: keeping a small\nlist of recently visited states and forbidding the algorithm to return to those states. Simulated\nannealing can also be used to escape from plateaux.\nAnother technique, called constraint weighting, can help concentrate the search on the\nCONSTRAINT\nWEIGHTING\nimportant constraints. Each constraint is given a numeric weight, Wi, initially all 1. At each\nstep of the search, the algorithm chooses a variable/value pair to change that will result in the\nlowest total weight of all violated constraints. The weights are then adjusted by incrementing\nthe weight of each constraint that is violated by the current assignment. This has two benefits:\nit adds topography to plateaux, making sure that it is possible to improve from the current\nstate, and it also, over time, adds weight to the constraints that are proving difficult to solve.\nAnother advantage of local search is that it can be used in an online setting when the\nproblem changes. This is particularly important in scheduling problems. A week’s airline\nschedule may involve thousands of flights and tens of thousands of personnel assignments,\nbut bad weather at one airport can render the schedule infeasible. We would like to repair the\nschedule with a minimum number of changes. This can be easily done with a local search\nalgorithm starting from the current schedule. A backtracking search with the new set of\nconstraints usually requires much more time and might find a solution with many changes\nfrom the current schedule.\n6.5\nTHE STRUCTURE OF PROBLEMS\nIn this section, we examine ways in which the structure of the problem, as represented by\nthe constraint graph, can be used to find solutions quickly. Most of the approaches here also\napply to other problems besides CSPs, such as probabilistic reasoning. After all, the only way\nwe can possibly hope to deal with the real world is to decompose it into many subproblems.\nLooking again at the constraint graph for Australia (Figure 6.1(b), repeated as Figure 6.12(a)),\none fact stands out: Tasmania is not connected to the mainland.3 Intuitively, it is obvious that\ncoloring Tasmania and coloring the mainland are independent subproblems—any solution\nINDEPENDENT\nSUBPROBLEMS\nfor the mainland combined with any solution for Tasmania yields a solution for the whole\nmap. Independence can be ascertained simply by finding connected components of the\nCONNECTED\nCOMPONENT\nconstraint graph. Each component corresponds to a subproblem CSPi. If assignment Si is\na solution of CSPi, then \u0016\ni Si is a solution of \u0016\ni CSPi. Why is this important? Consider\nthe following: suppose each CSPi has c variables from the total of n variables, where c is\na constant. Then there are n/c subproblems, each of which takes at most dc work to solve,\n3 A careful cartographer or patriotic Tasmanian might object that Tasmania should not be colored the same as\nits nearest mainland neighbor, to avoid the impression that it might be part of that state.\nSection 6.5.\nThe Structure of Problems\n223\nwhere d is the size of the domain. Hence, the total work is O(dcn/c), which is linear in n;\nwithout the decomposition, the total work is O(dn), which is exponential in n. Let’s make\nthis more concrete: dividing a Boolean CSP with 80 variables into four subproblems reduces\nthe worst-case solution time from the lifetime of the universe down to less than a second.\nCompletely independent subproblems are delicious, then, but rare. Fortunately, some\nother graph structures are also easy to solve. For example, a constraint graph is a tree when\nany two variables are connected by only one path. We show that any tree-structured CSP can\nbe solved in time linear in the number of variables.4 The key is a new notion of consistency,\ncalled directed arc consistency or DAC. A CSP is defined to be directed arc-consistent under\nDIRECTED ARC\nCONSISTENCY\nan ordering of variables X1, X2, . . . , Xn if and only if every Xi is arc-consistent with each\nXj for j > i.\nTo solve a tree-structured CSP, first pick any variable to be the root of the tree, and\nchoose an ordering of the variables such that each variable appears after its parent in the tree.\nSuch an ordering is called a topological sort. Figure 6.10(a) shows a sample tree and (b)\nTOPOLOGICAL SORT\nshows one possible ordering. Any tree with n nodes has n−1 arcs, so we can make this graph\ndirected arc-consistent in O(n) steps, each of which must compare up to d possible domain\nvalues for two variables, for a total time of O(nd2). Once we have a directed arc-consistent\ngraph, we can just march down the list of variables and choose any remaining value. Since\neach link from a parent to its child is arc consistent, we know that for any value we choose for\nthe parent, there will be a valid value left to choose for the child. That means we won’t have\nto backtrack; we can move linearly through the variables. The complete algorithm is shown\nin Figure 6.11.\nA\nC\nB\nD\nE\nF\n(a)\nA\nC\nB\nD\nE\nF\n(b)\nFigure 6.10\n(a) The constraint graph of a tree-structured CSP. (b) A linear ordering of the\nvariables consistent with the tree with A as the root. This is known as a topological sort of\nthe variables.\nNow that we have an efficient algorithm for trees, we can consider whether more general\nconstraint graphs can be reduced to trees somehow. There are two primary ways to do this,\none based on removing nodes and one based on collapsing nodes together.\nThe first approach involves assigning values to some variables so that the remaining\nvariables form a tree.\nConsider the constraint graph for Australia, shown again in Fig-\nure 6.12(a). If we could delete South Australia, the graph would become a tree, as in (b).\nFortunately, we can do this (in the graph, not the continent) by fixing a value for SA and\n4 Sadly, very few regions of the world have tree-structured maps, although Sulawesi comes close.\n224\nChapter\n6.\nConstraint Satisfaction Problems\nfunction TREE-CSP-SOLVER(csp) returns a solution, or failure\ninputs: csp, a CSP with components X, D, C\nn ←number of variables in X\nassignment ←an empty assignment\nroot ←any variable in X\nX ←TOPOLOGICALSORT(X ,root)\nfor j = n down to 2 do\nMAKE-ARC-CONSISTENT(PARENT(Xj),Xj)\nif it cannot be made consistent then return failure\nfor i = 1 to n do\nassignment[Xi] ←any consistent value from Di\nif there is no consistent value then return failure\nreturn assignment\nFigure 6.11\nThe TREE-CSP-SOLVER algorithm for solving tree-structured CSPs. If the\nCSP has a solution, we will find it in linear time; if not, we will detect a contradiction.\nWA\nNT\nSA\nQ\nNSW\nV\nT\nWA\nNT\nQ\nNSW\nV\nT\n(a)\n(b)\nFigure 6.12\n(a) The original constraint graph from Figure 6.1. (b) The constraint graph\nafter the removal of SA.\ndeleting from the domains of the other variables any values that are inconsistent with the\nvalue chosen for SA.\nNow, any solution for the CSP after SA and its constraints are removed will be con-\nsistent with the value chosen for SA. (This works for binary CSPs; the situation is more\ncomplicated with higher-order constraints.) Therefore, we can solve the remaining tree with\nthe algorithm given above and thus solve the whole problem. Of course, in the general case\n(as opposed to map coloring), the value chosen for SA could be the wrong one, so we would\nneed to try each possible value. The general algorithm is as follows:\nSection 6.5.\nThe Structure of Problems\n225\n1. Choose a subset S of the CSP’s variables such that the constraint graph becomes a tree\nafter removal of S. S is called a cycle cutset.\nCYCLE CUTSET\n2. For each possible assignment to the variables in S that satisfies all constraints on S,\n(a) remove from the domains of the remaining variables any values that are inconsis-\ntent with the assignment for S, and\n(b) If the remaining CSP has a solution, return it together with the assignment for S.\nIf the cycle cutset has size c, then the total run time is O(dc · (n −c)d2): we have to try each\nof the dc combinations of values for the variables in S, and for each combination we must\nsolve a tree problem of size n −c. If the graph is “nearly a tree,” then c will be small and the\nsavings over straight backtracking will be huge. In the worst case, however, c can be as large\nas (n −2). Finding the smallest cycle cutset is NP-hard, but several efficient approximation\nalgorithms are known. The overall algorithmic approach is called cutset conditioning; it\nCUTSET\nCONDITIONING\ncomes up again in Chapter 14, where it is used for reasoning about probabilities.\nThe second approach is based on constructing a tree decomposition of the constraint\nTREE\nDECOMPOSITION\ngraph into a set of connected subproblems. Each subproblem is solved independently, and the\nresulting solutions are then combined. Like most divide-and-conquer algorithms, this works\nwell if no subproblem is too large. Figure 6.13 shows a tree decomposition of the map-\ncoloring problem into five subproblems. A tree decomposition must satisfy the following\nthree requirements:\n• Every variable in the original problem appears in at least one of the subproblems.\n• If two variables are connected by a constraint in the original problem, they must appear\ntogether (along with the constraint) in at least one of the subproblems.\n• If a variable appears in two subproblems in the tree, it must appear in every subproblem\nalong the path connecting those subproblems.\nThe first two conditions ensure that all the variables and constraints are represented in the\ndecomposition. The third condition seems rather technical, but simply reflects the constraint\nthat any given variable must have the same value in every subproblem in which it appears;\nthe links joining subproblems in the tree enforce this constraint. For example, SA appears in\nall four of the connected subproblems in Figure 6.13. You can verify from Figure 6.12 that\nthis decomposition makes sense.\nWe solve each subproblem independently; if any one has no solution, we know the en-\ntire problem has no solution. If we can solve all the subproblems, then we attempt to construct\na global solution as follows. First, we view each subproblem as a “mega-variable” whose do-\nmain is the set of all solutions for the subproblem. For example, the leftmost subproblem in\nFigure 6.13 is a map-coloring problem with three variables and hence has six solutions—one\nis {WA = red, SA = blue, NT = green}. Then, we solve the constraints connecting the\nsubproblems, using the efficient algorithm for trees given earlier. The constraints between\nsubproblems simply insist that the subproblem solutions agree on their shared variables. For\nexample, given the solution {WA = red, SA = blue, NT = green} for the first subproblem,\nthe only consistent solution for the next subproblem is {SA = blue, NT = green, Q = red}.\nA given constraint graph admits many tree decompositions; in choosing a decompo-\nsition, the aim is to make the subproblems as small as possible. The tree width of a tree\nTREE WIDTH\n226\nChapter\n6.\nConstraint Satisfaction Problems\nT\nWA\nNT\nSA\nNT\nSA\nQ\nSA\nQ\nNSW\nSA\nNSW\nV\nFigure 6.13\nA tree decomposition of the constraint graph in Figure 6.12(a).\ndecomposition of a graph is one less than the size of the largest subproblem; the tree width\nof the graph itself is defined to be the minimum tree width among all its tree decompositions.\nIf a graph has tree width w and we are given the corresponding tree decomposition, then the\nproblem can be solved in O(ndw+1) time. Hence, CSPs with constraint graphs of bounded\ntree width are solvable in polynomial time. Unfortunately, finding the decomposition with\nminimal tree width is NP-hard, but there are heuristic methods that work well in practice.\nSo far, we have looked at the structure of the constraint graph. There can be important\nstructure in the values of variables as well. Consider the map-coloring problem with n colors.\nFor every consistent solution, there is actually a set of n! solutions formed by permuting the\ncolor names. For example, on the Australia map we know that WA, NT, and SA must all have\ndifferent colors, but there are 3! = 6 ways to assign the three colors to these three regions.\nThis is called value symmetry. We would like to reduce the search space by a factor of\nVALUE SYMMETRY\nn! by breaking the symmetry. We do this by introducing a symmetry-breaking constraint.\nSYMMETRY-\nBREAKING\nCONSTRAINT\nFor our example, we might impose an arbitrary ordering constraint, NT < SA < WA, that\nrequires the three values to be in alphabetical order. This constraint ensures that only one of\nthe n! solutions is possible: {NT = blue, SA = green, WA = red}.\nFor map coloring, it was easy to find a constraint that eliminates the symmetry, and\nin general it is possible to find constraints that eliminate all but one symmetric solution in\npolynomial time, but it is NP-hard to eliminate all symmetry among intermediate sets of\nvalues during search. In practice, breaking value symmetry has proved to be important and\neffective on a wide range of problems.\nSection 6.6.\nSummary\n227\n6.6\nSUMMARY\n• Constraint satisfaction problems (CSPs) represent a state with a set of variable/value\npairs and represent the conditions for a solution by a set of constraints on the variables.\nMany important real-world problems can be described as CSPs.\n• A number of inference techniques use the constraints to infer which variable/value pairs\nare consistent and which are not. These include node, arc, path, and k-consistency.\n• Backtracking search, a form of depth-first search, is commonly used for solving CSPs.\nInference can be interwoven with search.\n• The minimum-remaining-values and degree heuristics are domain-independent meth-\nods for deciding which variable to choose next in a backtracking search. The least-\nconstraining-value heuristic helps in deciding which value to try first for a given\nvariable. Backtracking occurs when no legal assignment can be found for a variable.\nConflict-directed backjumping backtracks directly to the source of the problem.\n• Local search using the min-conflicts heuristic has also been applied to constraint satis-\nfaction problems with great success.\n• The complexity of solving a CSP is strongly related to the structure of its constraint\ngraph. Tree-structured problems can be solved in linear time. Cutset conditioning can\nreduce a general CSP to a tree-structured one and is quite efficient if a small cutset can\nbe found. Tree decomposition techniques transform the CSP into a tree of subproblems\nand are efficient if the tree width of the constraint graph is small.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nThe earliest work related to constraint satisfaction dealt largely with numerical constraints.\nEquational constraints with integer domains were studied by the Indian mathematician Brah-\nmagupta in the seventh century; they are often called Diophantine equations, after the Greek\nDIOPHANTINE\nEQUATIONS\nmathematician Diophantus (c. 200–284), who actually considered the domain of positive ra-\ntionals. Systematic methods for solving linear equations by variable elimination were studied\nby Gauss (1829); the solution of linear inequality constraints goes back to Fourier (1827).\nFinite-domain constraint satisfaction problems also have a long history. For example,\ngraph coloring (of which map coloring is a special case) is an old problem in mathematics.\nGRAPH COLORING\nThe four-color conjecture (that every planar graph can be colored with four or fewer colors)\nwas first made by Francis Guthrie, a student of De Morgan, in 1852. It resisted solution—\ndespite several published claims to the contrary—until a proof was devised by Appel and\nHaken (1977) (see the book Four Colors Suffice (Wilson, 2004)). Purists were disappointed\nthat part of the proof relied on a computer, so Georges Gonthier (2008), using the COQ\ntheorem prover, derived a formal proof that Appel and Haken’s proof was correct.\nSpecific classes of constraint satisfaction problems occur throughout the history of\ncomputer science. One of the most influential early examples was the SKETCHPAD sys-\n228\nChapter\n6.\nConstraint Satisfaction Problems\ntem (Sutherland, 1963), which solved geometric constraints in diagrams and was the fore-\nrunner of modern drawing programs and CAD tools. The identification of CSPs as a general\nclass is due to Ugo Montanari (1974). The reduction of higher-order CSPs to purely binary\nCSPs with auxiliary variables (see Exercise 6.6) is due originally to the 19th-century logician\nCharles Sanders Peirce. It was introduced into the CSP literature by Dechter (1990b) and\nwas elaborated by Bacchus and van Beek (1998). CSPs with preferences among solutions are\nstudied widely in the optimization literature; see Bistarelli et al. (1997) for a generalization\nof the CSP framework to allow for preferences. The bucket-elimination algorithm (Dechter,\n1999) can also be applied to optimization problems.\nConstraint propagation methods were popularized by Waltz’s (1975) success on poly-\nhedral line-labeling problems for computer vision. Waltz showed that, in many problems,\npropagation completely eliminates the need for backtracking. Montanari (1974) introduced\nthe notion of constraint networks and propagation by path consistency. Alan Mackworth\n(1977) proposed the AC-3 algorithm for enforcing arc consistency as well as the general idea\nof combining backtracking with some degree of consistency enforcement. AC-4, a more\nefficient arc-consistency algorithm, was developed by Mohr and Henderson (1986). Soon af-\nter Mackworth’s paper appeared, researchers began experimenting with the tradeoff between\nthe cost of consistency enforcement and the benefits in terms of search reduction. Haralick\nand Elliot (1980) favored the minimal forward-checking algorithm described by McGregor\n(1979), whereas Gaschnig (1979) suggested full arc-consistency checking after each vari-\nable assignment—an algorithm later called MAC by Sabin and Freuder (1994). The latter\npaper provides somewhat convincing evidence that, on harder CSPs, full arc-consistency\nchecking pays off. Freuder (1978, 1982) investigated the notion of k-consistency and its\nrelationship to the complexity of solving CSPs. Apt (1999) describes a generic algorithmic\nframework within which consistency propagation algorithms can be analyzed, and Bessi`ere\n(2006) presents a current survey.\nSpecial methods for handling higher-order or global constraints were developed first\nwithin the context of constraint logic programming. Marriott and Stuckey (1998) provide\nexcellent coverage of research in this area. The Alldiffconstraint was studied by Regin\n(1994), Stergiou and Walsh (1999), and van Hoeve (2001). Bounds constraints were incorpo-\nrated into constraint logic programming by Van Hentenryck et al. (1998). A survey of global\nconstraints is provided by van Hoeve and Katriel (2006).\nSudoku has become the most widely known CSP and was described as such by Simonis\n(2005). Agerbeck and Hansen (2008) describe some of the strategies and show that Sudoku\non an n2 × n2 board is in the class of NP-hard problems. Reeson et al. (2007) show an\ninteractive solver based on CSP techniques.\nThe idea of backtracking search goes back to Golomb and Baumert (1965), and its\napplication to constraint satisfaction is due to Bitner and Reingold (1975), although they trace\nthe basic algorithm back to the 19th century. Bitner and Reingold also introduced the MRV\nheuristic, which they called the most-constrained-variable heuristic. Brelaz (1979) used the\ndegree heuristic as a tiebreaker after applying the MRV heuristic. The resulting algorithm,\ndespite its simplicity, is still the best method for k-coloring arbitrary graphs. Haralick and\nElliot (1980) proposed the least-constraining-value heuristic.\nBibliographical and Historical Notes\n229\nThe basic backjumping method is due to John Gaschnig (1977, 1979). Kondrak and\nvan Beek (1997) showed that this algorithm is essentially subsumed by forward checking.\nConflict-directed backjumping was devised by Prosser (1993). The most general and pow-\nerful form of intelligent backtracking was actually developed very early on by Stallman and\nSussman (1977). Their technique of dependency-directed backtracking led to the develop-\nDEPENDENCY-\nDIRECTED\nBACKTRACKING\nment of truth maintenance systems (Doyle, 1979), which we discuss in Section 12.6.2. The\nconnection between the two areas is analyzed by de Kleer (1989).\nThe work of Stallman and Sussman also introduced the idea of constraint learning,\nin which partial results obtained by search can be saved and reused later in the search. The\nidea was formalized Dechter (1990a). Backmarking (Gaschnig, 1979) is a particularly sim-\nBACKMARKING\nple method in which consistent and inconsistent pairwise assignments are saved and used\nto avoid rechecking constraints. Backmarking can be combined with conflict-directed back-\njumping; Kondrak and van Beek (1997) present a hybrid algorithm that provably subsumes\neither method taken separately. The method of dynamic backtracking (Ginsberg, 1993) re-\nDYNAMIC\nBACKTRACKING\ntains successful partial assignments from later subsets of variables when backtracking over\nan earlier choice that does not invalidate the later success.\nEmpirical studies of several randomized backtracking methods were done by Gomes\net al. (2000) and Gomes and Selman (2001). Van Beek (2006) surveys backtracking.\nLocal search in constraint satisfaction problems was popularized by the work of Kirk-\npatrick et al. (1983) on simulated annealing (see Chapter 4), which is widely used for schedul-\ning problems. The min-conflicts heuristic was first proposed by Gu (1989) and was developed\nindependently by Minton et al. (1992). Sosic and Gu (1994) showed how it could be applied\nto solve the 3,000,000 queens problem in less than a minute. The astounding success of\nlocal search using min-conflicts on the n-queens problem led to a reappraisal of the nature\nand prevalence of “easy” and “hard” problems. Peter Cheeseman et al. (1991) explored the\ndifficulty of randomly generated CSPs and discovered that almost all such problems either\nare trivially easy or have no solutions. Only if the parameters of the problem generator are\nset in a certain narrow range, within which roughly half of the problems are solvable, do we\nfind “hard” problem instances. We discuss this phenomenon further in Chapter 7. Konolige\n(1994) showed that local search is inferior to backtracking search on problems with a certain\ndegree of local structure; this led to work that combined local search and inference, such as\nthat by Pinkas and Dechter (1995). Hoos and Tsang (2006) survey local search techniques.\nWork relating the structure and complexity of CSPs originates with Freuder (1985), who\nshowed that search on arc consistent trees works without any backtracking. A similar result,\nwith extensions to acyclic hypergraphs, was developed in the database community (Beeri\net al., 1983). Bayardo and Miranker (1994) present an algorithm for tree-structured CSPs\nthat runs in linear time without any preprocessing.\nSince those papers were published, there has been a great deal of progress in developing\nmore general results relating the complexity of solving a CSP to the structure of its constraint\ngraph. The notion of tree width was introduced by the graph theorists Robertson and Seymour\n(1986). Dechter and Pearl (1987, 1989), building on the work of Freuder, applied a related\nnotion (which they called induced width) to constraint satisfaction problems and developed\nthe tree decomposition approach sketched in Section 6.5. Drawing on this work and on results\n230\nChapter\n6.\nConstraint Satisfaction Problems\nfrom database theory, Gottlob et al. (1999a, 1999b) developed a notion, hypertree width, that\nis based on the characterization of the CSP as a hypergraph. In addition to showing that any\nCSP with hypertree width w can be solved in time O(nw+1 log n), they also showed that\nhypertree width subsumes all previously defined measures of “width” in the sense that there\nare cases where the hypertree width is bounded and the other measures are unbounded.\nInterest in look-back approaches to backtracking was rekindled by the work of Bayardo\nand Schrag (1997), whose RELSAT algorithm combined constraint learning and backjumping\nand was shown to outperform many other algorithms of the time. This led to AND/OR\nsearch algorithms applicable to both CSPs and probabilistic reasoning (Dechter and Ma-\nteescu, 2007). Brown et al. (1988) introduce the idea of symmetry breaking in CSPs, and\nGent et al. (2006) give a recent survey.\nThe field of distributed constraint satisfaction looks at solving CSPs when there is a\nDISTRIBUTED\nCONSTRAINT\nSATISFACTION\ncollection of agents, each of which controls a subset of the constraint variables. There have\nbeen annual workshops on this problem since 2000, and good coverage elsewhere (Collin\net al., 1999; Pearce et al., 2008; Shoham and Leyton-Brown, 2009).\nComparing CSP algorithms is mostly an empirical science: few theoretical results show\nthat one algorithm dominates another on all problems; instead, we need to run experiments\nto see which algorithms perform better on typical instances of problems. As Hooker (1995)\npoints out, we need to be careful to distinguish between competitive testing—as occurs in\ncompetitions among algorithms based on run time—and scientific testing, whose goal is to\nidentify the properties of an algorithm that determine its efficacy on a class of problems.\nThe recent textbooks by Apt (2003) and Dechter (2003), and the collection by Rossi\net al. (2006) are excellent resources on constraint processing. There are several good earlier\nsurveys, including those by Kumar (1992), Dechter and Frost (2002), and Bartak (2001); and\nthe encyclopedia articles by Dechter (1992) and Mackworth (1992). Pearson and Jeavons\n(1997) survey tractable classes of CSPs, covering both structural decomposition methods\nand methods that rely on properties of the domains or constraints themselves. Kondrak and\nvan Beek (1997) give an analytical survey of backtracking search algorithms, and Bacchus\nand van Run (1995) give a more empirical survey. Constraint programming is covered in the\nbooks by Apt (2003) and Fruhwirth and Abdennadher (2003). Several interesting applications\nare described in the collection edited by Freuder and Mackworth (1994). Papers on constraint\nsatisfaction appear regularly in Artificial Intelligence and in the specialist journal Constraints.\nThe primary conference venue is the International Conference on Principles and Practice of\nConstraint Programming, often called CP.\nEXERCISES\n6.1\nHow many solutions are there for the map-coloring problem in Figure 6.1? How many\nsolutions if four colors are allowed? Two colors?\n6.2\nConsider the problem of placing k knights on an n × n chessboard such that no two\nknights are attacking each other, where k is given and k ≤n2.\nExercises\n231\na. Choose a CSP formulation. In your formulation, what are the variables?\nb. What are the possible values of each variable?\nc. What sets of variables are constrained, and how?\nd. Now consider the problem of putting as many knights as possible on the board with-\nout any attacks. Explain how to solve this with local search by defining appropriate\nACTIONS and RESULT functions and a sensible objective function.\n6.3\nConsider the problem of constructing (not solving) crossword puzzles:5 fitting words\ninto a rectangular grid. The grid, which is given as part of the problem, specifies which\nsquares are blank and which are shaded. Assume that a list of words (i.e., a dictionary)\nis provided and that the task is to fill in the blank squares by using any subset of the list.\nFormulate this problem precisely in two ways:\na. As a general search problem. Choose an appropriate search algorithm and specify a\nheuristic function. Is it better to fill in blanks one letter at a time or one word at a time?\nb. As a constraint satisfaction problem. Should the variables be words or letters?\nWhich formulation do you think will be better? Why?\n6.4\nGive precise formulations for each of the following as constraint satisfaction problems:\na. Rectilinear floor-planning: find non-overlapping places in a large rectangle for a number\nof smaller rectangles.\nb. Class scheduling: There is a fixed number of professors and classrooms, a list of classes\nto be offered, and a list of possible time slots for classes. Each professor has a set of\nclasses that he or she can teach.\nc. Hamiltonian tour: given a network of cities connected by roads, choose an order to visit\nall cities in a country without repeating any.\n6.5\nSolve the cryptarithmetic problem in Figure 6.2 by hand, using the strategy of back-\ntracking with forward checking and the MRV and least-constraining-value heuristics.\n6.6\nShow how a single ternary constraint such as “A + B = C” can be turned into three\nbinary constraints by using an auxiliary variable. You may assume finite domains. (Hint:\nConsider a new variable that takes on values that are pairs of other values, and consider\nconstraints such as “X is the first element of the pair Y .”) Next, show how constraints with\nmore than three variables can be treated similarly. Finally, show how unary constraints can be\neliminated by altering the domains of variables. This completes the demonstration that any\nCSP can be transformed into a CSP with only binary constraints.\n6.7\nConsider the following logic puzzle: In five houses, each with a different color, live five\npersons of different nationalities, each of whom prefers a different brand of candy, a different\ndrink, and a different pet. Given the following facts, the questions to answer are “Where does\nthe zebra live, and in which house do they drink water?”\n5 Ginsberg et al. (1990) discuss several methods for constructing crossword puzzles. Littman et al. (1999) tackle\nthe harder problem of solving them.\n232\nChapter\n6.\nConstraint Satisfaction Problems\nThe Englishman lives in the red house.\nThe Spaniard owns the dog.\nThe Norwegian lives in the first house on the left.\nThe green house is immediately to the right of the ivory house.\nThe man who eats Hershey bars lives in the house next to the man with the fox.\nKit Kats are eaten in the yellow house.\nThe Norwegian lives next to the blue house.\nThe Smarties eater owns snails.\nThe Snickers eater drinks orange juice.\nThe Ukrainian drinks tea.\nThe Japanese eats Milky Ways.\nKit Kats are eaten in a house next to the house where the horse is kept.\nCoffee is drunk in the green house.\nMilk is drunk in the middle house.\nDiscuss different representations of this problem as a CSP. Why would one prefer one repre-\nsentation over another?\n6.8\nConsider the graph with 8 nodes A1, A2, A3, A4, H, T, F1, F2. Ai is connected to\nAi+1 for all i, each Ai is connected to H, H is connected to T, and T is connected to each\nFi. Find a 3-coloring of this graph by hand using the following strategy: backtracking with\nconflict-directed backjumping, the variable order A1, H, A4, F1, A2, F2, A3, T, and the\nvalue order R, G, B.\n6.9\nExplain why it is a good heuristic to choose the variable that is most constrained but the\nvalue that is least constraining in a CSP search.\n6.10\nGenerate random instances of map-coloring problems as follows: scatter n points on\nthe unit square; select a point X at random, connect X by a straight line to the nearest point\nY such that X is not already connected to Y and the line crosses no other line; repeat the\nprevious step until no more connections are possible. The points represent regions on the\nmap and the lines connect neighbors. Now try to find k-colorings of each map, for both\nk = 3 and k = 4, using min-conflicts, backtracking, backtracking with forward checking, and\nbacktracking with MAC. Construct a table of average run times for each algorithm for values\nof n up to the largest you can manage. Comment on your results.\n6.11\nUse the AC-3 algorithm to show that arc consistency can detect the inconsistency of\nthe partial assignment {WA = green, V = red} for the problem shown in Figure 6.1.\n6.12\nWhat is the worst-case complexity of running AC-3 on a tree-structured CSP?\n6.13\nAC-3 puts back on the queue every arc (Xk, Xi) whenever any value is deleted from\nthe domain of Xi, even if each value of Xk is consistent with several remaining values of Xi.\nSuppose that, for every arc (Xk, Xi), we keep track of the number of remaining values of Xi\nthat are consistent with each value of Xk. Explain how to update these numbers efficiently\nand hence show that arc consistency can be enforced in total time O(n2d2).\nExercises\n233\n6.14\nThe TREE-CSP-SOLVER (Figure 6.10) makes arcs consistent starting at the leaves and\nworking backwards towards the root. Why does it do that? What would happen if it went in\nthe opposite direction?\n6.15\nWe introduced Sudoku as a CSP to be solved by search over partial assignments be-\ncause that is the way people generally undertake solving Sudoku problems. It is also possible,\nof course, to attack these problems with local search over complete assignments. How well\nwould a local solver using the min-conflicts heuristic do on Sudoku problems?\n6.16\nDefine in your own words the terms constraint, backtracking search, arc consistency,\nbackjumping, min-conflicts, and cycle cutset.\n6.17\nSuppose that a graph is known to have a cycle cutset of no more than k nodes. Describe\na simple algorithm for finding a minimal cycle cutset whose run time is not much more than\nO(nk) for a CSP with n variables. Search the literature for methods for finding approximately\nminimal cycle cutsets in time that is polynomial in the size of the cutset. Does the existence\nof such algorithms make the cycle cutset method practical?\n" } ] }, "Part III": { "part": "Part III", "chapters": [ { "chapter": 7, "text": "7\nLOGICAL AGENTS\nIn which we design agents that can form representations of a complex world, use a\nprocess of inference to derive new representations about the world, and use these\nnew representations to deduce what to do.\nHumans, it seems, know things; and what they know helps them do things.\nThese are\nnot empty statements. They make strong claims about how the intelligence of humans is\nachieved—not by purely reflex mechanisms but by processes of reasoning that operate on\nREASONING\ninternal representations of knowledge. In AI, this approach to intelligence is embodied in\nREPRESENTATION\nknowledge-based agents.\nKNOWLEDGE-BASED\nAGENTS\nThe problem-solving agents of Chapters 3 and 4 know things, but only in a very limited,\ninflexible sense. For example, the transition model for the 8-puzzle—knowledge of what the\nactions do—is hidden inside the domain-specific code of the RESULT function. It can be\nused to predict the outcome of actions but not to deduce that two tiles cannot occupy the\nsame space or that states with odd parity cannot be reached from states with even parity. The\natomic representations used by problem-solving agents are also very limiting. In a partially\nobservable environment, an agent’s only choice for representing what it knows about the\ncurrent state is to list all possible concrete states—a hopeless prospect in large environments.\nChapter 6 introduced the idea of representing states as assignments of values to vari-\nables; this is a step in the right direction, enabling some parts of the agent to work in a\ndomain-independent way and allowing for more efficient algorithms. In this chapter and\nthose that follow, we take this step to its logical conclusion, so to speak—we develop logic\nLOGIC\nas a general class of representations to support knowledge-based agents. Such agents can\ncombine and recombine information to suit myriad purposes. Often, this process can be quite\nfar removed from the needs of the moment—as when a mathematician proves a theorem or\nan astronomer calculates the earth’s life expectancy. Knowledge-based agents can accept new\ntasks in the form of explicitly described goals; they can achieve competence quickly by being\ntold or learning new knowledge about the environment; and they can adapt to changes in the\nenvironment by updating the relevant knowledge.\nWe begin in Section 7.1 with the overall agent design. Section 7.2 introduces a sim-\nple new environment, the wumpus world, and illustrates the operation of a knowledge-based\nagent without going into any technical detail. Then we explain the general principles of logic\n234\nSection 7.1.\nKnowledge-Based Agents\n235\nin Section 7.3 and the specifics of propositional logic in Section 7.4. While less expressive\nthan first-order logic (Chapter 8), propositional logic illustrates all the basic concepts of\nlogic; it also comes with well-developed inference technologies, which we describe in sec-\ntions 7.5 and 7.6. Finally, Section 7.7 combines the concept of knowledge-based agents with\nthe technology of propositional logic to build some simple agents for the wumpus world.\n7.1\nKNOWLEDGE-BASED AGENTS\nThe central component of a knowledge-based agent is its knowledge base, or KB. A knowl-\nKNOWLEDGE BASE\nedge base is a set of sentences. (Here “sentence” is used as a technical term. It is related\nSENTENCE\nbut not identical to the sentences of English and other natural languages.) Each sentence is\nexpressed in a language called a knowledge representation language and represents some\nKNOWLEDGE\nREPRESENTATION\nLANGUAGE\nassertion about the world. Sometimes we dignify a sentence with the name axiom, when the\nAXIOM\nsentence is taken as given without being derived from other sentences.\nThere must be a way to add new sentences to the knowledge base and a way to query\nwhat is known. The standard names for these operations are TELL and ASK, respectively.\nBoth operations may involve inference—that is, deriving new sentences from old. Inference\nINFERENCE\nmust obey the requirement that when one ASKs a question of the knowledge base, the answer\nshould follow from what has been told (or TELLed) to the knowledge base previously. Later\nin this chapter, we will be more precise about the crucial word “follow.” For now, take it to\nmean that the inference process should not make things up as it goes along.\nFigure 7.1 shows the outline of a knowledge-based agent program. Like all our agents,\nit takes a percept as input and returns an action. The agent maintains a knowledge base, KB,\nwhich may initially contain some background knowledge.\nBACKGROUND\nKNOWLEDGE\nEach time the agent program is called, it does three things. First, it TELLs the knowl-\nedge base what it perceives. Second, it ASKs the knowledge base what action it should\nperform. In the process of answering this query, extensive reasoning may be done about\nthe current state of the world, about the outcomes of possible action sequences, and so on.\nThird, the agent program TELLs the knowledge base which action was chosen, and the agent\nexecutes the action.\nThe details of the representation language are hidden inside three functions that imple-\nment the interface between the sensors and actuators on one side and the core representation\nand reasoning system on the other. MAKE-PERCEPT-SENTENCE constructs a sentence as-\nserting that the agent perceived the given percept at the given time. MAKE-ACTION-QUERY\nconstructs a sentence that asks what action should be done at the current time. Finally,\nMAKE-ACTION-SENTENCE constructs a sentence asserting that the chosen action was ex-\necuted. The details of the inference mechanisms are hidden inside TELL and ASK. Later\nsections will reveal these details.\nThe agent in Figure 7.1 appears quite similar to the agents with internal state described\nin Chapter 2. Because of the definitions of TELL and ASK, however, the knowledge-based\nagent is not an arbitrary program for calculating actions. It is amenable to a description at\n236\nChapter\n7.\nLogical Agents\nfunction KB-AGENT(percept) returns an action\npersistent: KB, a knowledge base\nt, a counter, initially 0, indicating time\nTELL(KB, MAKE-PERCEPT-SENTENCE(percept,t))\naction ←ASK(KB, MAKE-ACTION-QUERY(t))\nTELL(KB, MAKE-ACTION-SENTENCE(action,t))\nt ←t + 1\nreturn action\nFigure 7.1\nA generic knowledge-based agent. Given a percept, the agent adds the percept\nto its knowledge base, asks the knowledge base for the best action, and tells the knowledge\nbase that it has in fact taken that action.\nthe knowledge level, where we need specify only what the agent knows and what its goals\nKNOWLEDGE LEVEL\nare, in order to fix its behavior. For example, an automated taxi might have the goal of\ntaking a passenger from San Francisco to Marin County and might know that the Golden\nGate Bridge is the only link between the two locations. Then we can expect it to cross the\nGolden Gate Bridge because it knows that that will achieve its goal. Notice that this analysis\nis independent of how the taxi works at the implementation level. It doesn’t matter whether\nIMPLEMENTATION\nLEVEL\nits geographical knowledge is implemented as linked lists or pixel maps, or whether it reasons\nby manipulating strings of symbols stored in registers or by propagating noisy signals in a\nnetwork of neurons.\nA knowledge-based agent can be built simply by TELLing it what it needs to know.\nStarting with an empty knowledge base, the agent designer can TELL sentences one by one\nuntil the agent knows how to operate in its environment. This is called the declarative ap-\nDECLARATIVE\nproach to system building. In contrast, the procedural approach encodes desired behaviors\ndirectly as program code. In the 1970s and 1980s, advocates of the two approaches engaged\nin heated debates. We now understand that a successful agent often combines both declarative\nand procedural elements in its design, and that declarative knowledge can often be compiled\ninto more efficient procedural code.\nWe can also provide a knowledge-based agent with mechanisms that allow it to learn\nfor itself. These mechanisms, which are discussed in Chapter 18, create general knowledge\nabout the environment from a series of percepts. A learning agent can be fully autonomous.\n7.2\nTHE WUMPUS WORLD\nIn this section we describe an environment in which knowledge-based agents can show their\nworth. The wumpus world is a cave consisting of rooms connected by passageways. Lurking\nWUMPUS WORLD\nsomewhere in the cave is the terrible wumpus, a beast that eats anyone who enters its room.\nThe wumpus can be shot by an agent, but the agent has only one arrow. Some rooms contain\nSection 7.2.\nThe Wumpus World\n237\nbottomless pits that will trap anyone who wanders into these rooms (except for the wumpus,\nwhich is too big to fall in). The only mitigating feature of this bleak environment is the\npossibility of finding a heap of gold. Although the wumpus world is rather tame by modern\ncomputer game standards, it illustrates some important points about intelligence.\nA sample wumpus world is shown in Figure 7.2. The precise definition of the task\nenvironment is given, as suggested in Section 2.3, by the PEAS description:\n• Performance measure: +1000 for climbing out of the cave with the gold, –1000 for\nfalling into a pit or being eaten by the wumpus, –1 for each action taken and –10 for\nusing up the arrow. The game ends either when the agent dies or when the agent climbs\nout of the cave.\n• Environment: A 4 × 4 grid of rooms. The agent always starts in the square labeled\n[1,1], facing to the right. The locations of the gold and the wumpus are chosen ran-\ndomly, with a uniform distribution, from the squares other than the start square. In\naddition, each square other than the start can be a pit, with probability 0.2.\n• Actuators: The agent can move Forward, TurnLeft by 90◦, or TurnRight by 90◦. The\nagent dies a miserable death if it enters a square containing a pit or a live wumpus. (It\nis safe, albeit smelly, to enter a square with a dead wumpus.) If an agent tries to move\nforward and bumps into a wall, then the agent does not move. The action Grab can be\nused to pick up the gold if it is in the same square as the agent. The action Shoot can\nbe used to fire an arrow in a straight line in the direction the agent is facing. The arrow\ncontinues until it either hits (and hence kills) the wumpus or hits a wall. The agent has\nonly one arrow, so only the first Shoot action has any effect. Finally, the action Climb\ncan be used to climb out of the cave, but only from square [1,1].\n• Sensors: The agent has five sensors, each of which gives a single bit of information:\n– In the square containing the wumpus and in the directly (not diagonally) adjacent\nsquares, the agent will perceive a Stench.\n– In the squares directly adjacent to a pit, the agent will perceive a Breeze.\n– In the square where the gold is, the agent will perceive a Glitter.\n– When an agent walks into a wall, it will perceive a Bump.\n– When the wumpus is killed, it emits a woeful Scream that can be perceived any-\nwhere in the cave.\nThe percepts will be given to the agent program in the form of a list of five symbols;\nfor example, if there is a stench and a breeze, but no glitter, bump, or scream, the agent\nprogram will get [Stench, Breeze, None, None, None].\nWe can characterize the wumpus environment along the various dimensions given in Chap-\nter 2. Clearly, it is discrete, static, and single-agent. (The wumpus doesn’t move, fortunately.)\nIt is sequential, because rewards may come only after many actions are taken. It is partially\nobservable, because some aspects of the state are not directly perceivable: the agent’s lo-\ncation, the wumpus’s state of health, and the availability of an arrow. As for the locations\nof the pits and the wumpus: we could treat them as unobserved parts of the state that hap-\npen to be immutable—in which case, the transition model for the environment is completely\n238\nChapter\n7.\nLogical Agents\nPIT\n1\n2\n3\n4\n1\n2\n3\n4\nSTART\nStench\nStench\nB\nr\nee\nz\ne\nGold\nPIT\nPIT\nB\nr\nee\nz\ne\nB\nr\nee\nz\ne\nB\nr\nee\nz\ne\nB\nr\nee\nz\ne\nB\nr\nee\nz\ne\nStench\nFigure 7.2\nA typical wumpus world. The agent is in the bottom left corner, facing right.\nknown; or we could say that the transition model itself is unknown because the agent doesn’t\nknow which Forward actions are fatal—in which case, discovering the locations of pits and\nwumpus completes the agent’s knowledge of the transition model.\nFor an agent in the environment, the main challenge is its initial ignorance of the con-\nfiguration of the environment; overcoming this ignorance seems to require logical reasoning.\nIn most instances of the wumpus world, it is possible for the agent to retrieve the gold safely.\nOccasionally, the agent must choose between going home empty-handed and risking death to\nfind the gold. About 21% of the environments are utterly unfair, because the gold is in a pit\nor surrounded by pits.\nLet us watch a knowledge-based wumpus agent exploring the environment shown in\nFigure 7.2. We use an informal knowledge representation language consisting of writing\ndown symbols in a grid (as in Figures 7.3 and 7.4).\nThe agent’s initial knowledge base contains the rules of the environment, as described\npreviously; in particular, it knows that it is in [1,1] and that [1,1] is a safe square; we denote\nthat with an “A” and “OK,” respectively, in square [1,1].\nThe first percept is [None, None, None, None, None], from which the agent can con-\nclude that its neighboring squares, [1,2] and [2,1], are free of dangers—they are OK. Fig-\nure 7.3(a) shows the agent’s state of knowledge at this point.\nA cautious agent will move only into a square that it knows to be OK. Let us suppose\nthe agent decides to move forward to [2,1]. The agent perceives a breeze (denoted by “B”) in\n[2,1], so there must be a pit in a neighboring square. The pit cannot be in [1,1], by the rules of\nthe game, so there must be a pit in [2,2] or [3,1] or both. The notation “P?” in Figure 7.3(b)\nindicates a possible pit in those squares. At this point, there is only one known square that is\nOK and that has not yet been visited. So the prudent agent will turn around, go back to [1,1],\nand then proceed to [1,2].\nThe agent perceives a stench in [1,2], resulting in the state of knowledge shown in\nFigure 7.4(a). The stench in [1,2] means that there must be a wumpus nearby. But the\nSection 7.2.\nThe Wumpus World\n239\nA\nB\nG\nP\nS\nW\n = Agent\n = Breeze\n = Glitter, Gold\n = Pit\n = Stench\n = Wumpus\nOK = Safe square\nV\n = Visited\nA\nOK\n 1,1\n 2,1\n 3,1\n 4,1\n 1,2\n 2,2\n 3,2\n 4,2\n 1,3\n 2,3\n 3,3\n 4,3\n 1,4\n 2,4\n 3,4\n 4,4\nOK\nOK\nB\nP?\nP?\nA\nOK\nOK\nOK\n 1,1\n 2,1\n 3,1\n 4,1\n 1,2\n 2,2\n 3,2\n 4,2\n 1,3\n 2,3\n 3,3\n 4,3\n 1,4\n 2,4\n 3,4\n 4,4\nV\n(a)\n(b)\nFigure 7.3\nThe first step taken by the agent in the wumpus world. (a) The initial sit-\nuation, after percept [None, None, None, None, None]. (b) After one move, with percept\n[None, Breeze, None, None, None].\nB\nB\nP!\nA\nOK\nOK\nOK\n 1,1\n 2,1\n 3,1\n 4,1\n 1,2\n 2,2\n 3,2\n 4,2\n 1,3\n 2,3\n 3,3\n 4,3\n 1,4\n 2,4\n 3,4\n 4,4\nV\nOK\nW!\nV\nP!\nA\nOK\nOK\nOK\n 1,1\n 2,1\n 3,1\n 4,1\n 1,2\n 2,2\n 3,2\n 4,2\n 1,3\n 2,3\n 3,3\n 4,3\n 1,4\n 2,4\n 3,4\n 4,4\nV\nS\nOK\nW!\nV\nV\nV\nB\nS G\nP?\nP?\n(b)\n(a)\nS\nA\nB\nG\nP\nS\nW\n = Agent\n = Breeze\n = Glitter, Gold\n = Pit\n = Stench\n = Wumpus\nOK = Safe square\nV\n = Visited\nFigure 7.4\nTwo later stages in the progress of the agent.\n(a) After the third move,\nwith percept [Stench, None, None, None, None]. (b) After the fifth move, with percept\n[Stench, Breeze, Glitter, None, None].\nwumpus cannot be in [1,1], by the rules of the game, and it cannot be in [2,2] (or the agent\nwould have detected a stench when it was in [2,1]). Therefore, the agent can infer that the\nwumpus is in [1,3]. The notation W! indicates this inference. Moreover, the lack of a breeze\nin [1,2] implies that there is no pit in [2,2]. Yet the agent has already inferred that there must\nbe a pit in either [2,2] or [3,1], so this means it must be in [3,1]. This is a fairly difficult\ninference, because it combines knowledge gained at different times in different places and\nrelies on the lack of a percept to make one crucial step.\n240\nChapter\n7.\nLogical Agents\nThe agent has now proved to itself that there is neither a pit nor a wumpus in [2,2], so it\nis OK to move there. We do not show the agent’s state of knowledge at [2,2]; we just assume\nthat the agent turns and moves to [2,3], giving us Figure 7.4(b). In [2,3], the agent detects a\nglitter, so it should grab the gold and then return home.\nNote that in each case for which the agent draws a conclusion from the available in-\nformation, that conclusion is guaranteed to be correct if the available information is correct.\nThis is a fundamental property of logical reasoning. In the rest of this chapter, we describe\nhow to build logical agents that can represent information and draw conclusions such as those\ndescribed in the preceding paragraphs.\n7.3\nLOGIC\nThis section summarizes the fundamental concepts of logical representation and reasoning.\nThese beautiful ideas are independent of any of logic’s particular forms. We therefore post-\npone the technical details of those forms until the next section, using instead the familiar\nexample of ordinary arithmetic.\nIn Section 7.1, we said that knowledge bases consist of sentences. These sentences\nare expressed according to the syntax of the representation language, which specifies all the\nSYNTAX\nsentences that are well formed. The notion of syntax is clear enough in ordinary arithmetic:\n“x + y = 4” is a well-formed sentence, whereas “x4y+ =” is not.\nA logic must also define the semantics or meaning of sentences. The semantics defines\nSEMANTICS\nthe truth of each sentence with respect to each possible world. For example, the semantics\nTRUTH\nPOSSIBLE WORLD\nfor arithmetic specifies that the sentence “x + y = 4” is true in a world where x is 2 and y\nis 2, but false in a world where x is 1 and y is 1. In standard logics, every sentence must be\neither true or false in each possible world—there is no “in between.”1\nWhen we need to be precise, we use the term model in place of “possible world.”\nMODEL\nWhereas possible worlds might be thought of as (potentially) real environments that the agent\nmight or might not be in, models are mathematical abstractions, each of which simply fixes\nthe truth or falsehood of every relevant sentence. Informally, we may think of a possible world\nas, for example, having x men and y women sitting at a table playing bridge, and the sentence\nx + y = 4 is true when there are four people in total. Formally, the possible models are just\nall possible assignments of real numbers to the variables x and y. Each such assignment fixes\nthe truth of any sentence of arithmetic whose variables are x and y. If a sentence α is true in\nmodel m, we say that m satisfies α or sometimes m is a model of α. We use the notation\nSATISFACTION\nM(α) to mean the set of all models of α.\nNow that we have a notion of truth, we are ready to talk about logical reasoning. This\ninvolves the relation of logical entailment between sentences—the idea that a sentence fol-\nENTAILMENT\nlows logically from another sentence. In mathematical notation, we write\nα |= β\n1 Fuzzy logic, discussed in Chapter 14, allows for degrees of truth.\nSection 7.3.\nLogic\n241\n1\n2\n3\n1\n2\nPIT\n1\n2\n3\n1\n2\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\n1\n2\n3\n1\n2\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\n1\n2\n3\n1\n2\nKB\nα1\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\n1\n2\n3\n1\n2\nPIT\n1\n2\n3\n1\n2\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\n1\n2\n3\n1\n2\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\n1\n2\n3\n1\n2\nPIT\nPIT\n1\n2\n3\n1\n2\nKB\nB\nr\neez\ne\nα2\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\nB\nr\neez\ne\n(a)\n(b)\nFigure 7.5\nPossible models for the presence of pits in squares [1,2], [2,2], and [3,1]. The\nKB corresponding to the observations of nothing in [1,1] and a breeze in [2,1] is shown by\nthe solid line. (a) Dotted line shows models of α1 (no pit in [1,2]). (b) Dotted line shows\nmodels of α2 (no pit in [2,2]).\nto mean that the sentence α entails the sentence β. The formal definition of entailment is this:\nα |= β if and only if, in every model in which α is true, β is also true. Using the notation just\nintroduced, we can write\nα |= β if and only if M(α) ⊆M(β) .\n(Note the direction of the ⊆here: if α |= β, then α is a stronger assertion than β: it rules out\nmore possible worlds.) The relation of entailment is familiar from arithmetic; we are happy\nwith the idea that the sentence x = 0 entails the sentence xy = 0. Obviously, in any model\nwhere x is zero, it is the case that xy is zero (regardless of the value of y).\nWe can apply the same kind of analysis to the wumpus-world reasoning example given\nin the preceding section. Consider the situation in Figure 7.3(b): the agent has detected\nnothing in [1,1] and a breeze in [2,1]. These percepts, combined with the agent’s knowledge\nof the rules of the wumpus world, constitute the KB. The agent is interested (among other\nthings) in whether the adjacent squares [1,2], [2,2], and [3,1] contain pits. Each of the three\nsquares might or might not contain a pit, so (for the purposes of this example) there are 23 = 8\npossible models. These eight models are shown in Figure 7.5.2\nThe KB can be thought of as a set of sentences or as a single sentence that asserts all\nthe individual sentences. The KB is false in models that contradict what the agent knows—\nfor example, the KB is false in any model in which [1,2] contains a pit, because there is\nno breeze in [1,1]. There are in fact just three models in which the KB is true, and these are\n2 Although the figure shows the models as partial wumpus worlds, they are really nothing more than assignments\nof true and false to the sentences “there is a pit in [1,2]” etc. Models, in the mathematical sense, do not need to\nhave ’orrible ’airy wumpuses in them.\n242\nChapter\n7.\nLogical Agents\nshown surrounded by a solid line in Figure 7.5. Now let us consider two possible conclusions:\nα1 = “There is no pit in [1,2].”\nα2 = “There is no pit in [2,2].”\nWe have surrounded the models of α1 and α2 with dotted lines in Figures 7.5(a) and 7.5(b),\nrespectively. By inspection, we see the following:\nin every model in which KB is true, α1 is also true.\nHence, KB |= α1: there is no pit in [1,2]. We can also see that\nin some models in which KB is true, α2 is false.\nHence, KB ̸|= α2: the agent cannot conclude that there is no pit in [2,2]. (Nor can it conclude\nthat there is a pit in [2,2].)3\nThe preceding example not only illustrates entailment but also shows how the definition\nof entailment can be applied to derive conclusions—that is, to carry out logical inference.\nLOGICAL INFERENCE\nThe inference algorithm illustrated in Figure 7.5 is called model checking, because it enu-\nMODEL CHECKING\nmerates all possible models to check that α is true in all models in which KB is true, that is,\nthat M(KB) ⊆M(α).\nIn understanding entailment and inference, it might help to think of the set of all conse-\nquences of KB as a haystack and of α as a needle. Entailment is like the needle being in the\nhaystack; inference is like finding it. This distinction is embodied in some formal notation: if\nan inference algorithm i can derive α from KB, we write\nKB ⊢i α ,\nwhich is pronounced “α is derived from KB by i” or “i derives α from KB.”\nAn inference algorithm that derives only entailed sentences is called sound or truth-\nSOUND\npreserving. Soundness is a highly desirable property. An unsound inference procedure es-\nTRUTH-PRESERVING\nsentially makes things up as it goes along—it announces the discovery of nonexistent needles.\nIt is easy to see that model checking, when it is applicable,4 is a sound procedure.\nThe property of completeness is also desirable: an inference algorithm is complete if\nCOMPLETENESS\nit can derive any sentence that is entailed. For real haystacks, which are finite in extent,\nit seems obvious that a systematic examination can always decide whether the needle is in\nthe haystack. For many knowledge bases, however, the haystack of consequences is infinite,\nand completeness becomes an important issue.5 Fortunately, there are complete inference\nprocedures for logics that are sufficiently expressive to handle many knowledge bases.\nWe have described a reasoning process whose conclusions are guaranteed to be true\nin any world in which the premises are true; in particular, if KB is true in the real world,\nthen any sentence α derived from KB by a sound inference procedure is also true in the real\nworld. So, while an inference process operates on “syntax”—internal physical configurations\nsuch as bits in registers or patterns of electrical blips in brains—the process corresponds\n3 The agent can calculate the probability that there is a pit in [2,2]; Chapter 13 shows how.\n4 Model checking works if the space of models is finite—for example, in wumpus worlds of fixed size. For\narithmetic, on the other hand, the space of models is infinite: even if we restrict ourselves to the integers, there\nare infinitely many pairs of values for x and y in the sentence x + y = 4.\n5 Compare with the case of infinite search spaces in Chapter 3, where depth-first search is not complete.\nSection 7.4.\nPropositional Logic: A Very Simple Logic\n243\nFollows\nSentences\nSentence\nEntails\nSemantics\nSemantics\nRepresentation\nWorld\nAspects of the\n real world\nAspect of the\n real world\nFigure 7.6\nSentences are physical configurations of the agent, and reasoning is a process\nof constructing new physical configurations from old ones. Logical reasoning should en-\nsure that the new configurations represent aspects of the world that actually follow from the\naspects that the old configurations represent.\nto the real-world relationship whereby some aspect of the real world is the case6 by virtue\nof other aspects of the real world being the case. This correspondence between world and\nrepresentation is illustrated in Figure 7.6.\nThe final issue to consider is grounding—the connection between logical reasoning\nGROUNDING\nprocesses and the real environment in which the agent exists. In particular, how do we know\nthat KB is true in the real world? (After all, KB is just “syntax” inside the agent’s head.)\nThis is a philosophical question about which many, many books have been written. (See\nChapter 26.) A simple answer is that the agent’s sensors create the connection. For example,\nour wumpus-world agent has a smell sensor. The agent program creates a suitable sentence\nwhenever there is a smell. Then, whenever that sentence is in the knowledge base, it is\ntrue in the real world. Thus, the meaning and truth of percept sentences are defined by the\nprocesses of sensing and sentence construction that produce them. What about the rest of the\nagent’s knowledge, such as its belief that wumpuses cause smells in adjacent squares? This\nis not a direct representation of a single percept, but a general rule—derived, perhaps, from\nperceptual experience but not identical to a statement of that experience. General rules like\nthis are produced by a sentence construction process called learning, which is the subject\nof Part V. Learning is fallible. It could be the case that wumpuses cause smells except on\nFebruary 29 in leap years, which is when they take their baths. Thus, KB may not be true in\nthe real world, but with good learning procedures, there is reason for optimism.\n7.4\nPROPOSITIONAL LOGIC: A VERY SIMPLE LOGIC\nWe now present a simple but powerful logic called propositional logic. We cover the syntax\nPROPOSITIONAL\nLOGIC\nof propositional logic and its semantics—the way in which the truth of sentences is deter-\nmined. Then we look at entailment—the relation between a sentence and another sentence\nthat follows from it—and see how this leads to a simple algorithm for logical inference. Ev-\nerything takes place, of course, in the wumpus world.\n6 As Wittgenstein (1922) put it in his famous Tractatus: “The world is everything that is the case.”\n244\nChapter\n7.\nLogical Agents\n7.4.1\nSyntax\nThe syntax of propositional logic defines the allowable sentences. The atomic sentences\nATOMIC SENTENCES\nconsist of a single proposition symbol. Each such symbol stands for a proposition that can\nPROPOSITION\nSYMBOL\nbe true or false. We use symbols that start with an uppercase letter and may contain other\nletters or subscripts, for example: P, Q, R, W1,3 and North. The names are arbitrary but\nare often chosen to have some mnemonic value—we use W1,3 to stand for the proposition\nthat the wumpus is in [1,3]. (Remember that symbols such as W1,3 are atomic, i.e., W, 1,\nand 3 are not meaningful parts of the symbol.) There are two proposition symbols with fixed\nmeanings: True is the always-true proposition and False is the always-false proposition.\nComplex sentences are constructed from simpler sentences, using parentheses and logical\nCOMPLEX\nSENTENCES\nconnectives. There are five connectives in common use:\nLOGICAL\nCONNECTIVES\n¬ (not). A sentence such as ¬W1,3 is called the negation of W1,3. A literal is either an\nNEGATION\nLITERAL\natomic sentence (a positive literal) or a negated atomic sentence (a negative literal).\n∧(and). A sentence whose main connective is ∧, such as W1,3 ∧P3,1, is called a con-\njunction; its parts are the conjuncts. (The ∧looks like an “A” for “And.”)\nCONJUNCTION\n∨(or). A sentence using ∨, such as (W1,3∧P3,1)∨W2,2, is a disjunction of the disjuncts\nDISJUNCTION\n(W1,3 ∧P3,1) and W2,2. (Historically, the ∨comes from the Latin “vel,” which means\n“or.” For most people, it is easier to remember ∨as an upside-down ∧.)\n⇒(implies). A sentence such as (W1,3 ∧P3,1) ⇒¬W2,2 is called an implication (or con-\nIMPLICATION\nditional). Its premise or antecedent is (W1,3 ∧P3,1), and its conclusion or consequent\nPREMISE\nCONCLUSION\nis ¬W2,2. Implications are also known as rules or if–then statements. The implication\nRULES\nsymbol is sometimes written in other books as ⊃or →.\n⇔(if and only if). The sentence W1,3 ⇔¬W2,2 is a biconditional. Some other books\nBICONDITIONAL\nwrite this as ≡.\nSentence\n→\nAtomicSentence | ComplexSentence\nAtomicSentence\n→\nTrue | False | P | Q | R | . . .\nComplexSentence\n→\n( Sentence ) | [ Sentence ]\n|\n¬ Sentence\n|\nSentence ∧Sentence\n|\nSentence ∨Sentence\n|\nSentence\n⇒Sentence\n|\nSentence\n⇔\nSentence\nOPERATOR PRECEDENCE\n:\n¬, ∧, ∨, ⇒, ⇔\nFigure 7.7\nA BNF (Backus–Naur Form) grammar of sentences in propositional logic,\nalong with operator precedences, from highest to lowest.\nSection 7.4.\nPropositional Logic: A Very Simple Logic\n245\nFigure 7.7 gives a formal grammar of propositional logic; see page 1060 if you are not\nfamiliar with the BNF notation. The BNF grammar by itself is ambiguous; a sentence with\nseveral operators can be parsed by the grammar in multiple ways. To eliminate the ambiguity\nwe define a precedence for each operator. The “not” operator (¬) has the highest precedence,\nwhich means that in the sentence ¬A ∧B the ¬ binds most tightly, giving us the equivalent\nof (¬A)∧B rather than ¬(A∧B). (The notation for ordinary arithmetic is the same: −2+4\nis 2, not –6.) When in doubt, use parentheses to make sure of the right interpretation. Square\nbrackets mean the same thing as parentheses; the choice of square brackets or parentheses is\nsolely to make it easier for a human to read a sentence.\n7.4.2\nSemantics\nHaving specified the syntax of propositional logic, we now specify its semantics. The se-\nmantics defines the rules for determining the truth of a sentence with respect to a particular\nmodel. In propositional logic, a model simply fixes the truth value—true or false—for ev-\nTRUTH VALUE\nery proposition symbol. For example, if the sentences in the knowledge base make use of the\nproposition symbols P1,2, P2,2, and P3,1, then one possible model is\nm1 = {P1,2 = false, P2,2 = false, P3,1 = true} .\nWith three proposition symbols, there are 23 = 8 possible models—exactly those depicted\nin Figure 7.5. Notice, however, that the models are purely mathematical objects with no\nnecessary connection to wumpus worlds. P1,2 is just a symbol; it might mean “there is a pit\nin [1,2]” or “I’m in Paris today and tomorrow.”\nThe semantics for propositional logic must specify how to compute the truth value of\nany sentence, given a model. This is done recursively. All sentences are constructed from\natomic sentences and the five connectives; therefore, we need to specify how to compute the\ntruth of atomic sentences and how to compute the truth of sentences formed with each of the\nfive connectives. Atomic sentences are easy:\n• True is true in every model and False is false in every model.\n• The truth value of every other proposition symbol must be specified directly in the\nmodel. For example, in the model m1 given earlier, P1,2 is false.\nFor complex sentences, we have five rules, which hold for any subsentences P and Q in any\nmodel m (here “iff” means “if and only if”):\n• ¬P is true iff P is false in m.\n• P ∧Q is true iff both P and Q are true in m.\n• P ∨Q is true iff either P or Q is true in m.\n• P ⇒Q is true unless P is true and Q is false in m.\n• P ⇔Q is true iff P and Q are both true or both false in m.\nThe rules can also be expressed with truth tables that specify the truth value of a complex\nTRUTH TABLE\nsentence for each possible assignment of truth values to its components. Truth tables for the\nfive connectives are given in Figure 7.8. From these tables, the truth value of any sentence s\ncan be computed with respect to any model m by a simple recursive evaluation. For example,\n246\nChapter\n7.\nLogical Agents\nP\nQ\n¬P\nP ∧Q\nP ∨Q\nP ⇒Q\nP ⇔Q\nfalse\nfalse\ntrue\nfalse\nfalse\ntrue\ntrue\nfalse\ntrue\ntrue\nfalse\ntrue\ntrue\nfalse\ntrue\nfalse\nfalse\nfalse\ntrue\nfalse\nfalse\ntrue\ntrue\nfalse\ntrue\ntrue\ntrue\ntrue\nFigure 7.8\nTruth tables for the five logical connectives. To use the table to compute, for\nexample, the value of P ∨Q when P is true and Q is false, first look on the left for the row\nwhere P is true and Q is false (the third row). Then look in that row under the P ∨Q column\nto see the result: true.\nthe sentence ¬P1,2 ∧(P2,2 ∨P3,1), evaluated in m1, gives true ∧(false ∨true) = true ∧\ntrue = true. Exercise 7.3 asks you to write the algorithm PL-TRUE?(s, m), which computes\nthe truth value of a propositional logic sentence s in a model m.\nThe truth tables for “and,” “or,” and “not” are in close accord with our intuitions about\nthe English words. The main point of possible confusion is that P ∨Q is true when P is true\nor Q is true or both. A different connective, called “exclusive or” (“xor” for short), yields\nfalse when both disjuncts are true.7 There is no consensus on the symbol for exclusive or;\nsome choices are ˙∨or ̸= or ⊕.\nThe truth table for ⇒may not quite fit one’s intuitive understanding of “P implies Q”\nor “if P then Q.” For one thing, propositional logic does not require any relation of causation\nor relevance between P and Q. The sentence “5 is odd implies Tokyo is the capital of Japan”\nis a true sentence of propositional logic (under the normal interpretation), even though it is\na decidedly odd sentence of English. Another point of confusion is that any implication is\ntrue whenever its antecedent is false. For example, “5 is even implies Sam is smart” is true,\nregardless of whether Sam is smart. This seems bizarre, but it makes sense if you think of\n“P ⇒Q” as saying, “If P is true, then I am claiming that Q is true. Otherwise I am making\nno claim.” The only way for this sentence to be false is if P is true but Q is false.\nThe biconditional, P ⇔Q, is true whenever both P ⇒Q and Q ⇒P are true. In\nEnglish, this is often written as “P if and only if Q.” Many of the rules of the wumpus world\nare best written using ⇔. For example, a square is breezy if a neighboring square has a pit,\nand a square is breezy only if a neighboring square has a pit. So we need a biconditional,\nB1,1 ⇔(P1,2 ∨P2,1) ,\nwhere B1,1 means that there is a breeze in [1,1].\n7.4.3\nA simple knowledge base\nNow that we have defined the semantics for propositional logic, we can construct a knowledge\nbase for the wumpus world. We focus first on the immutable aspects of the wumpus world,\nleaving the mutable aspects for a later section. For now, we need the following symbols for\neach [x, y] location:\n7 Latin has a separate word, aut, for exclusive or.\nSection 7.4.\nPropositional Logic: A Very Simple Logic\n247\nPx,y is true if there is a pit in [x, y].\nWx,y is true if there is a wumpus in [x, y], dead or alive.\nBx,y is true if the agent perceives a breeze in [x, y].\nSx,y is true if the agent perceives a stench in [x, y].\nThe sentences we write will suffice to derive ¬P1,2 (there is no pit in [1,2]), as was done\ninformally in Section 7.3. We label each sentence Ri so that we can refer to them:\n• There is no pit in [1,1]:\nR1 :\n¬P1,1 .\n• A square is breezy if and only if there is a pit in a neighboring square. This has to be\nstated for each square; for now, we include just the relevant squares:\nR2 :\nB1,1\n⇔\n(P1,2 ∨P2,1) .\nR3 :\nB2,1\n⇔\n(P1,1 ∨P2,2 ∨P3,1) .\n• The preceding sentences are true in all wumpus worlds. Now we include the breeze\npercepts for the first two squares visited in the specific world the agent is in, leading up\nto the situation in Figure 7.3(b).\nR4 :\n¬B1,1 .\nR5 :\nB2,1 .\n7.4.4\nA simple inference procedure\nOur goal now is to decide whether KB |= α for some sentence α. For example, is ¬P1,2\nentailed by our KB? Our first algorithm for inference is a model-checking approach that is a\ndirect implementation of the definition of entailment: enumerate the models, and check that\nα is true in every model in which KB is true. Models are assignments of true or false to\nevery proposition symbol. Returning to our wumpus-world example, the relevant proposi-\ntion symbols are B1,1, B2,1, P1,1, P1,2, P2,1, P2,2, and P3,1. With seven symbols, there are\n27 = 128 possible models; in three of these, KB is true (Figure 7.9). In those three models,\n¬P1,2 is true, hence there is no pit in [1,2]. On the other hand, P2,2 is true in two of the three\nmodels and false in one, so we cannot yet tell whether there is a pit in [2,2].\nFigure 7.9 reproduces in a more precise form the reasoning illustrated in Figure 7.5. A\ngeneral algorithm for deciding entailment in propositional logic is shown in Figure 7.10. Like\nthe BACKTRACKING-SEARCH algorithm on page 215, TT-ENTAILS? performs a recursive\nenumeration of a finite space of assignments to symbols. The algorithm is sound because it\nimplements directly the definition of entailment, and complete because it works for any KB\nand α and always terminates—there are only finitely many models to examine.\nOf course, “finitely many” is not always the same as “few.” If KB and α contain n\nsymbols in all, then there are 2n models. Thus, the time complexity of the algorithm is\nO(2n). (The space complexity is only O(n) because the enumeration is depth-first.) Later in\nthis chapter we show algorithms that are much more efficient in many cases. Unfortunately,\npropositional entailment is co-NP-complete (i.e., probably no easier than NP-complete—see\nAppendix A), so every known inference algorithm for propositional logic has a worst-case\ncomplexity that is exponential in the size of the input.\n248\nChapter\n7.\nLogical Agents\nB1,1\nB2,1\nP1,1\nP1,2\nP2,1\nP2,2\nP3,1\nR1\nR2\nR3\nR4\nR5\nKB\nfalse\nfalse\nfalse\nfalse\nfalse\nfalse\nfalse\ntrue\ntrue\ntrue\ntrue\nfalse\nfalse\nfalse\nfalse\nfalse\nfalse\nfalse\nfalse\ntrue\ntrue\ntrue\nfalse\ntrue\nfalse\nfalse\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\nfalse\ntrue\nfalse\nfalse\nfalse\nfalse\nfalse\ntrue\ntrue\nfalse\ntrue\ntrue\nfalse\nfalse\ntrue\nfalse\nfalse\nfalse\nfalse\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\nfalse\ntrue\nfalse\nfalse\nfalse\ntrue\nfalse\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\nfalse\ntrue\nfalse\nfalse\nfalse\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\nfalse\ntrue\nfalse\nfalse\ntrue\nfalse\nfalse\ntrue\nfalse\nfalse\ntrue\ntrue\nfalse\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\ntrue\nfalse\ntrue\ntrue\nfalse\ntrue\nfalse\nFigure 7.9\nA truth table constructed for the knowledge base given in the text. KB is true\nif R1 through R5 are true, which occurs in just 3 of the 128 rows (the ones underlined in the\nright-hand column). In all 3 rows, P1,2 is false, so there is no pit in [1,2]. On the other hand,\nthere might (or might not) be a pit in [2,2].\nfunction TT-ENTAILS?(KB,α) returns true or false\ninputs: KB, the knowledge base, a sentence in propositional logic\nα, the query, a sentence in propositional logic\nsymbols ←a list of the proposition symbols in KB and α\nreturn TT-CHECK-ALL(KB,α,symbols,{ })\nfunction TT-CHECK-ALL(KB,α,symbols,model) returns true or false\nif EMPTY?(symbols) then\nif PL-TRUE?(KB,model) then return PL-TRUE?(α,model)\nelse return true // when KB is false, always return true\nelse do\nP ←FIRST(symbols)\nrest ←REST(symbols)\nreturn (TT-CHECK-ALL(KB,α,rest,model ∪{P = true})\nand\nTT-CHECK-ALL(KB,α,rest,model ∪{P = false }))\nFigure 7.10\nA truth-table enumeration algorithm for deciding propositional entailment.\n(TT stands for truth table.) PL-TRUE? returns true if a sentence holds within a model. The\nvariable model represents a partial model—an assignment to some of the symbols. The key-\nword “and” is used here as a logical operation on its two arguments, returning true or false.\nSection 7.5.\nPropositional Theorem Proving\n249\n(α ∧β) ≡(β ∧α)\ncommutativity of ∧\n(α ∨β) ≡(β ∨α)\ncommutativity of ∨\n((α ∧β) ∧γ) ≡(α ∧(β ∧γ))\nassociativity of ∧\n((α ∨β) ∨γ) ≡(α ∨(β ∨γ))\nassociativity of ∨\n¬(¬α) ≡α\ndouble-negation elimination\n(α ⇒β) ≡(¬β ⇒¬α)\ncontraposition\n(α ⇒β) ≡(¬α ∨β)\nimplication elimination\n(α ⇔β) ≡((α ⇒β) ∧(β ⇒α))\nbiconditional elimination\n¬(α ∧β) ≡(¬α ∨¬β)\nDe Morgan\n¬(α ∨β) ≡(¬α ∧¬β)\nDe Morgan\n(α ∧(β ∨γ)) ≡((α ∧β) ∨(α ∧γ))\ndistributivity of ∧over ∨\n(α ∨(β ∧γ)) ≡((α ∨β) ∧(α ∨γ))\ndistributivity of ∨over ∧\nFigure 7.11\nStandard logical equivalences. The symbols α, β, and γ stand for arbitrary\nsentences of propositional logic.\n7.5\nPROPOSITIONAL THEOREM PROVING\nSo far, we have shown how to determine entailment by model checking: enumerating models\nand showing that the sentence must hold in all models. In this section, we show how entail-\nment can be done by theorem proving—applying rules of inference directly to the sentences\nTHEOREM PROVING\nin our knowledge base to construct a proof of the desired sentence without consulting models.\nIf the number of models is large but the length of the proof is short, then theorem proving can\nbe more efficient than model checking.\nBefore we plunge into the details of theorem-proving algorithms, we will need some\nadditional concepts related to entailment. The first concept is logical equivalence: two sen-\nLOGICAL\nEQUIVALENCE\ntences α and β are logically equivalent if they are true in the same set of models. We write\nthis as α ≡β. For example, we can easily show (using truth tables) that P ∧Q and Q ∧P\nare logically equivalent; other equivalences are shown in Figure 7.11. These equivalences\nplay much the same role in logic as arithmetic identities do in ordinary mathematics. An\nalternative definition of equivalence is as follows: any two sentences α and β are equivalent\nonly if each of them entails the other:\nα ≡β\nif and only if\nα |= β and β |= α .\nThe second concept we will need is validity. A sentence is valid if it is true in all models. For\nVALIDITY\nexample, the sentence P ∨¬P is valid. Valid sentences are also known as tautologies—they\nTAUTOLOGY\nare necessarily true. Because the sentence True is true in all models, every valid sentence\nis logically equivalent to True. What good are valid sentences? From our definition of\nentailment, we can derive the deduction theorem, which was known to the ancient Greeks:\nDEDUCTION\nTHEOREM\nFor any sentences α and β, α |= β if and only if the sentence (α ⇒β) is valid.\n(Exercise 7.5 asks for a proof.) Hence, we can decide if α |= β by checking that (α ⇒β) is\ntrue in every model—which is essentially what the inference algorithm in Figure 7.10 does—\n250\nChapter\n7.\nLogical Agents\nor by proving that (α ⇒β) is equivalent to True. Conversely, the deduction theorem states\nthat every valid implication sentence describes a legitimate inference.\nThe final concept we will need is satisfiability. A sentence is satisfiable if it is true\nSATISFIABILITY\nin, or satisfied by, some model. For example, the knowledge base given earlier, (R1 ∧R2 ∧\nR3 ∧R4 ∧R5), is satisfiable because there are three models in which it is true, as shown\nin Figure 7.9. Satisfiability can be checked by enumerating the possible models until one is\nfound that satisfies the sentence. The problem of determining the satisfiability of sentences\nin propositional logic—the SAT problem—was the first problem proved to be NP-complete.\nSAT\nMany problems in computer science are really satisfiability problems. For example, all the\nconstraint satisfaction problems in Chapter 6 ask whether the constraints are satisfiable by\nsome assignment.\nValidity and satisfiability are of course connected: α is valid iff ¬α is unsatisfiable;\ncontrapositively, α is satisfiable iff ¬α is not valid. We also have the following useful result:\nα |= β if and only if the sentence (α ∧¬β) is unsatisfiable.\nProving β from α by checking the unsatisfiability of (α ∧¬β) corresponds exactly to the\nstandard mathematical proof technique of reductio ad absurdum (literally, “reduction to an\nREDUCTIO AD\nABSURDUM\nabsurd thing”). It is also called proof by refutation or proof by contradiction. One assumes a\nREFUTATION\nCONTRADICTION\nsentence β to be false and shows that this leads to a contradiction with known axioms α. This\ncontradiction is exactly what is meant by saying that the sentence (α ∧¬β) is unsatisfiable.\n7.5.1\nInference and proofs\nThis section covers inference rules that can be applied to derive a proof—a chain of conclu-\nINFERENCE RULES\nPROOF\nsions that leads to the desired goal. The best-known rule is called Modus Ponens (Latin for\nMODUS PONENS\nmode that affirms) and is written\nα ⇒β,\nα\nβ\n.\nThe notation means that, whenever any sentences of the form α ⇒β and α are given, then\nthe sentence β can be inferred. For example, if (WumpusAhead ∧WumpusAlive) ⇒Shoot\nand (WumpusAhead ∧WumpusAlive) are given, then Shoot can be inferred.\nAnother useful inference rule is And-Elimination, which says that, from a conjunction,\nAND-ELIMINATION\nany of the conjuncts can be inferred:\nα ∧β\nα\n.\nFor example, from (WumpusAhead ∧WumpusAlive), WumpusAlive can be inferred.\nBy considering the possible truth values of α and β, one can show easily that Modus\nPonens and And-Elimination are sound once and for all. These rules can then be used in\nany particular instances where they apply, generating sound inferences without the need for\nenumerating models.\nAll of the logical equivalences in Figure 7.11 can be used as inference rules. For exam-\nple, the equivalence for biconditional elimination yields the two inference rules\nα ⇔β\n(α ⇒β) ∧(β ⇒α)\nand\n(α ⇒β) ∧(β ⇒α)\nα ⇔β\n.\nSection 7.5.\nPropositional Theorem Proving\n251\nNot all inference rules work in both directions like this. For example, we cannot run Modus\nPonens in the opposite direction to obtain α ⇒β and α from β.\nLet us see how these inference rules and equivalences can be used in the wumpus world.\nWe start with the knowledge base containing R1 through R5 and show how to prove ¬P1,2,\nthat is, there is no pit in [1,2]. First, we apply biconditional elimination to R2 to obtain\nR6 :\n(B1,1 ⇒(P1,2 ∨P2,1)) ∧((P1,2 ∨P2,1) ⇒B1,1) .\nThen we apply And-Elimination to R6 to obtain\nR7 :\n((P1,2 ∨P2,1) ⇒B1,1) .\nLogical equivalence for contrapositives gives\nR8 :\n(¬B1,1 ⇒¬(P1,2 ∨P2,1)) .\nNow we can apply Modus Ponens with R8 and the percept R4 (i.e., ¬B1,1), to obtain\nR9 :\n¬(P1,2 ∨P2,1) .\nFinally, we apply De Morgan’s rule, giving the conclusion\nR10 :\n¬P1,2 ∧¬P2,1 .\nThat is, neither [1,2] nor [2,1] contains a pit.\nWe found this proof by hand, but we can apply any of the search algorithms in Chapter 3\nto find a sequence of steps that constitutes a proof. We just need to define a proof problem as\nfollows:\n• INITIAL STATE: the initial knowledge base.\n• ACTIONS: the set of actions consists of all the inference rules applied to all the sen-\ntences that match the top half of the inference rule.\n• RESULT: the result of an action is to add the sentence in the bottom half of the inference\nrule.\n• GOAL: the goal is a state that contains the sentence we are trying to prove.\nThus, searching for proofs is an alternative to enumerating models. In many practical cases\nfinding a proof can be more efficient because the proof can ignore irrelevant propositions, no\nmatter how many of them there are. For example, the proof given earlier leading to ¬P1,2 ∧\n¬P2,1 does not mention the propositions B2,1, P1,1, P2,2, or P3,1. They can be ignored\nbecause the goal proposition, P1,2, appears only in sentence R2; the other propositions in R2\nappear only in R4 and R2; so R1, R3, and R5 have no bearing on the proof. The same would\nhold even if we added a million more sentences to the knowledge base; the simple truth-table\nalgorithm, on the other hand, would be overwhelmed by the exponential explosion of models.\nOne final property of logical systems is monotonicity, which says that the set of en-\nMONOTONICITY\ntailed sentences can only increase as information is added to the knowledge base.8 For any\nsentences α and β,\nif\nKB |= α\nthen\nKB ∧β |= α .\n8 Nonmonotonic logics, which violate the monotonicity property, capture a common property of human rea-\nsoning: changing one’s mind. They are discussed in Section 12.6.\n252\nChapter\n7.\nLogical Agents\nFor example, suppose the knowledge base contains the additional assertion β stating that there\nare exactly eight pits in the world. This knowledge might help the agent draw additional con-\nclusions, but it cannot invalidate any conclusion α already inferred—such as the conclusion\nthat there is no pit in [1,2]. Monotonicity means that inference rules can be applied whenever\nsuitable premises are found in the knowledge base—the conclusion of the rule must follow\nregardless of what else is in the knowledge base.\n7.5.2\nProof by resolution\nWe have argued that the inference rules covered so far are sound, but we have not discussed\nthe question of completeness for the inference algorithms that use them. Search algorithms\nsuch as iterative deepening search (page 89) are complete in the sense that they will find\nany reachable goal, but if the available inference rules are inadequate, then the goal is not\nreachable—no proof exists that uses only those inference rules. For example, if we removed\nthe biconditional elimination rule, the proof in the preceding section would not go through.\nThe current section introduces a single inference rule, resolution, that yields a complete\ninference algorithm when coupled with any complete search algorithm.\nWe begin by using a simple version of the resolution rule in the wumpus world. Let us\nconsider the steps leading up to Figure 7.4(a): the agent returns from [2,1] to [1,1] and then\ngoes to [1,2], where it perceives a stench, but no breeze. We add the following facts to the\nknowledge base:\nR11 :\n¬B1,2 .\nR12 :\nB1,2 ⇔(P1,1 ∨P2,2 ∨P1,3) .\nBy the same process that led to R10 earlier, we can now derive the absence of pits in [2,2]\nand [1,3] (remember that [1,1] is already known to be pitless):\nR13 :\n¬P2,2 .\nR14 :\n¬P1,3 .\nWe can also apply biconditional elimination to R3, followed by Modus Ponens with R5, to\nobtain the fact that there is a pit in [1,1], [2,2], or [3,1]:\nR15 :\nP1,1 ∨P2,2 ∨P3,1 .\nNow comes the first application of the resolution rule: the literal ¬P2,2 in R13 resolves with\nthe literal P2,2 in R15 to give the resolvent\nRESOLVENT\nR16 :\nP1,1 ∨P3,1 .\nIn English; if there’s a pit in one of [1,1], [2,2], and [3,1] and it’s not in [2,2], then it’s in [1,1]\nor [3,1]. Similarly, the literal ¬P1,1 in R1 resolves with the literal P1,1 in R16 to give\nR17 :\nP3,1 .\nIn English: if there’s a pit in [1,1] or [3,1] and it’s not in [1,1], then it’s in [3,1]. These last\ntwo inference steps are examples of the unit resolution inference rule,\nUNIT RESOLUTION\nℓ1 ∨· · · ∨ℓk,\nm\nℓ1 ∨· · · ∨ℓi−1 ∨ℓi+1 ∨· · · ∨ℓk\n,\nwhere each ℓis a literal and ℓi and m are complementary literals (i.e., one is the negation\nCOMPLEMENTARY\nLITERALS\nSection 7.5.\nPropositional Theorem Proving\n253\nof the other). Thus, the unit resolution rule takes a clause—a disjunction of literals—and a\nCLAUSE\nliteral and produces a new clause. Note that a single literal can be viewed as a disjunction of\none literal, also known as a unit clause.\nUNIT CLAUSE\nThe unit resolution rule can be generalized to the full resolution rule,\nRESOLUTION\nℓ1 ∨· · · ∨ℓk,\nm1 ∨· · · ∨mn\nℓ1 ∨· · · ∨ℓi−1 ∨ℓi+1 ∨· · · ∨ℓk ∨m1 ∨· · · ∨mj−1 ∨mj+1 ∨· · · ∨mn\n,\nwhere ℓi and mj are complementary literals. This says that resolution takes two clauses and\nproduces a new clause containing all the literals of the two original clauses except the two\ncomplementary literals. For example, we have\nP1,1 ∨P3,1,\n¬P1,1 ∨¬P2,2\nP3,1 ∨¬P2,2\n.\nThere is one more technical aspect of the resolution rule: the resulting clause should contain\nonly one copy of each literal.9 The removal of multiple copies of literals is called factoring.\nFACTORING\nFor example, if we resolve (A ∨B) with (A ∨¬B), we obtain (A ∨A), which is reduced to\njust A.\nThe soundness of the resolution rule can be seen easily by considering the literal ℓi that\nis complementary to literal mj in the other clause. If ℓi is true, then mj is false, and hence\nm1 ∨· · · ∨mj−1 ∨mj+1 ∨· · · ∨mn must be true, because m1 ∨· · · ∨mn is given. If ℓi is\nfalse, then ℓ1 ∨· · · ∨ℓi−1 ∨ℓi+1 ∨· · · ∨ℓk must be true because ℓ1 ∨· · · ∨ℓk is given. Now\nℓi is either true or false, so one or other of these conclusions holds—exactly as the resolution\nrule states.\nWhat is more surprising about the resolution rule is that it forms the basis for a family\nof complete inference procedures. A resolution-based theorem prover can, for any sentences\nα and β in propositional logic, decide whether α |= β. The next two subsections explain\nhow resolution accomplishes this.\nConjunctive normal form\nThe resolution rule applies only to clauses (that is, disjunctions of literals), so it would seem\nto be relevant only to knowledge bases and queries consisting of clauses. How, then, can\nit lead to a complete inference procedure for all of propositional logic? The answer is that\nevery sentence of propositional logic is logically equivalent to a conjunction of clauses. A\nsentence expressed as a conjunction of clauses is said to be in conjunctive normal form or\nCONJUNCTIVE\nNORMAL FORM\nCNF (see Figure 7.14). We now describe a procedure for converting to CNF. We illustrate\nthe procedure by converting the sentence B1,1 ⇔(P1,2 ∨P2,1) into CNF. The steps are as\nfollows:\n1. Eliminate ⇔, replacing α ⇔β with (α ⇒β) ∧(β ⇒α).\n(B1,1 ⇒(P1,2 ∨P2,1)) ∧((P1,2 ∨P2,1) ⇒B1,1) .\n2. Eliminate ⇒, replacing α ⇒β with ¬α ∨β:\n(¬B1,1 ∨P1,2 ∨P2,1) ∧(¬(P1,2 ∨P2,1) ∨B1,1) .\n9 If a clause is viewed as a set of literals, then this restriction is automatically respected. Using set notation for\nclauses makes the resolution rule much cleaner, at the cost of introducing additional notation.\n254\nChapter\n7.\nLogical Agents\n3. CNF requires ¬ to appear only in literals, so we “move ¬ inwards” by repeated appli-\ncation of the following equivalences from Figure 7.11:\n¬(¬α) ≡α (double-negation elimination)\n¬(α ∧β) ≡(¬α ∨¬β) (De Morgan)\n¬(α ∨β) ≡(¬α ∧¬β) (De Morgan)\nIn the example, we require just one application of the last rule:\n(¬B1,1 ∨P1,2 ∨P2,1) ∧((¬P1,2 ∧¬P2,1) ∨B1,1) .\n4. Now we have a sentence containing nested ∧and ∨operators applied to literals. We\napply the distributivity law from Figure 7.11, distributing ∨over ∧wherever possible.\n(¬B1,1 ∨P1,2 ∨P2,1) ∧(¬P1,2 ∨B1,1) ∧(¬P2,1 ∨B1,1) .\nThe original sentence is now in CNF, as a conjunction of three clauses. It is much harder to\nread, but it can be used as input to a resolution procedure.\nA resolution algorithm\nInference procedures based on resolution work by using the principle of proof by contradic-\ntion introduced on page 250. That is, to show that KB |= α, we show that (KB ∧¬α) is\nunsatisfiable. We do this by proving a contradiction.\nA resolution algorithm is shown in Figure 7.12. First, (KB ∧¬α) is converted into\nCNF. Then, the resolution rule is applied to the resulting clauses. Each pair that contains\ncomplementary literals is resolved to produce a new clause, which is added to the set if it is\nnot already present. The process continues until one of two things happens:\n• there are no new clauses that can be added, in which case KB does not entail α; or,\n• two clauses resolve to yield the empty clause, in which case KB entails α.\nThe empty clause—a disjunction of no disjuncts—is equivalent to False because a disjunction\nis true only if at least one of its disjuncts is true. Another way to see that an empty clause\nrepresents a contradiction is to observe that it arises only from resolving two complementary\nunit clauses such as P and ¬P.\nWe can apply the resolution procedure to a very simple inference in the wumpus world.\nWhen the agent is in [1,1], there is no breeze, so there can be no pits in neighboring squares.\nThe relevant knowledge base is\nKB = R2 ∧R4 = (B1,1 ⇔(P1,2 ∨P2,1)) ∧¬B1,1\nand we wish to prove α which is, say, ¬P1,2. When we convert (KB ∧¬α) into CNF, we\nobtain the clauses shown at the top of Figure 7.13. The second row of the figure shows\nclauses obtained by resolving pairs in the first row. Then, when P1,2 is resolved with ¬P1,2,\nwe obtain the empty clause, shown as a small square. Inspection of Figure 7.13 reveals that\nmany resolution steps are pointless. For example, the clause B1,1 ∨¬B1,1 ∨P1,2 is equivalent\nto True ∨P1,2 which is equivalent to True. Deducing that True is true is not very helpful.\nTherefore, any clause in which two complementary literals appear can be discarded.\nSection 7.5.\nPropositional Theorem Proving\n255\nfunction PL-RESOLUTION(KB,α) returns true or false\ninputs: KB, the knowledge base, a sentence in propositional logic\nα, the query, a sentence in propositional logic\nclauses ←the set of clauses in the CNF representation of KB ∧¬α\nnew ←{ }\nloop do\nfor each pair of clauses Ci, Cj in clauses do\nresolvents ←PL-RESOLVE(Ci,Cj)\nif resolvents contains the empty clause then return true\nnew ←new ∪resolvents\nif new ⊆clauses then return false\nclauses ←clauses ∪new\nFigure 7.12\nA simple resolution algorithm for propositional logic.\nThe function\nPL-RESOLVE returns the set of all possible clauses obtained by resolving its two inputs.\n¬P2,1 B1,1\n¬B1,1 P1,2 P2,1\n¬P1,2 B1,1\n¬B1,1\nP1,2\n¬P2,1\n¬P1,2\nP1,2 P2,1 ¬P2,1\n¬B1,1 P2,1 B1,1\nP1,2 P2,1 ¬P1,2\n¬B1,1 P1,2 B1,1\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\nFigure 7.13\nPartial application of PL-RESOLUTION to a simple inference in the wumpus\nworld. ¬P1,2 is shown to follow from the first four clauses in the top row.\nCompleteness of resolution\nTo conclude our discussion of resolution, we now show why PL-RESOLUTION is complete.\nTo do this, we introduce the resolution closure RC (S) of a set of clauses S, which is the set\nRESOLUTION\nCLOSURE\nof all clauses derivable by repeated application of the resolution rule to clauses in S or their\nderivatives. The resolution closure is what PL-RESOLUTION computes as the final value of\nthe variable clauses. It is easy to see that RC (S) must be finite, because there are only finitely\nmany distinct clauses that can be constructed out of the symbols P1, . . . , Pk that appear in S.\n(Notice that this would not be true without the factoring step that removes multiple copies of\nliterals.) Hence, PL-RESOLUTION always terminates.\nThe completeness theorem for resolution in propositional logic is called the ground\nresolution theorem:\nGROUND\nRESOLUTION\nTHEOREM\nIf a set of clauses is unsatisfiable, then the resolution closure of those clauses\ncontains the empty clause.\nThis theorem is proved by demonstrating its contrapositive: if the closure RC(S) does not\n256\nChapter\n7.\nLogical Agents\ncontain the empty clause, then S is satisfiable. In fact, we can construct a model for S with\nsuitable truth values for P1, . . . , Pk. The construction procedure is as follows:\nFor i from 1 to k,\n– If a clause in RC(S) contains the literal ¬Pi and all its other literals are false under\nthe assignment chosen for P1, . . . , Pi−1, then assign false to Pi.\n– Otherwise, assign true to Pi.\nThis assignment to P1, . . . , Pk is a model of S. To see this, assume the opposite—that, at\nsome stage i in the sequence, assigning symbol Pi causes some clause C to become false.\nFor this to happen, it must be the case that all the other literals in C must already have been\nfalsified by assignments to P1, . . . , Pi−1. Thus, C must now look like either (false ∨false ∨\n· · · false∨Pi) or like (false∨false∨· · · false∨¬Pi). If just one of these two is in RC(S), then\nthe algorithm will assign the appropriate truth value to Pi to make C true, so C can only be\nfalsified if both of these clauses are in RC(S). Now, since RC(S) is closed under resolution,\nit will contain the resolvent of these two clauses, and that resolvent will have all of its literals\nalready falsified by the assignments to P1, . . . , Pi−1. This contradicts our assumption that\nthe first falsified clause appears at stage i. Hence, we have proved that the construction never\nfalsifies a clause in RC(S); that is, it produces a model of RC(S) and thus a model of S\nitself (since S is contained in RC(S)).\n7.5.3\nHorn clauses and definite clauses\nThe completeness of resolution makes it a very important inference method. In many practical\nsituations, however, the full power of resolution is not needed. Some real-world knowledge\nbases satisfy certain restrictions on the form of sentences they contain, which enables them\nto use a more restricted and efficient inference algorithm.\nOne such restricted form is the definite clause, which is a disjunction of literals of\nDEFINITE CLAUSE\nwhich exactly one is positive. For example, the clause (¬L1,1 ∨¬Breeze ∨B1,1) is a definite\nclause, whereas (¬B1,1 ∨P1,2 ∨P2,1) is not.\nSlightly more general is the Horn clause, which is a disjunction of literals of which at\nHORN CLAUSE\nmost one is positive. So all definite clauses are Horn clauses, as are clauses with no positive\nliterals; these are called goal clauses. Horn clauses are closed under resolution: if you resolve\nGOAL CLAUSES\ntwo Horn clauses, you get back a Horn clause.\nKnowledge bases containing only definite clauses are interesting for three reasons:\n1. Every definite clause can be written as an implication whose premise is a conjunction\nof positive literals and whose conclusion is a single positive literal. (See Exercise 7.13.)\nFor example, the definite clause (¬L1,1 ∨¬Breeze ∨B1,1) can be written as the im-\nplication (L1,1 ∧Breeze) ⇒B1,1. In the implication form, the sentence is easier to\nunderstand: it says that if the agent is in [1,1] and there is a breeze, then [1,1] is breezy.\nIn Horn form, the premise is called the body and the conclusion is called the head. A\nBODY\nHEAD\nsentence consisting of a single positive literal, such as L1,1, is called a fact. It too can\nFACT\nbe written in implication form as True ⇒L1,1, but it is simpler to write just L1,1.\nSection 7.5.\nPropositional Theorem Proving\n257\nCNFSentence\n→\nClause1 ∧· · · ∧Clausen\nClause\n→\nLiteral 1 ∨· · · ∨Literal m\nLiteral\n→\nSymbol | ¬Symbol\nSymbol\n→\nP | Q | R | . . .\nHornClauseForm\n→\nDefiniteClauseForm | GoalClauseForm\nDefiniteClauseForm\n→\n(Symbol 1 ∧· · · ∧Symbol l) ⇒Symbol\nGoalClauseForm\n→\n(Symbol 1 ∧· · · ∧Symbol l) ⇒False\nFigure 7.14\nA grammar for conjunctive normal form, Horn clauses, and definite clauses.\nA clause such as A ∧B ⇒C is still a definite clause when it is written as ¬A ∨¬B ∨C,\nbut only the former is considered the canonical form for definite clauses. One more class is\nthe k-CNF sentence, which is a CNF sentence where each clause has at most k literals.\n2. Inference with Horn clauses can be done through the forward-chaining and backward-\nFORWARD-CHAINING\nchaining algorithms, which we explain next. Both of these algorithms are natural,\nBACKWARD-\nCHAINING\nin that the inference steps are obvious and easy for humans to follow. This type of\ninference is the basis for logic programming, which is discussed in Chapter 9.\n3. Deciding entailment with Horn clauses can be done in time that is linear in the size of\nthe knowledge base—a pleasant surprise.\n7.5.4\nForward and backward chaining\nThe forward-chaining algorithm PL-FC-ENTAILS?(KB, q) determines if a single proposi-\ntion symbol q—the query—is entailed by a knowledge base of definite clauses. It begins\nfrom known facts (positive literals) in the knowledge base. If all the premises of an implica-\ntion are known, then its conclusion is added to the set of known facts. For example, if L1,1\nand Breeze are known and (L1,1 ∧Breeze) ⇒B1,1 is in the knowledge base, then B1,1 can\nbe added. This process continues until the query q is added or until no further inferences can\nbe made. The detailed algorithm is shown in Figure 7.15; the main point to remember is that\nit runs in linear time.\nThe best way to understand the algorithm is through an example and a picture. Fig-\nure 7.16(a) shows a simple knowledge base of Horn clauses with A and B as known facts.\nFigure 7.16(b) shows the same knowledge base drawn as an AND–OR graph (see Chap-\nter 4). In AND–OR graphs, multiple links joined by an arc indicate a conjunction—every\nlink must be proved—while multiple links without an arc indicate a disjunction—any link\ncan be proved. It is easy to see how forward chaining works in the graph. The known leaves\n(here, A and B) are set, and inference propagates up the graph as far as possible. Wher-\never a conjunction appears, the propagation waits until all the conjuncts are known before\nproceeding. The reader is encouraged to work through the example in detail.\n258\nChapter\n7.\nLogical Agents\nfunction PL-FC-ENTAILS?(KB,q) returns true or false\ninputs: KB, the knowledge base, a set of propositional definite clauses\nq, the query, a proposition symbol\ncount ←a table, where count[c] is the number of symbols in c’s premise\ninferred ←a table, where inferred[s] is initially false for all symbols\nagenda ←a queue of symbols, initially symbols known to be true in KB\nwhile agenda is not empty do\np ←POP(agenda)\nif p = q then return true\nif inferred[p] = false then\ninferred[p] ←true\nfor each clause c in KB where p is in c.PREMISE do\ndecrement count[c]\nif count[c] = 0 then add c.CONCLUSION to agenda\nreturn false\nFigure 7.15\nThe forward-chaining algorithm for propositional logic. The agenda keeps\ntrack of symbols known to be true but not yet “processed.” The count table keeps track of\nhow many premises of each implication are as yet unknown. Whenever a new symbol p from\nthe agenda is processed, the count is reduced by one for each implication in whose premise\np appears (easily identified in constant time with appropriate indexing.) If a count reaches\nzero, all the premises of the implication are known, so its conclusion can be added to the\nagenda. Finally, we need to keep track of which symbols have been processed; a symbol that\nis already in the set of inferred symbols need not be added to the agenda again. This avoids\nredundant work and prevents loops caused by implications such as P ⇒Q and Q ⇒P.\nIt is easy to see that forward chaining is sound: every inference is essentially an appli-\ncation of Modus Ponens. Forward chaining is also complete: every entailed atomic sentence\nwill be derived. The easiest way to see this is to consider the final state of the inferred table\n(after the algorithm reaches a fixed point where no new inferences are possible). The table\nFIXED POINT\ncontains true for each symbol inferred during the process, and false for all other symbols.\nWe can view the table as a logical model; moreover, every definite clause in the original KB is\ntrue in this model. To see this, assume the opposite, namely that some clause a1∧. . .∧ak ⇒b\nis false in the model. Then a1 ∧. . . ∧ak must be true in the model and b must be false in\nthe model. But this contradicts our assumption that the algorithm has reached a fixed point!\nWe can conclude, therefore, that the set of atomic sentences inferred at the fixed point defines\na model of the original KB. Furthermore, any atomic sentence q that is entailed by the KB\nmust be true in all its models and in this model in particular. Hence, every entailed atomic\nsentence q must be inferred by the algorithm.\nForward chaining is an example of the general concept of data-driven reasoning—that\nDATA-DRIVEN\nis, reasoning in which the focus of attention starts with the known data. It can be used within\nan agent to derive conclusions from incoming percepts, often without a specific query in\nmind. For example, the wumpus agent might TELL its percepts to the knowledge base using\nSection 7.6.\nEffective Propositional Model Checking\n259\nP ⇒Q\nL ∧M ⇒P\nB ∧L ⇒M\nA ∧P ⇒L\nA ∧B ⇒L\nA\nB\nQ\nP\nM\nL\nB\nA\n(a)\n(b)\nFigure 7.16\n(a) A set of Horn clauses. (b) The corresponding AND–OR graph.\nan incremental forward-chaining algorithm in which new facts can be added to the agenda to\ninitiate new inferences. In humans, a certain amount of data-driven reasoning occurs as new\ninformation arrives. For example, if I am indoors and hear rain starting to fall, it might occur\nto me that the picnic will be canceled. Yet it will probably not occur to me that the seventeenth\npetal on the largest rose in my neighbor’s garden will get wet; humans keep forward chaining\nunder careful control, lest they be swamped with irrelevant consequences.\nThe backward-chaining algorithm, as its name suggests, works backward from the\nquery. If the query q is known to be true, then no work is needed. Otherwise, the algorithm\nfinds those implications in the knowledge base whose conclusion is q. If all the premises of\none of those implications can be proved true (by backward chaining), then q is true. When\napplied to the query Q in Figure 7.16, it works back down the graph until it reaches a set of\nknown facts, A and B, that forms the basis for a proof. The algorithm is essentially identical\nto the AND-OR-GRAPH-SEARCH algorithm in Figure 4.11. As with forward chaining, an\nefficient implementation runs in linear time.\nBackward chaining is a form of goal-directed reasoning. It is useful for answering\nGOAL-DIRECTED\nREASONING\nspecific questions such as “What shall I do now?” and “Where are my keys?” Often, the cost\nof backward chaining is much less than linear in the size of the knowledge base, because the\nprocess touches only relevant facts.\n7.6\nEFFECTIVE PROPOSITIONAL MODEL CHECKING\nIn this section, we describe two families of efficient algorithms for general propositional\ninference based on model checking: One approach based on backtracking search, and one\non local hill-climbing search. These algorithms are part of the “technology” of propositional\nlogic. This section can be skimmed on a first reading of the chapter.\n260\nChapter\n7.\nLogical Agents\nThe algorithms we describe are for checking satisfiability: the SAT problem. (As noted\nearlier, testing entailment, α |= β, can be done by testing unsatisfiability of α ∧¬β.) We\nhave already noted the connection between finding a satisfying model for a logical sentence\nand finding a solution for a constraint satisfaction problem, so it is perhaps not surprising that\nthe two families of algorithms closely resemble the backtracking algorithms of Section 6.3\nand the local search algorithms of Section 6.4. They are, however, extremely important in\ntheir own right because so many combinatorial problems in computer science can be reduced\nto checking the satisfiability of a propositional sentence. Any improvement in satisfiability\nalgorithms has huge consequences for our ability to handle complexity in general.\n7.6.1\nA complete backtracking algorithm\nThe first algorithm we consider is often called the Davis–Putnam algorithm, after the sem-\nDAVIS–PUTNAM\nALGORITHM\ninal paper by Martin Davis and Hilary Putnam (1960). The algorithm is in fact the version\ndescribed by Davis, Logemann, and Loveland (1962), so we will call it DPLL after the ini-\ntials of all four authors. DPLL takes as input a sentence in conjunctive normal form—a set\nof clauses. Like BACKTRACKING-SEARCH and TT-ENTAILS?, it is essentially a recursive,\ndepth-first enumeration of possible models. It embodies three improvements over the simple\nscheme of TT-ENTAILS?:\n• Early termination: The algorithm detects whether the sentence must be true or false,\neven with a partially completed model. A clause is true if any literal is true, even if\nthe other literals do not yet have truth values; hence, the sentence as a whole could be\njudged true even before the model is complete. For example, the sentence (A ∨B) ∧\n(A ∨C) is true if A is true, regardless of the values of B and C. Similarly, a sentence\nis false if any clause is false, which occurs when each of its literals is false. Again, this\ncan occur long before the model is complete. Early termination avoids examination of\nentire subtrees in the search space.\n• Pure symbol heuristic: A pure symbol is a symbol that always appears with the same\nPURE SYMBOL\n“sign” in all clauses. For example, in the three clauses (A ∨¬B), (¬B ∨¬C), and\n(C ∨A), the symbol A is pure because only the positive literal appears, B is pure\nbecause only the negative literal appears, and C is impure. It is easy to see that if\na sentence has a model, then it has a model with the pure symbols assigned so as to\nmake their literals true, because doing so can never make a clause false. Note that, in\ndetermining the purity of a symbol, the algorithm can ignore clauses that are already\nknown to be true in the model constructed so far. For example, if the model contains\nB = false, then the clause (¬B ∨¬C) is already true, and in the remaining clauses C\nappears only as a positive literal; therefore C becomes pure.\n• Unit clause heuristic: A unit clause was defined earlier as a clause with just one lit-\neral. In the context of DPLL, it also means clauses in which all literals but one are\nalready assigned false by the model. For example, if the model contains B = true,\nthen (¬B ∨¬C) simplifies to ¬C, which is a unit clause. Obviously, for this clause\nto be true, C must be set to false. The unit clause heuristic assigns all such symbols\nbefore branching on the remainder. One important consequence of the heuristic is that\nSection 7.6.\nEffective Propositional Model Checking\n261\nfunction DPLL-SATISFIABLE?(s) returns true or false\ninputs: s, a sentence in propositional logic\nclauses ←the set of clauses in the CNF representation of s\nsymbols ←a list of the proposition symbols in s\nreturn DPLL(clauses,symbols,{ })\nfunction DPLL(clauses,symbols,model) returns true or false\nif every clause in clauses is true in model then return true\nif some clause in clauses is false in model then return false\nP,value ←FIND-PURE-SYMBOL(symbols,clauses,model)\nif P is non-null then return DPLL(clauses,symbols – P,model ∪{P=value})\nP,value ←FIND-UNIT-CLAUSE(clauses,model)\nif P is non-null then return DPLL(clauses,symbols – P,model ∪{P=value})\nP ←FIRST(symbols); rest ←REST(symbols)\nreturn DPLL(clauses,rest,model ∪{P=true}) or\nDPLL(clauses,rest,model ∪{P=false}))\nFigure 7.17\nThe DPLL algorithm for checking satisfiability of a sentence in propositional\nlogic. The ideas behind FIND-PURE-SYMBOL and FIND-UNIT-CLAUSE are described in\nthe text; each returns a symbol (or null) and the truth value to assign to that symbol. Like\nTT-ENTAILS?, DPLL operates over partial models.\nany attempt to prove (by refutation) a literal that is already in the knowledge base will\nsucceed immediately (Exercise 7.23). Notice also that assigning one unit clause can\ncreate another unit clause—for example, when C is set to false, (C ∨A) becomes a\nunit clause, causing true to be assigned to A. This “cascade” of forced assignments\nis called unit propagation. It resembles the process of forward chaining with definite\nUNIT PROPAGATION\nclauses, and indeed, if the CNF expression contains only definite clauses then DPLL\nessentially replicates forward chaining. (See Exercise 7.24.)\nThe DPLL algorithm is shown in Figure 7.17, which gives the the essential skeleton of the\nsearch process.\nWhat Figure 7.17 does not show are the tricks that enable SAT solvers to scale up to\nlarge problems. It is interesting that most of these tricks are in fact rather general, and we\nhave seen them before in other guises:\n1. Component analysis (as seen with Tasmania in CSPs): As DPLL assigns truth values\nto variables, the set of clauses may become separated into disjoint subsets, called com-\nponents, that share no unassigned variables. Given an efficient way to detect when this\noccurs, a solver can gain considerable speed by working on each component separately.\n2. Variable and value ordering (as seen in Section 6.3.1 for CSPs): Our simple imple-\nmentation of DPLL uses an arbitrary variable ordering and always tries the value true\nbefore false. The degree heuristic (see page 216) suggests choosing the variable that\nappears most frequently over all remaining clauses.\n262\nChapter\n7.\nLogical Agents\n3. Intelligent backtracking (as seen in Section 6.3 for CSPs): Many problems that can-\nnot be solved in hours of run time with chronological backtracking can be solved in\nseconds with intelligent backtracking that backs up all the way to the relevant point of\nconflict. All SAT solvers that do intelligent backtracking use some form of conflict\nclause learning to record conflicts so that they won’t be repeated later in the search.\nUsually a limited-size set of conflicts is kept, and rarely used ones are dropped.\n4. Random restarts (as seen on page 124 for hill-climbing): Sometimes a run appears not\nto be making progress. In this case, we can start over from the top of the search tree,\nrather than trying to continue. After restarting, different random choices (in variable\nand value selection) are made. Clauses that are learned in the first run are retained after\nthe restart and can help prune the search space. Restarting does not guarantee that a\nsolution will be found faster, but it does reduce the variance on the time to solution.\n5. Clever indexing (as seen in many algorithms): The speedup methods used in DPLL\nitself, as well as the tricks used in modern solvers, require fast indexing of such things\nas “the set of clauses in which variable Xi appears as a positive literal.” This task is\ncomplicated by the fact that the algorithms are interested only in the clauses that have\nnot yet been satisfied by previous assignments to variables, so the indexing structures\nmust be updated dynamically as the computation proceeds.\nWith these enhancements, modern solvers can handle problems with tens of millions of vari-\nables. They have revolutionized areas such as hardware verification and security protocol\nverification, which previously required laborious, hand-guided proofs.\n7.6.2\nLocal search algorithms\nWe have seen several local search algorithms so far in this book, including HILL-CLIMBING\n(page 122) and SIMULATED-ANNEALING (page 126). These algorithms can be applied di-\nrectly to satisfiability problems, provided that we choose the right evaluation function. Be-\ncause the goal is to find an assignment that satisfies every clause, an evaluation function that\ncounts the number of unsatisfied clauses will do the job. In fact, this is exactly the measure\nused by the MIN-CONFLICTS algorithm for CSPs (page 221). All these algorithms take steps\nin the space of complete assignments, flipping the truth value of one symbol at a time. The\nspace usually contains many local minima, to escape from which various forms of random-\nness are required. In recent years, there has been a great deal of experimentation to find a\ngood balance between greediness and randomness.\nOne of the simplest and most effective algorithms to emerge from all this work is called\nWALKSAT (Figure 7.18). On every iteration, the algorithm picks an unsatisfied clause and\npicks a symbol in the clause to flip. It chooses randomly between two ways to pick which\nsymbol to flip: (1) a “min-conflicts” step that minimizes the number of unsatisfied clauses in\nthe new state and (2) a “random walk” step that picks the symbol randomly.\nWhen WALKSAT returns a model, the input sentence is indeed satisfiable, but when\nit returns failure, there are two possible causes: either the sentence is unsatisfiable or we\nneed to give the algorithm more time. If we set max flips = ∞and p > 0, WALKSAT will\neventually return a model (if one exists), because the random-walk steps will eventually hit\nSection 7.6.\nEffective Propositional Model Checking\n263\nfunction WALKSAT(clauses,p,max flips) returns a satisfying model or failure\ninputs: clauses, a set of clauses in propositional logic\np, the probability of choosing to do a “random walk” move, typically around 0.5\nmax flips, number of flips allowed before giving up\nmodel ←a random assignment of true/false to the symbols in clauses\nfor i = 1 to max flips do\nif model satisfies clauses then return model\nclause ←a randomly selected clause from clauses that is false in model\nwith probability p flip the value in model of a randomly selected symbol from clause\nelse flip whichever symbol in clause maximizes the number of satisfied clauses\nreturn failure\nFigure 7.18\nThe WALKSAT algorithm for checking satisfiability by randomly flipping\nthe values of variables. Many versions of the algorithm exist.\nupon the solution. Alas, if max flips is infinity and the sentence is unsatisfiable, then the\nalgorithm never terminates!\nFor this reason, WALKSAT is most useful when we expect a solution to exist—for ex-\nample, the problems discussed in Chapters 3 and 6 usually have solutions. On the other hand,\nWALKSAT cannot always detect unsatisfiability, which is required for deciding entailment.\nFor example, an agent cannot reliably use WALKSAT to prove that a square is safe in the\nwumpus world. Instead, it can say, “I thought about it for an hour and couldn’t come up with\na possible world in which the square isn’t safe.” This may be a good empirical indicator that\nthe square is safe, but it’s certainly not a proof.\n7.6.3\nThe landscape of random SAT problems\nSome SAT problems are harder than others. Easy problems can be solved by any old algo-\nrithm, but because we know that SAT is NP-complete, at least some problem instances must\nrequire exponential run time. In Chapter 6, we saw some surprising discoveries about certain\nkinds of problems. For example, the n-queens problem—thought to be quite tricky for back-\ntracking search algorithms—turned out to be trivially easy for local search methods, such as\nmin-conflicts. This is because solutions are very densely distributed in the space of assign-\nments, and any initial assignment is guaranteed to have a solution nearby. Thus, n-queens is\neasy because it is underconstrained.\nUNDERCONSTRAINED\nWhen we look at satisfiability problems in conjunctive normal form, an undercon-\nstrained problem is one with relatively few clauses constraining the variables. For example,\nhere is a randomly generated 3-CNF sentence with five symbols and five clauses:\n(¬D ∨¬B ∨C) ∧(B ∨¬A ∨¬C) ∧(¬C ∨¬B ∨E)\n∧(E ∨¬D ∨B) ∧(B ∨E ∨¬C) .\nSixteen of the 32 possible assignments are models of this sentence, so, on average, it would\ntake just two random guesses to find a model. This is an easy satisfiability problem, as are\n264\nChapter\n7.\nLogical Agents\nmost such underconstrained problems. On the other hand, an overconstrained problem has\nmany clauses relative to the number of variables and is likely to have no solutions.\nTo go beyond these basic intuitions, we must define exactly how random sentences\nare generated. The notation CNFk(m, n) denotes a k-CNF sentence with m clauses and n\nsymbols, where the clauses are chosen uniformly, independently, and without replacement\nfrom among all clauses with k different literals, which are positive or negative at random. (A\nsymbol may not appear twice in a clause, nor may a clause appear twice in a sentence.)\nGiven a source of random sentences, we can measure the probability of satisfiability.\nFigure 7.19(a) plots the probability for CNF3(m, 50), that is, sentences with 50 variables\nand 3 literals per clause, as a function of the clause/symbol ratio, m/n. As we expect, for\nsmall m/n the probability of satisfiability is close to 1, and at large m/n the probability\nis close to 0. The probability drops fairly sharply around m/n = 4.3. Empirically, we find\nthat the “cliff” stays in roughly the same place (for k = 3) and gets sharper and sharper as n\nincreases. Theoretically, the satisfiability threshold conjecture says that for every k ≥3,\nSATISFIABILITY\nTHRESHOLD\nCONJECTURE\nthere is a threshold ratio rk such that, as n goes to infinity, the probability that CNFk(n, rn)\nis satisfiable becomes 1 for all values of r below the threshold, and 0 for all values above.\nThe conjecture remains unproven.\n0\n0.2\n0.4\n0.6\n0.8\n1\n0\n1\n2\n3\n4\n5\n6\n7\n8\nP(satisfiable)\nClause/symbol ratio m/n\n0\n200\n400\n600\n800\n1000\n1200\n1400\n1600\n1800\n2000\n0\n1\n2\n3\n4\n5\n6\n7\n8\nRuntime\nClause/symbol ratio m/n\nDPLL\nWalkSAT\n(a)\n(b)\nFigure 7.19\n(a) Graph showing the probability that a random 3-CNF sentence with n = 50\nsymbols is satisfiable, as a function of the clause/symbol ratio m/n. (b) Graph of the median\nrun time (measured in number of recursive calls to DPLL, a good proxy) on random 3-CNF\nsentences. The most difficult problems have a clause/symbol ratio of about 4.3.\nNow that we have a good idea where the satisfiable and unsatisfiable problems are, the\nnext question is, where are the hard problems? It turns out that they are also often at the\nthreshold value. Figure 7.19(b) shows that 50-symbol problems at the threshold value of 4.3\nare about 20 times more difficult to solve than those at a ratio of 3.3. The underconstrained\nproblems are easiest to solve (because it is so easy to guess a solution); the overconstrained\nproblems are not as easy as the underconstrained, but still are much easier than the ones right\nat the threshold.\nSection 7.7.\nAgents Based on Propositional Logic\n265\n7.7\nAGENTS BASED ON PROPOSITIONAL LOGIC\nIn this section, we bring together what we have learned so far in order to construct wumpus\nworld agents that use propositional logic. The first step is to enable the agent to deduce, to the\nextent possible, the state of the world given its percept history. This requires writing down a\ncomplete logical model of the effects of actions. We also show how the agent can keep track of\nthe world efficiently without going back into the percept history for each inference. Finally,\nwe show how the agent can use logical inference to construct plans that are guaranteed to\nachieve its goals.\n7.7.1\nThe current state of the world\nAs stated at the beginning of the chapter, a logical agent operates by deducing what to do\nfrom a knowledge base of sentences about the world. The knowledge base is composed of\naxioms—general knowledge about how the world works—and percept sentences obtained\nfrom the agent’s experience in a particular world. In this section, we focus on the problem of\ndeducing the current state of the wumpus world—where am I, is that square safe, and so on.\nWe began collecting axioms in Section 7.4.3. The agent knows that the starting square\ncontains no pit (¬P1,1) and no wumpus (¬W1,1). Furthermore, for each square, it knows that\nthe square is breezy if and only if a neighboring square has a pit; and a square is smelly if and\nonly if a neighboring square has a wumpus. Thus, we include a large collection of sentences\nof the following form:\nB1,1 ⇔(P1,2 ∨P2,1)\nS1,1 ⇔(W1,2 ∨W2,1)\n· · ·\nThe agent also knows that there is exactly one wumpus. This is expressed in two parts. First,\nwe have to say that there is at least one wumpus:\nW1,1 ∨W1,2 ∨· · · ∨W4,3 ∨W4,4 .\nThen, we have to say that there is at most one wumpus. For each pair of locations, we add a\nsentence saying that at least one of them must be wumpus-free:\n¬W1,1 ∨¬W1,2\n¬W1,1 ∨¬W1,3\n· · ·\n¬W4,3 ∨¬W4,4 .\nSo far, so good. Now let’s consider the agent’s percepts. If there is currently a stench, one\nmight suppose that a proposition Stench should be added to the knowledge base. This is not\nquite right, however: if there was no stench at the previous time step, then ¬Stench would al-\nready be asserted, and the new assertion would simply result in a contradiction. The problem\nis solved when we realize that a percept asserts something only about the current time. Thus,\nif the time step (as supplied to MAKE-PERCEPT-SENTENCE in Figure 7.1) is 4, then we add\n266\nChapter\n7.\nLogical Agents\nStench4 to the knowledge base, rather than Stench—neatly avoiding any contradiction with\n¬Stench3. The same goes for the breeze, bump, glitter, and scream percepts.\nThe idea of associating propositions with time steps extends to any aspect of the world\nthat changes over time. For example, the initial knowledge base includes L0\n1,1—the agent is in\nsquare [1, 1] at time 0—as well as FacingEast0, HaveArrow 0, and WumpusAlive0. We use\nthe word fluent (from the Latin fluens, flowing) to refer an aspect of the world that changes.\nFLUENT\n“Fluent” is a synonym for “state variable,” in the sense described in the discussion of factored\nrepresentations in Section 2.4.7 on page 57. Symbols associated with permanent aspects of\nthe world do not need a time superscript and are sometimes called atemporal variables.\nATEMPORAL\nVARIABLE\nWe can connect stench and breeze percepts directly to the properties of the squares\nwhere they are experienced through the location fluent as follows.10 For any time step t and\nany square [x, y], we assert\nLt\nx,y ⇒(Breezet ⇔Bx,y)\nLt\nx,y ⇒(Stencht ⇔Sx,y) .\nNow, of course, we need axioms that allow the agent to keep track of fluents such as Lt\nx,y.\nThese fluents change as the result of actions taken by the agent, so, in the terminology of\nChapter 3, we need to write down the transition model of the wumpus world as a set of\nlogical sentences.\nFirst, we need proposition symbols for the occurrences of actions. As with percepts,\nthese symbols are indexed by time; thus, Forward 0 means that the agent executes the Forward\naction at time 0. By convention, the percept for a given time step happens first, followed by\nthe action for that time step, followed by a transition to the next time step.\nTo describe how the world changes, we can try writing effect axioms that specify the\nEFFECT AXIOM\noutcome of an action at the next time step. For example, if the agent is at location [1, 1] facing\neast at time 0 and goes Forward, the result is that the agent is in square [2, 1] and no longer\nis in [1, 1]:\nL0\n1,1 ∧FacingEast0 ∧Forward 0 ⇒(L1\n2,1 ∧¬L1\n1,1) .\n(7.1)\nWe would need one such sentence for each possible time step, for each of the 16 squares,\nand each of the four orientations. We would also need similar sentences for the other actions:\nGrab, Shoot, Climb, TurnLeft, and TurnRight.\nLet us suppose that the agent does decide to move Forward at time 0 and asserts this\nfact into its knowledge base. Given the effect axiom in Equation (7.1), combined with the\ninitial assertions about the state at time 0, the agent can now deduce that it is in [2, 1]. That\nis, ASK(KB, L1\n2,1) = true. So far, so good. Unfortunately, the news elsewhere is less good:\nif we ASK(KB, HaveArrow 1), the answer is false, that is, the agent cannot prove it still\nhas the arrow; nor can it prove it doesn’t have it! The information has been lost because the\neffect axiom fails to state what remains unchanged as the result of an action. The need to do\nthis gives rise to the frame problem.11 One possible solution to the frame problem would\nFRAME PROBLEM\n10 Section 7.4.3 conveniently glossed over this requirement.\n11 The name “frame problem” comes from “frame of reference” in physics—the assumed stationary background\nwith respect to which motion is measured. It also has an analogy to the frames of a movie, in which normally\nmost of the background stays constant while changes occur in the foreground.\nSection 7.7.\nAgents Based on Propositional Logic\n267\nbe to add frame axioms explicitly asserting all the propositions that remain the same. For\nFRAME AXIOM\nexample, for each time t we would have\nForward t ⇒(HaveArrow t ⇔HaveArrow t+1)\nForward t ⇒(WumpusAlivet ⇔WumpusAlivet+1)\n· · ·\nwhere we explicitly mention every proposition that stays unchanged from time t to time\nt + 1 under the action Forward. Although the agent now knows that it still has the arrow\nafter moving forward and that the wumpus hasn’t died or come back to life, the proliferation\nof frame axioms seems remarkably inefficient. In a world with m different actions and n\nfluents, the set of frame axioms will be of size O(mn). This specific manifestation of the\nframe problem is sometimes called the representational frame problem. Historically, the\nREPRESENTATIONAL\nFRAME PROBLEM\nproblem was a significant one for AI researchers; we explore it further in the notes at the end\nof the chapter.\nThe representational frame problem is significant because the real world has very many\nfluents, to put it mildly. Fortunately for us humans, each action typically changes no more\nthan some small number k of those fluents—the world exhibits locality. Solving the repre-\nLOCALITY\nsentational frame problem requires defining the transition model with a set of axioms of size\nO(mk) rather than size O(mn). There is also an inferential frame problem: the problem\nINFERENTIAL FRAME\nPROBLEM\nof projecting forward the results of a t step plan of action in time O(kt) rather than O(nt).\nThe solution to the problem involves changing one’s focus from writing axioms about\nactions to writing axioms about fluents. Thus, for each fluent F, we will have an axiom that\ndefines the truth value of F t+1 in terms of fluents (including F itself) at time t and the actions\nthat may have occurred at time t. Now, the truth value of F t+1 can be set in one of two ways:\neither the action at time t causes F to be true at t + 1, or F was already true at time t and the\naction at time t does not cause it to be false. An axiom of this form is called a successor-state\naxiom and has this schema:\nSUCCESSOR-STATE\nAXIOM\nF t+1 ⇔ActionCausesF t ∨(F t ∧¬ActionCausesNotF t) .\nOne of the simplest successor-state axioms is the one for HaveArrow. Because there is no\naction for reloading, the ActionCausesF t part goes away and we are left with\nHaveArrow t+1 ⇔(HaveArrow t ∧¬Shoott) .\n(7.2)\nFor the agent’s location, the successor-state axioms are more elaborate. For example, Lt+1\n1,1\nis true if either (a) the agent moved Forward from [1, 2] when facing south, or from [2, 1]\nwhen facing west; or (b) Lt\n1,1 was already true and the action did not cause movement (either\nbecause the action was not Forward or because the action bumped into a wall). Written out\nin propositional logic, this becomes\nLt+1\n1,1\n⇔\n(Lt\n1,1 ∧(¬Forward t ∨Bumpt+1))\n∨(Lt\n1,2 ∧(Southt ∧Forward t))\n(7.3)\n∨(Lt\n2,1 ∧(Westt ∧Forward t)) .\nExercise 7.26 asks you to write out axioms for the remaining wumpus world fluents.\n268\nChapter\n7.\nLogical Agents\nGiven a complete set of successor-state axioms and the other axioms listed at the begin-\nning of this section, the agent will be able to ASK and answer any answerable question about\nthe current state of the world. For example, in Section 7.2 the initial sequence of percepts and\nactions is\n¬Stench0 ∧¬Breeze0 ∧¬Glitter 0 ∧¬Bump0 ∧¬Scream0 ; Forward 0\n¬Stench1 ∧Breeze1 ∧¬Glitter 1 ∧¬Bump1 ∧¬Scream1 ; TurnRight1\n¬Stench2 ∧Breeze2 ∧¬Glitter 2 ∧¬Bump2 ∧¬Scream2 ; TurnRight2\n¬Stench3 ∧Breeze3 ∧¬Glitter 3 ∧¬Bump3 ∧¬Scream3 ; Forward 3\n¬Stench4 ∧¬Breeze4 ∧¬Glitter 4 ∧¬Bump4 ∧¬Scream4 ; TurnRight4\n¬Stench5 ∧¬Breeze5 ∧¬Glitter 5 ∧¬Bump5 ∧¬Scream5 ; Forward 5\nStench6 ∧¬Breeze6 ∧¬Glitter 6 ∧¬Bump6 ∧¬Scream6\nAt this point, we have ASK(KB, L6\n1,2) = true, so the agent knows where it is. Moreover,\nASK(KB, W1,3) = true and ASK(KB, P3,1) = true, so the agent has found the wumpus and\none of the pits. The most important question for the agent is whether a square is OK to move\ninto, that is, the square contains no pit nor live wumpus. It’s convenient to add axioms for\nthis, having the form\nOK t\nx,y ⇔¬Px,y ∧¬(Wx,y ∧WumpusAlivet) .\nFinally, ASK(KB, OK 6\n2,2) = true, so the square [2, 2] is OK to move into. In fact, given a\nsound and complete inference algorithm such as DPLL, the agent can answer any answerable\nquestion about which squares are OK—and can do so in just a few milliseconds for small-to-\nmedium wumpus worlds.\nSolving the representational and inferential frame problems is a big step forward, but\na pernicious problem remains: we need to confirm that all the necessary preconditions of an\naction hold for it to have its intended effect. We said that the Forward action moves the agent\nahead unless there is a wall in the way, but there are many other unusual exceptions that could\ncause the action to fail: the agent might trip and fall, be stricken with a heart attack, be carried\naway by giant bats, etc. Specifying all these exceptions is called the qualification problem.\nQUALIFICATION\nPROBLEM\nThere is no complete solution within logic; system designers have to use good judgment in\ndeciding how detailed they want to be in specifying their model, and what details they want\nto leave out. We will see in Chapter 13 that probability theory allows us to summarize all the\nexceptions without explicitly naming them.\n7.7.2\nA hybrid agent\nThe ability to deduce various aspects of the state of the world can be combined fairly straight-\nforwardly with condition–action rules and with problem-solving algorithms from Chapters 3\nand 4 to produce a hybrid agent for the wumpus world. Figure 7.20 shows one possible way\nHYBRID AGENT\nto do this. The agent program maintains and updates a knowledge base as well as a current\nplan. The initial knowledge base contains the atemporal axioms—those that don’t depend\non t, such as the axiom relating the breeziness of squares to the presence of pits. At each\ntime step, the new percept sentence is added along with all the axioms that depend on t, such\nSection 7.7.\nAgents Based on Propositional Logic\n269\nas the successor-state axioms. (The next section explains why the agent doesn’t need axioms\nfor future time steps.) Then, the agent uses logical inference, by ASKing questions of the\nknowledge base, to work out which squares are safe and which have yet to be visited.\nThe main body of the agent program constructs a plan based on a decreasing priority of\ngoals. First, if there is a glitter, the program constructs a plan to grab the gold, follow a route\nback to the initial location, and climb out of the cave. Otherwise, if there is no current plan,\nthe program plans a route to the closest safe square that it has not visited yet, making sure\nthe route goes through only safe squares. Route planning is done with A∗search, not with\nASK. If there are no safe squares to explore, the next step—if the agent still has an arrow—is\nto try to make a safe square by shooting at one of the possible wumpus locations. These are\ndetermined by asking where ASK(KB, ¬Wx,y) is false—that is, where it is not known that\nthere is not a wumpus. The function PLAN-SHOT (not shown) uses PLAN-ROUTE to plan a\nsequence of actions that will line up this shot. If this fails, the program looks for a square to\nexplore that is not provably unsafe—that is, a square for which ASK(KB, ¬OK t\nx,y) returns\nfalse. If there is no such square, then the mission is impossible and the agent retreats to [1, 1]\nand climbs out of the cave.\n7.7.3\nLogical state estimation\nThe agent program in Figure 7.20 works quite well, but it has one major weakness: as time\ngoes by, the computational expense involved in the calls to ASK goes up and up. This happens\nmainly because the required inferences have to go back further and further in time and involve\nmore and more proposition symbols. Obviously, this is unsustainable—we cannot have an\nagent whose time to process each percept grows in proportion to the length of its life! What\nwe really need is a constant update time—that is, independent of t. The obvious answer is to\nsave, or cache, the results of inference, so that the inference process at the next time step can\nCACHING\nbuild on the results of earlier steps instead of having to start again from scratch.\nAs we saw in Section 4.4, the past history of percepts and all their ramifications can\nbe replaced by the belief state—that is, some representation of the set of all possible current\nstates of the world.12 The process of updating the belief state as new percepts arrive is called\nstate estimation. Whereas in Section 4.4 the belief state was an explicit list of states, here\nwe can use a logical sentence involving the proposition symbols associated with the current\ntime step, as well as the atemporal symbols. For example, the logical sentence\nWumpusAlive1 ∧L1\n2,1 ∧B2,1 ∧(P3,1 ∨P2,2)\n(7.4)\nrepresents the set of all states at time 1 in which the wumpus is alive, the agent is at [2, 1],\nthat square is breezy, and there is a pit in [3, 1] or [2, 2] or both.\nMaintaining an exact belief state as a logical formula turns out not to be easy. If there\nare n fluent symbols for time t, then there are 2n possible states—that is, assignments of truth\nvalues to those symbols. Now, the set of belief states is the powerset (set of all subsets) of the\nset of physical states. There are 2n physical states, hence 22n belief states. Even if we used\nthe most compact possible encoding of logical formulas, with each belief state represented\n12 We can think of the percept history itself as a representation of the belief state, but one that makes inference\nincreasingly expensive as the history gets longer.\n270\nChapter\n7.\nLogical Agents\nfunction HYBRID-WUMPUS-AGENT(percept) returns an action\ninputs: percept, a list, [stench,breeze,glitter,bump,scream]\npersistent: KB, a knowledge base, initially the atemporal “wumpus physics”\nt, a counter, initially 0, indicating time\nplan, an action sequence, initially empty\nTELL(KB, MAKE-PERCEPT-SENTENCE(percept,t))\nTELL the KB the temporal “physics” sentences for time t\nsafe ←{[x, y] : ASK(KB,OK t\nx,y) = true}\nif ASK(KB,Glitter t) = true then\nplan ←[Grab] + PLAN-ROUTE(current,{[1,1]},safe) + [Climb]\nif plan is empty then\nunvisited ←{[x, y] : ASK(KB, Lt′\nx,y) = false for all t′ ≤t}\nplan ←PLAN-ROUTE(current,unvisited ∩safe,safe)\nif plan is empty and ASK(KB,HaveArrow t) = true then\npossible wumpus ←{[x, y] : ASK(KB, ¬ Wx,y) = false}\nplan ←PLAN-SHOT(current,possible wumpus,safe)\nif plan is empty then // no choice but to take a risk\nnot unsafe ←{[x, y] : ASK(KB, ¬ OK t\nx,y) = false}\nplan ←PLAN-ROUTE(current,unvisited ∩not unsafe,safe)\nif plan is empty then\nplan ←PLAN-ROUTE(current,{[1, 1]},safe) + [Climb]\naction ←POP(plan)\nTELL(KB, MAKE-ACTION-SENTENCE(action,t))\nt ←t + 1\nreturn action\nfunction PLAN-ROUTE(current,goals,allowed) returns an action sequence\ninputs: current, the agent’s current position\ngoals, a set of squares; try to plan a route to one of them\nallowed, a set of squares that can form part of the route\nproblem ←ROUTE-PROBLEM(current,goals,allowed)\nreturn A*-GRAPH-SEARCH(problem)\nFigure 7.20\nA hybrid agent program for the wumpus world. It uses a propositional knowl-\nedge base to infer the state of the world, and a combination of problem-solving search and\ndomain-specific code to decide what actions to take.\nby a unique binary number, we would need numbers with log2(22n) = 2n bits to label the\ncurrent belief state. That is, exact state estimation may require logical formulas whose size is\nexponential in the number of symbols.\nOne very common and natural scheme for approximate state estimation is to represent\nbelief states as conjunctions of literals, that is, 1-CNF formulas. To do this, the agent program\nsimply tries to prove Xt and ¬Xt for each symbol Xt (as well as each atemporal symbol\nwhose truth value is not yet known), given the belief state at t −1. The conjunction of\nSection 7.7.\nAgents Based on Propositional Logic\n271\nFigure 7.21\nDepiction of a 1-CNF belief state (bold outline) as a simply representable,\nconservative approximation to the exact (wiggly) belief state (shaded region with dashed\noutline). Each possible world is shown as a circle; the shaded ones are consistent with all the\npercepts.\nprovable literals becomes the new belief state, and the previous belief state is discarded.\nIt is important to understand that this scheme may lose some information as time goes\nalong. For example, if the sentence in Equation (7.4) were the true belief state, then neither\nP3,1 nor P2,2 would be provable individually and neither would appear in the 1-CNF belief\nstate. (Exercise 7.27 explores one possible solution to this problem.) On the other hand,\nbecause every literal in the 1-CNF belief state is proved from the previous belief state, and\nthe initial belief state is a true assertion, we know that entire 1-CNF belief state must be\ntrue. Thus, the set of possible states represented by the 1-CNF belief state includes all states\nthat are in fact possible given the full percept history. As illustrated in Figure 7.21, the 1-\nCNF belief state acts as a simple outer envelope, or conservative approximation, around the\nCONSERVATIVE\nAPPROXIMATION\nexact belief state. We see this idea of conservative approximations to complicated sets as a\nrecurring theme in many areas of AI.\n7.7.4\nMaking plans by propositional inference\nThe agent in Figure 7.20 uses logical inference to determine which squares are safe, but uses\nA∗search to make plans. In this section, we show how to make plans by logical inference.\nThe basic idea is very simple:\n1. Construct a sentence that includes\n(a) Init0, a collection of assertions about the initial state;\n(b) Transition1, . . . , Transitiont, the successor-state axioms for all possible actions\nat each time up to some maximum time t;\n(c) the assertion that the goal is achieved at time t: HaveGoldt ∧ClimbedOutt.\n272\nChapter\n7.\nLogical Agents\n2. Present the whole sentence to a SAT solver. If the solver finds a satisfying model, then\nthe goal is achievable; if the sentence is unsatisfiable, then the planning problem is\nimpossible.\n3. Assuming a model is found, extract from the model those variables that represent ac-\ntions and are assigned true. Together they represent a plan to achieve the goals.\nA propositional planning procedure, SATPLAN, is shown in Figure 7.22. It implements the\nbasic idea just given, with one twist. Because the agent does not know how many steps it\nwill take to reach the goal, the algorithm tries each possible number of steps t, up to some\nmaximum conceivable plan length Tmax. In this way, it is guaranteed to find the shortest plan\nif one exists. Because of the way SATPLAN searches for a solution, this approach cannot\nbe used in a partially observable environment; SATPLAN would just set the unobservable\nvariables to the values it needs to create a solution.\nfunction SATPLAN(init, transition, goal,T max) returns solution or failure\ninputs: init, transition, goal, constitute a description of the problem\nT max, an upper limit for plan length\nfor t = 0 to T max do\ncnf ←TRANSLATE-TO-SAT(init, transition, goal,t)\nmodel ←SAT-SOLVER(cnf )\nif model is not null then\nreturn EXTRACT-SOLUTION(model)\nreturn failure\nFigure 7.22\nThe SATPLAN algorithm. The planning problem is translated into a CNF\nsentence in which the goal is asserted to hold at a fixed time step t and axioms are included\nfor each time step up to t. If the satisfiability algorithm finds a model, then a plan is extracted\nby looking at those proposition symbols that refer to actions and are assigned true in the\nmodel. If no model exists, then the process is repeated with the goal moved one step later.\nThe key step in using SATPLAN is the construction of the knowledge base. It might\nseem, on casual inspection, that the wumpus world axioms in Section 7.7.1 suffice for steps\n1(a) and 1(b) above. There is, however, a significant difference between the requirements for\nentailment (as tested by ASK) and those for satisfiability. Consider, for example, the agent’s\nlocation, initially [1, 1], and suppose the agent’s unambitious goal is to be in [2, 1] at time 1.\nThe initial knowledge base contains L0\n1,1 and the goal is L1\n2,1. Using ASK, we can prove L1\n2,1\nif Forward0 is asserted, and, reassuringly, we cannot prove L1\n2,1 if, say, Shoot0 is asserted\ninstead. Now, SATPLAN will find the plan [Forward0]; so far, so good. Unfortunately,\nSATPLAN also finds the plan [Shoot0]. How could this be? To find out, we inspect the model\nthat SATPLAN constructs: it includes the assignment L0\n2,1, that is, the agent can be in [2, 1]\nat time 1 by being there at time 0 and shooting. One might ask, “Didn’t we say the agent is in\n[1, 1] at time 0?” Yes, we did, but we didn’t tell the agent that it can’t be in two places at once!\nFor entailment, L0\n2,1 is unknown and cannot, therefore, be used in a proof; for satisfiability,\nSection 7.7.\nAgents Based on Propositional Logic\n273\non the other hand, L0\n2,1 is unknown and can, therefore, be set to whatever value helps to\nmake the goal true. For this reason, SATPLAN is a good debugging tool for knowledge bases\nbecause it reveals places where knowledge is missing. In this particular case, we can fix the\nknowledge base by asserting that, at each time step, the agent is in exactly one location, using\na collection of sentences similar to those used to assert the existence of exactly one wumpus.\nAlternatively, we can assert ¬L0\nx,y for all locations other than [1, 1]; the successor-state axiom\nfor location takes care of subsequent time steps. The same fixes also work to make sure the\nagent has only one orientation.\nSATPLAN has more surprises in store, however. The first is that it finds models with\nimpossible actions, such as shooting with no arrow. To understand why, we need to look more\ncarefully at what the successor-state axioms (such as Equation (7.3)) say about actions whose\npreconditions are not satisfied. The axioms do predict correctly that nothing will happen when\nsuch an action is executed (see Exercise 10.14), but they do not say that the action cannot be\nexecuted! To avoid generating plans with illegal actions, we must add precondition axioms\nPRECONDITION\nAXIOMS\nstating that an action occurrence requires the preconditions to be satisfied.13 For example, we\nneed to say, for each time t, that\nShoot t ⇒HaveArrow t .\nThis ensures that if a plan selects the Shoot action at any time, it must be the case that the\nagent has an arrow at that time.\nSATPLAN’s second surprise is the creation of plans with multiple simultaneous actions.\nFor example, it may come up with a model in which both Forward 0 and Shoot0 are true,\nwhich is not allowed. To eliminate this problem, we introduce action exclusion axioms: for\nACTION EXCLUSION\nAXIOM\nevery pair of actions At\ni and At\nj we add the axiom\n¬At\ni ∨¬At\nj .\nIt might be pointed out that walking forward and shooting at the same time is not so hard to\ndo, whereas, say, shooting and grabbing at the same time is rather impractical. By imposing\naction exclusion axioms only on pairs of actions that really do interfere with each other, we\ncan allow for plans that include multiple simultaneous actions—and because SATPLAN finds\nthe shortest legal plan, we can be sure that it will take advantage of this capability.\nTo summarize, SATPLAN finds models for a sentence containing the initial state, the\ngoal, the successor-state axioms, the precondition axioms, and the action exclusion axioms.\nIt can be shown that this collection of axioms is sufficient, in the sense that there are no\nlonger any spurious “solutions.” Any model satisfying the propositional sentence will be a\nvalid plan for the original problem. Modern SAT-solving technology makes the approach\nquite practical. For example, a DPLL-style solver has no difficulty in generating the 11-step\nsolution for the wumpus world instance shown in Figure 7.2.\nThis section has described a declarative approach to agent construction: the agent works\nby a combination of asserting sentences in the knowledge base and performing logical infer-\nence. This approach has some weaknesses hidden in phrases such as “for each time t” and\n13 Notice that the addition of precondition axioms means that we need not include preconditions for actions in\nthe successor-state axioms.\n274\nChapter\n7.\nLogical Agents\n“for each square [x, y].” For any practical agent, these phrases have to be implemented by\ncode that generates instances of the general sentence schema automatically for insertion into\nthe knowledge base. For a wumpus world of reasonable size—one comparable to a smallish\ncomputer game—we might need a 100 × 100 board and 1000 time steps, leading to knowl-\nedge bases with tens or hundreds of millions of sentences. Not only does this become rather\nimpractical, but it also illustrates a deeper problem: we know something about the wum-\npus world—namely, that the “physics” works the same way across all squares and all time\nsteps—that we cannot express directly in the language of propositional logic. To solve this\nproblem, we need a more expressive language, one in which phrases like “for each time t”\nand “for each square [x, y]” can be written in a natural way. First-order logic, described in\nChapter 8, is such a language; in first-order logic a wumpus world of any size and duration\ncan be described in about ten sentences rather than ten million or ten trillion.\n7.8\nSUMMARY\nWe have introduced knowledge-based agents and have shown how to define a logic with\nwhich such agents can reason about the world. The main points are as follows:\n• Intelligent agents need knowledge about the world in order to reach good decisions.\n• Knowledge is contained in agents in the form of sentences in a knowledge represen-\ntation language that are stored in a knowledge base.\n• A knowledge-based agent is composed of a knowledge base and an inference mecha-\nnism. It operates by storing sentences about the world in its knowledge base, using the\ninference mechanism to infer new sentences, and using these sentences to decide what\naction to take.\n• A representation language is defined by its syntax, which specifies the structure of\nsentences, and its semantics, which defines the truth of each sentence in each possible\nworld or model.\n• The relationship of entailment between sentences is crucial to our understanding of\nreasoning. A sentence α entails another sentence β if β is true in all worlds where\nα is true. Equivalent definitions include the validity of the sentence α ⇒β and the\nunsatisfiability of the sentence α ∧¬β.\n• Inference is the process of deriving new sentences from old ones. Sound inference algo-\nrithms derive only sentences that are entailed; complete algorithms derive all sentences\nthat are entailed.\n• Propositional logic is a simple language consisting of proposition symbols and logical\nconnectives. It can handle propositions that are known true, known false, or completely\nunknown.\n• The set of possible models, given a fixed propositional vocabulary, is finite, so en-\ntailment can be checked by enumerating models. Efficient model-checking inference\nalgorithms for propositional logic include backtracking and local search methods and\ncan often solve large problems quickly.\nBibliographical and Historical Notes\n275\n• Inference rules are patterns of sound inference that can be used to find proofs. The\nresolution rule yields a complete inference algorithm for knowledge bases that are\nexpressed in conjunctive normal form. Forward chaining and backward chaining\nare very natural reasoning algorithms for knowledge bases in Horn form.\n• Local search methods such as WALKSAT can be used to find solutions. Such algo-\nrithms are sound but not complete.\n• Logical state estimation involves maintaining a logical sentence that describes the set\nof possible states consistent with the observation history. Each update step requires\ninference using the transition model of the environment, which is built from successor-\nstate axioms that specify how each fluent changes.\n• Decisions within a logical agent can be made by SAT solving: finding possible models\nspecifying future action sequences that reach the goal. This approach works only for\nfully observable or sensorless environments.\n• Propositional logic does not scale to environments of unbounded size because it lacks\nthe expressive power to deal concisely with time, space, and universal patterns of rela-\ntionships among objects.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nJohn McCarthy’s paper “Programs with Common Sense” (McCarthy, 1958, 1968) promul-\ngated the notion of agents that use logical reasoning to mediate between percepts and actions.\nIt also raised the flag of declarativism, pointing out that telling an agent what it needs to know\nis an elegant way to build software. Allen Newell’s (1982) article “The Knowledge Level”\nmakes the case that rational agents can be described and analyzed at an abstract level defined\nby the knowledge they possess rather than the programs they run. The declarative and proce-\ndural approaches to AI are analyzed in depth by Boden (1977). The debate was revived by,\namong others, Brooks (1991) and Nilsson (1991), and continues to this day (Shaparau et al.,\n2008). Meanwhile, the declarative approach has spread into other areas of computer science\nsuch as networking (Loo et al., 2006).\nLogic itself had its origins in ancient Greek philosophy and mathematics. Various log-\nical principles—principles connecting the syntactic structure of sentences with their truth\nand falsity, with their meaning, or with the validity of arguments in which they figure—are\nscattered in the works of Plato. The first known systematic study of logic was carried out\nby Aristotle, whose work was assembled by his students after his death in 322 B.C. as a\ntreatise called the Organon. Aristotle’s syllogisms were what we would now call inference\nSYLLOGISM\nrules. Although the syllogisms included elements of both propositional and first-order logic,\nthe system as a whole lacked the compositional properties required to handle sentences of\narbitrary complexity.\nThe closely related Megarian and Stoic schools (originating in the fifth century B.C.\nand continuing for several centuries thereafter) began the systematic study of the basic logical\nconnectives. The use of truth tables for defining connectives is due to Philo of Megara. The\n276\nChapter\n7.\nLogical Agents\nStoics took five basic inference rules as valid without proof, including the rule we now call\nModus Ponens. They derived a number of other rules from these five, using, among other\nprinciples, the deduction theorem (page 249) and were much clearer about the notion of\nproof than was Aristotle. A good account of the history of Megarian and Stoic logic is given\nby Benson Mates (1953).\nThe idea of reducing logical inference to a purely mechanical process applied to a for-\nmal language is due to Wilhelm Leibniz (1646–1716), although he had limited success in im-\nplementing the ideas. George Boole (1847) introduced the first comprehensive and workable\nsystem of formal logic in his book The Mathematical Analysis of Logic. Boole’s logic was\nclosely modeled on the ordinary algebra of real numbers and used substitution of logically\nequivalent expressions as its primary inference method. Although Boole’s system still fell\nshort of full propositional logic, it was close enough that other mathematicians could quickly\nfill in the gaps. Schr¨oder (1877) described conjunctive normal form, while Horn form was\nintroduced much later by Alfred Horn (1951). The first comprehensive exposition of modern\npropositional logic (and first-order logic) is found in Gottlob Frege’s (1879) Begriffschrift\n(“Concept Writing” or “Conceptual Notation”).\nThe first mechanical device to carry out logical inferences was constructed by the third\nEarl of Stanhope (1753–1816). The Stanhope Demonstrator could handle syllogisms and\ncertain inferences in the theory of probability. William Stanley Jevons, one of those who\nimproved upon and extended Boole’s work, constructed his “logical piano” in 1869 to per-\nform inferences in Boolean logic. An entertaining and instructive history of these and other\nearly mechanical devices for reasoning is given by Martin Gardner (1968). The first pub-\nlished computer program for logical inference was the Logic Theorist of Newell, Shaw,\nand Simon (1957). This program was intended to model human thought processes. Mar-\ntin Davis (1957) had actually designed a program that came up with a proof in 1954, but the\nLogic Theorist’s results were published slightly earlier.\nTruth tables as a method of testing validity or unsatisfiability in propositional logic were\nintroduced independently by Emil Post (1921) and Ludwig Wittgenstein (1922). In the 1930s,\na great deal of progress was made on inference methods for first-order logic. In particular,\nG¨odel (1930) showed that a complete procedure for inference in first-order logic could be\nobtained via a reduction to propositional logic, using Herbrand’s theorem (Herbrand, 1930).\nWe take up this history again in Chapter 9; the important point here is that the development\nof efficient propositional algorithms in the 1960s was motivated largely by the interest of\nmathematicians in an effective theorem prover for first-order logic. The Davis–Putnam algo-\nrithm (Davis and Putnam, 1960) was the first effective algorithm for propositional resolution\nbut was in most cases much less efficient than the DPLL backtracking algorithm introduced\ntwo years later (1962). The full resolution rule and a proof of its completeness appeared in a\nseminal paper by J. A. Robinson (1965), which also showed how to do first-order reasoning\nwithout resort to propositional techniques.\nStephen Cook (1971) showed that deciding satisfiability of a sentence in propositional\nlogic (the SAT problem) is NP-complete. Since deciding entailment is equivalent to decid-\ning unsatisfiability, it is co-NP-complete. Many subsets of propositional logic are known for\nwhich the satisfiability problem is polynomially solvable; Horn clauses are one such subset.\nBibliographical and Historical Notes\n277\nThe linear-time forward-chaining algorithm for Horn clauses is due to Dowling and Gallier\n(1984), who describe their algorithm as a dataflow process similar to the propagation of sig-\nnals in a circuit.\nEarly theoretical investigations showed that DPLL has polynomial average-case com-\nplexity for certain natural distributions of problems. This potentially exciting fact became\nless exciting when Franco and Paull (1983) showed that the same problems could be solved\nin constant time simply by guessing random assignments. The random-generation method\ndescribed in the chapter produces much harder problems. Motivated by the empirical success\nof local search on these problems, Koutsoupias and Papadimitriou (1992) showed that a sim-\nple hill-climbing algorithm can solve almost all satisfiability problem instances very quickly,\nsuggesting that hard problems are rare. Moreover, Sch¨oning (1999) exhibited a randomized\nhill-climbing algorithm whose worst-case expected run time on 3-SAT problems (that is, sat-\nisfiability of 3-CNF sentences) is O(1.333n)—still exponential, but substantially faster than\nprevious worst-case bounds. The current record is O(1.324n) (Iwama and Tamaki, 2004).\nAchlioptas et al. (2004) and Alekhnovich et al. (2005) exhibit families of 3-SAT instances\nfor which all known DPLL-like algorithms require exponential running time.\nOn the practical side, efficiency gains in propositional solvers have been marked. Given\nten minutes of computing time, the original DPLL algorithm in 1962 could only solve prob-\nlems with no more than 10 or 15 variables. By 1995 the SATZ solver (Li and Anbulagan,\n1997) could handle 1,000 variables, thanks to optimized data structures for indexing vari-\nables. Two crucial contributions were the watched literal indexing technique of Zhang and\nStickel (1996), which makes unit propagation very efficient, and the introduction of clause\n(i.e., constraint) learning techniques from the CSP community by Bayardo and Schrag (1997).\nUsing these ideas, and spurred by the prospect of solving industrial-scale circuit verification\nproblems, Moskewicz et al. (2001) developed the CHAFF solver, which could handle prob-\nlems with millions of variables. Beginning in 2002, SAT competitions have been held reg-\nularly; most of the winning entries have either been descendants of CHAFF or have used the\nsame general approach. RSAT (Pipatsrisawat and Darwiche, 2007), the 2007 winner, falls in\nthe latter category. Also noteworthy is MINISAT (Een and S¨orensson, 2003), an open-source\nimplementation available at http://minisat.se that is designed to be easily modified\nand improved. The current landscape of solvers is surveyed by Gomes et al. (2008).\nLocal search algorithms for satisfiability were tried by various authors throughout the\n1980s; all of the algorithms were based on the idea of minimizing the number of unsatisfied\nclauses (Hansen and Jaumard, 1990). A particularly effective algorithm was developed by\nGu (1989) and independently by Selman et al. (1992), who called it GSAT and showed that\nit was capable of solving a wide range of very hard problems very quickly. The WALKSAT\nalgorithm described in the chapter is due to Selman et al. (1996).\nThe “phase transition” in satisfiability of random k-SAT problems was first observed\nby Simon and Dubois (1989) and has given rise to a great deal of theoretical and empirical\nresearch—due, in part, to the obvious connection to phase transition phenomena in statistical\nphysics. Cheeseman et al. (1991) observed phase transitions in several CSPs and conjecture\nthat all NP-hard problems have a phase transition. Crawford and Auton (1993) located the\n3-SAT transition at a clause/variable ratio of around 4.26, noting that this coincides with a\n278\nChapter\n7.\nLogical Agents\nsharp peak in the run time of their SAT solver. Cook and Mitchell (1997) provide an excellent\nsummary of the early literature on the problem.\nThe current state of theoretical understanding is summarized by Achlioptas (2009).\nThe satisfiability threshold conjecture states that, for each k, there is a sharp satisfiability\nSATISFIABILITY\nTHRESHOLD\nCONJECTURE\nthreshold rk, such that as the number of variables n →∞, instances below the threshold are\nsatisfiable with probability 1, while those above the threshold are unsatisfiable with proba-\nbility 1. The conjecture was not quite proved by Friedgut (1999): a sharp threshold exists but\nits location might depend on n even as n →∞. Despite significant progress in asymptotic\nanalysis of the threshold location for large k (Achlioptas and Peres, 2004; Achlioptas et al.,\n2007), all that can be proved for k = 3 is that it lies in the range [3.52,4.51]. Current theory\nsuggests that a peak in the run time of a SAT solver is not necessarily related to the satisfia-\nbility threshold, but instead to a phase transition in the solution distribution and structure of\nSAT instances. Empirical results due to Coarfa et al. (2003) support this view. In fact, al-\ngorithms such as survey propagation (Parisi and Zecchina, 2002; Maneva et al., 2007) take\nSURVEY\nPROPAGATION\nadvantage of special properties of random SAT instances near the satisfiability threshold and\ngreatly outperform general SAT solvers on such instances.\nThe best sources for information on satisfiability, both theoretical and practical, are the\nHandbook of Satisfiability (Biere et al., 2009) and the regular International Conferences on\nTheory and Applications of Satisfiability Testing, known as SAT.\nThe idea of building agents with propositional logic can be traced back to the seminal\npaper of McCulloch and Pitts (1943), which initiated the field of neural networks. Con-\ntrary to popular supposition, the paper was concerned with the implementation of a Boolean\ncircuit-based agent design in the brain. Circuit-based agents, which perform computation by\npropagating signals in hardware circuits rather than running algorithms in general-purpose\ncomputers, have received little attention in AI, however. The most notable exception is the\nwork of Stan Rosenschein (Rosenschein, 1985; Kaelbling and Rosenschein, 1990), who de-\nveloped ways to compile circuit-based agents from declarative descriptions of the task envi-\nronment. (Rosenschein’s approach is described at some length in the second edition of this\nbook.) The work of Rod Brooks (1986, 1989) demonstrates the effectiveness of circuit-based\ndesigns for controlling robots—a topic we take up in Chapter 25. Brooks (1991) argues\nthat circuit-based designs are all that is needed for AI—that representation and reasoning\nare cumbersome, expensive, and unnecessary. In our view, neither approach is sufficient by\nitself. Williams et al. (2003) show how a hybrid agent design not too different from our\nwumpus agent has been used to control NASA spacecraft, planning sequences of actions and\ndiagnosing and recovering from faults.\nThe general problem of keeping track of a partially observable environment was intro-\nduced for state-based representations in Chapter 4. Its instantiation for propositional repre-\nsentations was studied by Amir and Russell (2003), who identified several classes of envi-\nronments that admit efficient state-estimation algorithms and showed that for several other\nclasses the problem is intractable. The temporal-projection problem, which involves deter-\nTEMPORAL-\nPROJECTION\nmining what propositions hold true after an action sequence is executed, can be seen as a\nspecial case of state estimation with empty percepts. Many authors have studied this problem\nbecause of its importance in planning; some important hardness results were established by\nExercises\n279\nLiberatore (1997). The idea of representing a belief state with propositions can be traced to\nWittgenstein (1922).\nLogical state estimation, of course, requires a logical representation of the effects of\nactions—a key problem in AI since the late 1950s. The dominant proposal has been the sit-\nuation calculus formalism (McCarthy, 1963), which is couched within first-order logic. We\ndiscuss situation calculus, and various extensions and alternatives, in Chapters 10 and 12. The\napproach taken in this chapter—using temporal indices on propositional variables—is more\nrestrictive but has the benefit of simplicity. The general approach embodied in the SATPLAN\nalgorithm was proposed by Kautz and Selman (1992). Later generations of SATPLAN were\nable to take advantage of the advances in SAT solvers, described earlier, and remain among\nthe most effective ways of solving difficult problems (Kautz, 2006).\nThe frame problem was first recognized by McCarthy and Hayes (1969). Many re-\nsearchers considered the problem unsolvable within first-order logic, and it spurred a great\ndeal of research into nonmonotonic logics. Philosophers from Dreyfus (1972) to Crockett\n(1994) have cited the frame problem as one symptom of the inevitable failure of the entire\nAI enterprise. The solution of the frame problem with successor-state axioms is due to Ray\nReiter (1991). Thielscher (1999) identifies the inferential frame problem as a separate idea\nand provides a solution. In retrospect, one can see that Rosenschein’s (1985) agents were\nusing circuits that implemented successor-state axioms, but Rosenschein did not notice that\nthe frame problem was thereby largely solved. Foo (2001) explains why the discrete-event\ncontrol theory models typically used by engineers do not have to explicitly deal with the\nframe problem: because they are dealing with prediction and control, not with explanation\nand reasoning about counterfactual situations.\nModern propositional solvers have wide applicability in industrial applications. The ap-\nplication of propositional inference in the synthesis of computer hardware is now a standard\ntechnique having many large-scale deployments (Nowick et al., 1993). The SATMC satisfi-\nability checker was used to detect a previously unknown vulnerability in a Web browser user\nsign-on protocol (Armando et al., 2008).\nThe wumpus world was invented by Gregory Yob (1975). Ironically, Yob developed it\nbecause he was bored with games played on a rectangular grid: the topology of his original\nwumpus world was a dodecahedron, and we put it back in the boring old grid. Michael\nGenesereth was the first to suggest that the wumpus world be used as an agent testbed.\nEXERCISES\n7.1\nSuppose the agent has progressed to the point shown in Figure 7.4(a), page 239, having\nperceived nothing in [1,1], a breeze in [2,1], and a stench in [1,2], and is now concerned with\nthe contents of [1,3], [2,2], and [3,1]. Each of these can contain a pit, and at most one can\ncontain a wumpus. Following the example of Figure 7.5, construct the set of possible worlds.\n(You should find 32 of them.) Mark the worlds in which the KB is true and those in which\n280\nChapter\n7.\nLogical Agents\neach of the following sentences is true:\nα2 = “There is no pit in [2,2].”\nα3 = “There is a wumpus in [1,3].”\nHence show that KB |= α2 and KB |= α3.\n7.2\n(Adapted from Barwise and Etchemendy (1993).) Given the following, can you prove\nthat the unicorn is mythical? How about magical? Horned?\nIf the unicorn is mythical, then it is immortal, but if it is not mythical, then it is a\nmortal mammal. If the unicorn is either immortal or a mammal, then it is horned.\nThe unicorn is magical if it is horned.\n7.3\nConsider the problem of deciding whether a propositional logic sentence is true in a\ngiven model.\na. Write a recursive algorithm PL-TRUE?(s, m) that returns true if and only if the sen-\ntence s is true in the model m (where m assigns a truth value for every symbol in s).\nThe algorithm should run in time linear in the size of the sentence. (Alternatively, use a\nversion of this function from the online code repository.)\nb. Give three examples of sentences that can be determined to be true or false in a partial\nmodel that does not specify a truth value for some of the symbols.\nc. Show that the truth value (if any) of a sentence in a partial model cannot be determined\nefficiently in general.\nd. Modify your PL-TRUE? algorithm so that it can sometimes judge truth from partial\nmodels, while retaining its recursive structure and linear run time. Give three examples\nof sentences whose truth in a partial model is not detected by your algorithm.\ne. Investigate whether the modified algorithm makes TT-ENTAILS? more efficient.\n7.4\nWhich of the following are correct?\na. False |= True.\nb. True |= False.\nc. (A ∧B) |= (A ⇔B).\nd. A ⇔B |= A ∨B.\ne. A ⇔B |= ¬A ∨B.\nf. (A ∧B) ⇒C |= (A ⇒C) ∨(B ⇒C).\ng. (C ∨(¬A ∧¬B)) ≡((A ⇒C) ∧(B ⇒C)).\nh. (A ∨B) ∧(¬C ∨¬D ∨E) |= (A ∨B).\ni. (A ∨B) ∧(¬C ∨¬D ∨E) |= (A ∨B) ∧(¬D ∨E).\nj. (A ∨B) ∧¬(A ⇒B) is satisfiable.\nk. (A ⇔B) ∧(¬A ∨B) is satisfiable.\nl. (A ⇔B) ⇔C has the same number of models as (A ⇔B) for any fixed set of\nproposition symbols that includes A, B, C.\nExercises\n281\n7.5\nProve each of the following assertions:\na. α is valid if and only if True |= α.\nb. For any α, False |= α.\nc. α |= β if and only if the sentence (α ⇒β) is valid.\nd. α ≡β if and only if the sentence (α ⇔β) is valid.\ne. α |= β if and only if the sentence (α ∧¬β) is unsatisfiable.\n7.6\nProve, or find a counterexample to, each of the following assertions:\na. If α |= γ or β |= γ (or both) then (α ∧β) |= γ\nb. If α |= (β ∧γ) then α |= β and α |= γ.\nc. If α |= (β ∨γ) then α |= β or α |= γ (or both).\n7.7\nConsider a vocabulary with only four propositions, A, B, C, and D. How many models\nare there for the following sentences?\na. B ∨C.\nb. ¬A ∨¬B ∨¬C ∨¬D.\nc. (A ⇒B) ∧A ∧¬B ∧C ∧D.\n7.8\nWe have defined four binary logical connectives.\na. Are there any others that might be useful?\nb. How many binary connectives can there be?\nc. Why are some of them not very useful?\n7.9\nUsing a method of your choice, verify each of the equivalences in Figure 7.11 (page 249).\n7.10\nDecide whether each of the following sentences is valid, unsatisfiable, or neither. Ver-\nify your decisions using truth tables or the equivalence rules of Figure 7.11 (page 249).\na. Smoke ⇒Smoke\nb. Smoke ⇒Fire\nc. (Smoke ⇒Fire) ⇒(¬Smoke ⇒¬Fire)\nd. Smoke ∨Fire ∨¬Fire\ne. ((Smoke ∧Heat) ⇒Fire) ⇔((Smoke ⇒Fire) ∨(Heat ⇒Fire))\nf. (Smoke ⇒Fire) ⇒((Smoke ∧Heat) ⇒Fire)\ng. Big ∨Dumb ∨(Big ⇒Dumb)\n7.11\nAny propositional logic sentence is logically equivalent to the assertion that each pos-\nsible world in which it would be false is not the case. From this observation, prove that any\nsentence can be written in CNF.\n7.12\nUse resolution to prove the sentence ¬A∧¬B from the clauses in Exercise 7.20.\n7.13\nThis exercise looks into the relationship between clauses and implication sentences.\n282\nChapter\n7.\nLogical Agents\na. Show that the clause (¬P1 ∨· · · ∨¬Pm ∨Q) is logically equivalent to the implication\nsentence (P1 ∧· · · ∧Pm) ⇒Q.\nb. Show that every clause (regardless of the number of positive literals) can be written in\nthe form (P1 ∧· · · ∧Pm) ⇒(Q1 ∨· · · ∨Qn), where the Ps and Qs are proposition\nsymbols. A knowledge base consisting of such sentences is in implicative normal\nform or Kowalski form (Kowalski, 1979).\nIMPLICATIVE\nNORMAL FORM\nc. Write down the full resolution rule for sentences in implicative normal form.\n7.14\nAccording to some political pundits, a person who is radical (R) is electable (E) if\nhe/she is conservative (C), but otherwise is not electable.\na. Which of the following are correct representations of this assertion?\n(i) (R ∧E) ⇐⇒C\n(ii) R ⇒(E ⇐⇒C)\n(iii) R ⇒((C ⇒E) ∨¬E)\nb. Which of the sentences in (a) can be expressed in Horn form?\n7.15\nThis question considers representing satisfiability (SAT) problems as CSPs.\na. Draw the constraint graph corresponding to the SAT problem\n(¬X1 ∨X2) ∧(¬X2 ∨X3) ∧. . . ∧(¬Xn−1 ∨Xn)\nfor the particular case n = 5.\nb. How many solutions are there for this general SAT problem as a function of n?\nc. Suppose we apply BACKTRACKING-SEARCH (page 215) to find all solutions to a SAT\nCSP of the type given in (a). (To find all solutions to a CSP, we simply modify the\nbasic algorithm so it continues searching after each solution is found.) Assume that\nvariables are ordered X1, . . . , Xn and false is ordered before true. How much time\nwill the algorithm take to terminate? (Write an O(·) expression as a function of n.)\nd. We know that SAT problems in Horn form can be solved in linear time by forward\nchaining (unit propagation). We also know that every tree-structured binary CSP with\ndiscrete, finite domains can be solved in time linear in the number of variables (Sec-\ntion 6.5). Are these two facts connected? Discuss.\n7.16\nExplain why every nonempty propositional clause, by itself, is satisfiable. Prove rig-\norously that every set of five 3-SAT clauses is satisfiable, provided that each clause mentions\nexactly three distinct variables. What is the smallest set of such clauses that is unsatisfiable?\nConstruct such a set.\n7.17\nA propositional 2-CNF expression is a conjunction of clauses, each containing exactly\n2 literals, e.g.,\n(A ∨B) ∧(¬A ∨C) ∧(¬B ∨D) ∧(¬C ∨G) ∧(¬D ∨G) .\na. Prove using resolution that the above sentence entails G.\nExercises\n283\nb. Two clauses are semantically distinct if they are not logically equivalent. How many\nsemantically distinct 2-CNF clauses can be constructed from n proposition symbols?\nc. Using your answer to (b), prove that propositional resolution always terminates in time\npolynomial in n given a 2-CNF sentence containing no more than n distinct symbols.\nd. Explain why your argument in (c) does not apply to 3-CNF.\n7.18\nConsider the following sentence:\n[(Food ⇒Party) ∨(Drinks ⇒Party)] ⇒[(Food ∧Drinks) ⇒Party] .\na. Determine, using enumeration, whether this sentence is valid, satisfiable (but not valid),\nor unsatisfiable.\nb. Convert the left-hand and right-hand sides of the main implication into CNF, showing\neach step, and explain how the results confirm your answer to (a).\nc. Prove your answer to (a) using resolution.\n7.19\nA sentence is in disjunctive normal form (DNF) if it is the disjunction of conjunctions\nDISJUNCTIVE\nNORMAL FORM\nof literals. For example, the sentence (A ∧B ∧¬C) ∨(¬A ∧C) ∨(B ∧¬C) is in DNF.\na. Any propositional logic sentence is logically equivalent to the assertion that some pos-\nsible world in which it would be true is in fact the case. From this observation, prove\nthat any sentence can be written in DNF.\nb. Construct an algorithm that converts any sentence in propositional logic into DNF.\n(Hint: The algorithm is similar to the algorithm for conversion to CNF given in Sec-\ntion 7.5.2.)\nc. Construct a simple algorithm that takes as input a sentence in DNF and returns a satis-\nfying assignment if one exists, or reports that no satisfying assignment exists.\nd. Apply the algorithms in (b) and (c) to the following set of sentences:\nA ⇒B\nB ⇒C\nC ⇒¬A .\ne. Since the algorithm in (b) is very similar to the algorithm for conversion to CNF, and\nsince the algorithm in (c) is much simpler than any algorithm for solving a set of sen-\ntences in CNF, why is this technique not used in automated reasoning?\n7.20\nConvert the following set of sentences to clausal form.\nS1: A ⇔(B ∨E).\nS2: E ⇒D.\nS3: C ∧F ⇒¬B.\nS4: E ⇒B.\nS5: B ⇒F.\nS6: B ⇒C\nGive a trace of the execution of DPLL on the conjunction of these clauses.\n284\nChapter\n7.\nLogical Agents\n7.21\nIs a randomly generated 4-CNF sentence with n symbols and m clauses more or less\nlikely to be solvable than a randomly generated 3-CNF sentence with n symbols and m\nclauses? Explain.\n7.22\nMinesweeper, the well-known computer game, is closely related to the wumpus world.\nA minesweeper world is a rectangular grid of N squares with M invisible mines scattered\namong them. Any square may be probed by the agent; instant death follows if a mine is\nprobed. Minesweeper indicates the presence of mines by revealing, in each probed square,\nthe number of mines that are directly or diagonally adjacent. The goal is to probe every\nunmined square.\na. Let Xi,j be true iff square [i, j] contains a mine. Write down the assertion that exactly\ntwo mines are adjacent to [1,1] as a sentence involving some logical combination of\nXi,j propositions.\nb. Generalize your assertion from (a) by explaining how to construct a CNF sentence\nasserting that k of n neighbors contain mines.\nc. Explain precisely how an agent can use DPLL to prove that a given square does (or\ndoes not) contain a mine, ignoring the global constraint that there are exactly M mines\nin all.\nd. Suppose that the global constraint is constructed from your method from part (b). How\ndoes the number of clauses depend on M and N? Suggest a way to modify DPLL so\nthat the global constraint does not need to be represented explicitly.\ne. Are any conclusions derived by the method in part (c) invalidated when the global\nconstraint is taken into account?\nf. Give examples of configurations of probe values that induce long-range dependencies\nsuch that the contents of a given unprobed square would give information about the\ncontents of a far-distant square. (Hint: consider an N × 1 board.)\n7.23\nHow long does it take to prove KB |= α using DPLL when α is a literal already\ncontained in KB? Explain.\n7.24\nTrace the behavior of DPLL on the knowledge base in Figure 7.16 when trying to\nprove Q, and compare this behavior with that of the forward-chaining algorithm.\n7.25\nWrite a successor-state axiom for the Locked predicate, which applies to doors, as-\nsuming the only actions available are Lock and Unlock.\n7.26\nSection 7.7.1 provides some of the successor-state axioms required for the wumpus\nworld. Write down axioms for all remaining fluent symbols.\n7.27\nModify the HYBRID-WUMPUS-AGENT to use the 1-CNF logical state estimation\nmethod described on page 271. We noted on that page that such an agent will not be able\nto acquire, maintain, and use more complex beliefs such as the disjunction P3,1 ∨P2,2. Sug-\ngest a method for overcoming this problem by defining additional proposition symbols, and\ntry it out in the wumpus world. Does it improve the performance of the agent?\n" }, { "chapter": 8, "text": "8\nFIRST-ORDER LOGIC\nIn which we notice that the world is blessed with many objects, some of which are\nrelated to other objects, and in which we endeavor to reason about them.\nIn Chapter 7, we showed how a knowledge-based agent could represent the world in which it\noperates and deduce what actions to take. We used propositional logic as our representation\nlanguage because it sufficed to illustrate the basic concepts of logic and knowledge-based\nagents. Unfortunately, propositional logic is too puny a language to represent knowledge\nof complex environments in a concise way. In this chapter, we examine first-order logic,1\nFIRST-ORDER LOGIC\nwhich is sufficiently expressive to represent a good deal of our commonsense knowledge.\nIt also either subsumes or forms the foundation of many other representation languages and\nhas been studied intensively for many decades. We begin in Section 8.1 with a discussion of\nrepresentation languages in general; Section 8.2 covers the syntax and semantics of first-order\nlogic; Sections 8.3 and 8.4 illustrate the use of first-order logic for simple representations.\n8.1\nREPRESENTATION REVISITED\nIn this section, we discuss the nature of representation languages. Our discussion motivates\nthe development of first-order logic, a much more expressive language than the propositional\nlogic introduced in Chapter 7. We look at propositional logic and at other kinds of languages\nto understand what works and what fails. Our discussion will be cursory, compressing cen-\nturies of thought, trial, and error into a few paragraphs.\nProgramming languages (such as C++ or Java or Lisp) are by far the largest class of\nformal languages in common use. Programs themselves represent, in a direct sense, only\ncomputational processes. Data structures within programs can represent facts; for example,\na program could use a 4 × 4 array to represent the contents of the wumpus world. Thus, the\nprogramming language statement World[2,2] ←Pit is a fairly natural way to assert that there\nis a pit in square [2,2]. (Such representations might be considered ad hoc; database systems\nwere developed precisely to provide a more general, domain-independent way to store and\n1 Also called first-order predicate calculus, sometimes abbreviated as FOL or FOPC.\n285\n286\nChapter\n8.\nFirst-Order Logic\nretrieve facts.) What programming languages lack is any general mechanism for deriving\nfacts from other facts; each update to a data structure is done by a domain-specific procedure\nwhose details are derived by the programmer from his or her own knowledge of the domain.\nThis procedural approach can be contrasted with the declarative nature of propositional logic,\nin which knowledge and inference are separate, and inference is entirely domain independent.\nA second drawback of data structures in programs (and of databases, for that matter)\nis the lack of any easy way to say, for example, “There is a pit in [2,2] or [3,1]” or “If the\nwumpus is in [1,1] then he is not in [2,2].” Programs can store a single value for each variable,\nand some systems allow the value to be “unknown,” but they lack the expressiveness required\nto handle partial information.\nPropositional logic is a declarative language because its semantics is based on a truth\nrelation between sentences and possible worlds. It also has sufficient expressive power to\ndeal with partial information, using disjunction and negation. Propositional logic has a third\nproperty that is desirable in representation languages, namely, compositionality. In a com-\nCOMPOSITIONALITY\npositional language, the meaning of a sentence is a function of the meaning of its parts. For\nexample, the meaning of “S1,4 ∧S1,2” is related to the meanings of “S1,4” and “S1,2.” It\nwould be very strange if “S1,4” meant that there is a stench in square [1,4] and “S1,2” meant\nthat there is a stench in square [1,2], but “S1,4 ∧S1,2” meant that France and Poland drew 1–1\nin last week’s ice hockey qualifying match. Clearly, noncompositionality makes life much\nmore difficult for the reasoning system.\nAs we saw in Chapter 7, however, propositional logic lacks the expressive power to\nconcisely describe an environment with many objects. For example, we were forced to write\na separate rule about breezes and pits for each square, such as\nB1,1 ⇔(P1,2 ∨P2,1) .\nIn English, on the other hand, it seems easy enough to say, once and for all, “Squares adjacent\nto pits are breezy.” The syntax and semantics of English somehow make it possible to describe\nthe environment concisely.\n8.1.1\nThe language of thought\nNatural languages (such as English or Spanish) are very expressive indeed. We managed to\nwrite almost this whole book in natural language, with only occasional lapses into other lan-\nguages (including logic, mathematics, and the language of diagrams). There is a long tradi-\ntion in linguistics and the philosophy of language that views natural language as a declarative\nknowledge representation language. If we could uncover the rules for natural language, we\ncould use it in representation and reasoning systems and gain the benefit of the billions of\npages that have been written in natural language.\nThe modern view of natural language is that it serves a as a medium for communication\nrather than pure representation. When a speaker points and says, “Look!” the listener comes\nto know that, say, Superman has finally appeared over the rooftops. Yet we would not want\nto say that the sentence “Look!” represents that fact. Rather, the meaning of the sentence\ndepends both on the sentence itself and on the context in which the sentence was spoken.\nClearly, one could not store a sentence such as “Look!” in a knowledge base and expect to\nSection 8.1.\nRepresentation Revisited\n287\nrecover its meaning without also storing a representation of the context—which raises the\nquestion of how the context itself can be represented. Natural languages also suffer from\nambiguity, a problem for a representation language. As Pinker (1995) puts it: “When people\nAMBIGUITY\nthink about spring, surely they are not confused as to whether they are thinking about a season\nor something that goes boing—and if one word can correspond to two thoughts, thoughts\ncan’t be words.”\nThe famous Sapir–Whorf hypothesis claims that our understanding of the world is\nstrongly influenced by the language we speak. Whorf (1956) wrote “We cut nature up, orga-\nnize it into concepts, and ascribe significances as we do, largely because we are parties to an\nagreement to organize it this way—an agreement that holds throughout our speech commu-\nnity and is codified in the patterns of our language.” It is certainly true that different speech\ncommunities divide up the world differently. The French have two words “chaise” and “fau-\nteuil,” for a concept that English speakers cover with one: “chair.” But English speakers\ncan easily recognize the category fauteuil and give it a name—roughly “open-arm chair”—so\ndoes language really make a difference? Whorf relied mainly on intuition and speculation,\nbut in the intervening years we actually have real data from anthropological, psychological\nand neurological studies.\nFor example, can you remember which of the following two phrases formed the opening\nof Section 8.1?\n“In this section, we discuss the nature of representation languages . . .”\n“This section covers the topic of knowledge representation languages . . .”\nWanner (1974) did a similar experiment and found that subjects made the right choice at\nchance level—about 50% of the time—but remembered the content of what they read with\nbetter than 90% accuracy. This suggests that people process the words to form some kind of\nnonverbal representation.\nMore interesting is the case in which a concept is completely absent in a language.\nSpeakers of the Australian aboriginal language Guugu Yimithirr have no words for relative\ndirections, such as front, back, right, or left. Instead they use absolute directions, saying,\nfor example, the equivalent of “I have a pain in my north arm.” This difference in language\nmakes a difference in behavior: Guugu Yimithirr speakers are better at navigating in open\nterrain, while English speakers are better at placing the fork to the right of the plate.\nLanguage also seems to influence thought through seemingly arbitrary grammatical\nfeatures such as the gender of nouns. For example, “bridge” is masculine in Spanish and\nfeminine in German. Boroditsky (2003) asked subjects to choose English adjectives to de-\nscribe a photograph of a particular bridge. Spanish speakers chose big, dangerous, strong,\nand towering, whereas German speakers chose beautiful, elegant, fragile, and slender. Words\ncan serve as anchor points that affect how we perceive the world. Loftus and Palmer (1974)\nshowed experimental subjects a movie of an auto accident. Subjects who were asked “How\nfast were the cars going when they contacted each other?” reported an average of 32 mph,\nwhile subjects who were asked the question with the word “smashed” instead of “contacted”\nreported 41mph for the same cars in the same movie.\n288\nChapter\n8.\nFirst-Order Logic\nIn a first-order logic reasoning system that uses CNF, we can see that the linguistic form\n“¬(A ∨B)” and “¬A ∧¬B” are the same because we can look inside the system and see\nthat the two sentences are stored as the same canonical CNF form. Can we do that with the\nhuman brain? Until recently the answer was “no,” but now it is “maybe.” Mitchell et al.\n(2008) put subjects in an fMRI (functional magnetic resonance imaging) machine, showed\nthem words such as “celery,” and imaged their brains. The researchers were then able to train\na computer program to predict, from a brain image, what word the subject had been presented\nwith. Given two choices (e.g., “celery” or “airplane”), the system predicts correctly 77% of\nthe time. The system can even predict at above-chance levels for words it has never seen\nan fMRI image of before (by considering the images of related words) and for people it has\nnever seen before (proving that fMRI reveals some level of common representation across\npeople). This type of work is still in its infancy, but fMRI (and other imaging technology\nsuch as intracranial electrophysiology (Sahin et al., 2009)) promises to give us much more\nconcrete ideas of what human knowledge representations are like.\nFrom the viewpoint of formal logic, representing the same knowledge in two different\nways makes absolutely no difference; the same facts will be derivable from either represen-\ntation. In practice, however, one representation might require fewer steps to derive a conclu-\nsion, meaning that a reasoner with limited resources could get to the conclusion using one\nrepresentation but not the other. For nondeductive tasks such as learning from experience,\noutcomes are necessarily dependent on the form of the representations used. We show in\nChapter 18 that when a learning program considers two possible theories of the world, both\nof which are consistent with all the data, the most common way of breaking the tie is to choose\nthe most succinct theory—and that depends on the language used to represent theories. Thus,\nthe influence of language on thought is unavoidable for any agent that does learning.\n8.1.2\nCombining the best of formal and natural languages\nWe can adopt the foundation of propositional logic—a declarative, compositional semantics\nthat is context-independent and unambiguous—and build a more expressive logic on that\nfoundation, borrowing representational ideas from natural language while avoiding its draw-\nbacks. When we look at the syntax of natural language, the most obvious elements are nouns\nand noun phrases that refer to objects (squares, pits, wumpuses) and verbs and verb phrases\nOBJECT\nthat refer to relations among objects (is breezy, is adjacent to, shoots). Some of these rela-\nRELATION\ntions are functions—relations in which there is only one “value” for a given “input.” It is\nFUNCTION\neasy to start listing examples of objects, relations, and functions:\n• Objects: people, houses, numbers, theories, Ronald McDonald, colors, baseball games,\nwars, centuries . . .\n• Relations: these can be unary relations or properties such as red, round, bogus, prime,\nPROPERTY\nmultistoried . . ., or more general n-ary relations such as brother of, bigger than, inside,\npart of, has color, occurred after, owns, comes between, . . .\n• Functions: father of, best friend, third inning of, one more than, beginning of . . .\nIndeed, almost any assertion can be thought of as referring to objects and properties or rela-\ntions. Some examples follow:\nSection 8.1.\nRepresentation Revisited\n289\n• “One plus two equals three.”\nObjects: one, two, three, one plus two; Relation: equals; Function: plus. (“One plus\ntwo” is a name for the object that is obtained by applying the function “plus” to the\nobjects “one” and “two.” “Three” is another name for this object.)\n• “Squares neighboring the wumpus are smelly.”\nObjects: wumpus, squares; Property: smelly; Relation: neighboring.\n• “Evil King John ruled England in 1200.”\nObjects: John, England, 1200; Relation: ruled; Properties: evil, king.\nThe language of first-order logic, whose syntax and semantics we define in the next section,\nis built around objects and relations. It has been so important to mathematics, philosophy, and\nartificial intelligence precisely because those fields—and indeed, much of everyday human\nexistence—can be usefully thought of as dealing with objects and the relations among them.\nFirst-order logic can also express facts about some or all of the objects in the universe. This\nenables one to represent general laws or rules, such as the statement “Squares neighboring\nthe wumpus are smelly.”\nThe primary difference between propositional and first-order logic lies in the ontologi-\ncal commitment made by each language—that is, what it assumes about the nature of reality.\nONTOLOGICAL\nCOMMITMENT\nMathematically, this commitment is expressed through the nature of the formal models with\nrespect to which the truth of sentences is defined. For example, propositional logic assumes\nthat there are facts that either hold or do not hold in the world. Each fact can be in one\nof two states: true or false, and each model assigns true or false to each proposition sym-\nbol (see Section 7.4.2).2 First-order logic assumes more; namely, that the world consists of\nobjects with certain relations among them that do or do not hold. The formal models are\ncorrespondingly more complicated than those for propositional logic. Special-purpose logics\nmake still further ontological commitments; for example, temporal logic assumes that facts\nTEMPORAL LOGIC\nhold at particular times and that those times (which may be points or intervals) are ordered.\nThus, special-purpose logics give certain kinds of objects (and the axioms about them) “first\nclass” status within the logic, rather than simply defining them within the knowledge base.\nHigher-order logic views the relations and functions referred to by first-order logic as ob-\nHIGHER-ORDER\nLOGIC\njects in themselves. This allows one to make assertions about all relations—for example, one\ncould wish to define what it means for a relation to be transitive. Unlike most special-purpose\nlogics, higher-order logic is strictly more expressive than first-order logic, in the sense that\nsome sentences of higher-order logic cannot be expressed by any finite number of first-order\nlogic sentences.\nA logic can also be characterized by its epistemological commitments—the possible\nEPISTEMOLOGICAL\nCOMMITMENT\nstates of knowledge that it allows with respect to each fact. In both propositional and first-\norder logic, a sentence represents a fact and the agent either believes the sentence to be true,\nbelieves it to be false, or has no opinion. These logics therefore have three possible states\nof knowledge regarding any sentence. Systems using probability theory, on the other hand,\n2 In contrast, facts in fuzzy logic have a degree of truth between 0 and 1. For example, the sentence “Vienna is\na large city” might be true in our world only to degree 0.6 in fuzzy logic.\n290\nChapter\n8.\nFirst-Order Logic\ncan have any degree of belief, ranging from 0 (total disbelief) to 1 (total belief).3 For ex-\nample, a probabilistic wumpus-world agent might believe that the wumpus is in [1,3] with\nprobability 0.75. The ontological and epistemological commitments of five different logics\nare summarized in Figure 8.1.\nLanguage\nOntological Commitment\nEpistemological Commitment\n(What exists in the world)\n(What an agent believes about facts)\nPropositional logic\nfacts\ntrue/false/unknown\nFirst-order logic\nfacts, objects, relations\ntrue/false/unknown\nTemporal logic\nfacts, objects, relations, times\ntrue/false/unknown\nProbability theory\nfacts\ndegree of belief ∈[0, 1]\nFuzzy logic\nfacts with degree of truth ∈[0, 1]\nknown interval value\nFigure 8.1\nFormal languages and their ontological and epistemological commitments.\nIn the next section, we will launch into the details of first-order logic. Just as a student of\nphysics requires some familiarity with mathematics, a student of AI must develop a talent for\nworking with logical notation. On the other hand, it is also important not to get too concerned\nwith the specifics of logical notation—after all, there are dozens of different versions. The\nmain things to keep hold of are how the language facilitates concise representations and how\nits semantics leads to sound reasoning procedures.\n8.2\nSYNTAX AND SEMANTICS OF FIRST-ORDER LOGIC\nWe begin this section by specifying more precisely the way in which the possible worlds\nof first-order logic reflect the ontological commitment to objects and relations. Then we\nintroduce the various elements of the language, explaining their semantics as we go along.\n8.2.1\nModels for first-order logic\nRecall from Chapter 7 that the models of a logical language are the formal structures that\nconstitute the possible worlds under consideration. Each model links the vocabulary of the\nlogical sentences to elements of the possible world, so that the truth of any sentence can\nbe determined. Thus, models for propositional logic link proposition symbols to predefined\ntruth values. Models for first-order logic are much more interesting. First, they have objects\nin them! The domain of a model is the set of objects or domain elements it contains. The do-\nDOMAIN\nDOMAIN ELEMENTS\nmain is required to be nonempty—every possible world must contain at least one object. (See\nExercise 8.7 for a discussion of empty worlds.) Mathematically speaking, it doesn’t matter\nwhat these objects are—all that matters is how many there are in each particular model—but\nfor pedagogical purposes we’ll use a concrete example. Figure 8.2 shows a model with five\n3 It is important not to confuse the degree of belief in probability theory with the degree of truth in fuzzy logic.\nIndeed, some fuzzy systems allow uncertainty (degree of belief) about degrees of truth.\nSection 8.2.\nSyntax and Semantics of First-Order Logic\n291\nobjects: Richard the Lionheart, King of England from 1189 to 1199; his younger brother, the\nevil King John, who ruled from 1199 to 1215; the left legs of Richard and John; and a crown.\nThe objects in the model may be related in various ways. In the figure, Richard and\nJohn are brothers. Formally speaking, a relation is just the set of tuples of objects that are\nTUPLE\nrelated. (A tuple is a collection of objects arranged in a fixed order and is written with angle\nbrackets surrounding the objects.) Thus, the brotherhood relation in this model is the set\n{ ⟨Richard the Lionheart, King John⟩, ⟨King John, Richard the Lionheart⟩} .\n(8.1)\n(Here we have named the objects in English, but you may, if you wish, mentally substitute the\npictures for the names.) The crown is on King John’s head, so the “on head” relation contains\njust one tuple, ⟨the crown, King John⟩. The “brother” and “on head” relations are binary\nrelations—that is, they relate pairs of objects. The model also contains unary relations, or\nproperties: the “person” property is true of both Richard and John; the “king” property is true\nonly of John (presumably because Richard is dead at this point); and the “crown” property is\ntrue only of the crown.\nCertain kinds of relationships are best considered as functions, in that a given object\nmust be related to exactly one object in this way. For example, each person has one left leg,\nso the model has a unary “left leg” function that includes the following mappings:\n⟨Richard the Lionheart⟩→Richard’s left leg\n⟨King John⟩→John’s left leg .\n(8.2)\nStrictly speaking, models in first-order logic require total functions, that is, there must be a\nTOTAL FUNCTIONS\nvalue for every input tuple. Thus, the crown must have a left leg and so must each of the left\nlegs. There is a technical solution to this awkward problem involving an additional “invisible”\nR\nJ\n$\nleft leg\non head\nbrother\nbrother\nperson\nperson\nking\ncrown\nleft leg\nFigure 8.2\nA model containing five objects, two binary relations, three unary relations\n(indicated by labels on the objects), and one unary function, left-leg.\n292\nChapter\n8.\nFirst-Order Logic\nobject that is the left leg of everything that has no left leg, including itself. Fortunately, as\nlong as one makes no assertions about the left legs of things that have no left legs, these\ntechnicalities are of no import.\nSo far, we have described the elements that populate models for first-order logic. The\nother essential part of a model is the link between those elements and the vocabulary of the\nlogical sentences, which we explain next.\n8.2.2\nSymbols and interpretations\nWe turn now to the syntax of first-order logic. The impatient reader can obtain a complete\ndescription from the formal grammar in Figure 8.3.\nThe basic syntactic elements of first-order logic are the symbols that stand for objects,\nrelations, and functions. The symbols, therefore, come in three kinds: constant symbols,\nCONSTANT SYMBOL\nwhich stand for objects; predicate symbols, which stand for relations; and function sym-\nPREDICATE SYMBOL\nbols, which stand for functions. We adopt the convention that these symbols will begin with\nFUNCTION SYMBOL\nuppercase letters. For example, we might use the constant symbols Richard and John; the\npredicate symbols Brother, OnHead, Person, King, and Crown; and the function symbol\nLeftLeg. As with proposition symbols, the choice of names is entirely up to the user. Each\npredicate and function symbol comes with an arity that fixes the number of arguments.\nARITY\nAs in propositional logic, every model must provide the information required to deter-\nmine if any given sentence is true or false. Thus, in addition to its objects, relations, and\nfunctions, each model includes an interpretation that specifies exactly which objects, rela-\nINTERPRETATION\ntions and functions are referred to by the constant, predicate, and function symbols. One\npossible interpretation for our example—which a logician would call the intended interpre-\ntation—is as follows:\nINTENDED\nINTERPRETATION\n• Richard refers to Richard the Lionheart and John refers to the evil King John.\n• Brother refers to the brotherhood relation, that is, the set of tuples of objects given in\nEquation (8.1); OnHead refers to the “on head” relation that holds between the crown\nand King John; Person, King, and Crown refer to the sets of objects that are persons,\nkings, and crowns.\n• LeftLeg refers to the “left leg” function, that is, the mapping given in Equation (8.2).\nThere are many other possible interpretations, of course. For example, one interpretation\nmaps Richard to the crown and John to King John’s left leg. There are five objects in\nthe model, so there are 25 possible interpretations just for the constant symbols Richard\nand John. Notice that not all the objects need have a name—for example, the intended\ninterpretation does not name the crown or the legs. It is also possible for an object to have\nseveral names; there is an interpretation under which both Richard and John refer to the\ncrown.4 If you find this possibility confusing, remember that, in propositional logic, it is\nperfectly possible to have a model in which Cloudy and Sunny are both true; it is the job of\nthe knowledge base to rule out models that are inconsistent with our knowledge.\n4 Later, in Section 8.2.8, we examine a semantics in which every object has exactly one name.\nSection 8.2.\nSyntax and Semantics of First-Order Logic\n293\nSentence\n→\nAtomicSentence | ComplexSentence\nAtomicSentence\n→\nPredicate | Predicate(Term, . . .) | Term = Term\nComplexSentence\n→\n( Sentence ) | [ Sentence ]\n|\n¬ Sentence\n|\nSentence ∧Sentence\n|\nSentence ∨Sentence\n|\nSentence\n⇒Sentence\n|\nSentence\n⇔\nSentence\n|\nQuantifier Variable, . . . Sentence\nTerm\n→\nFunction(Term, . . .)\n|\nConstant\n|\nVariable\nQuantifier\n→\n∀| ∃\nConstant\n→\nA | X1 | John | · · ·\nVariable\n→\na | x | s | · · ·\nPredicate\n→\nTrue | False | After | Loves | Raining | · · ·\nFunction\n→\nMother | LeftLeg | · · ·\nOPERATOR PRECEDENCE\n:\n¬, =, ∧, ∨, ⇒, ⇔\nFigure 8.3\nThe syntax of first-order logic with equality, specified in Backus–Naur form\n(see page 1060 if you are not familiar with this notation). Operator precedences are specified,\nfrom highest to lowest. The precedence of quantifiers is such that a quantifier holds over\neverything to the right of it.\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\n. . .\n. . .\n. . .\nFigure 8.4\nSome members of the set of all models for a language with two constant sym-\nbols, R and J, and one binary relation symbol. The interpretation of each constant symbol is\nshown by a gray arrow. Within each model, the related objects are connected by arrows.\n294\nChapter\n8.\nFirst-Order Logic\nIn summary, a model in first-order logic consists of a set of objects and an interpretation\nthat maps constant symbols to objects, predicate symbols to relations on those objects, and\nfunction symbols to functions on those objects. Just as with propositional logic, entailment,\nvalidity, and so on are defined in terms of all possible models. To get an idea of what the\nset of all possible models looks like, see Figure 8.4. It shows that models vary in how many\nobjects they contain—from one up to infinity—and in the way the constant symbols map\nto objects. If there are two constant symbols and one object, then both symbols must refer\nto the same object; but this can still happen even with more objects. When there are more\nobjects than constant symbols, some of the objects will have no names. Because the number\nof possible models is unbounded, checking entailment by the enumeration of all possible\nmodels is not feasible for first-order logic (unlike propositional logic). Even if the number of\nobjects is restricted, the number of combinations can be very large. (See Exercise 8.5.) For\nthe example in Figure 8.4, there are 137,506,194,466 models with six or fewer objects.\n8.2.3\nTerms\nA term is a logical expression that refers to an object. Constant symbols are therefore terms,\nTERM\nbut it is not always convenient to have a distinct symbol to name every object. For example,\nin English we might use the expression “King John’s left leg” rather than giving a name\nto his leg. This is what function symbols are for: instead of using a constant symbol, we\nuse LeftLeg(John). In the general case, a complex term is formed by a function symbol\nfollowed by a parenthesized list of terms as arguments to the function symbol. It is important\nto remember that a complex term is just a complicated kind of name. It is not a “subroutine\ncall” that “returns a value.” There is no LeftLeg subroutine that takes a person as input and\nreturns a leg. We can reason about left legs (e.g., stating the general rule that everyone has one\nand then deducing that John must have one) without ever providing a definition of LeftLeg.\nThis is something that cannot be done with subroutines in programming languages.5\nThe formal semantics of terms is straightforward. Consider a term f(t1, . . . , tn). The\nfunction symbol f refers to some function in the model (call it F); the argument terms refer\nto objects in the domain (call them d1, . . . , dn); and the term as a whole refers to the object\nthat is the value of the function F applied to d1, . . . , dn. For example, suppose the LeftLeg\nfunction symbol refers to the function shown in Equation (8.2) and John refers to King John,\nthen LeftLeg(John) refers to King John’s left leg. In this way, the interpretation fixes the\nreferent of every term.\n8.2.4\nAtomic sentences\nNow that we have both terms for referring to objects and predicate symbols for referring to\nrelations, we can put them together to make atomic sentences that state facts. An atomic\n5 λ-expressions provide a useful notation in which new function symbols are constructed “on the fly.” For\nexample, the function that squares its argument can be written as (λx x × x) and can be applied to arguments\njust like any other function symbol. A λ-expression can also be defined and used as a predicate symbol. (See\nChapter 22.) The lambda operator in Lisp plays exactly the same role. Notice that the use of λ in this way does\nnot increase the formal expressive power of first-order logic, because any sentence that includes a λ-expression\ncan be rewritten by “plugging in” its arguments to yield an equivalent sentence.\nSection 8.2.\nSyntax and Semantics of First-Order Logic\n295\nsentence (or atom for short) is formed from a predicate symbol optionally followed by a\nATOMIC SENTENCE\nATOM\nparenthesized list of terms, such as\nBrother(Richard, John).\nThis states, under the intended interpretation given earlier, that Richard the Lionheart is the\nbrother of King John.6 Atomic sentences can have complex terms as arguments. Thus,\nMarried(Father(Richard), Mother(John))\nstates that Richard the Lionheart’s father is married to King John’s mother (again, under a\nsuitable interpretation).\nAn atomic sentence is true in a given model if the relation referred to by the predicate\nsymbol holds among the objects referred to by the arguments.\n8.2.5\nComplex sentences\nWe can use logical connectives to construct more complex sentences, with the same syntax\nand semantics as in propositional calculus. Here are four sentences that are true in the model\nof Figure 8.2 under our intended interpretation:\n¬Brother(LeftLeg(Richard), John)\nBrother(Richard, John) ∧Brother(John, Richard)\nKing(Richard) ∨King(John)\n¬King(Richard) ⇒King(John) .\n8.2.6\nQuantifiers\nOnce we have a logic that allows objects, it is only natural to want to express properties of\nentire collections of objects, instead of enumerating the objects by name. Quantifiers let us\nQUANTIFIER\ndo this. First-order logic contains two standard quantifiers, called universal and existential.\nUniversal quantification (∀)\nRecall the difficulty we had in Chapter 7 with the expression of general rules in proposi-\ntional logic. Rules such as “Squares neighboring the wumpus are smelly” and “All kings\nare persons” are the bread and butter of first-order logic. We deal with the first of these in\nSection 8.3. The second rule, “All kings are persons,” is written in first-order logic as\n∀x King(x) ⇒Person(x) .\n∀is usually pronounced “For all . . .”. (Remember that the upside-down A stands for “all.”)\nThus, the sentence says, “For all x, if x is a king, then x is a person.” The symbol x is called\na variable. By convention, variables are lowercase letters. A variable is a term all by itself,\nVARIABLE\nand as such can also serve as the argument of a function—for example, LeftLeg(x). A term\nwith no variables is called a ground term.\nGROUND TERM\nIntuitively, the sentence ∀x P, where P is any logical expression, says that P is true\nfor every object x. More precisely, ∀x P is true in a given model if P is true in all possible\nextended interpretations constructed from the interpretation given in the model, where each\nEXTENDED\nINTERPRETATION\n6 We usually follow the argument-ordering convention that P(x, y) is read as “x is a P of y.”\n296\nChapter\n8.\nFirst-Order Logic\nextended interpretation specifies a domain element to which x refers.\nThis sounds complicated, but it is really just a careful way of stating the intuitive mean-\ning of universal quantification. Consider the model shown in Figure 8.2 and the intended\ninterpretation that goes with it. We can extend the interpretation in five ways:\nx →Richard the Lionheart,\nx →King John,\nx →Richard’s left leg,\nx →John’s left leg,\nx →the crown.\nThe universally quantified sentence ∀x King(x) ⇒Person(x) is true in the original model\nif the sentence King(x) ⇒Person(x) is true under each of the five extended interpreta-\ntions. That is, the universally quantified sentence is equivalent to asserting the following five\nsentences:\nRichard the Lionheart is a king ⇒Richard the Lionheart is a person.\nKing John is a king ⇒King John is a person.\nRichard’s left leg is a king ⇒Richard’s left leg is a person.\nJohn’s left leg is a king ⇒John’s left leg is a person.\nThe crown is a king ⇒the crown is a person.\nLet us look carefully at this set of assertions. Since, in our model, King John is the only\nking, the second sentence asserts that he is a person, as we would hope. But what about\nthe other four sentences, which appear to make claims about legs and crowns? Is that part\nof the meaning of “All kings are persons”? In fact, the other four assertions are true in the\nmodel, but make no claim whatsoever about the personhood qualifications of legs, crowns,\nor indeed Richard. This is because none of these objects is a king. Looking at the truth table\nfor ⇒(Figure 7.8 on page 246), we see that the implication is true whenever its premise is\nfalse—regardless of the truth of the conclusion. Thus, by asserting the universally quantified\nsentence, which is equivalent to asserting a whole list of individual implications, we end\nup asserting the conclusion of the rule just for those objects for whom the premise is true\nand saying nothing at all about those individuals for whom the premise is false. Thus, the\ntruth-table definition of ⇒turns out to be perfect for writing general rules with universal\nquantifiers.\nA common mistake, made frequently even by diligent readers who have read this para-\ngraph several times, is to use conjunction instead of implication. The sentence\n∀x King(x) ∧Person(x)\nwould be equivalent to asserting\nRichard the Lionheart is a king ∧Richard the Lionheart is a person,\nKing John is a king ∧King John is a person,\nRichard’s left leg is a king ∧Richard’s left leg is a person,\nand so on. Obviously, this does not capture what we want.\nSection 8.2.\nSyntax and Semantics of First-Order Logic\n297\nExistential quantification (∃)\nUniversal quantification makes statements about every object. Similarly, we can make a state-\nment about some object in the universe without naming it, by using an existential quantifier.\nTo say, for example, that King John has a crown on his head, we write\n∃x Crown(x) ∧OnHead(x, John) .\n∃x is pronounced “There exists an x such that . . .” or “For some x . . .”.\nIntuitively, the sentence ∃x P says that P is true for at least one object x. More\nprecisely, ∃x P is true in a given model if P is true in at least one extended interpretation\nthat assigns x to a domain element. That is, at least one of the following is true:\nRichard the Lionheart is a crown ∧Richard the Lionheart is on John’s head;\nKing John is a crown ∧King John is on John’s head;\nRichard’s left leg is a crown ∧Richard’s left leg is on John’s head;\nJohn’s left leg is a crown ∧John’s left leg is on John’s head;\nThe crown is a crown ∧the crown is on John’s head.\nThe fifth assertion is true in the model, so the original existentially quantified sentence is\ntrue in the model. Notice that, by our definition, the sentence would also be true in a model\nin which King John was wearing two crowns. This is entirely consistent with the original\nsentence “King John has a crown on his head.” 7\nJust as ⇒appears to be the natural connective to use with ∀, ∧is the natural connective\nto use with ∃. Using ∧as the main connective with ∀led to an overly strong statement in\nthe example in the previous section; using ⇒with ∃usually leads to a very weak statement,\nindeed. Consider the following sentence:\n∃x Crown(x) ⇒OnHead(x, John) .\nOn the surface, this might look like a reasonable rendition of our sentence. Applying the\nsemantics, we see that the sentence says that at least one of the following assertions is true:\nRichard the Lionheart is a crown ⇒Richard the Lionheart is on John’s head;\nKing John is a crown ⇒King John is on John’s head;\nRichard’s left leg is a crown ⇒Richard’s left leg is on John’s head;\nand so on. Now an implication is true if both premise and conclusion are true, or if its premise\nis false. So if Richard the Lionheart is not a crown, then the first assertion is true and the\nexistential is satisfied. So, an existentially quantified implication sentence is true whenever\nany object fails to satisfy the premise; hence such sentences really do not say much at all.\nNested quantifiers\nWe will often want to express more complex sentences using multiple quantifiers. The sim-\nplest case is where the quantifiers are of the same type. For example, “Brothers are siblings”\ncan be written as\n∀x ∀y Brother(x, y) ⇒Sibling(x, y) .\n7 There is a variant of the existential quantifier, usually written ∃1 or ∃!, that means “There exists exactly one.”\nThe same meaning can be expressed using equality statements.\n298\nChapter\n8.\nFirst-Order Logic\nConsecutive quantifiers of the same type can be written as one quantifier with several vari-\nables. For example, to say that siblinghood is a symmetric relationship, we can write\n∀x, y Sibling(x, y) ⇔Sibling(y, x) .\nIn other cases we will have mixtures. “Everybody loves somebody” means that for every\nperson, there is someone that person loves:\n∀x ∃y Loves(x, y) .\nOn the other hand, to say “There is someone who is loved by everyone,” we write\n∃y ∀x Loves(x, y) .\nThe order of quantification is therefore very important. It becomes clearer if we insert paren-\ntheses. ∀x (∃y Loves(x, y)) says that everyone has a particular property, namely, the prop-\nerty that they love someone. On the other hand, ∃y (∀x Loves(x, y)) says that someone in\nthe world has a particular property, namely the property of being loved by everybody.\nSome confusion can arise when two quantifiers are used with the same variable name.\nConsider the sentence\n∀x (Crown(x) ∨(∃x Brother(Richard, x))) .\nHere the x in Brother(Richard, x) is existentially quantified. The rule is that the variable\nbelongs to the innermost quantifier that mentions it; then it will not be subject to any other\nquantification. Another way to think of it is this: ∃x Brother(Richard, x) is a sentence\nabout Richard (that he has a brother), not about x; so putting a ∀x outside it has no effect. It\ncould equally well have been written ∃z Brother(Richard, z). Because this can be a source\nof confusion, we will always use different variable names with nested quantifiers.\nConnections between ∀and ∃\nThe two quantifiers are actually intimately connected with each other, through negation. As-\nserting that everyone dislikes parsnips is the same as asserting there does not exist someone\nwho likes them, and vice versa:\n∀x ¬Likes(x, Parsnips)\nis equivalent to\n¬∃x Likes(x, Parsnips) .\nWe can go one step further: “Everyone likes ice cream” means that there is no one who does\nnot like ice cream:\n∀x Likes(x, IceCream)\nis equivalent to\n¬∃x ¬Likes(x, IceCream) .\nBecause ∀is really a conjunction over the universe of objects and ∃is a disjunction, it should\nnot be surprising that they obey De Morgan’s rules. The De Morgan rules for quantified and\nunquantified sentences are as follows:\n∀x ¬P\n≡¬∃x P\n¬(P ∨Q) ≡¬P ∧¬Q\n¬∀x P\n≡∃x ¬P\n¬(P ∧Q) ≡¬P ∨¬Q\n∀x P\n≡¬∃x ¬P\nP ∧Q\n≡¬(¬P ∨¬Q)\n∃x P\n≡¬∀x ¬P\nP ∨Q\n≡¬(¬P ∧¬Q) .\nThus, we do not really need both ∀and ∃, just as we do not really need both ∧and ∨. Still,\nreadability is more important than parsimony, so we will keep both of the quantifiers.\nSection 8.2.\nSyntax and Semantics of First-Order Logic\n299\n8.2.7\nEquality\nFirst-order logic includes one more way to make atomic sentences, other than using a predi-\ncate and terms as described earlier. We can use the equality symbol to signify that two terms\nEQUALITY SYMBOL\nrefer to the same object. For example,\nFather(John) = Henry\nsays that the object referred to by Father(John) and the object referred to by Henry are the\nsame. Because an interpretation fixes the referent of any term, determining the truth of an\nequality sentence is simply a matter of seeing that the referents of the two terms are the same\nobject.\nThe equality symbol can be used to state facts about a given function, as we just did for\nthe Father symbol. It can also be used with negation to insist that two terms are not the same\nobject. To say that Richard has at least two brothers, we would write\n∃x, y Brother(x, Richard) ∧Brother(y, Richard) ∧¬(x = y) .\nThe sentence\n∃x, y Brother(x, Richard) ∧Brother(y, Richard)\ndoes not have the intended meaning. In particular, it is true in the model of Figure 8.2, where\nRichard has only one brother. To see this, consider the extended interpretation in which both\nx and y are assigned to King John. The addition of ¬(x = y) rules out such models. The\nnotation x ̸= y is sometimes used as an abbreviation for ¬(x = y).\n8.2.8\nAn alternative semantics?\nContinuing the example from the previous section, suppose that we believe that Richard has\ntwo brothers, John and Geoffrey.8 Can we capture this state of affairs by asserting\nBrother(John, Richard) ∧Brother(Geoffrey, Richard) ?\n(8.3)\nNot quite. First, this assertion is true in a model where Richard has only one brother—\nwe need to add John ̸= Geoffrey. Second, the sentence doesn’t rule out models in which\nRichard has many more brothers besides John and Geoffrey. Thus, the correct translation of\n“Richard’s brothers are John and Geoffrey” is as follows:\nBrother(John, Richard) ∧Brother(Geoffrey, Richard) ∧John ̸= Geoffrey\n∧∀x Brother(x, Richard) ⇒(x = John ∨x = Geoffrey) .\nFor many purposes, this seems much more cumbersome than the corresponding natural-\nlanguage expression. As a consequence, humans may make mistakes in translating their\nknowledge into first-order logic, resulting in unintuitive behaviors from logical reasoning\nsystems that use the knowledge. Can we devise a semantics that allows a more straightfor-\nward logical expression?\nOne proposal that is very popular in database systems works as follows. First, we insist\nthat every constant symbol refer to a distinct object—the so-called unique-names assump-\ntion. Second, we assume that atomic sentences not known to be true are in fact false—the\nUNIQUE-NAMES\nASSUMPTION\nclosed-world assumption. Finally, we invoke domain closure, meaning that each model\nCLOSED-WORLD\nASSUMPTION\nDOMAIN CLOSURE\n8 Actually he had four, the others being William and Henry.\n300\nChapter\n8.\nFirst-Order Logic\n. . .\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nR\nJ\nFigure 8.5\nSome members of the set of all models for a language with two constant sym-\nbols, R and J, and one binary relation symbol, under database semantics. The interpretation\nof the constant symbols is fixed, and there is a distinct object for each constant symbol.\ncontains no more domain elements than those named by the constant symbols. Under the\nresulting semantics, which we call database semantics to distinguish it from the standard\nDATABASE\nSEMANTICS\nsemantics of first-order logic, the sentence Equation (8.3) does indeed state that Richard’s\ntwo brothers are John and Geoffrey. Database semantics is also used in logic programming\nsystems, as explained in Section 9.4.5.\nIt is instructive to consider the set of all possible models under database semantics for\nthe same case as shown in Figure 8.4. Figure 8.5 shows some of the models, ranging from\nthe model with no tuples satisfying the relation to the model with all tuples satisfying the\nrelation. With two objects, there are four possible two-element tuples, so there are 24 = 16\ndifferent subsets of tuples that can satisfy the relation. Thus, there are 16 possible models in\nall—a lot fewer than the infinitely many models for the standard first-order semantics. On the\nother hand, the database semantics requires definite knowledge of what the world contains.\nThis example brings up an important point: there is no one “correct” semantics for\nlogic. The usefulness of any proposed semantics depends on how concise and intuitive it\nmakes the expression of the kinds of knowledge we want to write down, and on how easy\nand natural it is to develop the corresponding rules of inference. Database semantics is most\nuseful when we are certain about the identity of all the objects described in the knowledge\nbase and when we have all the facts at hand; in other cases, it is quite awkward. For the rest\nof this chapter, we assume the standard semantics while noting instances in which this choice\nleads to cumbersome expressions.\n8.3\nUSING FIRST-ORDER LOGIC\nNow that we have defined an expressive logical language, it is time to learn how to use it. The\nbest way to do this is through examples. We have seen some simple sentences illustrating the\nvarious aspects of logical syntax; in this section, we provide more systematic representations\nof some simple domains. In knowledge representation, a domain is just some part of the\nDOMAIN\nworld about which we wish to express some knowledge.\nWe begin with a brief description of the TELL/ASK interface for first-order knowledge\nbases. Then we look at the domains of family relationships, numbers, sets, and lists, and at\nSection 8.3.\nUsing First-Order Logic\n301\nthe wumpus world. The next section contains a more substantial example (electronic circuits)\nand Chapter 12 covers everything in the universe.\n8.3.1\nAssertions and queries in first-order logic\nSentences are added to a knowledge base using TELL, exactly as in propositional logic. Such\nsentences are called assertions. For example, we can assert that John is a king, Richard is a\nASSERTION\nperson, and all kings are persons:\nTELL(KB, King(John)) .\nTELL(KB, Person(Richard)) .\nTELL(KB, ∀x King(x) ⇒Person(x)) .\nWe can ask questions of the knowledge base using ASK. For example,\nASK(KB, King(John))\nreturns true. Questions asked with ASK are called queries or goals. Generally speaking, any\nQUERY\nGOAL\nquery that is logically entailed by the knowledge base should be answered affirmatively. For\nexample, given the two preceding assertions, the query\nASK(KB, Person(John))\nshould also return true. We can ask quantified queries, such as\nASK(KB, ∃x Person(x)) .\nThe answer is true, but this is perhaps not as helpful as we would like. It is rather like\nanswering “Can you tell me the time?” with “Yes.” If we want to know what value of x\nmakes the sentence true, we will need a different function, ASKVARS, which we call with\nASKVARS(KB, Person(x))\nand which yields a stream of answers. In this case there will be two answers: {x/John} and\n{x/Richard}. Such an answer is called a substitution or binding list. ASKVARS is usually\nSUBSTITUTION\nBINDING LIST\nreserved for knowledge bases consisting solely of Horn clauses, because in such knowledge\nbases every way of making the query true will bind the variables to specific values. That is\nnot the case with first-order logic; if KB has been told King(John) ∨King(Richard), then\nthere is no binding to x for the query ∃x King(x), even though the query is true.\n8.3.2\nThe kinship domain\nThe first example we consider is the domain of family relationships, or kinship. This domain\nincludes facts such as “Elizabeth is the mother of Charles” and “Charles is the father of\nWilliam” and rules such as “One’s grandmother is the mother of one’s parent.”\nClearly, the objects in our domain are people. We have two unary predicates, Male and\nFemale. Kinship relations—parenthood, brotherhood, marriage, and so on—are represented\nby binary predicates: Parent, Sibling, Brother, Sister, Child, Daughter, Son, Spouse,\nWife, Husband, Grandparent, Grandchild, Cousin, Aunt, and Uncle. We use functions\nfor Mother and Father, because every person has exactly one of each of these (at least\naccording to nature’s design).\n302\nChapter\n8.\nFirst-Order Logic\nWe can go through each function and predicate, writing down what we know in terms\nof the other symbols. For example, one’s mother is one’s female parent:\n∀m, c Mother(c) = m ⇔Female(m) ∧Parent(m, c) .\nOne’s husband is one’s male spouse:\n∀w, h Husband(h, w) ⇔Male(h) ∧Spouse(h, w) .\nMale and female are disjoint categories:\n∀x Male(x) ⇔¬Female(x) .\nParent and child are inverse relations:\n∀p, c Parent(p, c) ⇔Child(c, p) .\nA grandparent is a parent of one’s parent:\n∀g, c Grandparent(g, c) ⇔∃p Parent(g, p) ∧Parent(p, c) .\nA sibling is another child of one’s parents:\n∀x, y Sibling(x, y) ⇔x ̸= y ∧∃p Parent(p, x) ∧Parent(p, y) .\nWe could go on for several more pages like this, and Exercise 8.14 asks you to do just that.\nEach of these sentences can be viewed as an axiom of the kinship domain, as explained\nin Section 7.1. Axioms are commonly associated with purely mathematical domains—we\nwill see some axioms for numbers shortly—but they are needed in all domains. They provide\nthe basic factual information from which useful conclusions can be derived. Our kinship\naxioms are also definitions; they have the form ∀x, y P(x, y) ⇔\n. . .. The axioms define\nDEFINITION\nthe Mother function and the Husband, Male, Parent, Grandparent, and Sibling predicates\nin terms of other predicates. Our definitions “bottom out” at a basic set of predicates (Child,\nSpouse, and Female) in terms of which the others are ultimately defined. This is a natural\nway in which to build up the representation of a domain, and it is analogous to the way in\nwhich software packages are built up by successive definitions of subroutines from primitive\nlibrary functions. Notice that there is not necessarily a unique set of primitive predicates;\nwe could equally well have used Parent, Spouse, and Male. In some domains, as we show,\nthere is no clearly identifiable basic set.\nNot all logical sentences about a domain are axioms. Some are theorems—that is, they\nTHEOREM\nare entailed by the axioms. For example, consider the assertion that siblinghood is symmetric:\n∀x, y Sibling(x, y) ⇔Sibling(y, x) .\nIs this an axiom or a theorem? In fact, it is a theorem that follows logically from the axiom\nthat defines siblinghood. If we ASK the knowledge base this sentence, it should return true.\nFrom a purely logical point of view, a knowledge base need contain only axioms and\nno theorems, because the theorems do not increase the set of conclusions that follow from\nthe knowledge base. From a practical point of view, theorems are essential to reduce the\ncomputational cost of deriving new sentences. Without them, a reasoning system has to start\nfrom first principles every time, rather like a physicist having to rederive the rules of calculus\nfor every new problem.\nSection 8.3.\nUsing First-Order Logic\n303\nNot all axioms are definitions. Some provide more general information about certain\npredicates without constituting a definition. Indeed, some predicates have no complete defi-\nnition because we do not know enough to characterize them fully. For example, there is no\nobvious definitive way to complete the sentence\n∀x Person(x) ⇔. . .\nFortunately, first-order logic allows us to make use of the Person predicate without com-\npletely defining it. Instead, we can write partial specifications of properties that every person\nhas and properties that make something a person:\n∀x Person(x) ⇒. . .\n∀x . . . ⇒Person(x) .\nAxioms can also be “just plain facts,” such as Male(Jim) and Spouse(Jim, Laura).\nSuch facts form the descriptions of specific problem instances, enabling specific questions\nto be answered. The answers to these questions will then be theorems that follow from\nthe axioms. Often, one finds that the expected answers are not forthcoming—for example,\nfrom Spouse(Jim, Laura) one expects (under the laws of many countries) to be able to infer\n¬Spouse(George, Laura); but this does not follow from the axioms given earlier—even after\nwe add Jim ̸= George as suggested in Section 8.2.8. This is a sign that an axiom is missing.\nExercise 8.8 asks the reader to supply it.\n8.3.3\nNumbers, sets, and lists\nNumbers are perhaps the most vivid example of how a large theory can be built up from\na tiny kernel of axioms. We describe here the theory of natural numbers or non-negative\nNATURAL NUMBERS\nintegers. We need a predicate NatNum that will be true of natural numbers; we need one\nconstant symbol, 0; and we need one function symbol, S (successor). The Peano axioms\nPEANO AXIOMS\ndefine natural numbers and addition.9 Natural numbers are defined recursively:\nNatNum(0) .\n∀n NatNum(n) ⇒NatNum(S(n)) .\nThat is, 0 is a natural number, and for every object n, if n is a natural number, then S(n) is\na natural number. So the natural numbers are 0, S(0), S(S(0)), and so on. (After reading\nSection 8.2.8, you will notice that these axioms allow for other natural numbers besides the\nusual ones; see Exercise 8.12.) We also need axioms to constrain the successor function:\n∀n 0 ̸= S(n) .\n∀m, n m ̸= n ⇒S(m) ̸= S(n) .\nNow we can define addition in terms of the successor function:\n∀m NatNum(m) ⇒+ (0, m) = m .\n∀m, n NatNum(m) ∧NatNum(n) ⇒+ (S(m), n) = S(+(m, n)) .\nThe first of these axioms says that adding 0 to any natural number m gives m itself. Notice\nthe use of the binary function symbol “+” in the term +(m, 0); in ordinary mathematics, the\nterm would be written m + 0 using infix notation. (The notation we have used for first-order\nINFIX\n9 The Peano axioms also include the principle of induction, which is a sentence of second-order logic rather\nthan of first-order logic. The importance of this distinction is explained in Chapter 9.\n304\nChapter\n8.\nFirst-Order Logic\nlogic is called prefix.) To make our sentences about numbers easier to read, we allow the use\nPREFIX\nof infix notation. We can also write S(n) as n + 1, so the second axiom becomes\n∀m, n NatNum(m) ∧NatNum(n) ⇒(m + 1) + n = (m + n) + 1 .\nThis axiom reduces addition to repeated application of the successor function.\nThe use of infix notation is an example of syntactic sugar, that is, an extension to or\nSYNTACTIC SUGAR\nabbreviation of the standard syntax that does not change the semantics. Any sentence that\nuses sugar can be “desugared” to produce an equivalent sentence in ordinary first-order logic.\nOnce we have addition, it is straightforward to define multiplication as repeated addi-\ntion, exponentiation as repeated multiplication, integer division and remainders, prime num-\nbers, and so on. Thus, the whole of number theory (including cryptography) can be built up\nfrom one constant, one function, one predicate and four axioms.\nThe domain of sets is also fundamental to mathematics as well as to commonsense\nSET\nreasoning. (In fact, it is possible to define number theory in terms of set theory.) We want to\nbe able to represent individual sets, including the empty set. We need a way to build up sets\nby adding an element to a set or taking the union or intersection of two sets. We will want\nto know whether an element is a member of a set and we will want to distinguish sets from\nobjects that are not sets.\nWe will use the normal vocabulary of set theory as syntactic sugar. The empty set is a\nconstant written as { }. There is one unary predicate, Set, which is true of sets. The binary\npredicates are x ∈s (x is a member of set s) and s1 ⊆s2 (set s1 is a subset, not necessarily\nproper, of set s2). The binary functions are s1 ∩s2 (the intersection of two sets), s1 ∪s2\n(the union of two sets), and {x|s} (the set resulting from adjoining element x to set s). One\npossible set of axioms is as follows:\n1. The only sets are the empty set and those made by adjoining something to a set:\n∀s Set(s) ⇔(s = { }) ∨(∃x, s2 Set(s2) ∧s = {x|s2}) .\n2. The empty set has no elements adjoined into it. In other words, there is no way to\ndecompose { } into a smaller set and an element:\n¬∃x, s {x|s} = { } .\n3. Adjoining an element already in the set has no effect:\n∀x, s x ∈s ⇔s = {x|s} .\n4. The only members of a set are the elements that were adjoined into it. We express\nthis recursively, saying that x is a member of s if and only if s is equal to some set s2\nadjoined with some element y, where either y is the same as x or x is a member of s2:\n∀x, s x ∈s ⇔∃y, s2 (s = {y|s2} ∧(x = y ∨x ∈s2)) .\n5. A set is a subset of another set if and only if all of the first set’s members are members\nof the second set:\n∀s1, s2 s1 ⊆s2 ⇔(∀x x ∈s1 ⇒x ∈s2) .\n6. Two sets are equal if and only if each is a subset of the other:\n∀s1, s2 (s1 = s2) ⇔(s1 ⊆s2 ∧s2 ⊆s1) .\nSection 8.3.\nUsing First-Order Logic\n305\n7. An object is in the intersection of two sets if and only if it is a member of both sets:\n∀x, s1, s2 x ∈(s1 ∩s2) ⇔(x ∈s1 ∧x ∈s2) .\n8. An object is in the union of two sets if and only if it is a member of either set:\n∀x, s1, s2 x ∈(s1 ∪s2) ⇔(x ∈s1 ∨x ∈s2) .\nLists are similar to sets. The differences are that lists are ordered and the same element can\nLIST\nappear more than once in a list. We can use the vocabulary of Lisp for lists: Nil is the constant\nlist with no elements; Cons, Append, First, and Rest are functions; and Find is the pred-\nicate that does for lists what Member does for sets. List? is a predicate that is true only of\nlists. As with sets, it is common to use syntactic sugar in logical sentences involving lists. The\nempty list is [ ]. The term Cons(x, y), where y is a nonempty list, is written [x|y]. The term\nCons(x, Nil) (i.e., the list containing the element x) is written as [x]. A list of several ele-\nments, such as [A, B, C], corresponds to the nested term Cons(A, Cons(B, Cons(C, Nil))).\nExercise 8.16 asks you to write out the axioms for lists.\n8.3.4\nThe wumpus world\nSome propositional logic axioms for the wumpus world were given in Chapter 7. The first-\norder axioms in this section are much more concise, capturing in a natural way exactly what\nwe want to say.\nRecall that the wumpus agent receives a percept vector with five elements. The corre-\nsponding first-order sentence stored in the knowledge base must include both the percept and\nthe time at which it occurred; otherwise, the agent will get confused about when it saw what.\nWe use integers for time steps. A typical percept sentence would be\nPercept([Stench, Breeze, Glitter, None, None], 5) .\nHere, Percept is a binary predicate, and Stench and so on are constants placed in a list. The\nactions in the wumpus world can be represented by logical terms:\nTurn(Right), Turn(Left), Forward, Shoot, Grab, Climb .\nTo determine which is best, the agent program executes the query\nASKVARS(∃a BestAction(a, 5)) ,\nwhich returns a binding list such as {a/Grab}. The agent program can then return Grab as\nthe action to take. The raw percept data implies certain facts about the current state. For\nexample:\n∀t, s, g, m, c Percept([s, Breeze, g, m, c], t) ⇒Breeze(t) ,\n∀t, s, b, m, c Percept([s, b, Glitter, m, c], t) ⇒Glitter(t) ,\nand so on. These rules exhibit a trivial form of the reasoning process called perception, which\nwe study in depth in Chapter 24. Notice the quantification over time t. In propositional logic,\nwe would need copies of each sentence for each time step.\nSimple “reflex” behavior can also be implemented by quantified implication sentences.\nFor example, we have\n∀t Glitter(t) ⇒BestAction(Grab, t) .\n306\nChapter\n8.\nFirst-Order Logic\nGiven the percept and rules from the preceding paragraphs, this would yield the desired con-\nclusion BestAction(Grab, 5)—that is, Grab is the right thing to do.\nWe have represented the agent’s inputs and outputs; now it is time to represent the\nenvironment itself. Let us begin with objects. Obvious candidates are squares, pits, and the\nwumpus. We could name each square—Square1,2 and so on—but then the fact that Square1,2\nand Square1,3 are adjacent would have to be an “extra” fact, and we would need one such\nfact for each pair of squares. It is better to use a complex term in which the row and column\nappear as integers; for example, we can simply use the list term [1, 2]. Adjacency of any two\nsquares can be defined as\n∀x, y, a, b Adjacent([x, y], [a, b]) ⇔\n(x = a ∧(y = b −1 ∨y = b + 1)) ∨(y = b ∧(x = a −1 ∨x = a + 1)) .\nWe could name each pit, but this would be inappropriate for a different reason: there is no\nreason to distinguish among pits.10 It is simpler to use a unary predicate Pit that is true of\nsquares containing pits. Finally, since there is exactly one wumpus, a constant Wumpus is\njust as good as a unary predicate (and perhaps more dignified from the wumpus’s viewpoint).\nThe agent’s location changes over time, so we write At(Agent, s, t) to mean that the\nagent is at square s at time t. We can fix the wumpus’s location with ∀t At(Wumpus, [2, 2], t).\nWe can then say that objects can only be at one location at a time:\n∀x, s1, s2, t At(x, s1, t) ∧At(x, s2, t) ⇒s1 = s2 .\nGiven its current location, the agent can infer properties of the square from properties of its\ncurrent percept. For example, if the agent is at a square and perceives a breeze, then that\nsquare is breezy:\n∀s, t At(Agent, s, t) ∧Breeze(t) ⇒Breezy(s) .\nIt is useful to know that a square is breezy because we know that the pits cannot move about.\nNotice that Breezy has no time argument.\nHaving discovered which places are breezy (or smelly) and, very important, not breezy\n(or not smelly), the agent can deduce where the pits are (and where the wumpus is). Whereas\npropositional logic necessitates a separate axiom for each square (see R2 and R3 on page 247)\nand would need a different set of axioms for each geographical layout of the world, first-order\nlogic just needs one axiom:\n∀s Breezy(s) ⇔∃r Adjacent(r, s) ∧Pit(r) .\n(8.4)\nSimilarly, in first-order logic we can quantify over time, so we need just one successor-state\naxiom for each predicate, rather than a different copy for each time step. For example, the\naxiom for the arrow (Equation (7.2) on page 267) becomes\n∀t HaveArrow(t + 1) ⇔(HaveArrow(t) ∧¬Action(Shoot, t)) .\nFrom these two example sentences, we can see that the first-order logic formulation is no\nless concise than the original English-language description given in Chapter 7. The reader\n10 Similarly, most of us do not name each bird that flies overhead as it migrates to warmer regions in winter. An\nornithologist wishing to study migration patterns, survival rates, and so on does name each bird, by means of a\nring on its leg, because individual birds must be tracked.\nSection 8.4.\nKnowledge Engineering in First-Order Logic\n307\nis invited to construct analogous axioms for the agent’s location and orientation; in these\ncases, the axioms quantify over both space and time. As in the case of propositional state\nestimation, an agent can use logical inference with axioms of this kind to keep track of aspects\nof the world that are not directly observed. Chapter 10 goes into more depth on the subject of\nfirst-order successor-state axioms and their uses for constructing plans.\n8.4\nKNOWLEDGE ENGINEERING IN FIRST-ORDER LOGIC\nThe preceding section illustrated the use of first-order logic to represent knowledge in three\nsimple domains. This section describes the general process of knowledge-base construction—\na process called knowledge engineering. A knowledge engineer is someone who investigates\nKNOWLEDGE\nENGINEERING\na particular domain, learns what concepts are important in that domain, and creates a formal\nrepresentation of the objects and relations in the domain. We illustrate the knowledge engi-\nneering process in an electronic circuit domain that should already be fairly familiar, so that\nwe can concentrate on the representational issues involved. The approach we take is suitable\nfor developing special-purpose knowledge bases whose domain is carefully circumscribed\nand whose range of queries is known in advance. General-purpose knowledge bases, which\ncover a broad range of human knowledge and are intended to support tasks such as natural\nlanguage understanding, are discussed in Chapter 12.\n8.4.1\nThe knowledge-engineering process\nKnowledge engineering projects vary widely in content, scope, and difficulty, but all such\nprojects include the following steps:\n1. Identify the task. The knowledge engineer must delineate the range of questions that\nthe knowledge base will support and the kinds of facts that will be available for each\nspecific problem instance. For example, does the wumpus knowledge base need to be\nable to choose actions or is it required to answer questions only about the contents\nof the environment? Will the sensor facts include the current location? The task will\ndetermine what knowledge must be represented in order to connect problem instances to\nanswers. This step is analogous to the PEAS process for designing agents in Chapter 2.\n2. Assemble the relevant knowledge. The knowledge engineer might already be an expert\nin the domain, or might need to work with real experts to extract what they know—a\nprocess called knowledge acquisition. At this stage, the knowledge is not represented\nKNOWLEDGE\nACQUISITION\nformally. The idea is to understand the scope of the knowledge base, as determined by\nthe task, and to understand how the domain actually works.\nFor the wumpus world, which is defined by an artificial set of rules, the relevant\nknowledge is easy to identify. (Notice, however, that the definition of adjacency was\nnot supplied explicitly in the wumpus-world rules.) For real domains, the issue of\nrelevance can be quite difficult—for example, a system for simulating VLSI designs\nmight or might not need to take into account stray capacitances and skin effects.\n308\nChapter\n8.\nFirst-Order Logic\n3. Decide on a vocabulary of predicates, functions, and constants. That is, translate the\nimportant domain-level concepts into logic-level names. This involves many questions\nof knowledge-engineering style. Like programming style, this can have a significant\nimpact on the eventual success of the project. For example, should pits be represented\nby objects or by a unary predicate on squares? Should the agent’s orientation be a\nfunction or a predicate? Should the wumpus’s location depend on time? Once the\nchoices have been made, the result is a vocabulary that is known as the ontology of\nONTOLOGY\nthe domain. The word ontology means a particular theory of the nature of being or\nexistence. The ontology determines what kinds of things exist, but does not determine\ntheir specific properties and interrelationships.\n4. Encode general knowledge about the domain. The knowledge engineer writes down\nthe axioms for all the vocabulary terms. This pins down (to the extent possible) the\nmeaning of the terms, enabling the expert to check the content. Often, this step reveals\nmisconceptions or gaps in the vocabulary that must be fixed by returning to step 3 and\niterating through the process.\n5. Encode a description of the specific problem instance. If the ontology is well thought\nout, this step will be easy. It will involve writing simple atomic sentences about in-\nstances of concepts that are already part of the ontology. For a logical agent, problem\ninstances are supplied by the sensors, whereas a “disembodied” knowledge base is sup-\nplied with additional sentences in the same way that traditional programs are supplied\nwith input data.\n6. Pose queries to the inference procedure and get answers. This is where the reward is:\nwe can let the inference procedure operate on the axioms and problem-specific facts to\nderive the facts we are interested in knowing. Thus, we avoid the need for writing an\napplication-specific solution algorithm.\n7. Debug the knowledge base. Alas, the answers to queries will seldom be correct on\nthe first try. More precisely, the answers will be correct for the knowledge base as\nwritten, assuming that the inference procedure is sound, but they will not be the ones\nthat the user is expecting. For example, if an axiom is missing, some queries will not be\nanswerable from the knowledge base. A considerable debugging process could ensue.\nMissing axioms or axioms that are too weak can be easily identified by noticing places\nwhere the chain of reasoning stops unexpectedly. For example, if the knowledge base\nincludes a diagnostic rule (see Exercise 8.13) for finding the wumpus,\n∀s Smelly(s) ⇒Adjacent(Home(Wumpus), s) ,\ninstead of the biconditional, then the agent will never be able to prove the absence of\nwumpuses. Incorrect axioms can be identified because they are false statements about\nthe world. For example, the sentence\n∀x NumOfLegs(x, 4) ⇒Mammal(x)\nis false for reptiles, amphibians, and, more importantly, tables. The falsehood of this\nsentence can be determined independently of the rest of the knowledge base. In contrast,\nSection 8.4.\nKnowledge Engineering in First-Order Logic\n309\na typical error in a program looks like this:\noffset = position + 1 .\nIt is impossible to tell whether this statement is correct without looking at the rest of the\nprogram to see whether, for example, offset is used to refer to the current position,\nor to one beyond the current position, or whether the value of position is changed\nby another statement and so offset should also be changed again.\nTo understand this seven-step process better, we now apply it to an extended example—the\ndomain of electronic circuits.\n8.4.2\nThe electronic circuits domain\nWe will develop an ontology and knowledge base that allow us to reason about digital circuits\nof the kind shown in Figure 8.6. We follow the seven-step process for knowledge engineering.\nIdentify the task\nThere are many reasoning tasks associated with digital circuits. At the highest level, one\nanalyzes the circuit’s functionality. For example, does the circuit in Figure 8.6 actually add\nproperly? If all the inputs are high, what is the output of gate A2? Questions about the\ncircuit’s structure are also interesting. For example, what are all the gates connected to the\nfirst input terminal? Does the circuit contain feedback loops? These will be our tasks in this\nsection. There are more detailed levels of analysis, including those related to timing delays,\ncircuit area, power consumption, production cost, and so on. Each of these levels would\nrequire additional knowledge.\nAssemble the relevant knowledge\nWhat do we know about digital circuits? For our purposes, they are composed of wires and\ngates. Signals flow along wires to the input terminals of gates, and each gate produces a\n1\n2\n3\n1\n2\nX1\nX2\nA1\nA2\nO1\nC1\nFigure 8.6\nA digital circuit C1, purporting to be a one-bit full adder. The first two inputs\nare the two bits to be added, and the third input is a carry bit. The first output is the sum, and\nthe second output is a carry bit for the next adder. The circuit contains two XOR gates, two\nAND gates, and one OR gate.\n310\nChapter\n8.\nFirst-Order Logic\nsignal on the output terminal that flows along another wire. To determine what these signals\nwill be, we need to know how the gates transform their input signals. There are four types\nof gates: AND, OR, and XOR gates have two input terminals, and NOT gates have one. All\ngates have one output terminal. Circuits, like gates, have input and output terminals.\nTo reason about functionality and connectivity, we do not need to talk about the wires\nthemselves, the paths they take, or the junctions where they come together. All that matters\nis the connections between terminals—we can say that one output terminal is connected to\nanother input terminal without having to say what actually connects them. Other factors such\nas the size, shape, color, or cost of the various components are irrelevant to our analysis.\nIf our purpose were something other than verifying designs at the gate level, the ontol-\nogy would be different. For example, if we were interested in debugging faulty circuits, then\nit would probably be a good idea to include the wires in the ontology, because a faulty wire\ncan corrupt the signal flowing along it. For resolving timing faults, we would need to include\ngate delays. If we were interested in designing a product that would be profitable, then the\ncost of the circuit and its speed relative to other products on the market would be important.\nDecide on a vocabulary\nWe now know that we want to talk about circuits, terminals, signals, and gates. The next step\nis to choose functions, predicates, and constants to represent them. First, we need to be able\nto distinguish gates from each other and from other objects. Each gate is represented as an\nobject named by a constant, about which we assert that it is a gate with, say, Gate(X1). The\nbehavior of each gate is determined by its type: one of the constants AND, OR, XOR, or\nNOT. Because a gate has exactly one type, a function is appropriate: Type(X1) = XOR.\nCircuits, like gates, are identified by a predicate: Circuit(C1).\nNext we consider terminals, which are identified by the predicate Terminal(x). A gate\nor circuit can have one or more input terminals and one or more output terminals. We use the\nfunction In(1, X1) to denote the first input terminal for gate X1. A similar function Out is\nused for output terminals. The function Arity(c, i, j) says that circuit c has i input and j out-\nput terminals. The connectivity between gates can be represented by a predicate, Connected,\nwhich takes two terminals as arguments, as in Connected(Out(1, X1), In(1, X2)).\nFinally, we need to know whether a signal is on or off. One possibility is to use a unary\npredicate, On(t), which is true when the signal at a terminal is on. This makes it a little\ndifficult, however, to pose questions such as “What are all the possible values of the signals\nat the output terminals of circuit C1 ?” We therefore introduce as objects two signal values, 1\nand 0, and a function Signal(t) that denotes the signal value for the terminal t.\nEncode general knowledge of the domain\nOne sign that we have a good ontology is that we require only a few general rules, which can\nbe stated clearly and concisely. These are all the axioms we will need:\n1. If two terminals are connected, then they have the same signal:\n∀t1, t2 Terminal(t1) ∧Terminal(t2) ∧Connected(t1, t2) ⇒\nSignal(t1) = Signal(t2) .\nSection 8.4.\nKnowledge Engineering in First-Order Logic\n311\n2. The signal at every terminal is either 1 or 0:\n∀t Terminal(t) ⇒Signal(t) = 1 ∨Signal(t) = 0 .\n3. Connected is commutative:\n∀t1, t2 Connected(t1, t2) ⇔Connected(t2, t1) .\n4. There are four types of gates:\n∀g Gate(g) ∧k = Type(g) ⇒k = AND ∨k = OR ∨k = XOR ∨k = NOT .\n5. An AND gate’s output is 0 if and only if any of its inputs is 0:\n∀g Gate(g) ∧Type(g) = AND ⇒\nSignal(Out(1, g)) = 0 ⇔∃n Signal(In(n, g)) = 0 .\n6. An OR gate’s output is 1 if and only if any of its inputs is 1:\n∀g Gate(g) ∧Type(g) = OR ⇒\nSignal(Out(1, g)) = 1 ⇔∃n Signal(In(n, g)) = 1 .\n7. An XOR gate’s output is 1 if and only if its inputs are different:\n∀g Gate(g) ∧Type(g) = XOR ⇒\nSignal(Out(1, g)) = 1 ⇔Signal(In(1, g)) ̸= Signal(In(2, g)) .\n8. A NOT gate’s output is different from its input:\n∀g Gate(g) ∧(Type(g) = NOT) ⇒\nSignal(Out(1, g)) ̸= Signal(In(1, g)) .\n9. The gates (except for NOT) have two inputs and one output.\n∀g Gate(g) ∧Type(g) = NOT ⇒Arity(g, 1, 1) .\n∀g Gate(g) ∧k = Type(g) ∧(k = AND ∨k = OR ∨k = XOR) ⇒\nArity(g, 2, 1)\n10. A circuit has terminals, up to its input and output arity, and nothing beyond its arity:\n∀c, i, j Circuit(c) ∧Arity(c, i, j) ⇒\n∀n (n ≤i ⇒Terminal(In(c, n))) ∧(n > i ⇒In(c, n) = Nothing) ∧\n∀n (n ≤j ⇒Terminal(Out(c, n))) ∧(n > j ⇒Out(c, n) = Nothing)\n11. Gates, terminals, signals, gate types, and Nothing are all distinct.\n∀g, t Gate(g) ∧Terminal(t) ⇒\ng ̸= t ̸= 1 ̸= 0 ̸= OR ̸= AND ̸= XOR ̸= NOT ̸= Nothing .\n12. Gates are circuits.\n∀g Gate(g) ⇒Circuit(g)\nEncode the specific problem instance\nThe circuit shown in Figure 8.6 is encoded as circuit C1 with the following description. First,\nwe categorize the circuit and its component gates:\nCircuit(C1) ∧Arity(C1, 3, 2)\nGate(X1) ∧Type(X1) = XOR\nGate(X2) ∧Type(X2) = XOR\nGate(A1) ∧Type(A1) = AND\nGate(A2) ∧Type(A2) = AND\nGate(O1) ∧Type(O1) = OR .\n312\nChapter\n8.\nFirst-Order Logic\nThen, we show the connections between them:\nConnected(Out(1, X1), In(1, X2))\nConnected(In(1, C1), In(1, X1))\nConnected(Out(1, X1), In(2, A2))\nConnected(In(1, C1), In(1, A1))\nConnected(Out(1, A2), In(1, O1))\nConnected(In(2, C1), In(2, X1))\nConnected(Out(1, A1), In(2, O1))\nConnected(In(2, C1), In(2, A1))\nConnected(Out(1, X2), Out(1, C1)) Connected(In(3, C1), In(2, X2))\nConnected(Out(1, O1), Out(2, C1)) Connected(In(3, C1), In(1, A2)) .\nPose queries to the inference procedure\nWhat combinations of inputs would cause the first output of C1 (the sum bit) to be 0 and the\nsecond output of C1 (the carry bit) to be 1?\n∃i1, i2, i3 Signal(In(1, C1)) = i1 ∧Signal(In(2, C1)) = i2 ∧Signal(In(3, C1)) = i3\n∧Signal(Out(1, C1)) = 0 ∧Signal(Out(2, C1)) = 1 .\nThe answers are substitutions for the variables i1, i2, and i3 such that the resulting sentence\nis entailed by the knowledge base. ASKVARS will give us three such substitutions:\n{i1/1, i2/1, i3/0}\n{i1/1, i2/0, i3/1}\n{i1/0, i2/1, i3/1} .\nWhat are the possible sets of values of all the terminals for the adder circuit?\n∃i1, i2, i3, o1, o2 Signal(In(1, C1)) = i1 ∧Signal(In(2, C1)) = i2\n∧Signal(In(3, C1)) = i3 ∧Signal(Out(1, C1)) = o1 ∧Signal(Out(2, C1)) = o2 .\nThis final query will return a complete input–output table for the device, which can be used\nto check that it does in fact add its inputs correctly. This is a simple example of circuit\nverification. We can also use the definition of the circuit to build larger digital systems, for\nCIRCUIT\nVERIFICATION\nwhich the same kind of verification procedure can be carried out. (See Exercise 8.26.) Many\ndomains are amenable to the same kind of structured knowledge-base development, in which\nmore complex concepts are defined on top of simpler concepts.\nDebug the knowledge base\nWe can perturb the knowledge base in various ways to see what kinds of erroneous behaviors\nemerge. For example, suppose we fail to read Section 8.2.8 and hence forget to assert that\n1 ̸= 0. Suddenly, the system will be unable to prove any outputs for the circuit, except for\nthe input cases 000 and 110. We can pinpoint the problem by asking for the outputs of each\ngate. For example, we can ask\n∃i1, i2, o Signal(In(1, C1)) = i1 ∧Signal(In(2, C1)) = i2 ∧Signal(Out(1, X1)) ,\nwhich reveals that no outputs are known at X1 for the input cases 10 and 01. Then, we look\nat the axiom for XOR gates, as applied to X1:\nSignal(Out(1, X1)) = 1 ⇔Signal(In(1, X1)) ̸= Signal(In(2, X1)) .\nIf the inputs are known to be, say, 1 and 0, then this reduces to\nSignal(Out(1, X1)) = 1 ⇔1 ̸= 0 .\nNow the problem is apparent: the system is unable to infer that Signal(Out(1, X1)) = 1, so\nwe need to tell it that 1 ̸= 0.\nSection 8.5.\nSummary\n313\n8.5\nSUMMARY\nThis chapter has introduced first-order logic, a representation language that is far more pow-\nerful than propositional logic. The important points are as follows:\n• Knowledge representation languages should be declarative, compositional, expressive,\ncontext independent, and unambiguous.\n• Logics differ in their ontological commitments and epistemological commitments.\nWhile propositional logic commits only to the existence of facts, first-order logic com-\nmits to the existence of objects and relations and thereby gains expressive power.\n• The syntax of first-order logic builds on that of propositional logic. It adds terms to\nrepresent objects, and has universal and existential quantifiers to construct assertions\nabout all or some of the possible values of the quantified variables.\n• A possible world, or model, for first-order logic includes a set of objects and an inter-\npretation that maps constant symbols to objects, predicate symbols to relations among\nobjects, and function symbols to functions on objects.\n• An atomic sentence is true just when the relation named by the predicate holds between\nthe objects named by the terms. Extended interpretations, which map quantifier vari-\nables to objects in the model, define the truth of quantified sentences.\n• Developing a knowledge base in first-order logic requires a careful process of analyzing\nthe domain, choosing a vocabulary, and encoding the axioms required to support the\ndesired inferences.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nAlthough Aristotle’s logic deals with generalizations over objects, it fell far short of the ex-\npressive power of first-order logic. A major barrier to its further development was its concen-\ntration on one-place predicates to the exclusion of many-place relational predicates. The first\nsystematic treatment of relations was given by Augustus De Morgan (1864), who cited the\nfollowing example to show the sorts of inferences that Aristotle’s logic could not handle: “All\nhorses are animals; therefore, the head of a horse is the head of an animal.” This inference\nis inaccessible to Aristotle because any valid rule that can support this inference must first\nanalyze the sentence using the two-place predicate “x is the head of y.” The logic of relations\nwas studied in depth by Charles Sanders Peirce (1870, 2004).\nTrue first-order logic dates from the introduction of quantifiers in Gottlob Frege’s (1879)\nBegriffschrift (“Concept Writing” or “Conceptual Notation”). Peirce (1883) also developed\nfirst-order logic independently of Frege, although slightly later. Frege’s ability to nest quan-\ntifiers was a big step forward, but he used an awkward notation. The present notation for\nfirst-order logic is due substantially to Giuseppe Peano (1889), but the semantics is virtually\nidentical to Frege’s. Oddly enough, Peano’s axioms were due in large measure to Grassmann\n(1861) and Dedekind (1888).\n314\nChapter\n8.\nFirst-Order Logic\nLeopold L¨owenheim (1915) gave a systematic treatment of model theory for first-order\nlogic, including the first proper treatment of the equality symbol. L¨owenheim’s results were\nfurther extended by Thoralf Skolem (1920). Alfred Tarski (1935, 1956) gave an explicit\ndefinition of truth and model-theoretic satisfaction in first-order logic, using set theory.\nMcCarthy (1958) was primarily responsible for the introduction of first-order logic as a\ntool for building AI systems. The prospects for logic-based AI were advanced significantly by\nRobinson’s (1965) development of resolution, a complete procedure for first-order inference\ndescribed in Chapter 9. The logicist approach took root at Stanford University. Cordell Green\n(1969a, 1969b) developed a first-order reasoning system, QA3, leading to the first attempts to\nbuild a logical robot at SRI (Fikes and Nilsson, 1971). First-order logic was applied by Zohar\nManna and Richard Waldinger (1971) for reasoning about programs and later by Michael\nGenesereth (1984) for reasoning about circuits. In Europe, logic programming (a restricted\nform of first-order reasoning) was developed for linguistic analysis (Colmerauer et al., 1973)\nand for general declarative systems (Kowalski, 1974). Computational logic was also well\nentrenched at Edinburgh through the LCF (Logic for Computable Functions) project (Gordon\net al., 1979). These developments are chronicled further in Chapters 9 and 12.\nPractical applications built with first-order logic include a system for evaluating the\nmanufacturing requirements for electronic products (Mannion, 2002), a system for reasoning\nabout policies for file access and digital rights management (Halpern and Weissman, 2008),\nand a system for the automated composition of Web services (McIlraith and Zeng, 2001).\nReactions to the Whorf hypothesis (Whorf, 1956) and the problem of language and\nthought in general, appear in several recent books (Gumperz and Levinson, 1996; Bowerman\nand Levinson, 2001; Pinker, 2003; Gentner and Goldin-Meadow, 2003). The “theory” theory\n(Gopnik and Glymour, 2002; Tenenbaum et al., 2007) views children’s learning about the\nworld as analogous to the construction of scientific theories. Just as the predictions of a\nmachine learning algorithm depend strongly on the vocabulary supplied to it, so will the\nchild’s formulation of theories depend on the linguistic environment in which learning occurs.\nThere are a number of good introductory texts on first-order logic, including some by\nleading figures in the history of logic: Alfred Tarski (1941), Alonzo Church (1956), and\nW.V. Quine (1982) (which is one of the most readable). Enderton (1972) gives a more math-\nematically oriented perspective. A highly formal treatment of first-order logic, along with\nmany more advanced topics in logic, is provided by Bell and Machover (1977). Manna and\nWaldinger (1985) give a readable introduction to logic from a computer science perspec-\ntive, as do Huth and Ryan (2004), who concentrate on program verification. Barwise and\nEtchemendy (2002) take an approach similar to the one used here. Smullyan (1995) presents\nresults concisely, using the tableau format. Gallier (1986) provides an extremely rigorous\nmathematical exposition of first-order logic, along with a great deal of material on its use in\nautomated reasoning. Logical Foundations of Artificial Intelligence (Genesereth and Nilsson,\n1987) is both a solid introduction to logic and the first systematic treatment of logical agents\nwith percepts and actions, and there are two good handbooks: van Bentham and ter Meulen\n(1997) and Robinson and Voronkov (2001). The journal of record for the field of pure math-\nematical logic is the Journal of Symbolic Logic, whereas the Journal of Applied Logic deals\nwith concerns closer to those of artificial intelligence.\nExercises\n315\nEXERCISES\n8.1\nA logical knowledge base represents the world using a set of sentences with no explicit\nstructure. An analogical representation, on the other hand, has physical structure that corre-\nsponds directly to the structure of the thing represented. Consider a road map of your country\nas an analogical representation of facts about the country—it represents facts with a map lan-\nguage. The two-dimensional structure of the map corresponds to the two-dimensional surface\nof the area.\na. Give five examples of symbols in the map language.\nb. An explicit sentence is a sentence that the creator of the representation actually writes\ndown. An implicit sentence is a sentence that results from explicit sentences because\nof properties of the analogical representation. Give three examples each of implicit and\nexplicit sentences in the map language.\nc. Give three examples of facts about the physical structure of your country that cannot be\nrepresented in the map language.\nd. Give two examples of facts that are much easier to express in the map language than in\nfirst-order logic.\ne. Give two other examples of useful analogical representations. What are the advantages\nand disadvantages of each of these languages?\n8.2\nConsider a knowledge base containing just two sentences: P(a) and P(b). Does this\nknowledge base entail ∀x P(x)? Explain your answer in terms of models.\n8.3\nIs the sentence ∃x, y x = y valid? Explain.\n8.4\nWrite down a logical sentence such that every world in which it is true contains exactly\none object.\n8.5\nConsider a symbol vocabulary that contains c constant symbols, pk predicate symbols\nof each arity k, and fk function symbols of each arity k, where 1 ≤k ≤A. Let the domain\nsize be fixed at D. For any given model, each predicate or function symbol is mapped onto a\nrelation or function, respectively, of the same arity. You may assume that the functions in the\nmodel allow some input tuples to have no value for the function (i.e., the value is the invisible\nobject). Derive a formula for the number of possible models for a domain with D elements.\nDon’t worry about eliminating redundant combinations.\n8.6\nWhich of the following are valid (necessarily true) sentences?\na. (∃x x = x) ⇒(∀y ∃z y = z).\nb. ∀x P(x) ∨¬P(x).\nc. ∀x Smart(x) ∨(x = x).\n8.7\nConsider a version of the semantics for first-order logic in which models with empty\ndomains are allowed. Give at least two examples of sentences that are valid according to the\n316\nChapter\n8.\nFirst-Order Logic\nstandard semantics but not according to the new semantics. Discuss which outcome makes\nmore intuitive sense for your examples.\n8.8\nDoes the fact ¬Spouse(George, Laura) follow from the facts Jim ̸= George and\nSpouse(Jim, Laura)? If so, give a proof; if not, supply additional axioms as needed. What\nhappens if we use Spouse as a unary function symbol instead of a binary predicate?\n8.9\nThis exercise uses the function MapColor and predicates In(x, y), Borders(x, y), and\nCountry(x), whose arguments are geographical regions, along with constant symbols for\nvarious regions. In each of the following we give an English sentence and a number of can-\ndidate logical expressions. For each of the logical expressions, state whether it (1) correctly\nexpresses the English sentence; (2) is syntactically invalid and therefore meaningless; or (3)\nis syntactically valid but does not express the meaning of the English sentence.\na. Paris and Marseilles are both in France.\n(i) In(Paris ∧Marseilles, France).\n(ii) In(Paris, France) ∧In(Marseilles, France).\n(iii) In(Paris, France) ∨In(Marseilles, France).\nb. There is a country that borders both Iraq and Pakistan.\n(i) ∃c Country(c) ∧Border(c, Iraq) ∧Border(c, Pakistan).\n(ii) ∃c Country(c) ⇒[Border(c, Iraq) ∧Border(c, Pakistan)].\n(iii) [∃c\nCountry(c)] ⇒[Border(c, Iraq) ∧Border(c, Pakistan)].\n(iv) ∃c Border(Country(c), Iraq ∧Pakistan).\nc. All countries that border Ecuador are in South America.\n(i) ∀c Country(c) ∧Border(c, Ecuador) ⇒In(c, SouthAmerica).\n(ii) ∀c Country(c) ⇒[Border(c, Ecuador) ⇒In(c, SouthAmerica)].\n(iii) ∀c [Country(c) ⇒Border(c, Ecuador)] ⇒In(c, SouthAmerica).\n(iv) ∀c Country(c) ∧Border(c, Ecuador) ∧In(c, SouthAmerica).\nd. No region in South America borders any region in Europe.\n(i) ¬[∃c, d In(c, SouthAmerica) ∧In(d, Europe) ∧Borders(c, d)].\n(ii) ∀c, d [In(c, SouthAmerica) ∧In(d, Europe)] ⇒¬Borders(c, d)].\n(iii) ¬∀c In(c, SouthAmerica) ⇒∃d In(d, Europe) ∧¬Borders(c, d).\n(iv) ∀c In(c, SouthAmerica) ⇒∀d In(d, Europe) ⇒¬Borders(c, d).\ne. No two adjacent countries have the same map color.\n(i) ∀x, y ¬Country(x) ∨¬Country(y) ∨¬Borders(x, y) ∨\n¬(MapColor(x) = MapColor(y)).\n(ii) ∀x, y (Country(x) ∧Country(y) ∧Borders(x, y) ∧¬(x = y)) ⇒\n¬(MapColor(x) = MapColor(y)).\n(iii) ∀x, y Country(x) ∧Country(y) ∧Borders(x, y) ∧\n¬(MapColor(x) = MapColor(y)).\n(iv) ∀x, y (Country(x) ∧Country(y) ∧Borders(x, y)) ⇒MapColor(x ̸= y).\nExercises\n317\n8.10\nConsider a vocabulary with the following symbols:\nOccupation(p, o): Predicate. Person p has occupation o.\nCustomer(p1, p2): Predicate. Person p1 is a customer of person p2.\nBoss(p1, p2): Predicate. Person p1 is a boss of person p2.\nDoctor, Surgeon, Lawyer, Actor: Constants denoting occupations.\nEmily, Joe: Constants denoting people.\nUse these symbols to write the following assertions in first-order logic:\na. Emily is either a surgeon or a lawyer.\nb. Joe is an actor, but he also holds another job.\nc. All surgeons are doctors.\nd. Joe does not have a lawyer (i.e., is not a customer of any lawyer).\ne. Emily has a boss who is a lawyer.\nf. There exists a lawyer all of whose customers are doctors.\ng. Every surgeon has a lawyer.\n8.11\nComplete the following exercises about logical senntences:\na. Translate into good, natural English (no xs or ys!):\n∀x, y, l SpeaksLanguage(x, l) ∧SpeaksLanguage(y, l)\n⇒Understands(x, y) ∧Understands(y, x).\nb. Explain why this sentence is entailed by the sentence\n∀x, y, l SpeaksLanguage(x, l) ∧SpeaksLanguage(y, l)\n⇒Understands(x, y).\nc. Translate into first-order logic the following sentences:\n(i) Understanding leads to friendship.\n(ii) Friendship is transitive.\nRemember to define all predicates, functions, and constants you use.\n8.12\nRewrite the first two Peano axioms in Section 8.3.3 as a single axiom that defines\nNatNum(x) so as to exclude the possibility of natural numbers except for those generated by\nthe successor function.\n8.13\nEquation (8.4) on page 306 defines the conditions under which a square is breezy. Here\nwe consider two other ways to describe this aspect of the wumpus world.\na. We can write diagnostic rules leading from observed effects to hidden causes. For find-\nDIAGNOSTIC RULE\ning pits, the obvious diagnostic rules say that if a square is breezy, some adjacent square\nmust contain a pit; and if a square is not breezy, then no adjacent square contains a pit.\nWrite these two rules in first-order logic and show that their conjunction is logically\nequivalent to Equation (8.4).\nb. We can write causal rules leading from cause to effect. One obvious causal rule is that\nCAUSAL RULE\na pit causes all adjacent squares to be breezy. Write this rule in first-order logic, explain\nwhy it is incomplete compared to Equation (8.4), and supply the missing axiom.\n318\nChapter\n8.\nFirst-Order Logic\nBeatrice\nAndrew\nEugenie\nWilliam Harry\nCharles\nDiana\nMum\nGeorge\nPhilip\nElizabeth\nMargaret\nKydd\nSpencer\nPeter\nMark\nZara\nAnne\nSarah\nEdward\nSophie\nLouise\nJames\nFigure 8.7\nA typical family tree. The symbol “▷◁” connects spouses and arrows point to\nchildren.\n8.14\nWrite axioms describing the predicates Grandchild, Greatgrandparent , Ancestor,\nBrother, Sister, Daughter, Son, FirstCousin, BrotherInLaw, SisterInLaw, Aunt, and\nUncle. Find out the proper definition of mth cousin n times removed, and write the def-\ninition in first-order logic. Now write down the basic facts depicted in the family tree in\nFigure 8.7. Using a suitable logical reasoning system, TELL it all the sentences you have\nwritten down, and ASK it who are Elizabeth’s grandchildren, Diana’s brothers-in-law, Zara’s\ngreat-grandparents, and Eugenie’s ancestors.\n8.15\nExplain what is wrong with the following proposed definition of the set membership\npredicate ∈:\n∀x, s x ∈{x|s}\n∀x, s x ∈s ⇒∀y x ∈{y|s} .\n8.16\nUsing the set axioms as examples, write axioms for the list domain, including all the\nconstants, functions, and predicates mentioned in the chapter.\n8.17\nExplain what is wrong with the following proposed definition of adjacent squares in\nthe wumpus world:\n∀x, y Adjacent([x, y], [x + 1, y]) ∧Adjacent([x, y], [x, y + 1]) .\n8.18\nWrite out the axioms required for reasoning about the wumpus’s location, using a\nconstant symbol Wumpus and a binary predicate At(Wumpus, Location). Remember that\nthere is only one wumpus.\n8.19\nAssuming predicates Parent(p, q) and Female(p) and constants Joan and Kevin,\nwith the obvious meanings, express each of the following sentences in first-order logic. (You\nmay use the abbreviation ∃1 to mean “there exists exactly one.”)\na. Joan has a daughter (possibly more than one, and possibly sons as well).\nb. Joan has exactly one daughter (but may have sons as well).\nc. Joan has exactly one child, a daughter.\nd. Joan and Kevin have exactly one child together.\ne. Joan has at least one child with Kevin, and no children with anyone else.\nExercises\n319\n8.20\nArithmetic assertions can be written in first-order logic with the predicate symbol <,\nthe function symbols + and ×, and the constant symbols 0 and 1. Additional predicates can\nalso be defined with biconditionals.\na. Represent the property “x is an even number.”\nb. Represent the property “x is prime.”\nc. Goldbach’s conjecture is the conjecture (unproven as yet) that every even number is\nequal to the sum of two primes. Represent this conjecture as a logical sentence.\n8.21\nIn Chapter 6, we used equality to indicate the relation between a variable and its value.\nFor instance, we wrote WA = red to mean that Western Australia is colored red. Repre-\nsenting this in first-order logic, we must write more verbosely ColorOf (WA) = red. What\nincorrect inference could be drawn if we wrote sentences such as WA = red directly as logical\nassertions?\n8.22\nWrite in first-order logic the assertion that every key and at least one of every pair of\nsocks will eventually be lost forever, using only the following vocabulary: Key(x), x is a key;\nSock(x), x is a sock; Pair(x, y), x and y are a pair; Now, the current time; Before(t1, t2),\ntime t1 comes before time t2; Lost(x, t), object x is lost at time t.\n8.23\nFor each of the following sentences in English, decide if the accompanying first-order\nlogic sentence is a good translation. If not, explain why not and correct it. (Some sentences\nmay have more than one error!)\na. No two people have the same social security number.\n¬∃x, y, n Person(x) ∧Person(y) ⇒[HasSS#(x, n) ∧HasSS#(y, n)].\nb. John’s social security number is the same as Mary’s.\n∃n HasSS#(John, n) ∧HasSS#(Mary, n).\nc. Everyone’s social security number has nine digits.\n∀x, n Person(x) ⇒[HasSS#(x, n) ∧Digits(n, 9)].\nd. Rewrite each of the above (uncorrected) sentences using a function symbol SS# instead\nof the predicate HasSS#.\n8.24\nRepresent the following sentences in first-order logic, using a consistent vocabulary\n(which you must define):\na. Some students took French in spring 2001.\nb. Every student who takes French passes it.\nc. Only one student took Greek in spring 2001.\nd. The best score in Greek is always higher than the best score in French.\ne. Every person who buys a policy is smart.\nf. No person buys an expensive policy.\ng. There is an agent who sells policies only to people who are not insured.\n320\nChapter\n8.\nFirst-Order Logic\nZ0\nZ1\nZ2\nZ3\nZ4\nX0\nY0\nX1\nY1\nX2\nY2\nX3\nY3\nAd0\nAd1\nAd2\nAd3\nX0\nX1\nX2\nX3\nZ0\nZ1\nZ2\nZ3\nZ4\nY0\nY1\nY2\nY3\n+\nFigure 8.8\nA four-bit adder. Each Adi is a one-bit adder, as in Figure 8.6 on page 309.\nh. There is a barber who shaves all men in town who do not shave themselves.\ni. A person born in the UK, each of whose parents is a UK citizen or a UK resident, is a\nUK citizen by birth.\nj. A person born outside the UK, one of whose parents is a UK citizen by birth, is a UK\ncitizen by descent.\nk. Politicians can fool some of the people all of the time, and they can fool all of the people\nsome of the time, but they can’t fool all of the people all of the time.\nl. All Greeks speak the same language. (Use Speaks(x, l) to mean that person x speaks\nlanguage l.)\n8.25\nWrite a general set of facts and axioms to represent the assertion “Wellington heard\nabout Napoleon’s death” and to correctly answer the question “Did Napoleon hear about\nWellington’s death?”\n8.26\nExtend the vocabulary from Section 8.4 to define addition for n-bit binary numbers.\nThen encode the description of the four-bit adder in Figure 8.8, and pose the queries needed\nto verify that it is in fact correct.\n8.27\nObtain a passport application for your country, identify the rules determining eligi-\nbility for a passport, and translate them into first-order logic, following the steps outlined in\nSection 8.4.\n8.28\nConsider a first-order logical knowledge base that describes worlds containing people,\nsongs, albums (e.g., “Meet the Beatles”) and disks (i.e., particular physical instances of CDs).\nThe vocabulary contains the following symbols:\nCopyOf (d, a): Predicate. Disk d is a copy of album a.\nOwns(p, d): Predicate. Person p owns disk d.\nSings(p, s, a): Album a includes a recording of song s sung by person p.\nWrote(p, s): Person p wrote song s.\nMcCartney, Gershwin, BHoliday, Joe, EleanorRigby, TheManILove, Revolver:\nConstants with the obvious meanings.\nExercises\n321\nExpress the following statements in first-order logic:\na. Gershwin wrote “The Man I Love.”\nb. Gershwin did not write “Eleanor Rigby.”\nc. Either Gershwin or McCartney wrote “The Man I Love.”\nd. Joe has written at least one song.\ne. Joe owns a copy of Revolver.\nf. Every song that McCartney sings on Revolver was written by McCartney.\ng. Gershwin did not write any of the songs on Revolver.\nh. Every song that Gershwin wrote has been recorded on some album. (Possibly different\nsongs are recorded on different albums.)\ni. There is a single album that contains every song that Joe has written.\nj. Joe owns a copy of an album that has Billie Holiday singing “The Man I Love.”\nk. Joe owns a copy of every album that has a song sung by McCartney. (Of course, each\ndifferent album is instantiated in a different physical CD.)\nl. Joe owns a copy of every album on which all the songs are sung by Billie Holiday.\n" }, { "chapter": 9, "text": "9\nINFERENCE IN\nFIRST-ORDER LOGIC\nIn which we define effective procedures for answering questions posed in first-\norder logic.\nChapter 7 showed how sound and complete inference can be achieved for propositional logic.\nIn this chapter, we extend those results to obtain algorithms that can answer any answer-\nable question stated in first-order logic. Section 9.1 introduces inference rules for quantifiers\nand shows how to reduce first-order inference to propositional inference, albeit at potentially\ngreat expense. Section 9.2 describes the idea of unification, showing how it can be used\nto construct inference rules that work directly with first-order sentences. We then discuss\nthree major families of first-order inference algorithms. Forward chaining and its applica-\ntions to deductive databases and production systems are covered in Section 9.3; backward\nchaining and logic programming systems are developed in Section 9.4. Forward and back-\nward chaining can be very efficient, but are applicable only to knowledge bases that can\nbe expressed as sets of Horn clauses. General first-order sentences require resolution-based\ntheorem proving, which is described in Section 9.5.\n9.1\nPROPOSITIONAL VS. FIRST-ORDER INFERENCE\nThis section and the next introduce the ideas underlying modern logical inference systems.\nWe begin with some simple inference rules that can be applied to sentences with quantifiers\nto obtain sentences without quantifiers. These rules lead naturally to the idea that first-order\ninference can be done by converting the knowledge base to propositional logic and using\npropositional inference, which we already know how to do. The next section points out an\nobvious shortcut, leading to inference methods that manipulate first-order sentences directly.\n9.1.1\nInference rules for quantifiers\nLet us begin with universal quantifiers. Suppose our knowledge base contains the standard\nfolkloric axiom stating that all greedy kings are evil:\n∀x King(x) ∧Greedy(x) ⇒Evil(x) .\n322\nSection 9.1.\nPropositional vs. First-Order Inference\n323\nThen it seems quite permissible to infer any of the following sentences:\nKing(John) ∧Greedy(John) ⇒Evil(John)\nKing(Richard) ∧Greedy(Richard) ⇒Evil(Richard)\nKing(Father(John)) ∧Greedy(Father(John)) ⇒Evil(Father(John)) .\n...\nThe rule of Universal Instantiation (UI for short) says that we can infer any sentence ob-\nUNIVERSAL\nINSTANTIATION\ntained by substituting a ground term (a term without variables) for the variable.1 To write\nGROUND TERM\nout the inference rule formally, we use the notion of substitutions introduced in Section 8.3.\nLet SUBST(θ, α) denote the result of applying the substitution θ to the sentence α. Then the\nrule is written\n∀v α\nSUBST({v/g}, α)\nfor any variable v and ground term g. For example, the three sentences given earlier are\nobtained with the substitutions {x/John}, {x/Richard}, and {x/Father(John)}.\nIn the rule for Existential Instantiation, the variable is replaced by a single new con-\nEXISTENTIAL\nINSTANTIATION\nstant symbol. The formal statement is as follows: for any sentence α, variable v, and constant\nsymbol k that does not appear elsewhere in the knowledge base,\n∃v α\nSUBST({v/k}, α) .\nFor example, from the sentence\n∃x Crown(x) ∧OnHead(x, John)\nwe can infer the sentence\nCrown(C1) ∧OnHead(C1, John)\nas long as C1 does not appear elsewhere in the knowledge base. Basically, the existential\nsentence says there is some object satisfying a condition, and applying the existential instan-\ntiation rule just gives a name to that object. Of course, that name must not already belong\nto another object. Mathematics provides a nice example: suppose we discover that there is a\nnumber that is a little bigger than 2.71828 and that satisfies the equation d(xy)/dy = xy for x.\nWe can give this number a name, such as e, but it would be a mistake to give it the name of\nan existing object, such as π. In logic, the new name is called a Skolem constant. Existen-\nSKOLEM CONSTANT\ntial Instantiation is a special case of a more general process called skolemization, which we\ncover in Section 9.5.\nWhereas Universal Instantiation can be applied many times to produce many different\nconsequences, Existential Instantiation can be applied once, and then the existentially quan-\ntified sentence can be discarded. For example, we no longer need ∃x Kill(x, Victim) once\nwe have added the sentence Kill(Murderer, Victim). Strictly speaking, the new knowledge\nbase is not logically equivalent to the old, but it can be shown to be inferentially equivalent\nINFERENTIAL\nEQUIVALENCE\nin the sense that it is satisfiable exactly when the original knowledge base is satisfiable.\n1 Do not confuse these substitutions with the extended interpretations used to define the semantics of quantifiers.\nThe substitution replaces a variable with a term (a piece of syntax) to produce a new sentence, whereas an\ninterpretation maps a variable to an object in the domain.\n324\nChapter\n9.\nInference in First-Order Logic\n9.1.2\nReduction to propositional inference\nOnce we have rules for inferring nonquantified sentences from quantified sentences, it be-\ncomes possible to reduce first-order inference to propositional inference. In this section we\ngive the main ideas; the details are given in Section 9.5.\nThe first idea is that, just as an existentially quantified sentence can be replaced by\none instantiation, a universally quantified sentence can be replaced by the set of all possible\ninstantiations. For example, suppose our knowledge base contains just the sentences\n∀x King(x) ∧Greedy(x) ⇒Evil(x)\nKing(John)\nGreedy(John)\nBrother(Richard, John) .\n(9.1)\nThen we apply UI to the first sentence using all possible ground-term substitutions from the\nvocabulary of the knowledge base—in this case, {x/John} and {x/Richard}. We obtain\nKing(John) ∧Greedy(John) ⇒Evil(John)\nKing(Richard) ∧Greedy(Richard) ⇒Evil(Richard) ,\nand we discard the universally quantified sentence. Now, the knowledge base is essentially\npropositional if we view the ground atomic sentences—King(John), Greedy(John), and\nso on—as proposition symbols. Therefore, we can apply any of the complete propositional\nalgorithms in Chapter 7 to obtain conclusions such as Evil(John).\nThis technique of propositionalization can be made completely general, as we show\nin Section 9.5; that is, every first-order knowledge base and query can be propositionalized\nin such a way that entailment is preserved. Thus, we have a complete decision procedure\nfor entailment . . . or perhaps not. There is a problem: when the knowledge base includes\na function symbol, the set of possible ground-term substitutions is infinite! For example, if\nthe knowledge base mentions the Father symbol, then infinitely many nested terms such as\nFather(Father(Father(John))) can be constructed. Our propositional algorithms will have\ndifficulty with an infinitely large set of sentences.\nFortunately, there is a famous theorem due to Jacques Herbrand (1930) to the effect\nthat if a sentence is entailed by the original, first-order knowledge base, then there is a proof\ninvolving just a finite subset of the propositionalized knowledge base. Since any such subset\nhas a maximum depth of nesting among its ground terms, we can find the subset by first\ngenerating all the instantiations with constant symbols (Richard and John), then all terms of\ndepth 1 (Father(Richard) and Father(John)), then all terms of depth 2, and so on, until we\nare able to construct a propositional proof of the entailed sentence.\nWe have sketched an approach to first-order inference via propositionalization that is\ncomplete—that is, any entailed sentence can be proved. This is a major achievement, given\nthat the space of possible models is infinite. On the other hand, we do not know until the\nproof is done that the sentence is entailed! What happens when the sentence is not entailed?\nCan we tell? Well, for first-order logic, it turns out that we cannot. Our proof procedure can\ngo on and on, generating more and more deeply nested terms, but we will not know whether\nit is stuck in a hopeless loop or whether the proof is just about to pop out. This is very much\nSection 9.2.\nUnification and Lifting\n325\nlike the halting problem for Turing machines. Alan Turing (1936) and Alonzo Church (1936)\nboth proved, in rather different ways, the inevitability of this state of affairs. The question of\nentailment for first-order logic is semidecidable—that is, algorithms exist that say yes to every\nentailed sentence, but no algorithm exists that also says no to every nonentailed sentence.\n9.2\nUNIFICATION AND LIFTING\nThe preceding section described the understanding of first-order inference that existed up\nto the early 1960s. The sharp-eyed reader (and certainly the computational logicians of the\nearly 1960s) will have noticed that the propositionalization approach is rather inefficient. For\nexample, given the query Evil(x) and the knowledge base in Equation (9.1), it seems per-\nverse to generate sentences such as King(Richard) ∧Greedy(Richard) ⇒Evil(Richard).\nIndeed, the inference of Evil(John) from the sentences\n∀x King(x) ∧Greedy(x) ⇒Evil(x)\nKing(John)\nGreedy(John)\nseems completely obvious to a human being. We now show how to make it completely\nobvious to a computer.\n9.2.1\nA first-order inference rule\nThe inference that John is evil—that is, that {x/John} solves the query Evil(x)—works like\nthis: to use the rule that greedy kings are evil, find some x such that x is a king and x is\ngreedy, and then infer that this x is evil. More generally, if there is some substitution θ that\nmakes each of the conjuncts of the premise of the implication identical to sentences already\nin the knowledge base, then we can assert the conclusion of the implication, after applying θ.\nIn this case, the substitution θ = {x/John} achieves that aim.\nWe can actually make the inference step do even more work. Suppose that instead of\nknowing Greedy(John), we know that everyone is greedy:\n∀y Greedy(y) .\n(9.2)\nThen we would still like to be able to conclude that Evil(John), because we know that\nJohn is a king (given) and John is greedy (because everyone is greedy). What we need for\nthis to work is to find a substitution both for the variables in the implication sentence and\nfor the variables in the sentences that are in the knowledge base. In this case, applying the\nsubstitution {x/John, y/John} to the implication premises King(x) and Greedy(x) and the\nknowledge-base sentences King(John) and Greedy(y) will make them identical. Thus, we\ncan infer the conclusion of the implication.\nThis inference process can be captured as a single inference rule that we call Gener-\nalized Modus Ponens:2 For atomic sentences pi, pi′, and q, where there is a substitution θ\nGENERALIZED\nMODUS PONENS\n326\nChapter\n9.\nInference in First-Order Logic\nsuch that SUBST(θ, pi′) = SUBST(θ, pi), for all i,\np1′, p2′, . . . , pn′, (p1 ∧p2 ∧. . . ∧pn ⇒q)\nSUBST(θ, q)\n.\nThere are n+1 premises to this rule: the n atomic sentences pi′ and the one implication. The\nconclusion is the result of applying the substitution θ to the consequent q. For our example:\np1′ is King(John)\np1 is King(x)\np2′ is Greedy(y)\np2 is Greedy(x)\nθ is {x/John, y/John}\nq is Evil(x)\nSUBST(θ, q) is Evil(John) .\nIt is easy to show that Generalized Modus Ponens is a sound inference rule. First, we observe\nthat, for any sentence p (whose variables are assumed to be universally quantified) and for\nany substitution θ,\np |= SUBST(θ, p)\nholds by Universal Instantiation. It holds in particular for a θ that satisfies the conditions of\nthe Generalized Modus Ponens rule. Thus, from p1′, . . . , pn′ we can infer\nSUBST(θ, p1′) ∧. . . ∧SUBST(θ, pn′)\nand from the implication p1 ∧. . . ∧pn ⇒q we can infer\nSUBST(θ, p1) ∧. . . ∧SUBST(θ, pn) ⇒SUBST(θ, q) .\nNow, θ in Generalized Modus Ponens is defined so that SUBST(θ, pi′) = SUBST(θ, pi), for\nall i; therefore the first of these two sentences matches the premise of the second exactly.\nHence, SUBST(θ, q) follows by Modus Ponens.\nGeneralized Modus Ponens is a lifted version of Modus Ponens—it raises Modus Po-\nLIFTING\nnens from ground (variable-free) propositional logic to first-order logic. We will see in the\nrest of this chapter that we can develop lifted versions of the forward chaining, backward\nchaining, and resolution algorithms introduced in Chapter 7. The key advantage of lifted\ninference rules over propositionalization is that they make only those substitutions that are\nrequired to allow particular inferences to proceed.\n9.2.2\nUnification\nLifted inference rules require finding substitutions that make different logical expressions\nlook identical. This process is called unification and is a key component of all first-order\nUNIFICATION\ninference algorithms. The UNIFY algorithm takes two sentences and returns a unifier for\nUNIFIER\nthem if one exists:\nUNIFY(p, q) = θ where SUBST(θ, p) = SUBST(θ, q) .\nLet us look at some examples of how UNIFY should behave. Suppose we have a query\nAskVars(Knows(John, x)): whom does John know? Answers to this query can be found\n2 Generalized Modus Ponens is more general than Modus Ponens (page 249) in the sense that the known facts\nand the premise of the implication need match only up to a substitution, rather than exactly. On the other hand,\nModus Ponens allows any sentence α as the premise, rather than just a conjunction of atomic sentences.\nSection 9.2.\nUnification and Lifting\n327\nby finding all sentences in the knowledge base that unify with Knows(John, x). Here are the\nresults of unification with four different sentences that might be in the knowledge base:\nUNIFY(Knows(John, x), Knows(John, Jane)) = {x/Jane}\nUNIFY(Knows(John, x), Knows(y, Bill)) = {x/Bill, y/John}\nUNIFY(Knows(John, x), Knows(y, Mother(y))) = {y/John, x/Mother(John)}\nUNIFY(Knows(John, x), Knows(x, Elizabeth)) = fail .\nThe last unification fails because x cannot take on the values John and Elizabeth at the\nsame time. Now, remember that Knows(x, Elizabeth) means “Everyone knows Elizabeth,”\nso we should be able to infer that John knows Elizabeth. The problem arises only because\nthe two sentences happen to use the same variable name, x. The problem can be avoided\nby standardizing apart one of the two sentences being unified, which means renaming its\nSTANDARDIZING\nAPART\nvariables to avoid name clashes. For example, we can rename x in Knows(x, Elizabeth) to\nx17 (a new variable name) without changing its meaning. Now the unification will work:\nUNIFY(Knows(John, x), Knows(x17, Elizabeth)) = {x/Elizabeth, x17/John} .\nExercise 9.12 delves further into the need for standardizing apart.\nThere is one more complication: we said that UNIFY should return a substitution\nthat makes the two arguments look the same. But there could be more than one such uni-\nfier. For example, UNIFY(Knows(John, x), Knows(y, z)) could return {y/John, x/z} or\n{y/John, x/John, z/John}. The first unifier gives Knows(John, z) as the result of unifi-\ncation, whereas the second gives Knows(John, John). The second result could be obtained\nfrom the first by an additional substitution {z/John}; we say that the first unifier is more\ngeneral than the second, because it places fewer restrictions on the values of the variables. It\nturns out that, for every unifiable pair of expressions, there is a single most general unifier (or\nMOST GENERAL\nUNIFIER\nMGU) that is unique up to renaming and substitution of variables. (For example, {x/John}\nand {y/John} are considered equivalent, as are {x/John, y/John} and {x/John, y/x}.) In\nthis case it is {y/John, x/z}.\nAn algorithm for computing most general unifiers is shown in Figure 9.1. The process\nis simple: recursively explore the two expressions simultaneously “side by side,” building up\na unifier along the way, but failing if two corresponding points in the structures do not match.\nThere is one expensive step: when matching a variable against a complex term, one must\ncheck whether the variable itself occurs inside the term; if it does, the match fails because no\nconsistent unifier can be constructed. For example, S(x) can’t unify with S(S(x)). This so-\ncalled occur check makes the complexity of the entire algorithm quadratic in the size of the\nOCCUR CHECK\nexpressions being unified. Some systems, including all logic programming systems, simply\nomit the occur check and sometimes make unsound inferences as a result; other systems use\nmore complex algorithms with linear-time complexity.\n9.2.3\nStorage and retrieval\nUnderlying the TELL and ASK functions used to inform and interrogate a knowledge base\nare the more primitive STORE and FETCH functions. STORE(s) stores a sentence s into the\nknowledge base and FETCH(q) returns all unifiers such that the query q unifies with some\n328\nChapter\n9.\nInference in First-Order Logic\nfunction UNIFY(x,y,θ) returns a substitution to make x and y identical\ninputs: x, a variable, constant, list, or compound expression\ny, a variable, constant, list, or compound expression\nθ, the substitution built up so far (optional, defaults to empty)\nif θ = failure then return failure\nelse if x = y then return θ\nelse if VARIABLE?(x) then return UNIFY-VAR(x,y,θ)\nelse if VARIABLE?(y) then return UNIFY-VAR(y,x,θ)\nelse if COMPOUND?(x) and COMPOUND?(y) then\nreturn UNIFY(x.ARGS,y.ARGS, UNIFY(x.OP,y.OP,θ))\nelse if LIST?(x) and LIST?(y) then\nreturn UNIFY(x.REST,y.REST, UNIFY(x.FIRST,y.FIRST,θ))\nelse return failure\nfunction UNIFY-VAR(var,x,θ) returns a substitution\nif {var/val} ∈θ then return UNIFY(val,x,θ)\nelse if {x/val} ∈θ then return UNIFY(var,val,θ)\nelse if OCCUR-CHECK?(var,x) then return failure\nelse return add {var/x} to θ\nFigure 9.1\nThe unification algorithm. The algorithm works by comparing the structures\nof the inputs, element by element. The substitution θ that is the argument to UNIFY is built\nup along the way and is used to make sure that later comparisons are consistent with bindings\nthat were established earlier. In a compound expression such as F(A, B), the OP field picks\nout the function symbol F and the ARGS field picks out the argument list (A, B).\nsentence in the knowledge base. The problem we used to illustrate unification—finding all\nfacts that unify with Knows(John, x)—is an instance of FETCHing.\nThe simplest way to implement STORE and FETCH is to keep all the facts in one long\nlist and unify each query against every element of the list. Such a process is inefficient, but\nit works, and it’s all you need to understand the rest of the chapter. The remainder of this\nsection outlines ways to make retrieval more efficient; it can be skipped on first reading.\nWe can make FETCH more efficient by ensuring that unifications are attempted only\nwith sentences that have some chance of unifying. For example, there is no point in trying\nto unify Knows(John, x) with Brother(Richard, John). We can avoid such unifications by\nindexing the facts in the knowledge base. A simple scheme called predicate indexing puts\nINDEXING\nPREDICATE\nINDEXING\nall the Knows facts in one bucket and all the Brother facts in another. The buckets can be\nstored in a hash table for efficient access.\nPredicate indexing is useful when there are many predicate symbols but only a few\nclauses for each symbol. Sometimes, however, a predicate has many clauses. For example,\nsuppose that the tax authorities want to keep track of who employs whom, using a predi-\ncate Employs(x, y). This would be a very large bucket with perhaps millions of employers\nSection 9.2.\nUnification and Lifting\n329\nEmploys(x,y)\nEmploys(x,Richard)\nEmploys(IBM,y)\nEmploys(IBM,Richard)\nEmploys(x,y)\nEmploys(John,John)\nEmploys(x,x)\nEmploys(x,John)\nEmploys(John,y)\n(a)\n(b)\nFigure 9.2\n(a) The subsumption lattice whose lowest node is Employs(IBM , Richard).\n(b) The subsumption lattice for the sentence Employs(John, John).\nand tens of millions of employees. Answering a query such as Employs(x, Richard) with\npredicate indexing would require scanning the entire bucket.\nFor this particular query, it would help if facts were indexed both by predicate and by\nsecond argument, perhaps using a combined hash table key. Then we could simply construct\nthe key from the query and retrieve exactly those facts that unify with the query. For other\nqueries, such as Employs(IBM , y), we would need to have indexed the facts by combining\nthe predicate with the first argument. Therefore, facts can be stored under multiple index\nkeys, rendering them instantly accessible to various queries that they might unify with.\nGiven a sentence to be stored, it is possible to construct indices for all possible queries\nthat unify with it. For the fact Employs(IBM , Richard), the queries are\nEmploys(IBM , Richard)\nDoes IBM employ Richard?\nEmploys(x, Richard)\nWho employs Richard?\nEmploys(IBM , y)\nWhom does IBM employ?\nEmploys(x, y)\nWho employs whom?\nThese queries form a subsumption lattice, as shown in Figure 9.2(a). The lattice has some\nSUBSUMPTION\nLATTICE\ninteresting properties. For example, the child of any node in the lattice is obtained from its\nparent by a single substitution; and the “highest” common descendant of any two nodes is\nthe result of applying their most general unifier. The portion of the lattice above any ground\nfact can be constructed systematically (Exercise 9.5). A sentence with repeated constants has\na slightly different lattice, as shown in Figure 9.2(b). Function symbols and variables in the\nsentences to be stored introduce still more interesting lattice structures.\nThe scheme we have described works very well whenever the lattice contains a small\nnumber of nodes. For a predicate with n arguments, however, the lattice contains O(2n)\nnodes. If function symbols are allowed, the number of nodes is also exponential in the size\nof the terms in the sentence to be stored. This can lead to a huge number of indices. At some\npoint, the benefits of indexing are outweighed by the costs of storing and maintaining all\nthe indices. We can respond by adopting a fixed policy, such as maintaining indices only on\nkeys composed of a predicate plus each argument, or by using an adaptive policy that creates\nindices to meet the demands of the kinds of queries being asked. For most AI systems, the\nnumber of facts to be stored is small enough that efficient indexing is considered a solved\nproblem. For commercial databases, where facts number in the billions, the problem has\nbeen the subject of intensive study and technology development..\n330\nChapter\n9.\nInference in First-Order Logic\n9.3\nFORWARD CHAINING\nA forward-chaining algorithm for propositional definite clauses was given in Section 7.5.\nThe idea is simple: start with the atomic sentences in the knowledge base and apply Modus\nPonens in the forward direction, adding new atomic sentences, until no further inferences\ncan be made. Here, we explain how the algorithm is applied to first-order definite clauses.\nDefinite clauses such as Situation ⇒Response are especially useful for systems that make\ninferences in response to newly arrived information. Many systems can be defined this way,\nand forward chaining can be implemented very efficiently.\n9.3.1\nFirst-order definite clauses\nFirst-order definite clauses closely resemble propositional definite clauses (page 256): they\nare disjunctions of literals of which exactly one is positive. A definite clause either is atomic\nor is an implication whose antecedent is a conjunction of positive literals and whose conse-\nquent is a single positive literal. The following are first-order definite clauses:\nKing(x) ∧Greedy(x) ⇒Evil(x) .\nKing(John) .\nGreedy(y) .\nUnlike propositional literals, first-order literals can include variables, in which case those\nvariables are assumed to be universally quantified. (Typically, we omit universal quantifiers\nwhen writing definite clauses.) Not every knowledge base can be converted into a set of\ndefinite clauses because of the single-positive-literal restriction, but many can. Consider the\nfollowing problem:\nThe law says that it is a crime for an American to sell weapons to hostile nations. The\ncountry Nono, an enemy of America, has some missiles, and all of its missiles were sold\nto it by Colonel West, who is American.\nWe will prove that West is a criminal. First, we will represent these facts as first-order definite\nclauses. The next section shows how the forward-chaining algorithm solves the problem.\n“. . . it is a crime for an American to sell weapons to hostile nations”:\nAmerican(x) ∧Weapon(y) ∧Sells(x, y, z) ∧Hostile(z) ⇒Criminal(x) .\n(9.3)\n“Nono . . . has some missiles.” The sentence ∃x Owns(Nono, x)∧Missile(x) is transformed\ninto two definite clauses by Existential Instantiation, introducing a new constant M1:\nOwns(Nono, M1)\n(9.4)\nMissile(M1)\n.\n(9.5)\n“All of its missiles were sold to it by Colonel West”:\nMissile(x) ∧Owns(Nono, x) ⇒Sells(West, x, Nono) .\n(9.6)\nWe will also need to know that missiles are weapons:\nMissile(x) ⇒Weapon(x)\n(9.7)\nSection 9.3.\nForward Chaining\n331\nand we must know that an enemy of America counts as “hostile”:\nEnemy(x, America) ⇒Hostile(x) .\n(9.8)\n“West, who is American . . .”:\nAmerican(West) .\n(9.9)\n“The country Nono, an enemy of America . . .”:\nEnemy(Nono, America) .\n(9.10)\nThis knowledge base contains no function symbols and is therefore an instance of the class\nof Datalog knowledge bases. Datalog is a language that is restricted to first-order definite\nDATALOG\nclauses with no function symbols. Datalog gets its name because it can represent the type of\nstatements typically made in relational databases. We will see that the absence of function\nsymbols makes inference much easier.\n9.3.2\nA simple forward-chaining algorithm\nThe first forward-chaining algorithm we consider is a simple one, shown in Figure 9.3. Start-\ning from the known facts, it triggers all the rules whose premises are satisfied, adding their\nconclusions to the known facts. The process repeats until the query is answered (assuming\nthat just one answer is required) or no new facts are added. Notice that a fact is not “new”\nif it is just a renaming of a known fact. One sentence is a renaming of another if they\nRENAMING\nare identical except for the names of the variables. For example, Likes(x, IceCream) and\nLikes(y, IceCream) are renamings of each other because they differ only in the choice of x\nor y; their meanings are identical: everyone likes ice cream.\nWe use our crime problem to illustrate how FOL-FC-ASK works. The implication\nsentences are (9.3), (9.6), (9.7), and (9.8). Two iterations are required:\n• On the first iteration, rule (9.3) has unsatisfied premises.\nRule (9.6) is satisfied with {x/M1}, and Sells(West, M1, Nono) is added.\nRule (9.7) is satisfied with {x/M1}, and Weapon(M1) is added.\nRule (9.8) is satisfied with {x/Nono}, and Hostile(Nono) is added.\n• On the second iteration, rule (9.3) is satisfied with {x/West, y/M1, z/Nono}, and\nCriminal(West) is added.\nFigure 9.4 shows the proof tree that is generated. Notice that no new inferences are possible\nat this point because every sentence that could be concluded by forward chaining is already\ncontained explicitly in the KB. Such a knowledge base is called a fixed point of the inference\nprocess. Fixed points reached by forward chaining with first-order definite clauses are similar\nto those for propositional forward chaining (page 258); the principal difference is that a first-\norder fixed point can include universally quantified atomic sentences.\nFOL-FC-ASK is easy to analyze. First, it is sound, because every inference is just an\napplication of Generalized Modus Ponens, which is sound. Second, it is complete for definite\nclause knowledge bases; that is, it answers every query whose answers are entailed by any\nknowledge base of definite clauses. For Datalog knowledge bases, which contain no function\nsymbols, the proof of completeness is fairly easy. We begin by counting the number of\n332\nChapter\n9.\nInference in First-Order Logic\nfunction FOL-FC-ASK(KB,α) returns a substitution or false\ninputs: KB, the knowledge base, a set of first-order definite clauses\nα, the query, an atomic sentence\nlocal variables: new, the new sentences inferred on each iteration\nrepeat until new is empty\nnew ←{ }\nfor each rule in KB do\n(p1 ∧. . . ∧pn ⇒q) ←STANDARDIZE-VARIABLES(rule)\nfor each θ such that SUBST(θ,p1 ∧. . . ∧pn) = SUBST(θ,p′\n1 ∧. . . ∧p′\nn)\nfor some p′\n1, . . . , p′\nn in KB\nq′ ←SUBST(θ,q)\nif q′ does not unify with some sentence already in KB or new then\nadd q′ to new\nφ ←UNIFY(q′,α)\nif φ is not fail then return φ\nadd new to KB\nreturn false\nFigure 9.3\nA conceptually straightforward, but very inefficient, forward-chaining algo-\nrithm. On each iteration, it adds to KB all the atomic sentences that can be inferred in one\nstep from the implication sentences and the atomic sentences already in KB. The function\nSTANDARDIZE-VARIABLES replaces all variables in its arguments with new ones that have\nnot been used before.\nHostile(Nono)\nEnemy(Nono,America)\nOwns(Nono,M1)\nMissile(M1)\nAmerican(West)\nWeapon(M1)\nCriminal(West)\nSells(West,M1,Nono)\nFigure 9.4\nThe proof tree generated by forward chaining on the crime example. The initial\nfacts appear at the bottom level, facts inferred on the first iteration in the middle level, and\nfacts inferred on the second iteration at the top level.\npossible facts that can be added, which determines the maximum number of iterations. Let k\nbe the maximum arity (number of arguments) of any predicate, p be the number of predicates,\nand n be the number of constant symbols. Clearly, there can be no more than pnk distinct\nground facts, so after this many iterations the algorithm must have reached a fixed point. Then\nwe can make an argument very similar to the proof of completeness for propositional forward\nSection 9.3.\nForward Chaining\n333\nchaining. (See page 258.) The details of how to make the transition from propositional to\nfirst-order completeness are given for the resolution algorithm in Section 9.5.\nFor general definite clauses with function symbols, FOL-FC-ASK can generate in-\nfinitely many new facts, so we need to be more careful. For the case in which an answer to\nthe query sentence q is entailed, we must appeal to Herbrand’s theorem to establish that the\nalgorithm will find a proof. (See Section 9.5 for the resolution case.) If the query has no\nanswer, the algorithm could fail to terminate in some cases. For example, if the knowledge\nbase includes the Peano axioms\nNatNum(0)\n∀n NatNum(n) ⇒NatNum(S(n)) ,\nthen forward chaining adds NatNum(S(0)), NatNum(S(S(0))), NatNum(S(S(S(0)))),\nand so on. This problem is unavoidable in general. As with general first-order logic, entail-\nment with definite clauses is semidecidable.\n9.3.3\nEfficient forward chaining\nThe forward-chaining algorithm in Figure 9.3 is designed for ease of understanding rather\nthan for efficiency of operation. There are three possible sources of inefficiency. First, the\n“inner loop” of the algorithm involves finding all possible unifiers such that the premise of\na rule unifies with a suitable set of facts in the knowledge base. This is often called pattern\nmatching and can be very expensive. Second, the algorithm rechecks every rule on every\nPATTERN MATCHING\niteration to see whether its premises are satisfied, even if very few additions are made to the\nknowledge base on each iteration. Finally, the algorithm might generate many facts that are\nirrelevant to the goal. We address each of these issues in turn.\nMatching rules against known facts\nThe problem of matching the premise of a rule against the facts in the knowledge base might\nseem simple enough. For example, suppose we want to apply the rule\nMissile(x) ⇒Weapon(x) .\nThen we need to find all the facts that unify with Missile(x); in a suitably indexed knowledge\nbase, this can be done in constant time per fact. Now consider a rule such as\nMissile(x) ∧Owns(Nono, x) ⇒Sells(West, x, Nono) .\nAgain, we can find all the objects owned by Nono in constant time per object; then, for each\nobject, we could check whether it is a missile. If the knowledge base contains many objects\nowned by Nono and very few missiles, however, it would be better to find all the missiles first\nand then check whether they are owned by Nono. This is the conjunct ordering problem:\nCONJUNCT\nORDERING\nfind an ordering to solve the conjuncts of the rule premise so that the total cost is minimized.\nIt turns out that finding the optimal ordering is NP-hard, but good heuristics are available.\nFor example, the minimum-remaining-values (MRV) heuristic used for CSPs in Chapter 6\nwould suggest ordering the conjuncts to look for missiles first if fewer missiles than objects\nare owned by Nono.\n334\nChapter\n9.\nInference in First-Order Logic\nWA\nNT\nSA\nQ\nNSW\nV\nT\nDiff(wa, nt) ∧Diff(wa, sa) ∧\nDiff(nt, q) ∧Diff(nt, sa) ∧\nDiff(q, nsw) ∧Diff(q, sa) ∧\nDiff(nsw, v) ∧Diff(nsw, sa) ∧\nDiff(v, sa) ⇒Colorable()\nDiff(Red, Blue)\nDiff(Red, Green)\nDiff(Green, Red) Diff(Green, Blue)\nDiff(Blue, Red)\nDiff(Blue, Green)\n(a)\n(b)\nFigure 9.5\n(a) Constraint graph for coloring the map of Australia. (b) The map-coloring\nCSP expressed as a single definite clause. Each map region is represented as a variable whose\nvalue can be one of the constants Red, Green or Blue.\nThe connection between pattern matching and constraint satisfaction is actually very\nclose. We can view each conjunct as a constraint on the variables that it contains—for ex-\nample, Missile(x) is a unary constraint on x. Extending this idea, we can express every\nfinite-domain CSP as a single definite clause together with some associated ground facts.\nConsider the map-coloring problem from Figure 6.1, shown again in Figure 9.5(a). An equiv-\nalent formulation as a single definite clause is given in Figure 9.5(b). Clearly, the conclusion\nColorable() can be inferred only if the CSP has a solution. Because CSPs in general include\n3-SAT problems as special cases, we can conclude that matching a definite clause against a\nset of facts is NP-hard.\nIt might seem rather depressing that forward chaining has an NP-hard matching problem\nin its inner loop. There are three ways to cheer ourselves up:\n• We can remind ourselves that most rules in real-world knowledge bases are small and\nsimple (like the rules in our crime example) rather than large and complex (like the\nCSP formulation in Figure 9.5). It is common in the database world to assume that\nboth the sizes of rules and the arities of predicates are bounded by a constant and to\nworry only about data complexity—that is, the complexity of inference as a function\nDATA COMPLEXITY\nof the number of ground facts in the knowledge base. It is easy to show that the data\ncomplexity of forward chaining is polynomial.\n• We can consider subclasses of rules for which matching is efficient. Essentially every\nDatalog clause can be viewed as defining a CSP, so matching will be tractable just\nwhen the corresponding CSP is tractable. Chapter 6 describes several tractable families\nof CSPs. For example, if the constraint graph (the graph whose nodes are variables\nand whose links are constraints) forms a tree, then the CSP can be solved in linear\ntime. Exactly the same result holds for rule matching. For instance, if we remove South\nSection 9.3.\nForward Chaining\n335\nAustralia from the map in Figure 9.5, the resulting clause is\nDiff(wa, nt) ∧Diff(nt, q) ∧Diff(q, nsw) ∧Diff(nsw, v) ⇒Colorable()\nwhich corresponds to the reduced CSP shown in Figure 6.12 on page 224. Algorithms\nfor solving tree-structured CSPs can be applied directly to the problem of rule matching.\n• We can try to to eliminate redundant rule-matching attempts in the forward-chaining\nalgorithm, as described next.\nIncremental forward chaining\nWhen we showed how forward chaining works on the crime example, we cheated; in partic-\nular, we omitted some of the rule matching done by the algorithm shown in Figure 9.3. For\nexample, on the second iteration, the rule\nMissile(x) ⇒Weapon(x)\nmatches against Missile(M1) (again), and of course the conclusion Weapon(M1) is already\nknown so nothing happens. Such redundant rule matching can be avoided if we make the\nfollowing observation: Every new fact inferred on iteration t must be derived from at least\none new fact inferred on iteration t −1.\nThis is true because any inference that does not\nrequire a new fact from iteration t −1 could have been done at iteration t −1 already.\nThis observation leads naturally to an incremental forward-chaining algorithm where,\nat iteration t, we check a rule only if its premise includes a conjunct pi that unifies with a fact\np′\ni newly inferred at iteration t −1. The rule-matching step then fixes pi to match with p′\ni, but\nallows the other conjuncts of the rule to match with facts from any previous iteration. This\nalgorithm generates exactly the same facts at each iteration as the algorithm in Figure 9.3, but\nis much more efficient.\nWith suitable indexing, it is easy to identify all the rules that can be triggered by any\ngiven fact, and indeed many real systems operate in an “update” mode wherein forward chain-\ning occurs in response to each new fact that is TELLed to the system. Inferences cascade\nthrough the set of rules until the fixed point is reached, and then the process begins again for\nthe next new fact.\nTypically, only a small fraction of the rules in the knowledge base are actually triggered\nby the addition of a given fact. This means that a great deal of redundant work is done in\nrepeatedly constructing partial matches that have some unsatisfied premises. Our crime ex-\nample is rather too small to show this effectively, but notice that a partial match is constructed\non the first iteration between the rule\nAmerican(x) ∧Weapon(y) ∧Sells(x, y, z) ∧Hostile(z) ⇒Criminal(x)\nand the fact American(West). This partial match is then discarded and rebuilt on the second\niteration (when the rule succeeds). It would be better to retain and gradually complete the\npartial matches as new facts arrive, rather than discarding them.\nThe rete algorithm3 was the first to address this problem. The algorithm preprocesses\nRETE\nthe set of rules in the knowledge base to construct a sort of dataflow network in which each\n3 Rete is Latin for net. The English pronunciation rhymes with treaty.\n336\nChapter\n9.\nInference in First-Order Logic\nnode is a literal from a rule premise. Variable bindings flow through the network and are\nfiltered out when they fail to match a literal. If two literals in a rule share a variable—for\nexample, Sells(x, y, z) ∧Hostile(z) in the crime example—then the bindings from each\nliteral are filtered through an equality node. A variable binding reaching a node for an n-\nary literal such as Sells(x, y, z) might have to wait for bindings for the other variables to be\nestablished before the process can continue. At any given point, the state of a rete network\ncaptures all the partial matches of the rules, avoiding a great deal of recomputation.\nRete networks, and various improvements thereon, have been a key component of so-\ncalled production systems, which were among the earliest forward-chaining systems in\nPRODUCTION\nSYSTEM\nwidespread use.4 The XCON system (originally called R1; McDermott, 1982) was built\nwith a production-system architecture. XCON contained several thousand rules for designing\nconfigurations of computer components for customers of the Digital Equipment Corporation.\nIt was one of the first clear commercial successes in the emerging field of expert systems.\nMany other similar systems have been built with the same underlying technology, which has\nbeen implemented in the general-purpose language OPS-5.\nProduction systems are also popular in cognitive architectures—that is, models of hu-\nCOGNITIVE\nARCHITECTURES\nman reasoning—such as ACT (Anderson, 1983) and SOAR (Laird et al., 1987). In such sys-\ntems, the “working memory” of the system models human short-term memory, and the pro-\nductions are part of long-term memory. On each cycle of operation, productions are matched\nagainst the working memory of facts. A production whose conditions are satisfied can add or\ndelete facts in working memory. In contrast to the typical situation in databases, production\nsystems often have many rules and relatively few facts. With suitably optimized matching\ntechnology, some modern systems can operate in real time with tens of millions of rules.\nIrrelevant facts\nThe final source of inefficiency in forward chaining appears to be intrinsic to the approach\nand also arises in the propositional context. Forward chaining makes all allowable inferences\nbased on the known facts, even if they are irrelevant to the goal at hand. In our crime example,\nthere were no rules capable of drawing irrelevant conclusions, so the lack of directedness was\nnot a problem. In other cases (e.g., if many rules describe the eating habits of Americans and\nthe prices of missiles), FOL-FC-ASK will generate many irrelevant conclusions.\nOne way to avoid drawing irrelevant conclusions is to use backward chaining, as de-\nscribed in Section 9.4. Another solution is to restrict forward chaining to a selected subset of\nrules, as in PL-FC-ENTAILS? (page 258). A third approach has emerged in the field of de-\nductive databases, which are large-scale databases, like relational databases, but which use\nDEDUCTIVE\nDATABASES\nforward chaining as the standard inference tool rather than SQL queries. The idea is to rewrite\nthe rule set, using information from the goal, so that only relevant variable bindings—those\nbelonging to a so-called magic set—are considered during forward inference. For example, if\nMAGIC SET\nthe goal is Criminal(West), the rule that concludes Criminal(x) will be rewritten to include\nan extra conjunct that constrains the value of x:\nMagic(x) ∧American(x) ∧Weapon(y) ∧Sells(x, y, z) ∧Hostile(z) ⇒Criminal(x) .\n4 The word production in production systems denotes a condition–action rule.\nSection 9.4.\nBackward Chaining\n337\nThe fact Magic(West) is also added to the KB. In this way, even if the knowledge base\ncontains facts about millions of Americans, only Colonel West will be considered during the\nforward inference process. The complete process for defining magic sets and rewriting the\nknowledge base is too complex to go into here, but the basic idea is to perform a sort of\n“generic” backward inference from the goal in order to work out which variable bindings\nneed to be constrained. The magic sets approach can therefore be thought of as a kind of\nhybrid between forward inference and backward preprocessing.\n9.4\nBACKWARD CHAINING\nThe second major family of logical inference algorithms uses the backward chaining ap-\nproach introduced in Section 7.5 for definite clauses. These algorithms work backward from\nthe goal, chaining through rules to find known facts that support the proof. We describe\nthe basic algorithm, and then we describe how it is used in logic programming, which is the\nmost widely used form of automated reasoning. We also see that backward chaining has some\ndisadvantages compared with forward chaining, and we look at ways to overcome them. Fi-\nnally, we look at the close connection between logic programming and constraint satisfaction\nproblems.\n9.4.1\nA backward-chaining algorithm\nFigure 9.6 shows a backward-chaining algorithm for definite clauses. FOL-BC-ASK(KB,\ngoal) will be proved if the knowledge base contains a clause of the form lhs ⇒goal, where\nlhs (left-hand side) is a list of conjuncts. An atomic fact like American(West) is considered\nas a clause whose lhs is the empty list. Now a query that contains variables might be proved\nin multiple ways. For example, the query Person(x) could be proved with the substitution\n{x/John} as well as with {x/Richard}. So we implement FOL-BC-ASK as a generator—\nGENERATOR\na function that returns multiple times, each time giving one possible result.\nBackward chaining is a kind of AND/OR search—the OR part because the goal query\ncan be proved by any rule in the knowledge base, and the AND part because all the conjuncts\nin the lhs of a clause must be proved. FOL-BC-OR works by fetching all clauses that might\nunify with the goal, standardizing the variables in the clause to be brand-new variables, and\nthen, if the rhs of the clause does indeed unify with the goal, proving every conjunct in the\nlhs, using FOL-BC-AND. That function in turn works by proving each of the conjuncts in\nturn, keeping track of the accumulated substitution as we go. Figure 9.7 is the proof tree for\nderiving Criminal(West) from sentences (9.3) through (9.10).\nBackward chaining, as we have written it, is clearly a depth-first search algorithm.\nThis means that its space requirements are linear in the size of the proof (neglecting, for\nnow, the space required to accumulate the solutions). It also means that backward chaining\n(unlike forward chaining) suffers from problems with repeated states and incompleteness. We\nwill discuss these problems and some potential solutions, but first we show how backward\nchaining is used in logic programming systems.\n338\nChapter\n9.\nInference in First-Order Logic\nfunction FOL-BC-ASK(KB,query) returns a generator of substitutions\nreturn FOL-BC-OR(KB,query,{ })\ngenerator FOL-BC-OR(KB,goal,θ) yields a substitution\nfor each rule (lhs ⇒rhs) in FETCH-RULES-FOR-GOAL(KB, goal) do\n(lhs, rhs) ←STANDARDIZE-VARIABLES((lhs, rhs))\nfor each θ′ in FOL-BC-AND(KB,lhs, UNIFY(rhs, goal, θ)) do\nyield θ′\ngenerator FOL-BC-AND(KB,goals,θ) yields a substitution\nif θ = failure then return\nelse if LENGTH(goals) = 0 then yield θ\nelse do\nfirst,rest ←FIRST(goals), REST(goals)\nfor each θ′ in FOL-BC-OR(KB, SUBST(θ, first), θ) do\nfor each θ′′ in FOL-BC-AND(KB,rest,θ′) do\nyield θ′′\nFigure 9.6\nA simple backward-chaining algorithm for first-order knowledge bases.\nHostile(Nono)\nEnemy(Nono,America)\nOwns(Nono,M1)\nMissile(M1)\nCriminal(West)\nMissile(y)\nWeapon(y)\nSells(West,M1,z)\nAmerican(West)\n{y/M1}\n{ }\n{ }\n{ }\n {z/Nono}\n{ }\nFigure 9.7\nProof tree constructed by backward chaining to prove that West is a criminal.\nThe tree should be read depth first, left to right. To prove Criminal(West), we have to prove\nthe four conjuncts below it. Some of these are in the knowledge base, and others require\nfurther backward chaining. Bindings for each successful unification are shown next to the\ncorresponding subgoal. Note that once one subgoal in a conjunction succeeds, its substitution\nis applied to subsequent subgoals. Thus, by the time FOL-BC-ASK gets to the last conjunct,\noriginally Hostile(z), z is already bound to Nono.\nSection 9.4.\nBackward Chaining\n339\n9.4.2\nLogic programming\nLogic programming is a technology that comes fairly close to embodying the declarative\nideal described in Chapter 7: that systems should be constructed by expressing knowledge in\na formal language and that problems should be solved by running inference processes on that\nknowledge. The ideal is summed up in Robert Kowalski’s equation,\nAlgorithm = Logic + Control .\nProlog is the most widely used logic programming language. It is used primarily as a rapid-\nPROLOG\nprototyping language and for symbol-manipulation tasks such as writing compilers (Van Roy,\n1990) and parsing natural language (Pereira and Warren, 1980). Many expert systems have\nbeen written in Prolog for legal, medical, financial, and other domains.\nProlog programs are sets of definite clauses written in a notation somewhat different\nfrom standard first-order logic. Prolog uses uppercase letters for variables and lowercase for\nconstants—the opposite of our convention for logic. Commas separate conjuncts in a clause,\nand the clause is written “backwards” from what we are used to; instead of A ∧B ⇒C in\nProlog we have C :- A, B. Here is a typical example:\ncriminal(X) :- american(X), weapon(Y), sells(X,Y,Z), hostile(Z).\nThe notation [E|L] denotes a list whose first element is E and whose rest is L. Here is a\nProlog program for append(X,Y,Z), which succeeds if list Z is the result of appending\nlists X and Y:\nappend([],Y,Y).\nappend([A|X],Y,[A|Z]) :- append(X,Y,Z).\nIn English, we can read these clauses as (1) appending an empty list with a list Y produces\nthe same list Y and (2) [A|Z] is the result of appending [A|X] onto Y, provided that Z is\nthe result of appending X onto Y. In most high-level languages we can write a similar recur-\nsive function that describes how to append two lists. The Prolog definition is actually much\nmore powerful, however, because it describes a relation that holds among three arguments,\nrather than a function computed from two arguments. For example, we can ask the query\nappend(X,Y,[1,2]): what two lists can be appended to give [1,2]? We get back the\nsolutions\nX=[]\nY=[1,2];\nX=[1]\nY=[2];\nX=[1,2] Y=[]\nThe execution of Prolog programs is done through depth-first backward chaining, where\nclauses are tried in the order in which they are written in the knowledge base. Some aspects\nof Prolog fall outside standard logical inference:\n• Prolog uses the database semantics of Section 8.2.8 rather than first-order semantics,\nand this is apparent in its treatment of equality and negation (see Section 9.4.5).\n• There is a set of built-in functions for arithmetic. Literals using these function symbols\nare “proved” by executing code rather than doing further inference. For example, the\n340\nChapter\n9.\nInference in First-Order Logic\ngoal “X is 4+3” succeeds with X bound to 7. On the other hand, the goal “5 is X+Y”\nfails, because the built-in functions do not do arbitrary equation solving.5\n• There are built-in predicates that have side effects when executed. These include input–\noutput predicates and the assert/retract predicates for modifying the knowledge\nbase. Such predicates have no counterpart in logic and can produce confusing results—\nfor example, if facts are asserted in a branch of the proof tree that eventually fails.\n• The occur check is omitted from Prolog’s unification algorithm. This means that some\nunsound inferences can be made; these are almost never a problem in practice.\n• Prolog uses depth-first backward-chaining search with no checks for infinite recursion.\nThis makes it very fast when given the right set of axioms, but incomplete when given\nthe wrong ones.\nProlog’s design represents a compromise between declarativeness and execution efficiency—\ninasmuch as efficiency was understood at the time Prolog was designed.\n9.4.3\nEfficient implementation of logic programs\nThe execution of a Prolog program can happen in two modes: interpreted and compiled.\nInterpretation essentially amounts to running the FOL-BC-ASK algorithm from Figure 9.6,\nwith the program as the knowledge base. We say “essentially” because Prolog interpreters\ncontain a variety of improvements designed to maximize speed. Here we consider only two.\nFirst, our implementation had to explicitly manage the iteration over possible results\ngenerated by each of the subfunctions.\nProlog interpreters have a global data structure,\na stack of choice points, to keep track of the multiple possibilities that we considered in\nCHOICE POINT\nFOL-BC-OR. This global stack is more efficient, and it makes debugging easier, because\nthe debugger can move up and down the stack.\nSecond, our simple implementation of FOL-BC-ASK spends a good deal of time gener-\nating substitutions. Instead of explicitly constructing substitutions, Prolog has logic variables\nthat remember their current binding. At any point in time, every variable in the program ei-\nther is unbound or is bound to some value. Together, these variables and values implicitly\ndefine the substitution for the current branch of the proof. Extending the path can only add\nnew variable bindings, because an attempt to add a different binding for an already bound\nvariable results in a failure of unification. When a path in the search fails, Prolog will back\nup to a previous choice point, and then it might have to unbind some variables. This is done\nby keeping track of all the variables that have been bound in a stack called the trail. As each\nTRAIL\nnew variable is bound by UNIFY-VAR, the variable is pushed onto the trail. When a goal fails\nand it is time to back up to a previous choice point, each of the variables is unbound as it is\nremoved from the trail.\nEven the most efficient Prolog interpreters require several thousand machine instruc-\ntions per inference step because of the cost of index lookup, unification, and building the\nrecursive call stack. In effect, the interpreter always behaves as if it has never seen the pro-\ngram before; for example, it has to find clauses that match the goal. A compiled Prolog\n5 Note that if the Peano axioms are provided, such goals can be solved by inference within a Prolog program.\nSection 9.4.\nBackward Chaining\n341\nprocedure APPEND(ax,y,az,continuation)\ntrail ←GLOBAL-TRAIL-POINTER()\nif ax = [ ] and UNIFY(y,az) then CALL(continuation)\nRESET-TRAIL(trail)\na, x, z ←NEW-VARIABLE(), NEW-VARIABLE(), NEW-VARIABLE()\nif UNIFY(ax,[a | x]) and UNIFY(az,[a | z]) then APPEND(x,y,z,continuation)\nFigure 9.8\nPseudocode representing the result of compiling the Append predicate. The\nfunction NEW-VARIABLE returns a new variable, distinct from all other variables used so far.\nThe procedure CALL(continuation) continues execution with the specified continuation.\nprogram, on the other hand, is an inference procedure for a specific set of clauses, so it knows\nwhat clauses match the goal. Prolog basically generates a miniature theorem prover for each\ndifferent predicate, thereby eliminating much of the overhead of interpretation. It is also pos-\nsible to open-code the unification routine for each different call, thereby avoiding explicit\nOPEN-CODE\nanalysis of term structure. (For details of open-coded unification, see Warren et al. (1977).)\nThe instruction sets of today’s computers give a poor match with Prolog’s semantics,\nso most Prolog compilers compile into an intermediate language rather than directly into ma-\nchine language. The most popular intermediate language is the Warren Abstract Machine,\nor WAM, named after David H. D. Warren, one of the implementers of the first Prolog com-\npiler. The WAM is an abstract instruction set that is suitable for Prolog and can be either\ninterpreted or translated into machine language. Other compilers translate Prolog into a high-\nlevel language such as Lisp or C and then use that language’s compiler to translate to machine\nlanguage. For example, the definition of the Append predicate can be compiled into the code\nshown in Figure 9.8. Several points are worth mentioning:\n• Rather than having to search the knowledge base for Append clauses, the clauses be-\ncome a procedure and the inferences are carried out simply by calling the procedure.\n• As described earlier, the current variable bindings are kept on a trail. The first step of the\nprocedure saves the current state of the trail, so that it can be restored by RESET-TRAIL\nif the first clause fails. This will undo any bindings generated by the first call to UNIFY.\n• The trickiest part is the use of continuations to implement choice points. You can think\nCONTINUATION\nof a continuation as packaging up a procedure and a list of arguments that together\ndefine what should be done next whenever the current goal succeeds. It would not\ndo just to return from a procedure like APPEND when the goal succeeds, because it\ncould succeed in several ways, and each of them has to be explored. The continuation\nargument solves this problem because it can be called each time the goal succeeds. In\nthe APPEND code, if the first argument is empty and the second argument unifies with\nthe third, then the APPEND predicate has succeeded. We then CALL the continuation,\nwith the appropriate bindings on the trail, to do whatever should be done next. For\nexample, if the call to APPEND were at the top level, the continuation would print the\nbindings of the variables.\n342\nChapter\n9.\nInference in First-Order Logic\nBefore Warren’s work on the compilation of inference in Prolog, logic programming was\ntoo slow for general use. Compilers by Warren and others allowed Prolog code to achieve\nspeeds that are competitive with C on a variety of standard benchmarks (Van Roy, 1990).\nOf course, the fact that one can write a planner or natural language parser in a few dozen\nlines of Prolog makes it somewhat more desirable than C for prototyping most small-scale AI\nresearch projects.\nParallelization can also provide substantial speedup. There are two principal sources of\nparallelism. The first, called OR-parallelism, comes from the possibility of a goal unifying\nOR-PARALLELISM\nwith many different clauses in the knowledge base. Each gives rise to an independent branch\nin the search space that can lead to a potential solution, and all such branches can be solved\nin parallel. The second, called AND-parallelism, comes from the possibility of solving\nAND-PARALLELISM\neach conjunct in the body of an implication in parallel. AND-parallelism is more difficult to\nachieve, because solutions for the whole conjunction require consistent bindings for all the\nvariables. Each conjunctive branch must communicate with the other branches to ensure a\nglobal solution.\n9.4.4\nRedundant inference and infinite loops\nWe now turn to the Achilles heel of Prolog: the mismatch between depth-first search and\nsearch trees that include repeated states and infinite paths. Consider the following logic pro-\ngram that decides if a path exists between two points on a directed graph:\npath(X,Z) :- link(X,Z).\npath(X,Z) :- path(X,Y), link(Y,Z).\nA simple three-node graph, described by the facts link(a,b) and link(b,c), is shown\nin Figure 9.9(a). With this program, the query path(a,c) generates the proof tree shown\nin Figure 9.10(a). On the other hand, if we put the two clauses in the order\npath(X,Z) :- path(X,Y), link(Y,Z).\npath(X,Z) :- link(X,Z).\nthen Prolog follows the infinite path shown in Figure 9.10(b). Prolog is therefore incomplete\nas a theorem prover for definite clauses—even for Datalog programs, as this example shows—\nbecause, for some knowledge bases, it fails to prove sentences that are entailed. Notice that\nforward chaining does not suffer from this problem: once path(a,b), path(b,c), and\npath(a,c) are inferred, forward chaining halts.\nDepth-first backward chaining also has problems with redundant computations. For\nexample, when finding a path from A1 to J4 in Figure 9.9(b), Prolog performs 877 inferences,\nmost of which involve finding all possible paths to nodes from which the goal is unreachable.\nThis is similar to the repeated-state problem discussed in Chapter 3. The total amount of\ninference can be exponential in the number of ground facts that are generated. If we apply\nforward chaining instead, at most n2 path(X,Y) facts can be generated linking n nodes.\nFor the problem in Figure 9.9(b), only 62 inferences are needed.\nForward chaining on graph search problems is an example of dynamic programming,\nDYNAMIC\nPROGRAMMING\nin which the solutions to subproblems are constructed incrementally from those of smaller\nSection 9.4.\nBackward Chaining\n343\n(a)\n(b)\nA\nB\nC\nA1\nJ4\nFigure 9.9\n(a) Finding a path from A to C can lead Prolog into an infinite loop. (b) A\ngraph in which each node is connected to two random successors in the next layer. Finding a\npath from A1 to J4 requires 877 inferences.\npath(a,c)\nfail\n{\n}\n/\nY b\n{ }\nlink(a,c)\npath(a,Y)\nlink(a,Y)\nlink(b,c)\npath(a,c)\npath(a,Y)\nlink(Y,c)\npath(a,Y’)\nlink(Y’,Y)\n(a)\n(b)\nFigure 9.10\n(a) Proof that a path exists from A to C. (b) Infinite proof tree generated\nwhen the clauses are in the “wrong” order.\nsubproblems and are cached to avoid recomputation. We can obtain a similar effect in a\nbackward chaining system using memoization—that is, caching solutions to subgoals as\nthey are found and then reusing those solutions when the subgoal recurs, rather than repeat-\ning the previous computation. This is the approach taken by tabled logic programming sys-\nTABLED LOGIC\nPROGRAMMING\ntems, which use efficient storage and retrieval mechanisms to perform memoization. Tabled\nlogic programming combines the goal-directedness of backward chaining with the dynamic-\nprogramming efficiency of forward chaining. It is also complete for Datalog knowledge\nbases, which means that the programmer need worry less about infinite loops. (It is still pos-\nsible to get an infinite loop with predicates like father(X,Y) that refer to a potentially\nunbounded number of objects.)\n9.4.5\nDatabase semantics of Prolog\nProlog uses database semantics, as discussed in Section 8.2.8. The unique names assumption\nsays that every Prolog constant and every ground term refers to a distinct object, and the\nclosed world assumption says that the only sentences that are true are those that are entailed\n344\nChapter\n9.\nInference in First-Order Logic\nby the knowledge base. There is no way to assert that a sentence is false in Prolog. This makes\nProlog less expressive than first-order logic, but it is part of what makes Prolog more efficient\nand more concise. Consider the following Prolog assertions about some course offerings:\nCourse(CS, 101), Course(CS, 102), Course(CS, 106), Course(EE, 101). (9.11)\nUnder the unique names assumption, CS and EE are different (as are 101, 102, and 106),\nso this means that there are four distinct courses. Under the closed-world assumption there\nare no other courses, so there are exactly four courses. But if these were assertions in FOL\nrather than in Prolog, then all we could say is that there are somewhere between one and\ninfinity courses. That’s because the assertions (in FOL) do not deny the possibility that other\nunmentioned courses are also offered, nor do they say that the courses mentioned are different\nfrom each other. If we wanted to translate Equation (9.11) into FOL, we would get this:\nCourse(d, n)\n⇔\n(d = CS ∧n = 101) ∨(d = CS ∧n = 102)\n∨(d = CS ∧n = 106) ∨(d = EE ∧n = 101) .\n(9.12)\nThis is called the completion of Equation (9.11). It expresses in FOL the idea that there are\nCOMPLETION\nat most four courses. To express in FOL the idea that there are at least four courses, we need\nto write the completion of the equality predicate:\nx = y\n⇔\n(x = CS ∧y = CS) ∨(x = EE ∧y = EE) ∨(x = 101 ∧y = 101)\n∨(x = 102 ∧y = 102) ∨(x = 106 ∧y = 106) .\nThe completion is useful for understanding database semantics, but for practical purposes, if\nyour problem can be described with database semantics, it is more efficient to reason with\nProlog or some other database semantics system, rather than translating into FOL and rea-\nsoning with a full FOL theorem prover.\n9.4.6\nConstraint logic programming\nIn our discussion of forward chaining (Section 9.3), we showed how constraint satisfaction\nproblems (CSPs) can be encoded as definite clauses. Standard Prolog solves such problems\nin exactly the same way as the backtracking algorithm given in Figure 6.5.\nBecause backtracking enumerates the domains of the variables, it works only for finite-\ndomain CSPs. In Prolog terms, there must be a finite number of solutions for any goal\nwith unbound variables. (For example, the goal diff(Q,SA), which says that Queensland\nand South Australia must be different colors, has six solutions if three colors are allowed.)\nInfinite-domain CSPs—for example, with integer or real-valued variables—require quite dif-\nferent algorithms, such as bounds propagation or linear programming.\nConsider the following example. We define triangle(X,Y,Z) as a predicate that\nholds if the three arguments are numbers that satisfy the triangle inequality:\ntriangle(X,Y,Z) :-\nX>0, Y>0, Z>0, X+Y>=Z, Y+Z>=X, X+Z>=Y.\nIf we ask Prolog the query triangle(3,4,5), it succeeds. On the other hand, if we\nask triangle(3,4,Z), no solution will be found, because the subgoal Z>=0 cannot be\nhandled by Prolog; we can’t compare an unbound value to 0.\nSection 9.5.\nResolution\n345\nConstraint logic programming (CLP) allows variables to be constrained rather than\nCONSTRAINT LOGIC\nPROGRAMMING\nbound. A CLP solution is the most specific set of constraints on the query variables that can\nbe derived from the knowledge base. For example, the solution to the triangle(3,4,Z)\nquery is the constraint 7 >= Z >= 1. Standard logic programs are just a special case of\nCLP in which the solution constraints must be equality constraints—that is, bindings.\nCLP systems incorporate various constraint-solving algorithms for the constraints al-\nlowed in the language. For example, a system that allows linear inequalities on real-valued\nvariables might include a linear programming algorithm for solving those constraints. CLP\nsystems also adopt a much more flexible approach to solving standard logic programming\nqueries. For example, instead of depth-first, left-to-right backtracking, they might use any of\nthe more efficient algorithms discussed in Chapter 6, including heuristic conjunct ordering,\nbackjumping, cutset conditioning, and so on. CLP systems therefore combine elements of\nconstraint satisfaction algorithms, logic programming, and deductive databases.\nSeveral systems that allow the programmer more control over the search order for in-\nference have been defined. The MRS language (Genesereth and Smith, 1981; Russell, 1985)\nallows the programmer to write metarules to determine which conjuncts are tried first. The\nMETARULE\nuser could write a rule saying that the goal with the fewest variables should be tried first or\ncould write domain-specific rules for particular predicates.\n9.5\nRESOLUTION\nThe last of our three families of logical systems is based on resolution. We saw on page 250\nthat propositional resolution using refutation is a complete inference procedure for proposi-\ntional logic. In this section, we describe how to extend resolution to first-order logic.\n9.5.1\nConjunctive normal form for first-order logic\nAs in the propositional case, first-order resolution requires that sentences be in conjunctive\nnormal form (CNF)—that is, a conjunction of clauses, where each clause is a disjunction of\nliterals.6 Literals can contain variables, which are assumed to be universally quantified. For\nexample, the sentence\n∀x American(x) ∧Weapon(y) ∧Sells(x, y, z) ∧Hostile(z) ⇒Criminal(x)\nbecomes, in CNF,\n¬American(x) ∨¬Weapon(y) ∨¬Sells(x, y, z) ∨¬Hostile(z) ∨Criminal(x) .\nEvery sentence of first-order logic can be converted into an inferentially equivalent CNF\nsentence. In particular, the CNF sentence will be unsatisfiable just when the original sentence\nis unsatisfiable, so we have a basis for doing proofs by contradiction on the CNF sentences.\n6 A clause can also be represented as an implication with a conjunction of atoms in the premise and a disjunction\nof atoms in the conclusion (Exercise 7.13). This is called implicative normal form or Kowalski form (especially\nwhen written with a right-to-left implication symbol (Kowalski, 1979)) and is often much easier to read.\n346\nChapter\n9.\nInference in First-Order Logic\nThe procedure for conversion to CNF is similar to the propositional case, which we saw\non page 253. The principal difference arises from the need to eliminate existential quantifiers.\nWe illustrate the procedure by translating the sentence “Everyone who loves all animals is\nloved by someone,” or\n∀x [∀y Animal(y) ⇒Loves(x, y)] ⇒[∃y Loves(y, x)] .\nThe steps are as follows:\n• Eliminate implications:\n∀x [¬∀y ¬Animal(y) ∨Loves(x, y)] ∨[∃y Loves(y, x)] .\n• Move ¬ inwards: In addition to the usual rules for negated connectives, we need rules\nfor negated quantifiers. Thus, we have\n¬∀x p\nbecomes\n∃x ¬p\n¬∃x p\nbecomes\n∀x ¬p .\nOur sentence goes through the following transformations:\n∀x [∃y ¬(¬Animal(y) ∨Loves(x, y))] ∨[∃y Loves(y, x)] .\n∀x [∃y ¬¬Animal(y) ∧¬Loves(x, y)] ∨[∃y Loves(y, x)] .\n∀x [∃y Animal(y) ∧¬Loves(x, y)] ∨[∃y Loves(y, x)] .\nNotice how a universal quantifier (∀y) in the premise of the implication has become\nan existential quantifier. The sentence now reads “Either there is some animal that x\ndoesn’t love, or (if this is not the case) someone loves x.” Clearly, the meaning of the\noriginal sentence has been preserved.\n• Standardize variables: For sentences like (∃x P(x))∨(∃x Q(x)) which use the same\nvariable name twice, change the name of one of the variables. This avoids confusion\nlater when we drop the quantifiers. Thus, we have\n∀x [∃y Animal(y) ∧¬Loves(x, y)] ∨[∃z Loves(z, x)] .\n• Skolemize: Skolemization is the process of removing existential quantifiers by elimi-\nSKOLEMIZATION\nnation. In the simple case, it is just like the Existential Instantiation rule of Section 9.1:\ntranslate ∃x P(x) into P(A), where A is a new constant. However, we can’t apply Ex-\nistential Instantiation to our sentence above because it doesn’t match the pattern ∃v α;\nonly parts of the sentence match the pattern. If we blindly apply the rule to the two\nmatching parts we get\n∀x [Animal(A) ∧¬Loves(x, A)] ∨Loves(B, x) ,\nwhich has the wrong meaning entirely: it says that everyone either fails to love a par-\nticular animal A or is loved by some particular entity B. In fact, our original sentence\nallows each person to fail to love a different animal or to be loved by a different person.\nThus, we want the Skolem entities to depend on x and z:\n∀x [Animal(F(x)) ∧¬Loves(x, F(x))] ∨Loves(G(z), x) .\nHere F and G are Skolem functions. The general rule is that the arguments of the\nSKOLEM FUNCTION\nSkolem function are all the universally quantified variables in whose scope the exis-\ntential quantifier appears. As with Existential Instantiation, the Skolemized sentence is\nsatisfiable exactly when the original sentence is satisfiable.\nSection 9.5.\nResolution\n347\n• Drop universal quantifiers: At this point, all remaining variables must be universally\nquantified. Moreover, the sentence is equivalent to one in which all the universal quan-\ntifiers have been moved to the left. We can therefore drop the universal quantifiers:\n[Animal(F(x)) ∧¬Loves(x, F(x))] ∨Loves(G(z), x) .\n• Distribute ∨over ∧:\n[Animal(F(x)) ∨Loves(G(z), x)] ∧[¬Loves(x, F(x)) ∨Loves(G(z), x)] .\nThis step may also require flattening out nested conjunctions and disjunctions.\nThe sentence is now in CNF and consists of two clauses. It is quite unreadable. (It may\nhelp to explain that the Skolem function F(x) refers to the animal potentially unloved by x,\nwhereas G(z) refers to someone who might love x.) Fortunately, humans seldom need look\nat CNF sentences—the translation process is easily automated.\n9.5.2\nThe resolution inference rule\nThe resolution rule for first-order clauses is simply a lifted version of the propositional reso-\nlution rule given on page 253. Two clauses, which are assumed to be standardized apart so\nthat they share no variables, can be resolved if they contain complementary literals. Propo-\nsitional literals are complementary if one is the negation of the other; first-order literals are\ncomplementary if one unifies with the negation of the other. Thus, we have\nℓ1 ∨· · · ∨ℓk,\nm1 ∨· · · ∨mn\nSUBST(θ, ℓ1 ∨· · · ∨ℓi−1 ∨ℓi+1 ∨· · · ∨ℓk ∨m1 ∨· · · ∨mj−1 ∨mj+1 ∨· · · ∨mn)\nwhere UNIFY(ℓi, ¬mj) = θ. For example, we can resolve the two clauses\n[Animal(F(x)) ∨Loves(G(x), x)]\nand\n[¬Loves(u, v) ∨¬Kills(u, v)]\nby eliminating the complementary literals Loves(G(x), x) and ¬Loves(u, v), with unifier\nθ = {u/G(x), v/x}, to produce the resolvent clause\n[Animal(F(x)) ∨¬Kills(G(x), x)] .\nThis rule is called the binary resolution rule because it resolves exactly two literals. The\nBINARY RESOLUTION\nbinary resolution rule by itself does not yield a complete inference procedure. The full reso-\nlution rule resolves subsets of literals in each clause that are unifiable. An alternative approach\nis to extend factoring—the removal of redundant literals—to the first-order case. Proposi-\ntional factoring reduces two literals to one if they are identical; first-order factoring reduces\ntwo literals to one if they are unifiable. The unifier must be applied to the entire clause. The\ncombination of binary resolution and factoring is complete.\n9.5.3\nExample proofs\nResolution proves that KB |= α by proving KB ∧¬α unsatisfiable, that is, by deriving the\nempty clause. The algorithmic approach is identical to the propositional case, described in\n348\nChapter\n9.\nInference in First-Order Logic\n¬American(x) ¬Weapon(y) ¬Sells(x,y,z) ¬Hostile(z) Criminal(x)\n¬Criminal(West)\n¬Enemy(Nono,America)\nEnemy(Nono,America)\n¬Missile(x) Weapon(x)\n¬Weapon(y) ¬Sells(West,y,z) ¬Hostile(z)\nMissile(M1)\n¬Missile(y) ¬Sells(West,y,z) ¬Hostile(z)\n¬Missile(x) ¬Owns(Nono,x) Sells(West,x,Nono)\n¬Sells(West,M1,z) ¬Hostile(z)\n¬American(West) ¬Weapon(y) ¬Sells(West,y,z) ¬Hostile(z)\nAmerican(West)\n¬Missile(M1) ¬Owns(Nono,M1) ¬Hostile(Nono)\nMissile(M1)\n¬Owns(Nono,M1) ¬Hostile(Nono)\nOwns(Nono,M1)\n¬Enemy(x,America) Hostile(x)\n¬Hostile(Nono)\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\n^\nFigure 9.11\nA resolution proof that West is a criminal. At each step, the literals that unify\nare in bold.\nFigure 7.12, so we need not repeat it here. Instead, we give two example proofs. The first is\nthe crime example from Section 9.3. The sentences in CNF are\n¬American(x) ∨¬Weapon(y) ∨¬Sells(x, y, z) ∨¬Hostile(z) ∨Criminal(x)\n¬Missile(x) ∨¬Owns(Nono, x) ∨Sells(West, x, Nono)\n¬Enemy(x, America) ∨Hostile(x)\n¬Missile(x) ∨Weapon(x)\nOwns(Nono, M1)\nMissile(M1)\nAmerican(West)\nEnemy(Nono, America) .\nWe also include the negated goal ¬Criminal(West). The resolution proof is shown in Fig-\nure 9.11. Notice the structure: single “spine” beginning with the goal clause, resolving against\nclauses from the knowledge base until the empty clause is generated. This is characteristic\nof resolution on Horn clause knowledge bases. In fact, the clauses along the main spine\ncorrespond exactly to the consecutive values of the goals variable in the backward-chaining\nalgorithm of Figure 9.6. This is because we always choose to resolve with a clause whose\npositive literal unified with the leftmost literal of the “current” clause on the spine; this is\nexactly what happens in backward chaining. Thus, backward chaining is just a special case\nof resolution with a particular control strategy to decide which resolution to perform next.\nOur second example makes use of Skolemization and involves clauses that are not def-\ninite clauses. This results in a somewhat more complex proof structure. In English, the\nproblem is as follows:\nEveryone who loves all animals is loved by someone.\nAnyone who kills an animal is loved by no one.\nJack loves all animals.\nEither Jack or Curiosity killed the cat, who is named Tuna.\nDid Curiosity kill the cat?\nSection 9.5.\nResolution\n349\nFirst, we express the original sentences, some background knowledge, and the negated goal\nG in first-order logic:\nA.\n∀x [∀y Animal(y) ⇒Loves(x, y)] ⇒[∃y Loves(y, x)]\nB.\n∀x [∃z Animal(z) ∧Kills(x, z)] ⇒[∀y ¬Loves(y, x)]\nC.\n∀x Animal(x) ⇒Loves(Jack, x)\nD.\nKills(Jack, Tuna) ∨Kills(Curiosity, Tuna)\nE.\nCat(Tuna)\nF.\n∀x Cat(x) ⇒Animal(x)\n¬G.\n¬Kills(Curiosity, Tuna)\nNow we apply the conversion procedure to convert each sentence to CNF:\nA1.\nAnimal(F(x)) ∨Loves(G(x), x)\nA2.\n¬Loves(x, F(x)) ∨Loves(G(x), x)\nB.\n¬Loves(y, x) ∨¬Animal(z) ∨¬Kills(x, z)\nC.\n¬Animal(x) ∨Loves(Jack, x)\nD.\nKills(Jack, Tuna) ∨Kills(Curiosity, Tuna)\nE.\nCat(Tuna)\nF.\n¬Cat(x) ∨Animal(x)\n¬G.\n¬Kills(Curiosity, Tuna)\nThe resolution proof that Curiosity killed the cat is given in Figure 9.12. In English, the proof\ncould be paraphrased as follows:\nSuppose Curiosity did not kill Tuna. We know that either Jack or Curiosity did; thus\nJack must have. Now, Tuna is a cat and cats are animals, so Tuna is an animal. Because\nanyone who kills an animal is loved by no one, we know that no one loves Jack. On the\nother hand, Jack loves all animals, so someone loves him; so we have a contradiction.\nTherefore, Curiosity killed the cat.\n¬Loves(y, Jack)\nLoves(G(Jack), Jack)\n¬Kills(Curiosity, Tuna)\nKills(Jack, Tuna) Kills(Curiosity, Tuna)\n¬Cat(x) Animal(x)\nCat(Tuna)\n¬Animal(F(Jack)) Loves(G(Jack), Jack)\nAnimal(F(x)) Loves(G(x), x) \n¬Loves(y, x) ¬Kills(x, Tuna)\nKills(Jack, Tuna)\n¬Loves(y, x) ¬Animal(z) ¬Kills(x, z)\nAnimal(Tuna)\n¬Loves(x,F(x)) Loves(G(x), x)\n¬Animal(x) Loves(Jack, x)\n^\n^\n^\n^\n^\n^\n^\n^\n^\nFigure 9.12\nA resolution proof that Curiosity killed the cat. Notice the use of factoring\nin the derivation of the clause Loves(G(Jack), Jack). Notice also in the upper right, the\nunification of Loves(x, F(x)) and Loves(Jack, x) can only succeed after the variables have\nbeen standardized apart.\n350\nChapter\n9.\nInference in First-Order Logic\nThe proof answers the question “Did Curiosity kill the cat?” but often we want to pose more\ngeneral questions, such as “Who killed the cat?” Resolution can do this, but it takes a little\nmore work to obtain the answer. The goal is ∃w Kills(w, Tuna), which, when negated,\nbecomes ¬Kills(w, Tuna) in CNF. Repeating the proof in Figure 9.12 with the new negated\ngoal, we obtain a similar proof tree, but with the substitution {w/Curiosity} in one of the\nsteps. So, in this case, finding out who killed the cat is just a matter of keeping track of the\nbindings for the query variables in the proof.\nUnfortunately, resolution can produce nonconstructive proofs for existential goals.\nNONCONSTRUCTIVE\nPROOF\nFor example, ¬Kills(w, Tuna) resolves with Kills(Jack, Tuna) ∨Kills(Curiosity, Tuna)\nto give Kills(Jack, Tuna), which resolves again with ¬Kills(w, Tuna) to yield the empty\nclause. Notice that w has two different bindings in this proof; resolution is telling us that,\nyes, someone killed Tuna—either Jack or Curiosity. This is no great surprise! One so-\nlution is to restrict the allowed resolution steps so that the query variables can be bound\nonly once in a given proof; then we need to be able to backtrack over the possible bind-\nings. Another solution is to add a special answer literal to the negated goal, which be-\nANSWER LITERAL\ncomes ¬Kills(w, Tuna) ∨Answer(w). Now, the resolution process generates an answer\nwhenever a clause is generated containing just a single answer literal. For the proof in Fig-\nure 9.12, this is Answer(Curiosity). The nonconstructive proof would generate the clause\nAnswer(Curiosity) ∨Answer(Jack), which does not constitute an answer.\n9.5.4\nCompleteness of resolution\nThis section gives a completeness proof of resolution. It can be safely skipped by those who\nare willing to take it on faith.\nWe show that resolution is refutation-complete, which means that if a set of sentences\nREFUTATION\nCOMPLETENESS\nis unsatisfiable, then resolution will always be able to derive a contradiction. Resolution\ncannot be used to generate all logical consequences of a set of sentences, but it can be used\nto establish that a given sentence is entailed by the set of sentences. Hence, it can be used to\nfind all answers to a given question, Q(x), by proving that KB ∧¬Q(x) is unsatisfiable.\nWe take it as given that any sentence in first-order logic (without equality) can be rewrit-\nten as a set of clauses in CNF. This can be proved by induction on the form of the sentence,\nusing atomic sentences as the base case (Davis and Putnam, 1960). Our goal therefore is to\nprove the following: if S is an unsatisfiable set of clauses, then the application of a finite\nnumber of resolution steps to S will yield a contradiction.\nOur proof sketch follows Robinson’s original proof with some simplifications from\nGenesereth and Nilsson (1987). The basic structure of the proof (Figure 9.13) is as follows:\n1. First, we observe that if S is unsatisfiable, then there exists a particular set of ground\ninstances of the clauses of S such that this set is also unsatisfiable (Herbrand’s theorem).\n2. We then appeal to the ground resolution theorem given in Chapter 7, which states that\npropositional resolution is complete for ground sentences.\n3. We then use a lifting lemma to show that, for any propositional resolution proof using\nthe set of ground sentences, there is a corresponding first-order resolution proof using\nthe first-order sentences from which the ground sentences were obtained.\nSection 9.5.\nResolution\n351\nResolution can find a contradiction in S'\nThere is a resolution proof for the contradiction in S'\nHerbrand’s theorem\nSome set S' of ground instances is unsatisfiable\nAny set of sentences S is representable in clausal form\nAssume S is unsatisfiable, and in clausal form\nLifting lemma\nGround resolution\ntheorem\nFigure 9.13\nStructure of a completeness proof for resolution.\nTo carry out the first step, we need three new concepts:\n• Herbrand universe: If S is a set of clauses, then HS, the Herbrand universe of S, is\nHERBRAND\nUNIVERSE\nthe set of all ground terms constructable from the following:\na. The function symbols in S, if any.\nb. The constant symbols in S, if any; if none, then the constant symbol A.\nFor example, if S contains just the clause ¬P(x, F(x, A)) ∨¬Q(x, A) ∨R(x, B), then\nHS is the following infinite set of ground terms:\n{A, B, F(A, A), F(A, B), F(B, A), F(B, B), F(A, F(A, A)), . . .} .\n• Saturation: If S is a set of clauses and P is a set of ground terms, then P(S), the\nSATURATION\nsaturation of S with respect to P, is the set of all ground clauses obtained by applying\nall possible consistent substitutions of ground terms in P with variables in S.\n• Herbrand base: The saturation of a set S of clauses with respect to its Herbrand uni-\nHERBRAND BASE\nverse is called the Herbrand base of S, written as HS(S). For example, if S contains\nsolely the clause just given, then HS(S) is the infinite set of clauses\n{¬P(A, F(A, A)) ∨¬Q(A, A) ∨R(A, B),\n¬P(B, F(B, A)) ∨¬Q(B, A) ∨R(B, B),\n¬P(F(A, A), F(F(A, A), A)) ∨¬Q(F(A, A), A) ∨R(F(A, A), B),\n¬P(F(A, B), F(F(A, B), A)) ∨¬Q(F(A, B), A) ∨R(F(A, B), B), . . . }\nThese definitions allow us to state a form of Herbrand’s theorem (Herbrand, 1930):\nHERBRAND’S\nTHEOREM\nIf a set S of clauses is unsatisfiable, then there exists a finite subset of HS(S) that\nis also unsatisfiable.\nLet S′ be this finite subset of ground sentences. Now, we can appeal to the ground resolution\ntheorem (page 255) to show that the resolution closure RC(S′) contains the empty clause.\nThat is, running propositional resolution to completion on S′ will derive a contradiction.\nNow that we have established that there is always a resolution proof involving some\nfinite subset of the Herbrand base of S, the next step is to show that there is a resolution\n352\nChapter\n9.\nInference in First-Order Logic\nG ¨ODEL’S INCOMPLETENESS THEOREM\nBy slightly extending the language of first-order logic to allow for the mathemat-\nical induction schema in arithmetic, Kurt G¨odel was able to show, in his incom-\npleteness theorem, that there are true arithmetic sentences that cannot be proved.\nThe proof of the incompleteness theorem is somewhat beyond the scope of\nthis book, occupying, as it does, at least 30 pages, but we can give a hint here. We\nbegin with the logical theory of numbers. In this theory, there is a single constant,\n0, and a single function, S (the successor function). In the intended model, S(0)\ndenotes 1, S(S(0)) denotes 2, and so on; the language therefore has names for all\nthe natural numbers. The vocabulary also includes the function symbols +, ×, and\nExpt (exponentiation) and the usual set of logical connectives and quantifiers. The\nfirst step is to notice that the set of sentences that we can write in this language can\nbe enumerated. (Imagine defining an alphabetical order on the symbols and then\narranging, in alphabetical order, each of the sets of sentences of length 1, 2, and\nso on.) We can then number each sentence α with a unique natural number #α\n(the G¨odel number). This is crucial: number theory contains a name for each of\nits own sentences. Similarly, we can number each possible proof P with a G¨odel\nnumber G(P), because a proof is simply a finite sequence of sentences.\nNow suppose we have a recursively enumerable set A of sentences that are\ntrue statements about the natural numbers. Recalling that A can be named by a\ngiven set of integers, we can imagine writing in our language a sentence α(j, A) of\nthe following sort:\n∀i i is not the G¨odel number of a proof of the sentence whose G¨odel\nnumber is j, where the proof uses only premises in A.\nThen let σ be the sentence α(#σ, A), that is, a sentence that states its own unprov-\nability from A. (That this sentence always exists is true but not entirely obvious.)\nNow we make the following ingenious argument: Suppose that σ is provable\nfrom A; then σ is false (because σ says it cannot be proved). But then we have a\nfalse sentence that is provable from A, so A cannot consist of only true sentences—\na violation of our premise. Therefore, σ is not provable from A. But this is exactly\nwhat σ itself claims; hence σ is a true sentence.\nSo, we have shown (barring 29 1\n2 pages) that for any set of true sentences of\nnumber theory, and in particular any set of basic axioms, there are other true sen-\ntences that cannot be proved from those axioms. This establishes, among other\nthings, that we can never prove all the theorems of mathematics within any given\nsystem of axioms. Clearly, this was an important discovery for mathematics. Its\nsignificance for AI has been widely debated, beginning with speculations by G¨odel\nhimself. We take up the debate in Chapter 26.\nSection 9.5.\nResolution\n353\nproof using the clauses of S itself, which are not necessarily ground clauses. We start by\nconsidering a single application of the resolution rule. Robinson stated this lemma:\nLet C1 and C2 be two clauses with no shared variables, and let C′\n1 and C′\n2 be\nground instances of C1 and C2. If C′ is a resolvent of C′\n1 and C′\n2, then there exists\na clause C such that (1) C is a resolvent of C1 and C2 and (2) C′ is a ground\ninstance of C.\nThis is called a lifting lemma, because it lifts a proof step from ground clauses up to general\nLIFTING LEMMA\nfirst-order clauses. In order to prove his basic lifting lemma, Robinson had to invent unifi-\ncation and derive all of the properties of most general unifiers. Rather than repeat the proof\nhere, we simply illustrate the lemma:\nC1 = ¬P(x, F(x, A)) ∨¬Q(x, A) ∨R(x, B)\nC2 = ¬N(G(y), z) ∨P(H(y), z)\nC′\n1 = ¬P(H(B), F(H(B), A)) ∨¬Q(H(B), A) ∨R(H(B), B)\nC′\n2 = ¬N(G(B), F(H(B), A)) ∨P(H(B), F(H(B), A))\nC′ = ¬N(G(B), F(H(B), A)) ∨¬Q(H(B), A) ∨R(H(B), B)\nC = ¬N(G(y), F(H(y), A)) ∨¬Q(H(y), A) ∨R(H(y), B) .\nWe see that indeed C′ is a ground instance of C. In general, for C′\n1 and C′\n2 to have any\nresolvents, they must be constructed by first applying to C1 and C2 the most general unifier\nof a pair of complementary literals in C1 and C2. From the lifting lemma, it is easy to derive\na similar statement about any sequence of applications of the resolution rule:\nFor any clause C′ in the resolution closure of S′ there is a clause C in the resolu-\ntion closure of S such that C′ is a ground instance of C and the derivation of C is\nthe same length as the derivation of C′.\nFrom this fact, it follows that if the empty clause appears in the resolution closure of S′, it\nmust also appear in the resolution closure of S. This is because the empty clause cannot be a\nground instance of any other clause. To recap: we have shown that if S is unsatisfiable, then\nthere is a finite derivation of the empty clause using the resolution rule.\nThe lifting of theorem proving from ground clauses to first-order clauses provides a vast\nincrease in power. This increase comes from the fact that the first-order proof need instantiate\nvariables only as far as necessary for the proof, whereas the ground-clause methods were\nrequired to examine a huge number of arbitrary instantiations.\n9.5.5\nEquality\nNone of the inference methods described so far in this chapter handle an assertion of the form\nx = y. Three distinct approaches can be taken. The first approach is to axiomatize equality—\nto write down sentences about the equality relation in the knowledge base. We need to say that\nequality is reflexive, symmetric, and transitive, and we also have to say that we can substitute\nequals for equals in any predicate or function. So we need three basic axioms, and then one\n354\nChapter\n9.\nInference in First-Order Logic\nfor each predicate and function:\n∀x x = x\n∀x, y x = y ⇒y = x\n∀x, y, z x = y ∧y = z ⇒x = z\n∀x, y x = y ⇒(P1(x) ⇔P1(y))\n∀x, y x = y ⇒(P2(x) ⇔P2(y))\n...\n∀w, x, y, z w = y ∧x = z ⇒(F1(w, x) = F1(y, z))\n∀w, x, y, z w = y ∧x = z ⇒(F2(w, x) = F2(y, z))\n...\nGiven these sentences, a standard inference procedure such as resolution can perform tasks\nrequiring equality reasoning, such as solving mathematical equations. However, these axioms\nwill generate a lot of conclusions, most of them not helpful to a proof. So there has been a\nsearch for more efficient ways of handling equality. One alternative is to add inference rules\nrather than axioms. The simplest rule, demodulation, takes a unit clause x = y and some\nclause α that contains the term x, and yields a new clause formed by substituting y for x\nwithin α. It works if the term within α unifies with x; it need not be exactly equal to x.\nNote that demodulation is directional; given x = y, the x always gets replaced with y, never\nvice versa. That means that demodulation can be used for simplifying expressions using\ndemodulators such as x + 0 = x or x1 = x. As another example, given\nFather(Father(x)) = PaternalGrandfather (x)\nBirthdate(Father(Father(Bella)), 1926)\nwe can conclude by demodulation\nBirthdate(PaternalGrandfather (Bella), 1926) .\nMore formally, we have\n• Demodulation: For any terms x, y, and z, where z appears somewhere in literal mi\nDEMODULATION\nand where UNIFY(x, z) = θ,\nx = y,\nm1 ∨· · · ∨mn\nSUB(SUBST(θ, x), SUBST(θ, y), m1 ∨· · · ∨mn) .\nwhere SUBST is the usual substitution of a binding list, and SUB(x, y, m) means to\nreplace x with y everywhere that x occurs within m.\nThe rule can also be extended to handle non-unit clauses in which an equality literal appears:\n• Paramodulation: For any terms x, y, and z, where z appears somewhere in literal mi,\nPARAMODULATION\nand where UNIFY(x, z) = θ,\nℓ1 ∨· · · ∨ℓk ∨x = y,\nm1 ∨· · · ∨mn\nSUB(SUBST(θ, x), SUBST(θ, y), SUBST(θ, ℓ1 ∨· · · ∨ℓk ∨m1 ∨· · · ∨mn) .\nFor example, from\nP(F(x, B), x) ∨Q(x)\nand\nF(A, y) = y ∨R(y)\nSection 9.5.\nResolution\n355\nwe have θ = UNIFY(F(A, y), F(x, B)) = {x/A, y/B}, and we can conclude by paramodu-\nlation the sentence\nP(B, A) ∨Q(A) ∨R(B) .\nParamodulation yields a complete inference procedure for first-order logic with equality.\nA third approach handles equality reasoning entirely within an extended unification\nalgorithm. That is, terms are unifiable if they are provably equal under some substitution,\nwhere “provably” allows for equality reasoning. For example, the terms 1 + 2 and 2 + 1\nnormally are not unifiable, but a unification algorithm that knows that x + y = y + x could\nunify them with the empty substitution. Equational unification of this kind can be done with\nEQUATIONAL\nUNIFICATION\nefficient algorithms designed for the particular axioms used (commutativity, associativity, and\nso on) rather than through explicit inference with those axioms. Theorem provers using this\ntechnique are closely related to the CLP systems described in Section 9.4.\n9.5.6\nResolution strategies\nWe know that repeated applications of the resolution inference rule will eventually find a\nproof if one exists. In this subsection, we examine strategies that help find proofs efficiently.\nUnit preference: This strategy prefers to do resolutions where one of the sentences is a single\nUNIT PREFERENCE\nliteral (also known as a unit clause). The idea behind the strategy is that we are trying to\nproduce an empty clause, so it might be a good idea to prefer inferences that produce shorter\nclauses. Resolving a unit sentence (such as P) with any other sentence (such as ¬P ∨¬Q∨R)\nalways yields a clause (in this case, ¬Q ∨R) that is shorter than the other clause. When\nthe unit preference strategy was first tried for propositional inference in 1964, it led to a\ndramatic speedup, making it feasible to prove theorems that could not be handled without the\npreference. Unit resolution is a restricted form of resolution in which every resolution step\nmust involve a unit clause. Unit resolution is incomplete in general, but complete for Horn\nclauses. Unit resolution proofs on Horn clauses resemble forward chaining.\nThe OTTER theorem prover (Organized Techniques for Theorem-proving and Effective\nResearch, McCune, 1992), uses a form of best-first search. Its heuristic function measures\nthe “weight” of each clause, where lighter clauses are preferred. The exact choice of heuristic\nis up to the user, but generally, the weight of a clause should be correlated with its size or\ndifficulty. Unit clauses are treated as light; the search can thus be seen as a generalization of\nthe unit preference strategy.\nSet of support: Preferences that try certain resolutions first are helpful, but in general it is\nSET OF SUPPORT\nmore effective to try to eliminate some potential resolutions altogether. For example, we can\ninsist that every resolution step involve at least one element of a special set of clauses—the\nset of support. The resolvent is then added into the set of support. If the set of support is\nsmall relative to the whole knowledge base, the search space will be reduced dramatically.\nWe have to be careful with this approach because a bad choice for the set of support\nwill make the algorithm incomplete. However, if we choose the set of support S so that the\nremainder of the sentences are jointly satisfiable, then set-of-support resolution is complete.\nFor example, one can use the negated query as the set of support, on the assumption that the\n356\nChapter\n9.\nInference in First-Order Logic\noriginal knowledge base is consistent. (After all, if it is not consistent, then the fact that the\nquery follows from it is vacuous.) The set-of-support strategy has the additional advantage of\ngenerating goal-directed proof trees that are often easy for humans to understand.\nInput resolution: In this strategy, every resolution combines one of the input sentences (from\nINPUT RESOLUTION\nthe KB or the query) with some other sentence. The proof in Figure 9.11 on page 348 uses\nonly input resolutions and has the characteristic shape of a single “spine” with single sen-\ntences combining onto the spine. Clearly, the space of proof trees of this shape is smaller\nthan the space of all proof graphs. In Horn knowledge bases, Modus Ponens is a kind of\ninput resolution strategy, because it combines an implication from the original KB with some\nother sentences. Thus, it is no surprise that input resolution is complete for knowledge bases\nthat are in Horn form, but incomplete in the general case. The linear resolution strategy is a\nLINEAR RESOLUTION\nslight generalization that allows P and Q to be resolved together either if P is in the original\nKB or if P is an ancestor of Q in the proof tree. Linear resolution is complete.\nSubsumption: The subsumption method eliminates all sentences that are subsumed by (that\nSUBSUMPTION\nis, more specific than) an existing sentence in the KB. For example, if P(x) is in the KB, then\nthere is no sense in adding P(A) and even less sense in adding P(A) ∨Q(B). Subsumption\nhelps keep the KB small and thus helps keep the search space small.\nPractical uses of resolution theorem provers\nTheorem provers can be applied to the problems involved in the synthesis and verification\nSYNTHESIS\nVERIFICATION\nof both hardware and software. Thus, theorem-proving research is carried out in the fields of\nhardware design, programming languages, and software engineering—not just in AI.\nIn the case of hardware, the axioms describe the interactions between signals and cir-\ncuit elements. (See Section 8.4.2 on page 309 for an example.) Logical reasoners designed\nspecially for verification have been able to verify entire CPUs, including their timing prop-\nerties (Srivas and Bickford, 1990). The AURA theorem prover has been applied to design\ncircuits that are more compact than any previous design (Wojciechowski and Wojcik, 1983).\nIn the case of software, reasoning about programs is quite similar to reasoning about\nactions, as in Chapter 7: axioms describe the preconditions and effects of each statement.\nThe formal synthesis of algorithms was one of the first uses of theorem provers, as outlined\nby Cordell Green (1969a), who built on earlier ideas by Herbert Simon (1963). The idea\nis to constructively prove a theorem to the effect that “there exists a program p satisfying a\ncertain specification.” Although fully automated deductive synthesis, as it is called, has not\nDEDUCTIVE\nSYNTHESIS\nyet become feasible for general-purpose programming, hand-guided deductive synthesis has\nbeen successful in designing several novel and sophisticated algorithms. Synthesis of special-\npurpose programs, such as scientific computing code, is also an active area of research.\nSimilar techniques are now being applied to software verification by systems such as the\nSPIN model checker (Holzmann, 1997). For example, the Remote Agent spacecraft control\nprogram was verified before and after flight (Havelund et al., 2000). The RSA public key\nencryption algorithm and the Boyer–Moore string-matching algorithm have been verified this\nway (Boyer and Moore, 1984).\nSection 9.6.\nSummary\n357\n9.6\nSUMMARY\nWe have presented an analysis of logical inference in first-order logic and a number of algo-\nrithms for doing it.\n• A first approach uses inference rules (universal instantiation and existential instan-\ntiation) to propositionalize the inference problem. Typically, this approach is slow,\nunless the domain is small.\n• The use of unification to identify appropriate substitutions for variables eliminates the\ninstantiation step in first-order proofs, making the process more efficient in many cases.\n• A lifted version of Modus Ponens uses unification to provide a natural and powerful\ninference rule, generalized Modus Ponens. The forward-chaining and backward-\nchaining algorithms apply this rule to sets of definite clauses.\n• Generalized Modus Ponens is complete for definite clauses, although the entailment\nproblem is semidecidable. For Datalog knowledge bases consisting of function-free\ndefinite clauses, entailment is decidable.\n• Forward chaining is used in deductive databases, where it can be combined with re-\nlational database operations. It is also used in production systems, which perform\nefficient updates with very large rule sets. Forward chaining is complete for Datalog\nand runs in polynomial time.\n• Backward chaining is used in logic programming systems, which employ sophisti-\ncated compiler technology to provide very fast inference. Backward chaining suffers\nfrom redundant inferences and infinite loops; these can be alleviated by memoization.\n• Prolog, unlike first-order logic, uses a closed world with the unique names assumption\nand negation as failure. These make Prolog a more practical programming language,\nbut bring it further from pure logic.\n• The generalized resolution inference rule provides a complete proof system for first-\norder logic, using knowledge bases in conjunctive normal form.\n• Several strategies exist for reducing the search space of a resolution system without\ncompromising completeness. One of the most important issues is dealing with equality;\nwe showed how demodulation and paramodulation can be used.\n• Efficient resolution-based theorem provers have been used to prove interesting mathe-\nmatical theorems and to verify and synthesize software and hardware.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nGottlob Frege, who developed full first-order logic in 1879, based his system of inference\non a collection of valid schemas plus a single inference rule, Modus Ponens. Whitehead\nand Russell (1910) expounded the so-called rules of passage (the actual term is from Her-\nbrand (1930)) that are used to move quantifiers to the front of formulas. Skolem constants\n358\nChapter\n9.\nInference in First-Order Logic\nand Skolem functions were introduced, appropriately enough, by Thoralf Skolem (1920).\nOddly enough, it was Skolem who introduced the Herbrand universe (Skolem, 1928).\nHerbrand’s theorem (Herbrand, 1930) has played a vital role in the development of\nautomated reasoning. Herbrand is also the inventor of unification. G¨odel (1930) built on\nthe ideas of Skolem and Herbrand to show that first-order logic has a complete proof pro-\ncedure. Alan Turing (1936) and Alonzo Church (1936) simultaneously showed, using very\ndifferent proofs, that validity in first-order logic was not decidable. The excellent text by\nEnderton (1972) explains all of these results in a rigorous yet understandable fashion.\nAbraham Robinson proposed that an automated reasoner could be built using proposi-\ntionalization and Herbrand’s theorem, and Paul Gilmore (1960) wrote the first program. Davis\nand Putnam (1960) introduced the propositionalization method of Section 9.1. Prawitz (1960)\ndeveloped the key idea of letting the quest for propositional inconsistency drive the search,\nand generating terms from the Herbrand universe only when they were necessary to estab-\nlish propositional inconsistency. After further development by other researchers, this idea led\nJ. A. Robinson (no relation) to develop resolution (Robinson, 1965).\nIn AI, resolution was adopted for question-answering systems by Cordell Green and\nBertram Raphael (1968). Early AI implementations put a good deal of effort into data struc-\ntures that would allow efficient retrieval of facts; this work is covered in AI programming\ntexts (Charniak et al., 1987; Norvig, 1992; Forbus and de Kleer, 1993). By the early 1970s,\nforward chaining was well established in AI as an easily understandable alternative to res-\nolution. AI applications typically involved large numbers of rules, so it was important to\ndevelop efficient rule-matching technology, particularly for incremental updates. The tech-\nnology for production systems was developed to support such applications. The production\nsystem language OPS-5 (Forgy, 1981; Brownston et al., 1985), incorporating the efficient\nrete match process (Forgy, 1982), was used for applications such as the R1 expert system for\nRETE\nminicomputer configuration (McDermott, 1982).\nThe SOAR cognitive architecture (Laird et al., 1987; Laird, 2008) was designed to han-\ndle very large rule sets—up to a million rules (Doorenbos, 1994). Example applications of\nSOAR include controlling simulated fighter aircraft (Jones et al., 1998), airspace manage-\nment (Taylor et al., 2007), AI characters for computer games (Wintermute et al., 2007), and\ntraining tools for soldiers (Wray and Jones, 2005).\nThe field of deductive databases began with a workshop in Toulouse in 1977 that\nbrought together experts in logical inference and database systems (Gallaire and Minker,\n1978). Influential work by Chandra and Harel (1980) and Ullman (1985) led to the adoption\nof Datalog as a standard language for deductive databases. The development of the magic sets\ntechnique for rule rewriting by Bancilhon et al. (1986) allowed forward chaining to borrow\nthe advantage of goal-directedness from backward chaining. Current work includes the idea\nof integrating multiple databases into a consistent dataspace (Halevy, 2007).\nBackward chaining for logical inference appeared first in Hewitt’s PLANNER lan-\nguage (1969). Meanwhile, in 1972, Alain Colmerauer had developed and implemented Pro-\nlog for the purpose of parsing natural language—Prolog’s clauses were intended initially\nas context-free grammar rules (Roussel, 1975; Colmerauer et al., 1973). Much of the the-\noretical background for logic programming was developed by Robert Kowalski, working\nBibliographical and Historical Notes\n359\nwith Colmerauer; see Kowalski (1988) and Colmerauer and Roussel (1993) for a historical\noverview. Efficient Prolog compilers are generally based on the Warren Abstract Machine\n(WAM) model of computation developed by David H. D. Warren (1983). Van Roy (1990)\nshowed that Prolog programs can be competitive with C programs in terms of speed.\nMethods for avoiding unnecessary looping in recursive logic programs were developed\nindependently by Smith et al. (1986) and Tamaki and Sato (1986). The latter paper also\nincluded memoization for logic programs, a method developed extensively as tabled logic\nprogramming by David S. Warren. Swift and Warren (1994) show how to extend the WAM\nto handle tabling, enabling Datalog programs to execute an order of magnitude faster than\nforward-chaining deductive database systems.\nEarly work on constraint logic programming was done by Jaffar and Lassez (1987).\nJaffar et al. (1992) developed the CLP(R) system for handling real-valued constraints. There\nare now commercial products for solving large-scale configuration and optimization problems\nwith constraint programming; one of the best known is ILOG (Junker, 2003). Answer set\nprogramming (Gelfond, 2008) extends Prolog, allowing disjunction and negation.\nTexts on logic programming and Prolog, including Shoham (1994), Bratko (2001),\nClocksin (2003), and Clocksin and Mellish (2003). Prior to 2000, the Journal of Logic Pro-\ngramming was the journal of record; it has now been replaced by Theory and Practice of\nLogic Programming. Logic programming conferences include the International Conference\non Logic Programming (ICLP) and the International Logic Programming Symposium (ILPS).\nResearch into mathematical theorem proving began even before the first complete\nfirst-order systems were developed. Herbert Gelernter’s Geometry Theorem Prover (Gelern-\nter, 1959) used heuristic search methods combined with diagrams for pruning false subgoals\nand was able to prove some quite intricate results in Euclidean geometry. The demodula-\ntion and paramodulation rules for equality reasoning were introduced by Wos et al. (1967)\nand Wos and Robinson (1968), respectively. These rules were also developed independently\nin the context of term-rewriting systems (Knuth and Bendix, 1970). The incorporation of\nequality reasoning into the unification algorithm is due to Gordon Plotkin (1972). Jouannaud\nand Kirchner (1991) survey equational unification from a term-rewriting perspective. An\noverview of unification is given by Baader and Snyder (2001).\nA number of control strategies have been proposed for resolution, beginning with the\nunit preference strategy (Wos et al., 1964). The set-of-support strategy was proposed by Wos\net al. (1965) to provide a degree of goal-directedness in resolution. Linear resolution first\nappeared in Loveland (1970). Genesereth and Nilsson (1987, Chapter 5) provide a short but\nthorough analysis of a wide variety of control strategies.\nA Computational Logic (Boyer and Moore, 1979) is the basic reference on the Boyer-\nMoore theorem prover. Stickel (1992) covers the Prolog Technology Theorem Prover (PTTP),\nwhich combines the advantages of Prolog compilation with the completeness of model elimi-\nnation. SETHEO (Letz et al., 1992) is another widely used theorem prover based on this ap-\nproach. LEANTAP (Beckert and Posegga, 1995) is an efficient theorem prover implemented\nin only 25 lines of Prolog. Weidenbach (2001) describes SPASS, one of the strongest current\ntheorem provers. The most successful theorem prover in recent annual competitions has been\nVAMPIRE (Riazanov and Voronkov, 2002). The COQ system (Bertot et al., 2004) and the E\n360\nChapter\n9.\nInference in First-Order Logic\nequational solver (Schulz, 2004) have also proven to be valuable tools for proving correct-\nness. Theorem provers have been used to automatically synthesize and verify software for\ncontrolling spacecraft (Denney et al., 2006), including NASA’s new Orion capsule (Lowry,\n2008). The design of the FM9001 32-bit microprocessor was proved correct by the NQTHM\nsystem (Hunt and Brock, 1992). The Conference on Automated Deduction (CADE) runs an\nannual contest for automated theorem provers. From 2002 through 2008, the most successful\nsystem has been VAMPIRE (Riazanov and Voronkov, 2002). Wiedijk (2003) compares the\nstrength of 15 mathematical provers. TPTP (Thousands of Problems for Theorem Provers)\nis a library of theorem-proving problems, useful for comparing the performance of systems\n(Sutcliffe and Suttner, 1998; Sutcliffe et al., 2006).\nTheorem provers have come up with novel mathematical results that eluded human\nmathematicians for decades, as detailed in the book Automated Reasoning and the Discov-\nery of Missing Elegant Proofs (Wos and Pieper, 2003). The SAM (Semi-Automated Math-\nematics) program was the first, proving a lemma in lattice theory (Guard et al., 1969). The\nAURA program has also answered open questions in several areas of mathematics (Wos and\nWinker, 1983). The Boyer–Moore theorem prover (Boyer and Moore, 1979) was used by\nNatarajan Shankar to give the first fully rigorous formal proof of G¨odel’s Incompleteness\nTheorem (Shankar, 1986). The NUPRL system proved Girard’s paradox (Howe, 1987) and\nHigman’s Lemma (Murthy and Russell, 1990). In 1933, Herbert Robbins proposed a simple\nset of axioms—the Robbins algebra—that appeared to define Boolean algebra, but no proof\nROBBINS ALGEBRA\ncould be found (despite serious work by Alfred Tarski and others). On October 10, 1996,\nafter eight days of computation, EQP (a version of OTTER) found a proof (McCune, 1997).\nMany early papers in mathematical logic are to be found in From Frege to G¨odel:\nA Source Book in Mathematical Logic (van Heijenoort, 1967). Textbooks geared toward\nautomated deduction include the classic Symbolic Logic and Mechanical Theorem Prov-\ning (Chang and Lee, 1973), as well as more recent works by Duffy (1991), Wos et al. (1992),\nBibel (1993), and Kaufmann et al. (2000). The principal journal for theorem proving is the\nJournal of Automated Reasoning; the main conferences are the annual Conference on Auto-\nmated Deduction (CADE) and the International Joint Conference on Automated Reasoning\n(IJCAR). The Handbook of Automated Reasoning (Robinson and Voronkov, 2001) collects\npapers in the field. MacKenzie’s Mechanizing Proof (2004) covers the history and technology\nof theorem proving for the popular audience.\nEXERCISES\n9.1\nProve that Universal Instantiation is sound and that Existential Instantiation produces\nan inferentially equivalent knowledge base.\n9.2\nFrom Likes(Jerry, IceCream) it seems reasonable to infer ∃x Likes(x, IceCream).\nWrite down a general inference rule, Existential Introduction, that sanctions this inference.\nEXISTENTIAL\nINTRODUCTION\nState carefully the conditions that must be satisfied by the variables and terms involved.\nExercises\n361\n9.3\nSuppose a knowledge base contains just one sentence, ∃x AsHighAs(x, Everest).\nWhich of the following are legitimate results of applying Existential Instantiation?\na. AsHighAs(Everest, Everest).\nb. AsHighAs(Kilimanjaro, Everest).\nc. AsHighAs(Kilimanjaro, Everest) ∧AsHighAs(BenNevis, Everest)\n(after two applications).\n9.4\nFor each pair of atomic sentences, give the most general unifier if it exists:\na. P(A, B, B), P(x, y, z).\nb. Q(y, G(A, B)), Q(G(x, x), y).\nc. Older(Father(y), y), Older(Father(x), John).\nd. Knows(Father(y), y), Knows(x, x).\n9.5\nConsider the subsumption lattices shown in Figure 9.2 (page 329).\na. Construct the lattice for the sentence Employs(Mother(John), Father(Richard)).\nb. Construct the lattice for the sentence Employs(IBM , y) (“Everyone works for IBM”).\nRemember to include every kind of query that unifies with the sentence.\nc. Assume that STORE indexes each sentence under every node in its subsumption lattice.\nExplain how FETCH should work when some of these sentences contain variables; use\nas examples the sentences in (a) and (b) and the query Employs(x, Father(x)).\n9.6\nWrite down logical representations for the following sentences, suitable for use with\nGeneralized Modus Ponens:\na. Horses, cows, and pigs are mammals.\nb. An offspring of a horse is a horse.\nc. Bluebeard is a horse.\nd. Bluebeard is Charlie’s parent.\ne. Offspring and parent are inverse relations.\nf. Every mammal has a parent.\n9.7\nThese questions concern concern issues with substitution and Skolemization.\na. Given the premise ∀x ∃y P(x, y), it is not valid to conclude that ∃q P(q, q). Give\nan example of a predicate P where the first is true but the second is false.\nb. Suppose that an inference engine is incorrectly written with the occurs check omitted,\nso that it allows a literal like P(x, F(x)) to be unified with P(q, q). (As mentioned,\nmost standard implementations of Prolog actually do allow this.) Show that such an\ninference engine will allow the conclusion ∃y P(q, q) to be inferred from the premise\n∀x ∃y P(x, y).\n362\nChapter\n9.\nInference in First-Order Logic\nc. Suppose that a procedure that converts first-order logic to clausal form incorrectly\nSkolemizes ∀x\n∃y\nP(x, y) to P(x, Sk0)—that is, it replaces y by a Skolem con-\nstant rather than by a Skolem function of x. Show that an inference engine that uses\nsuch a procedure will likewise allow ∃q\nP(q, q) to be inferred from the premise\n∀x ∃y P(x, y).\nd. A common error among students is to suppose that, in unification, one is allowed to\nsubstitute a term for a Skolem constant instead of for a variable. For instance, they will\nsay that the formulas P(Sk1) and P(A) can be unified under the substitution {Sk1/A}.\nGive an example where this leads to an invalid inference.\n9.8\nExplain how to write any given 3-SAT problem of arbitrary size using a single first-order\ndefinite clause and no more than 30 ground facts.\n9.9\nSuppose you are given the following axioms:\n1. 0 ≤3.\n2. 7 ≤9.\n3. ∀x\nx ≤x.\n4. ∀x\nx ≤x + 0.\n5. ∀x\nx + 0 ≤x.\n6. ∀x, y\nx + y ≤y + x.\n7. ∀w, x, y, z\nw ≤y ∧x ≤z ⇒w + x ≤y + z.\n8. ∀x, y, z\nx ≤y ∧y ≤z ⇒x ≤z\na. Give a backward-chaining proof of the sentence 7 ≤3 + 9. (Be sure, of course, to use\nonly the axioms given here, not anything else you may know about arithmetic.) Show\nonly the steps that leads to success, not the irrelevant steps.\nb. Give a forward-chaining proof of the sentence 7 ≤3 + 9. Again, show only the steps\nthat lead to success.\n9.10\nA popular children’s riddle is “Brothers and sisters have I none, but that man’s father\nis my father’s son.” Use the rules of the family domain (Section 8.3.2 on page 301) to show\nwho that man is. You may apply any of the inference methods described in this chapter. Why\ndo you think that this riddle is difficult?\n9.11\nSuppose we put into a logical knowledge base a segment of the U.S. census data list-\ning the age, city of residence, date of birth, and mother of every person, using social se-\ncurity numbers as identifying constants for each person. Thus, George’s age is given by\nAge(443-65-1282, 56). Which of the following indexing schemes S1–S5 enable an efficient\nsolution for which of the queries Q1–Q4 (assuming normal backward chaining)?\n• S1: an index for each atom in each position.\n• S2: an index for each first argument.\n• S3: an index for each predicate atom.\n• S4: an index for each combination of predicate and first argument.\nExercises\n363\n• S5: an index for each combination of predicate and second argument and an index for\neach first argument.\n• Q1: Age(443-44-4321, x)\n• Q2: ResidesIn(x, Houston)\n• Q3: Mother(x, y)\n• Q4: Age(x, 34) ∧ResidesIn(x, TinyTownUSA)\n9.12\nOne might suppose that we can avoid the problem of variable conflict in unification\nduring backward chaining by standardizing apart all of the sentences in the knowledge base\nonce and for all. Show that, for some sentences, this approach cannot work. (Hint: Consider\na sentence in which one part unifies with another.)\n9.13\nIn this exercise, use the sentences you wrote in Exercise 9.6 to answer a question by\nusing a backward-chaining algorithm.\na. Draw the proof tree generated by an exhaustive backward-chaining algorithm for the\nquery ∃h Horse(h), where clauses are matched in the order given.\nb. What do you notice about this domain?\nc. How many solutions for h actually follow from your sentences?\nd. Can you think of a way to find all of them? (Hint: See Smith et al. (1986).)\n9.14\nTrace the execution of the backward-chaining algorithm in Figure 9.6 (page 338) when\nit is applied to solve the crime problem (page 330). Show the sequence of values taken on by\nthe goals variable, and arrange them into a tree.\n9.15\nThe following Prolog code defines a predicate P. (Remember that uppercase terms are\nvariables, not constants, in Prolog.)\nP(X,[X|Y]).\nP(X,[Y|Z]) :- P(X,Z).\na. Show proof trees and solutions for the queries P(A,[2,1,3])and P(2,[1,A,3]).\nb. What standard list operation does P represent?\n9.16\nThis exercise looks at sorting in Prolog.\na. Write Prolog clauses that define the predicate sorted(L), which is true if and only if\nlist L is sorted in ascending order.\nb. Write a Prolog definition for the predicate perm(L,M), which is true if and only if L\nis a permutation of M.\nc. Define sort(L,M) (M is a sorted version of L) using perm and sorted.\nd. Run sort on longer and longer lists until you lose patience. What is the time complex-\nity of your program?\ne. Write a faster sorting algorithm, such as insertion sort or quicksort, in Prolog.\n364\nChapter\n9.\nInference in First-Order Logic\n9.17\nThis exercise looks at the recursive application of rewrite rules, using logic program-\nming. A rewrite rule (or demodulator in OTTER terminology) is an equation with a specified\ndirection. For example, the rewrite rule x + 0 →x suggests replacing any expression that\nmatches x+0 with the expression x. Rewrite rules are a key component of equational reason-\ning systems. Use the predicate rewrite(X,Y) to represent rewrite rules. For example, the\nearlier rewrite rule is written as rewrite(X+0,X). Some terms are primitive and cannot\nbe further simplified; thus, we write primitive(0) to say that 0 is a primitive term.\na. Write a definition of a predicate simplify(X,Y), that is true when Y is a simplified\nversion of X—that is, when no further rewrite rules apply to any subexpression of Y.\nb. Write a collection of rules for the simplification of expressions involving arithmetic\noperators, and apply your simplification algorithm to some sample expressions.\nc. Write a collection of rewrite rules for symbolic differentiation, and use them along with\nyour simplification rules to differentiate and simplify expressions involving arithmetic\nexpressions, including exponentiation.\n9.18\nThis exercise considers the implementation of search algorithms in Prolog. Suppose\nthat successor(X,Y) is true when state Y is a successor of state X; and that goal(X)\nis true when X is a goal state. Write a definition for solve(X,P), which means that P is a\npath (list of states) beginning with X, ending in a goal state, and consisting of a sequence of\nlegal steps as defined by successor. You will find that depth-first search is the easiest way\nto do this. How easy would it be to add heuristic search control?\n9.19\nSuppose a knowledge base contains just the following first-order Horn clauses:\nAncestor(Mother(x), x)\nAncestor(x, y) ∧Ancestor(y, z) ⇒Ancestor(x, z)\nConsider a forward chaining algorithm that, on the jth iteration, terminates if the KB contains\na sentence that unifies with the query, else adds to the KB every atomic sentence that can be\ninferred from the sentences already in the KB after iteration j −1.\na. For each of the following queries, say whether the algorithm will (1) give an answer (if\nso, write down that answer); or (2) terminate with no answer; or (3) never terminate.\n(i) Ancestor(Mother(y), John)\n(ii) Ancestor(Mother(Mother(y)), John)\n(iii) Ancestor(Mother(Mother(Mother(y))), Mother(y))\n(iv) Ancestor(Mother(John), Mother(Mother(John)))\nb. Can a resolution algorithm prove the sentence ¬Ancestor(John, John) from the orig-\ninal knowledge base? Explain how, or why not.\nc. Suppose we add the assertion that ¬(Mother(x) = x) and augment the resolution al-\ngorithm with inference rules for equality. Now what is the answer to (b)?\n9.20\nLet L be the first-order language with a single predicate S(p, q), meaning “p shaves q.”\nAssume a domain of people.\nExercises\n365\na. Consider the sentence “There exists a person P who shaves every one who does not\nshave themselves, and only people that do not shave themselves.” Express this in L.\nb. Convert the sentence in (a) to clausal form.\nc. Construct a resolution proof to show that the clauses in (b) are inherently inconsistent.\n(Note: you do not need any additional axioms.)\n9.21\nHow can resolution be used to show that a sentence is valid? Unsatisfiable?\n9.22\nConstruct an example of two clauses that can be resolved together in two different\nways giving two different outcomes.\n9.23\nFrom “Horses are animals,” it follows that “The head of a horse is the head of an\nanimal.” Demonstrate that this inference is valid by carrying out the following steps:\na. Translate the premise and the conclusion into the language of first-order logic. Use three\npredicates: HeadOf (h, x) (meaning “h is the head of x”), Horse(x), and Animal(x).\nb. Negate the conclusion, and convert the premise and the negated conclusion into con-\njunctive normal form.\nc. Use resolution to show that the conclusion follows from the premise.\n9.24\nHere are two sentences in the language of first-order logic:\n(A) ∀x ∃y (x ≥y)\n(B) ∃y ∀x (x ≥y)\na. Assume that the variables range over all the natural numbers 0, 1, 2, . . . , ∞and that the\n“≥” predicate means “is greater than or equal to.” Under this interpretation, translate\n(A) and (B) into English.\nb. Is (A) true under this interpretation?\nc. Is (B) true under this interpretation?\nd. Does (A) logically entail (B)?\ne. Does (B) logically entail (A)?\nf. Using resolution, try to prove that (A) follows from (B). Do this even if you think that\n(B) does not logically entail (A); continue until the proof breaks down and you cannot\nproceed (if it does break down). Show the unifying substitution for each resolution step.\nIf the proof fails, explain exactly where, how, and why it breaks down.\ng. Now try to prove that (B) follows from (A).\n9.25\nResolution can produce nonconstructive proofs for queries with variables, so we had\nto introduce special mechanisms to extract definite answers. Explain why this issue does not\narise with knowledge bases containing only definite clauses.\n9.26\nWe said in this chapter that resolution cannot be used to generate all logical conse-\nquences of a set of sentences. Can any algorithm do this?\n" }, { "chapter": 10, "text": "10\nCLASSICAL PLANNING\nIn which we see how an agent can take advantage of the structure of a problem to\nconstruct complex plans of action.\nWe have defined AI as the study of rational action, which means that planning—devising a\nplan of action to achieve one’s goals—is a critical part of AI. We have seen two examples\nof planning agents so far: the search-based problem-solving agent of Chapter 3 and the hy-\nbrid logical agent of Chapter 7. In this chapter we introduce a representation for planning\nproblems that scales up to problems that could not be handled by those earlier approaches.\nSection 10.1 develops an expressive yet carefully constrained language for representing\nplanning problems. Section 10.2 shows how forward and backward search algorithms can\ntake advantage of this representation, primarily through accurate heuristics that can be derived\nautomatically from the structure of the representation. (This is analogous to the way in which\neffective domain-independent heuristics were constructed for constraint satisfaction problems\nin Chapter 6.) Section 10.3 shows how a data structure called the planning graph can make the\nsearch for a plan more efficient. We then describe a few of the other approaches to planning,\nand conclude by comparing the various approaches.\nThis chapter covers fully observable, deterministic, static environments with single\nagents. Chapters 11 and 17 cover partially observable, stochastic, dynamic environments\nwith multiple agents.\n10.1\nDEFINITION OF CLASSICAL PLANNING\nThe problem-solving agent of Chapter 3 can find sequences of actions that result in a goal\nstate. But it deals with atomic representations of states and thus needs good domain-specific\nheuristics to perform well. The hybrid propositional logical agent of Chapter 7 can find plans\nwithout domain-specific heuristics because it uses domain-independent heuristics based on\nthe logical structure of the problem. But it relies on ground (variable-free) propositional\ninference, which means that it may be swamped when there are many actions and states. For\nexample, in the wumpus world, the simple action of moving a step forward had to be repeated\nfor all four agent orientations, T time steps, and n2 current locations.\n366\nSection 10.1.\nDefinition of Classical Planning\n367\nIn response to this, planning researchers have settled on a factored representation—\none in which a state of the world is represented by a collection of variables. We use a language\ncalled PDDL, the Planning Domain Definition Language, that allows us to express all 4Tn2\nPDDL\nactions with one action schema. There have been several versions of PDDL; we select a\nsimple version and alter its syntax to be consistent with the rest of the book.1 We now show\nhow PDDL describes the four things we need to define a search problem: the initial state, the\nactions that are available in a state, the result of applying an action, and the goal test.\nEach state is represented as a conjunction of fluents that are ground, functionless atoms.\nFor example, Poor ∧Unknown might represent the state of a hapless agent, and a state\nin a package delivery problem might be At(Truck 1, Melbourne) ∧At(Truck 2, Sydney).\nDatabase semantics is used: the closed-world assumption means that any fluents that are not\nmentioned are false, and the unique names assumption means that Truck 1 and Truck 2 are\ndistinct. The following fluents are not allowed in a state: At(x, y) (because it is non-ground),\n¬Poor (because it is a negation), and At(Father(Fred), Sydney) (because it uses a function\nsymbol). The representation of states is carefully designed so that a state can be treated\neither as a conjunction of fluents, which can be manipulated by logical inference, or as a set\nof fluents, which can be manipulated with set operations. The set semantics is sometimes\nSET SEMANTICS\neasier to deal with.\nActions are described by a set of action schemas that implicitly define the ACTIONS(s)\nand RESULT(s, a) functions needed to do a problem-solving search. We saw in Chapter 7 that\nany system for action description needs to solve the frame problem—to say what changes and\nwhat stays the same as the result of the action. Classical planning concentrates on problems\nwhere most actions leave most things unchanged. Think of a world consisting of a bunch of\nobjects on a flat surface. The action of nudging an object causes that object to change its lo-\ncation by a vector Δ. A concise description of the action should mention only Δ; it shouldn’t\nhave to mention all the objects that stay in place. PDDL does that by specifying the result of\nan action in terms of what changes; everything that stays the same is left unmentioned.\nA set of ground (variable-free) actions can be represented by a single action schema.\nACTION SCHEMA\nThe schema is a lifted representation—it lifts the level of reasoning from propositional logic\nto a restricted subset of first-order logic. For example, here is an action schema for flying a\nplane from one location to another:\nAction(Fly(p, from, to),\nPRECOND:At(p, from) ∧Plane(p) ∧Airport(from) ∧Airport(to)\nEFFECT:¬At(p, from) ∧At(p, to))\nThe schema consists of the action name, a list of all the variables used in the schema, a\nprecondition and an effect. Although we haven’t said yet how the action schema converts\nPRECONDITION\nEFFECT\ninto logical sentences, think of the variables as being universally quantified. We are free to\nchoose whatever values we want to instantiate the variables. For example, here is one ground\n1 PDDL was derived from the original STRIPS planning language(Fikes and Nilsson, 1971). which is slightly\nmore restricted than PDDL: STRIPS preconditions and goals cannot contain negative literals.\n368\nChapter\n10.\nClassical Planning\naction that results from substituting values for all the variables:\nAction(Fly(P1, SFO, JFK),\nPRECOND:At(P1, SFO) ∧Plane(P1) ∧Airport(SFO) ∧Airport(JFK)\nEFFECT:¬At(P1, SFO) ∧At(P1, JFK))\nThe precondition and effect of an action are each conjunctions of literals (positive or negated\natomic sentences). The precondition defines the states in which the action can be executed,\nand the effect defines the result of executing the action. An action a can be executed in state\ns if s entails the precondition of a. Entailment can also be expressed with the set semantics:\ns |= q iff every positive literal in q is in s and every negated literal in q is not. In formal\nnotation we say\n(a ∈ACTIONS(s)) ⇔s |= PRECOND(a) ,\nwhere any variables in a are universally quantified. For example,\n∀p, from, to (Fly(p, from, to) ∈ACTIONS(s)) ⇔\ns |= (At(p, from) ∧Plane(p) ∧Airport(from) ∧Airport(to))\nWe say that action a is applicable in state s if the preconditions are satisfied by s. When\nAPPLICABLE\nan action schema a contains variables, it may have multiple applicable instantiations. For\nexample, with the initial state defined in Figure 10.1, the Fly action can be instantiated as\nFly(P1, SFO, JFK) or as Fly(P2, JFK , SFO), both of which are applicable in the initial\nstate. If an action a has v variables, then, in a domain with k unique names of objects, it takes\nO(vk) time in the worst case to find the applicable ground actions.\nSometimes we want to propositionalize a PDDL problem—replace each action schema\nPROPOSITIONALIZE\nwith a set of ground actions and then use a propositional solver such as SATPLAN to find a\nsolution. However, this is impractical when v and k are large.\nThe result of executing action a in state s is defined as a state s′ which is represented\nby the set of fluents formed by starting with s, removing the fluents that appear as negative\nliterals in the action’s effects (what we call the delete list or DEL(a)), and adding the fluents\nDELETE LIST\nthat are positive literals in the action’s effects (what we call the add list or ADD(a)):\nADD LIST\nRESULT(s, a) = (s −DEL(a)) ∪ADD(a) .\n(10.1)\nFor example, with the action Fly(P1, SFO, JFK), we would remove At(P1, SFO) and add\nAt(P1, JFK). It is a requirement of action schemas that any variable in the effect must also\nappear in the precondition. That way, when the precondition is matched against the state s,\nall the variables will be bound, and RESULT(s, a) will therefore have only ground atoms. In\nother words, ground states are closed under the RESULT operation.\nAlso note that the fluents do not explicitly refer to time, as they did in Chapter 7. There\nwe needed superscripts for time, and successor-state axioms of the form\nF t+1 ⇔ActionCausesF t ∨(F t ∧¬ActionCausesNotF t) .\nIn PDDL the times and states are implicit in the action schemas: the precondition always\nrefers to time t and the effect to time t + 1.\nA set of action schemas serves as a definition of a planning domain. A specific problem\nwithin the domain is defined with the addition of an initial state and a goal. The initial\nSection 10.1.\nDefinition of Classical Planning\n369\nInit(At(C1, SFO) ∧At(C2, JFK) ∧At(P1, SFO) ∧At(P2, JFK)\n∧Cargo(C1) ∧Cargo(C2) ∧Plane(P1) ∧Plane(P2)\n∧Airport(JFK) ∧Airport(SFO))\nGoal(At(C1, JFK) ∧At(C2, SFO))\nAction(Load(c, p, a),\nPRECOND: At(c, a) ∧At(p, a) ∧Cargo(c) ∧Plane(p) ∧Airport(a)\nEFFECT: ¬ At(c, a) ∧In(c, p))\nAction(Unload(c, p, a),\nPRECOND: In(c, p) ∧At(p, a) ∧Cargo(c) ∧Plane(p) ∧Airport(a)\nEFFECT: At(c, a) ∧¬ In(c, p))\nAction(Fly(p, from, to),\nPRECOND: At(p, from) ∧Plane(p) ∧Airport(from) ∧Airport(to)\nEFFECT: ¬ At(p, from) ∧At(p, to))\nFigure 10.1\nA PDDL description of an air cargo transportation planning problem.\nstate is a conjunction of ground atoms. (As with all states, the closed-world assumption is\nINITIAL STATE\nused, which means that any atoms that are not mentioned are false.) The goal is just like a\nGOAL\nprecondition: a conjunction of literals (positive or negative) that may contain variables, such\nas At(p, SFO) ∧Plane(p). Any variables are treated as existentially quantified, so this goal\nis to have any plane at SFO. The problem is solved when we can find a sequence of actions\nthat end in a state s that entails the goal. For example, the state Rich ∧Famous ∧Miserable\nentails the goal Rich ∧Famous, and the state Plane(Plane1) ∧At(Plane1, SFO) entails\nthe goal At(p, SFO) ∧Plane(p).\nNow we have defined planning as a search problem: we have an initial state, an ACTIONS\nfunction, a RESULT function, and a goal test. We’ll look at some example problems before\ninvestigating efficient search algorithms.\n10.1.1\nExample: Air cargo transport\nFigure 10.1 shows an air cargo transport problem involving loading and unloading cargo and\nflying it from place to place. The problem can be defined with three actions: Load, Unload,\nand Fly. The actions affect two predicates: In(c, p) means that cargo c is inside plane p, and\nAt(x, a) means that object x (either plane or cargo) is at airport a. Note that some care must\nbe taken to make sure the At predicates are maintained properly. When a plane flies from\none airport to another, all the cargo inside the plane goes with it. In first-order logic it would\nbe easy to quantify over all objects that are inside the plane. But basic PDDL does not have\na universal quantifier, so we need a different solution. The approach we use is to say that a\npiece of cargo ceases to be At anywhere when it is In a plane; the cargo only becomes At the\nnew airport when it is unloaded. So At really means “available for use at a given location.”\nThe following plan is a solution to the problem:\n[Load(C1, P1, SFO), Fly(P1, SFO, JFK ), Unload(C1, P1, JFK),\nLoad(C2, P2, JFK ), Fly(P2, JFK , SFO), Unload(C2, P2, SFO)] .\n370\nChapter\n10.\nClassical Planning\nFinally, there is the problem of spurious actions such as Fly(P1, JFK, JFK ), which should\nbe a no-op, but which has contradictory effects (according to the definition, the effect would\ninclude At(P1, JFK) ∧¬At(P1, JFK)). It is common to ignore such problems, because\nthey seldom cause incorrect plans to be produced. The correct approach is to add inequality\npreconditions saying that the from and to airports must be different; see another example of\nthis in Figure 10.3.\n10.1.2\nExample: The spare tire problem\nConsider the problem of changing a flat tire (Figure 10.2). The goal is to have a good spare\ntire properly mounted onto the car’s axle, where the initial state has a flat tire on the axle and\na good spare tire in the trunk. To keep it simple, our version of the problem is an abstract\none, with no sticky lug nuts or other complications. There are just four actions: removing the\nspare from the trunk, removing the flat tire from the axle, putting the spare on the axle, and\nleaving the car unattended overnight. We assume that the car is parked in a particularly bad\nneighborhood, so that the effect of leaving it overnight is that the tires disappear. A solution\nto the problem is [Remove(Flat, Axle), Remove(Spare, Trunk), PutOn(Spare, Axle)].\nInit(Tire(Flat) ∧Tire(Spare) ∧At(Flat, Axle) ∧At(Spare, Trunk))\nGoal(At(Spare, Axle))\nAction(Remove(obj , loc),\nPRECOND: At(obj , loc)\nEFFECT: ¬ At(obj , loc) ∧At(obj , Ground))\nAction(PutOn(t, Axle),\nPRECOND: Tire(t) ∧At(t, Ground) ∧¬ At(Flat, Axle)\nEFFECT: ¬ At(t, Ground) ∧At(t, Axle))\nAction(LeaveOvernight,\nPRECOND:\nEFFECT: ¬ At(Spare, Ground) ∧¬ At(Spare, Axle) ∧¬ At(Spare, Trunk)\n∧¬ At(Flat, Ground) ∧¬ At(Flat, Axle) ∧¬ At(Flat, Trunk))\nFigure 10.2\nThe simple spare tire problem.\n10.1.3\nExample: The blocks world\nOne of the most famous planning domains is known as the blocks world. This domain\nBLOCKS WORLD\nconsists of a set of cube-shaped blocks sitting on a table.2 The blocks can be stacked, but\nonly one block can fit directly on top of another. A robot arm can pick up a block and move\nit to another position, either on the table or on top of another block. The arm can pick up\nonly one block at a time, so it cannot pick up a block that has another one on it. The goal will\nalways be to build one or more stacks of blocks, specified in terms of what blocks are on top\n2 The blocks world used in planning research is much simpler than SHRDLU’s version, shown on page 20.\nSection 10.1.\nDefinition of Classical Planning\n371\nInit(On(A, Table) ∧On(B, Table) ∧On(C, A)\n∧Block(A) ∧Block(B) ∧Block(C) ∧Clear(B) ∧Clear(C))\nGoal(On(A, B) ∧On(B, C))\nAction(Move(b, x, y),\nPRECOND: On(b, x) ∧Clear(b) ∧Clear(y) ∧Block(b) ∧Block(y) ∧\n(b̸=x) ∧(b̸=y) ∧(x̸=y),\nEFFECT: On(b, y) ∧Clear(x) ∧¬On(b, x) ∧¬Clear(y))\nAction(MoveToTable(b, x),\nPRECOND: On(b, x) ∧Clear(b) ∧Block(b) ∧(b̸=x),\nEFFECT: On(b, Table) ∧Clear(x) ∧¬On(b, x))\nFigure 10.3\nA planning problem in the blocks world: building a three-block tower. One\nsolution is the sequence [MoveToTable(C, A), Move(B, Table, C), Move(A, Table, B)].\nStart State\nGoal State\nB\nA\nC\nA\nB\nC\nFigure 10.4\nDiagram of the blocks-world problem in Figure 10.3.\nof what other blocks. For example, a goal might be to get block A on B and block B on C\n(see Figure 10.4).\nWe use On(b, x) to indicate that block b is on x, where x is either another block or the\ntable. The action for moving block b from the top of x to the top of y will be Move(b, x, y).\nNow, one of the preconditions on moving b is that no other block be on it. In first-order logic,\nthis would be ¬∃x On(x, b) or, alternatively, ∀x ¬On(x, b). Basic PDDL does not allow\nquantifiers, so instead we introduce a predicate Clear(x) that is true when nothing is on x.\n(The complete problem description is in Figure 10.3.)\nThe action Move moves a block b from x to y if both b and y are clear. After the move\nis made, b is still clear but y is not. A first attempt at the Move schema is\nAction(Move(b, x, y),\nPRECOND:On(b, x) ∧Clear(b) ∧Clear(y),\nEFFECT:On(b, y) ∧Clear(x) ∧¬On(b, x) ∧¬Clear(y)) .\nUnfortunately, this does not maintain Clear properly when x or y is the table. When x is the\nTable, this action has the effect Clear(Table), but the table should not become clear; and\nwhen y = Table, it has the precondition Clear(Table), but the table does not have to be clear\n372\nChapter\n10.\nClassical Planning\nfor us to move a block onto it. To fix this, we do two things. First, we introduce another\naction to move a block b from x to the table:\nAction(MoveToTable(b, x),\nPRECOND:On(b, x) ∧Clear(b),\nEFFECT:On(b, Table) ∧Clear(x) ∧¬On(b, x)) .\nSecond, we take the interpretation of Clear(x) to be “there is a clear space on x to hold a\nblock.” Under this interpretation, Clear(Table) will always be true. The only problem is that\nnothing prevents the planner from using Move(b, x, Table) instead of MoveToTable(b, x).\nWe could live with this problem—it will lead to a larger-than-necessary search space, but will\nnot lead to incorrect answers—or we could introduce the predicate Block and add Block(b)∧\nBlock(y) to the precondition of Move.\n10.1.4\nThe complexity of classical planning\nIn this subsection we consider the theoretical complexity of planning and distinguish two\ndecision problems. PlanSAT is the question of whether there exists any plan that solves a\nPLANSAT\nplanning problem. Bounded PlanSAT asks whether there is a solution of length k or less;\nBOUNDED PLANSAT\nthis can be used to find an optimal plan.\nThe first result is that both decision problems are decidable for classical planning. The\nproof follows from the fact that the number of states is finite. But if we add function symbols\nto the language, then the number of states becomes infinite, and PlanSAT becomes only\nsemidecidable: an algorithm exists that will terminate with the correct answer for any solvable\nproblem, but may not terminate on unsolvable problems. The Bounded PlanSAT problem\nremains decidable even in the presence of function symbols. For proofs of the assertions in\nthis section, see Ghallab et al. (2004).\nBoth PlanSAT and Bounded PlanSAT are in the complexity class PSPACE, a class that\nis larger (and hence more difficult) than NP and refers to problems that can be solved by a\ndeterministic Turing machine with a polynomial amount of space. Even if we make some\nrather severe restrictions, the problems remain quite difficult. For example, if we disallow\nnegative effects, both problems are still NP-hard. However, if we also disallow negative\npreconditions, PlanSAT reduces to the class P.\nThese worst-case results may seem discouraging. We can take solace in the fact that\nagents are usually not asked to find plans for arbitrary worst-case problem instances, but\nrather are asked for plans in specific domains (such as blocks-world problems with n blocks),\nwhich can be much easier than the theoretical worst case. For many domains (including the\nblocks world and the air cargo world), Bounded PlanSAT is NP-complete while PlanSAT is\nin P; in other words, optimal planning is usually hard, but sub-optimal planning is sometimes\neasy. To do well on easier-than-worst-case problems, we will need good search heuristics.\nThat’s the true advantage of the classical planning formalism: it has facilitated the develop-\nment of very accurate domain-independent heuristics, whereas systems based on successor-\nstate axioms in first-order logic have had less success in coming up with good heuristics.\nSection 10.2.\nAlgorithms for Planning as State-Space Search\n373\n10.2\nALGORITHMS FOR PLANNING AS STATE-SPACE SEARCH\nNow we turn our attention to planning algorithms. We saw how the description of a planning\nproblem defines a search problem: we can search from the initial state through the space\nof states, looking for a goal. One of the nice advantages of the declarative representation of\naction schemas is that we can also search backward from the goal, looking for the initial state.\nFigure 10.5 compares forward and backward searches.\n10.2.1\nForward (progression) state-space search\nNow that we have shown how a planning problem maps into a search problem, we can solve\nplanning problems with any of the heuristic search algorithms from Chapter 3 or a local\nsearch algorithm from Chapter 4 (provided we keep track of the actions used to reach the\ngoal). From the earliest days of planning research (around 1961) until around 1998 it was\nassumed that forward state-space search was too inefficient to be practical. It is not hard to\ncome up with reasons why.\nFirst, forward search is prone to exploring irrelevant actions. Consider the noble task\nof buying a copy of AI: A Modern Approach from an online bookseller. Suppose there is an\n(a)\n(b)\nAt(P1, A)\nFly(P1, A, B)\nFly(P2, A, B)\nFly(P1, A, B)\nFly(P2, A, B)\nAt(P2, A)\nAt(P1, B)\nAt(P2, A)\nAt(P1, A)\nAt(P2, B)\nAt(P1, B)\nAt(P2, B)\nAt(P1, B)\nAt(P2, A)\nAt(P1, A)\nAt(P2, B)\nFigure 10.5\nTwo approaches to searching for a plan. (a) Forward (progression) search\nthrough the space of states, starting in the initial state and using the problem’s actions to\nsearch forward for a member of the set of goal states. (b) Backward (regression) search\nthrough sets of relevant states, starting at the set of states representing the goal and using the\ninverse of the actions to search backward for the initial state.\n374\nChapter\n10.\nClassical Planning\naction schema Buy(isbn) with effect Own(isbn). ISBNs are 10 digits, so this action schema\nrepresents 10 billion ground actions. An uninformed forward-search algorithm would have\nto start enumerating these 10 billion actions to find one that leads to the goal.\nSecond, planning problems often have large state spaces. Consider an air cargo problem\nwith 10 airports, where each airport has 5 planes and 20 pieces of cargo. The goal is to move\nall the cargo at airport A to airport B. There is a simple solution to the problem: load the 20\npieces of cargo into one of the planes at A, fly the plane to B, and unload the cargo. Finding\nthe solution can be difficult because the average branching factor is huge: each of the 50\nplanes can fly to 9 other airports, and each of the 200 packages can be either unloaded (if\nit is loaded) or loaded into any plane at its airport (if it is unloaded). So in any state there\nis a minimum of 450 actions (when all the packages are at airports with no planes) and a\nmaximum of 10,450 (when all packages and planes are at the same airport). On average, let’s\nsay there are about 2000 possible actions per state, so the search graph up to the depth of the\nobvious solution has about 200041 nodes.\nClearly, even this relatively small problem instance is hopeless without an accurate\nheuristic. Although many real-world applications of planning have relied on domain-specific\nheuristics, it turns out (as we see in Section 10.2.3) that strong domain-independent heuristics\ncan be derived automatically; that is what makes forward search feasible.\n10.2.2\nBackward (regression) relevant-states search\nIn regression search we start at the goal and apply the actions backward until we find a\nsequence of steps that reaches the initial state. It is called relevant-states search because we\nRELEVANT-STATES\nonly consider actions that are relevant to the goal (or current state). As in belief-state search\n(Section 4.4), there is a set of relevant states to consider at each step, not just a single state.\nWe start with the goal, which is a conjunction of literals forming a description of a set of\nstates—for example, the goal ¬Poor ∧Famous describes those states in which Poor is false,\nFamous is true, and any other fluent can have any value. If there are n ground fluents in a\ndomain, then there are 2n ground states (each fluent can be true or false), but 3n descriptions\nof sets of goal states (each fluent can be positive, negative, or not mentioned).\nIn general, backward search works only when we know how to regress from a state\ndescription to the predecessor state description. For example, it is hard to search backwards\nfor a solution to the n-queens problem because there is no easy way to describe the states that\nare one move away from the goal. Happily, the PDDL representation was designed to make\nit easy to regress actions—if a domain can be expressed in PDDL, then we can do regression\nsearch on it. Given a ground goal description g and a ground action a, the regression from g\nover a gives us a state description g′ defined by\ng′ = (g −ADD(a)) ∪Precond(a) .\nThat is, the effects that were added by the action need not have been true before, and also\nthe preconditions must have held before, or else the action could not have been executed.\nNote that DEL(a) does not appear in the formula; that’s because while we know the fluents\nin DEL(a) are no longer true after the action, we don’t know whether or not they were true\nbefore, so there’s nothing to be said about them.\nSection 10.2.\nAlgorithms for Planning as State-Space Search\n375\nTo get the full advantage of backward search, we need to deal with partially uninstanti-\nated actions and states, not just ground ones. For example, suppose the goal is to deliver a spe-\ncific piece of cargo to SFO: At(C2, SFO). That suggests the action Unload(C2, p′, SFO):\nAction(Unload(C2, p′, SFO),\nPRECOND:In(C2, p′) ∧At(p′, SFO) ∧Cargo(C2) ∧Plane(p′) ∧Airport(SFO)\nEFFECT:At(C2, SFO) ∧¬In(C2, p′) .\n(Note that we have standardized variable names (changing p to p′ in this case) so that there\nwill be no confusion between variable names if we happen to use the same action schema\ntwice in a plan. The same approach was used in Chapter 9 for first-order logical inference.)\nThis represents unloading the package from an unspecified plane at SFO; any plane will do,\nbut we need not say which one now. We can take advantage of the power of first-order\nrepresentations: a single description summarizes the possibility of using any of the planes by\nimplicitly quantifying over p′. The regressed state description is\ng′ = In(C2, p′) ∧At(p′, SFO) ∧Cargo(C2) ∧Plane(p′) ∧Airport(SFO) .\nThe final issue is deciding which actions are candidates to regress over. In the forward direc-\ntion we chose actions that were applicable—those actions that could be the next step in the\nplan. In backward search we want actions that are relevant—those actions that could be the\nRELEVANCE\nlast step in a plan leading up to the current goal state.\nFor an action to be relevant to a goal it obviously must contribute to the goal: at least\none of the action’s effects (either positive or negative) must unify with an element of the goal.\nWhat is less obvious is that the action must not have any effect (positive or negative) that\nnegates an element of the goal. Now, if the goal is A ∧B ∧C and an action has the effect\nA∧B∧¬C then there is a colloquial sense in which that action is very relevant to the goal—it\ngets us two-thirds of the way there. But it is not relevant in the technical sense defined here,\nbecause this action could not be the final step of a solution—we would always need at least\none more step to achieve C.\nGiven the goal At(C2, SFO), several instantiations of Unload are relevant: we could\nchose any specific plane to unload from, or we could leave the plane unspecified by using\nthe action Unload(C2, p′, SFO). We can reduce the branching factor without ruling out any\nsolutions by always using the action formed by substituting the most general unifier into the\n(standardized) action schema.\nAs another example, consider the goal Own(0136042597), given an initial state with\n10 billion ISBNs, and the single action schema\nA = Action(Buy(i), PRECOND:ISBN (i), EFFECT:Own(i)) .\nAs we mentioned before, forward search without a heuristic would have to start enumer-\nating the 10 billion ground Buy actions. But with backward search, we would unify the\ngoal Own(0136042597) with the (standardized) effect Own(i′), yielding the substitution\nθ = {i′/0136042597}. Then we would regress over the action Subst(θ, A′) to yield the\npredecessor state description ISBN (0136042597). This is part of, and thus entailed by, the\ninitial state, so we are done.\n376\nChapter\n10.\nClassical Planning\nWe can make this more formal. Assume a goal description g which contains a goal\nliteral gi and an action schema A that is standardized to produce A′. If A′ has an effect literal\ne′\nj where Unify(gi, e′\nj) = θ and where we define a′ = SUBST(θ, A′) and if there is no effect\nin a′ that is the negation of a literal in g, then a′ is a relevant action towards g.\nBackward search keeps the branching factor lower than forward search, for most prob-\nlem domains. However, the fact that backward search uses state sets rather than individual\nstates makes it harder to come up with good heuristics. That is the main reason why the\nmajority of current systems favor forward search.\n10.2.3\nHeuristics for planning\nNeither forward nor backward search is efficient without a good heuristic function. Recall\nfrom Chapter 3 that a heuristic function h(s) estimates the distance from a state s to the\ngoal and that if we can derive an admissible heuristic for this distance—one that does not\noverestimate—then we can use A∗search to find optimal solutions. An admissible heuristic\ncan be derived by defining a relaxed problem that is easier to solve. The exact cost of a\nsolution to this easier problem then becomes the heuristic for the original problem.\nBy definition, there is no way to analyze an atomic state, and thus it it requires some\ningenuity by a human analyst to define good domain-specific heuristics for search problems\nwith atomic states. Planning uses a factored representation for states and action schemas.\nThat makes it possible to define good domain-independent heuristics and for programs to\nautomatically apply a good domain-independent heuristic for a given problem.\nThink of a search problem as a graph where the nodes are states and the edges are\nactions. The problem is to find a path connecting the initial state to a goal state. There are\ntwo ways we can relax this problem to make it easier: by adding more edges to the graph,\nmaking it strictly easier to find a path, or by grouping multiple nodes together, forming an\nabstraction of the state space that has fewer states, and thus is easier to search.\nWe look first at heuristics that add edges to the graph. For example, the ignore pre-\nconditions heuristic drops all preconditions from actions. Every action becomes applicable\nIGNORE\nPRECONDITIONS\nHEURISTIC\nin every state, and any single goal fluent can be achieved in one step (if there is an applica-\nble action—if not, the problem is impossible). This almost implies that the number of steps\nrequired to solve the relaxed problem is the number of unsatisfied goals—almost but not\nquite, because (1) some action may achieve multiple goals and (2) some actions may undo\nthe effects of others. For many problems an accurate heuristic is obtained by considering (1)\nand ignoring (2). First, we relax the actions by removing all preconditions and all effects\nexcept those that are literals in the goal. Then, we count the minimum number of actions\nrequired such that the union of those actions’ effects satisfies the goal. This is an instance\nof the set-cover problem. There is one minor irritation: the set-cover problem is NP-hard.\nSET-COVER\nPROBLEM\nFortunately a simple greedy algorithm is guaranteed to return a set covering whose size is\nwithin a factor of log n of the true minimum covering, where n is the number of literals in\nthe goal. Unfortunately, the greedy algorithm loses the guarantee of admissibility.\nIt is also possible to ignore only selected preconditions of actions. Consider the sliding-\nblock puzzle (8-puzzle or 15-puzzle) from Section 3.2. We could encode this as a planning\nSection 10.2.\nAlgorithms for Planning as State-Space Search\n377\nproblem involving tiles with a single schema Slide:\nAction(Slide(t, s1, s2),\nPRECOND:On(t, s1) ∧Tile(t) ∧Blank(s2) ∧Adjacent(s1, s2)\nEFFECT:On(t, s2) ∧Blank(s1) ∧¬On(t, s1) ∧¬Blank(s2))\nAs we saw in Section 3.6, if we remove the preconditions Blank(s2) ∧Adjacent(s1, s2)\nthen any tile can move in one action to any space and we get the number-of-misplaced-tiles\nheuristic. If we remove Blank(s2) then we get the Manhattan-distance heuristic. It is easy to\nsee how these heuristics could be derived automatically from the action schema description.\nThe ease of manipulating the schemas is the great advantage of the factored representation of\nplanning problems, as compared with the atomic representation of search problems.\nAnother possibility is the ignore delete lists heuristic. Assume for a moment that all\nIGNORE DELETE\nLISTS\ngoals and preconditions contain only positive literals3 We want to create a relaxed version of\nthe original problem that will be easier to solve, and where the length of the solution will serve\nas a good heuristic. We can do that by removing the delete lists from all actions (i.e., removing\nall negative literals from effects). That makes it possible to make monotonic progress towards\nthe goal—no action will ever undo progress made by another action. It turns out it is still NP-\nhard to find the optimal solution to this relaxed problem, but an approximate solution can be\nfound in polynomial time by hill-climbing. Figure 10.6 diagrams part of the state space for\ntwo planning problems using the ignore-delete-lists heuristic. The dots represent states and\nthe edges actions, and the height of each dot above the bottom plane represents the heuristic\nvalue. States on the bottom plane are solutions. In both these problems, there is a wide path\nto the goal. There are no dead ends, so no need for backtracking; a simple hillclimbing search\nwill easily find a solution to these problems (although it may not be an optimal solution).\nThe relaxed problems leave us with a simplified—but still expensive—planning prob-\nlem just to calculate the value of the heuristic function. Many planning problems have 10100\nstates or more, and relaxing the actions does nothing to reduce the number of states. There-\nfore, we now look at relaxations that decrease the number of states by forming a state ab-\nstraction—a many-to-one mapping from states in the ground representation of the problem\nSTATE ABSTRACTION\nto the abstract representation.\nThe easiest form of state abstraction is to ignore some fluents. For example, consider\nan air cargo problem with 10 airports, 50 planes, and 200 pieces of cargo. Each plane can\nbe at one of 10 airports and each package can be either in one of the planes or unloaded at\none of the airports. So there are 5010 × 20050+10 ≈10155 states. Now consider a particular\nproblem in that domain in which it happens that all the packages are at just 5 of the airports,\nand all packages at a given airport have the same destination. Then a useful abstraction of the\nproblem is to drop all the At fluents except for the ones involving one plane and one package\nat each of the 5 airports. Now there are only 510 × 55+10 ≈1017 states. A solution in this\nabstract state space will be shorter than a solution in the original space (and thus will be an\nadmissible heuristic), and the abstract solution is easy to extend to a solution to the original\nproblem (by adding additional Load and Unload actions).\n3 Many problems are written with this convention. For problems that aren’t, replace every negative literal ¬P\nin a goal or precondition with a new positive literal, P ′.\n378\nChapter\n10.\nClassical Planning\nFigure 10.6\nTwo state spaces from planning problems with the ignore-delete-lists heuris-\ntic. The height above the bottom plane is the heuristic score of a state; states on the bottom\nplane are goals. There are no local minima, so search for the goal is straightforward. From\nHoffmann (2005).\nA key idea in defining heuristics is decomposition: dividing a problem into parts, solv-\nDECOMPOSITION\ning each part independently, and then combining the parts. The subgoal independence as-\nSUBGOAL\nINDEPENDENCE\nsumption is that the cost of solving a conjunction of subgoals is approximated by the sum\nof the costs of solving each subgoal independently. The subgoal independence assumption\ncan be optimistic or pessimistic. It is optimistic when there are negative interactions between\nthe subplans for each subgoal—for example, when an action in one subplan deletes a goal\nachieved by another subplan. It is pessimistic, and therefore inadmissible, when subplans\ncontain redundant actions—for instance, two actions that could be replaced by a single action\nin the merged plan.\nSuppose the goal is a set of fluents G, which we divide into disjoint subsets G1, . . . , Gn.\nWe then find plans P1, . . . , Pn that solve the respective subgoals. What is an estimate of the\ncost of the plan for achieving all of G? We can think of each Cost(Pi) as a heuristic estimate,\nand we know that if we combine estimates by taking their maximum value, we always get an\nadmissible heuristic. So maxi COST(Pi) is admissible, and sometimes it is exactly correct:\nit could be that P1 serendipitously achieves all the Gi. But in most cases, in practice the\nestimate is too low. Could we sum the costs instead? For many problems that is a reasonable\nestimate, but it is not admissible. The best case is when we can determine that Gi and Gj are\nindependent. If the effects of Pi leave all the preconditions and goals of Pj unchanged, then\nthe estimate COST(Pi) + COST(Pj) is admissible, and more accurate than the max estimate.\nWe show in Section 10.3.1 that planning graphs can help provide better heuristic estimates.\nIt is clear that there is great potential for cutting down the search space by forming ab-\nstractions. The trick is choosing the right abstractions and using them in a way that makes\nthe total cost—defining an abstraction, doing an abstract search, and mapping the abstraction\nback to the original problem—less than the cost of solving the original problem. The tech-\nSection 10.3.\nPlanning Graphs\n379\nniques of pattern databases from Section 3.6.3 can be useful, because the cost of creating\nthe pattern database can be amortized over multiple problem instances.\nAn example of a system that makes use of effective heuristics is FF, or FASTFORWARD\n(Hoffmann, 2005), a forward state-space searcher that uses the ignore-delete-lists heuristic,\nestimating the heuristic with the help of a planning graph (see Section 10.3). FF then uses\nhill-climbing search (modified to keep track of the plan) with the heuristic to find a solution.\nWhen it hits a plateau or local maximum—when no action leads to a state with better heuristic\nscore—then FF uses iterative deepening search until it finds a state that is better, or it gives\nup and restarts hill-climbing.\n10.3\nPLANNING GRAPHS\nAll of the heuristics we have suggested can suffer from inaccuracies. This section shows\nhow a special data structure called a planning graph can be used to give better heuristic\nPLANNING GRAPH\nestimates. These heuristics can be applied to any of the search techniques we have seen so\nfar. Alternatively, we can search for a solution over the space formed by the planning graph,\nusing an algorithm called GRAPHPLAN.\nA planning problem asks if we can reach a goal state from the initial state. Suppose we\nare given a tree of all possible actions from the initial state to successor states, and their suc-\ncessors, and so on. If we indexed this tree appropriately, we could answer the planning ques-\ntion “can we reach state G from state S0” immediately, just by looking it up. Of course, the\ntree is of exponential size, so this approach is impractical. A planning graph is polynomial-\nsize approximation to this tree that can be constructed quickly. The planning graph can’t\nanswer definitively whether G is reachable from S0, but it can estimate how many steps it\ntakes to reach G. The estimate is always correct when it reports the goal is not reachable, and\nit never overestimates the number of steps, so it is an admissible heuristic.\nA planning graph is a directed graph organized into levels: first a level S0 for the initial\nLEVEL\nstate, consisting of nodes representing each fluent that holds in S0; then a level A0 consisting\nof nodes for each ground action that might be applicable in S0; then alternating levels Si\nfollowed by Ai; until we reach a termination condition (to be discussed later).\nRoughly speaking, Si contains all the literals that could hold at time i, depending on\nthe actions executed at preceding time steps. If it is possible that either P or ¬P could hold,\nthen both will be represented in Si. Also roughly speaking, Ai contains all the actions that\ncould have their preconditions satisfied at time i. We say “roughly speaking” because the\nplanning graph records only a restricted subset of the possible negative interactions among\nactions; therefore, a literal might show up at level Sj when actually it could not be true until\na later level, if at all. (A literal will never show up too late.) Despite the possible error, the\nlevel j at which a literal first appears is a good estimate of how difficult it is to achieve the\nliteral from the initial state.\nPlanning graphs work only for propositional planning problems—ones with no vari-\nables. As we mentioned on page 368, it is straightforward to propositionalize a set of ac-\n380\nChapter\n10.\nClassical Planning\nInit(Have(Cake))\nGoal(Have(Cake) ∧Eaten(Cake))\nAction(Eat(Cake)\nPRECOND: Have(Cake)\nEFFECT: ¬ Have(Cake) ∧Eaten(Cake))\nAction(Bake(Cake)\nPRECOND: ¬ Have(Cake)\nEFFECT: Have(Cake))\nFigure 10.7\nThe “have cake and eat cake too” problem.\nBake(Cake)\nEat(Cake)\nHave(Cake)\nS0\nA0\nS1\nA1\nS2\nHave(Cake)\nHave(Cake)\nHave(Cake)\nHave(Cake)\nEaten(Cake)\nEaten(Cake)\nEaten(Cake)\nEaten(Cake)\nEaten(Cake)\nEat(Cake)\n¬\n¬\n¬\n¬\n¬\nFigure 10.8\nThe planning graph for the “have cake and eat cake too” problem up to level\nS2. Rectangles indicate actions (small squares indicate persistence actions), and straight\nlines indicate preconditions and effects. Mutex links are shown as curved gray lines. Not all\nmutex links are shown, because the graph would be too cluttered. In general, if two literals\nare mutex at Si, then the persistence actions for those literals will be mutex at Ai and we\nneed not draw that mutex link.\ntion schemas. Despite the resulting increase in the size of the problem description, planning\ngraphs have proved to be effective tools for solving hard planning problems.\nFigure 10.7 shows a simple planning problem, and Figure 10.8 shows its planning\ngraph. Each action at level Ai is connected to its preconditions at Si and its effects at Si+1.\nSo a literal appears because an action caused it, but we also want to say that a literal can\npersist if no action negates it. This is represented by a persistence action (sometimes called\nPERSISTENCE\nACTION\na no-op). For every literal C, we add to the problem a persistence action with precondition C\nand effect C. Level A0 in Figure 10.8 shows one “real” action, Eat(Cake), along with two\npersistence actions drawn as small square boxes.\nLevel A0 contains all the actions that could occur in state S0, but just as important it\nrecords conflicts between actions that would prevent them from occurring together. The gray\nlines in Figure 10.8 indicate mutual exclusion (or mutex) links. For example, Eat(Cake) is\nMUTUAL EXCLUSION\nMUTEX\nmutually exclusive with the persistence of either Have(Cake) or ¬Eaten(Cake). We shall\nsee shortly how mutex links are computed.\nLevel S1 contains all the literals that could result from picking any subset of the actions\nin A0, as well as mutex links (gray lines) indicating literals that could not appear together,\nregardless of the choice of actions. For example, Have(Cake) and Eaten(Cake) are mutex:\nSection 10.3.\nPlanning Graphs\n381\ndepending on the choice of actions in A0, either, but not both, could be the result. In other\nwords, S1 represents a belief state: a set of possible states. The members of this set are all\nsubsets of the literals such that there is no mutex link between any members of the subset.\nWe continue in this way, alternating between state level Si and action level Ai until we\nreach a point where two consecutive levels are identical. At this point, we say that the graph\nhas leveled off. The graph in Figure 10.8 levels off at S2.\nLEVELED OFF\nWhat we end up with is a structure where every Ai level contains all the actions that are\napplicable in Si, along with constraints saying that two actions cannot both be executed at the\nsame level. Every Si level contains all the literals that could result from any possible choice\nof actions in Ai−1, along with constraints saying which pairs of literals are not possible.\nIt is important to note that the process of constructing the planning graph does not require\nchoosing among actions, which would entail combinatorial search. Instead, it just records the\nimpossibility of certain choices using mutex links.\nWe now define mutex links for both actions and literals. A mutex relation holds between\ntwo actions at a given level if any of the following three conditions holds:\n• Inconsistent effects: one action negates an effect of the other. For example, Eat(Cake)\nand the persistence of Have(Cake) have inconsistent effects because they disagree on\nthe effect Have(Cake).\n• Interference: one of the effects of one action is the negation of a precondition of the\nother. For example Eat(Cake) interferes with the persistence of Have(Cake) by negat-\ning its precondition.\n• Competing needs: one of the preconditions of one action is mutually exclusive with a\nprecondition of the other. For example, Bake(Cake) and Eat(Cake) are mutex because\nthey compete on the value of the Have(Cake) precondition.\nA mutex relation holds between two literals at the same level if one is the negation of the other\nor if each possible pair of actions that could achieve the two literals is mutually exclusive.\nThis condition is called inconsistent support. For example, Have(Cake) and Eaten(Cake)\nare mutex in S1 because the only way of achieving Have(Cake), the persistence action, is\nmutex with the only way of achieving Eaten(Cake), namely Eat(Cake). In S2 the two\nliterals are not mutex, because there are new ways of achieving them, such as Bake(Cake)\nand the persistence of Eaten(Cake), that are not mutex.\nA planning graph is polynomial in the size of the planning problem. For a planning\nproblem with l literals and a actions, each Si has no more than l nodes and l2 mutex links,\nand each Ai has no more than a + l nodes (including the no-ops), (a + l)2 mutex links, and\n2(al + l) precondition and effect links. Thus, an entire graph with n levels has a size of\nO(n(a + l)2). The time to build the graph has the same complexity.\n10.3.1\nPlanning graphs for heuristic estimation\nA planning graph, once constructed, is a rich source of information about the problem. First,\nif any goal literal fails to appear in the final level of the graph, then the problem is unsolvable.\nSecond, we can estimate the cost of achieving any goal literal gi from state s as the level at\nwhich gi first appears in the planning graph constructed from initial state s. We call this the\n382\nChapter\n10.\nClassical Planning\nlevel cost of gi. In Figure 10.8, Have(Cake) has level cost 0 and Eaten(Cake) has level cost\nLEVEL COST\n1. It is easy to show (Exercise 10.10) that these estimates are admissible for the individual\ngoals. The estimate might not always be accurate, however, because planning graphs allow\nseveral actions at each level, whereas the heuristic counts just the level and not the number\nof actions. For this reason, it is common to use a serial planning graph for computing\nSERIAL PLANNING\nGRAPH\nheuristics. A serial graph insists that only one action can actually occur at any given time\nstep; this is done by adding mutex links between every pair of nonpersistence actions. Level\ncosts extracted from serial graphs are often quite reasonable estimates of actual costs.\nTo estimate the cost of a conjunction of goals, there are three simple approaches. The\nmax-level heuristic simply takes the maximum level cost of any of the goals; this is admissi-\nMAX-LEVEL\nble, but not necessarily accurate.\nThe level sum heuristic, following the subgoal independence assumption, returns the\nLEVEL SUM\nsum of the level costs of the goals; this can be inadmissible but works well in practice\nfor problems that are largely decomposable. It is much more accurate than the number-\nof-unsatisfied-goals heuristic from Section 10.2. For our problem, the level-sum heuristic\nestimate for the conjunctive goal Have(Cake) ∧Eaten(Cake) will be 0 + 1 = 1, whereas\nthe correct answer is 2, achieved by the plan [Eat(Cake), Bake(Cake)]. That doesn’t seem\nso bad. A more serious error is that if Bake(Cake) were not in the set of actions, then the\nestimate would still be 1, when in fact the conjunctive goal would be impossible.\nFinally, the set-level heuristic finds the level at which all the literals in the conjunctive\nSET-LEVEL\ngoal appear in the planning graph without any pair of them being mutually exclusive. This\nheuristic gives the correct values of 2 for our original problem and infinity for the problem\nwithout Bake(Cake). It is admissible, it dominates the max-level heuristic, and it works\nextremely well on tasks in which there is a good deal of interaction among subplans. It is not\nperfect, of course; for example, it ignores interactions among three or more literals.\nAs a tool for generating accurate heuristics, we can view the planning graph as a relaxed\nproblem that is efficiently solvable. To understand the nature of the relaxed problem, we\nneed to understand exactly what it means for a literal g to appear at level Si in the planning\ngraph. Ideally, we would like it to be a guarantee that there exists a plan with i action levels\nthat achieves g, and also that if g does not appear, there is no such plan. Unfortunately,\nmaking that guarantee is as difficult as solving the original planning problem. So the planning\ngraph makes the second half of the guarantee (if g does not appear, there is no plan), but\nif g does appear, then all the planning graph promises is that there is a plan that possibly\nachieves g and has no “obvious” flaws. An obvious flaw is defined as a flaw that can be\ndetected by considering two actions or two literals at a time—in other words, by looking at\nthe mutex relations. There could be more subtle flaws involving three, four, or more actions,\nbut experience has shown that it is not worth the computational effort to keep track of these\npossible flaws. This is similar to a lesson learned from constraint satisfaction problems—that\nit is often worthwhile to compute 2-consistency before searching for a solution, but less often\nworthwhile to compute 3-consistency or higher. (See page 211.)\nOne example of an unsolvable problem that cannot be recognized as such by a planning\ngraph is the blocks-world problem where the goal is to get block A on B, B on C, and C on\nA. This is an impossible goal; a tower with the bottom on top of the top. But a planning graph\nSection 10.3.\nPlanning Graphs\n383\ncannot detect the impossibility, because any two of the three subgoals are achievable. There\nare no mutexes between any pair of literals, only between the three as a whole. To detect that\nthis problem is impossible, we would have to search over the planning graph.\n10.3.2\nThe GRAPHPLAN algorithm\nThis subsection shows how to extract a plan directly from the planning graph, rather than just\nusing the graph to provide a heuristic. The GRAPHPLAN algorithm (Figure 10.9) repeatedly\nadds a level to a planning graph with EXPAND-GRAPH. Once all the goals show up as non-\nmutex in the graph, GRAPHPLAN calls EXTRACT-SOLUTION to search for a plan that solves\nthe problem. If that fails, it expands another level and tries again, terminating with failure\nwhen there is no reason to go on.\nfunction GRAPHPLAN(problem) returns solution or failure\ngraph ←INITIAL-PLANNING-GRAPH(problem)\ngoals ←CONJUNCTS(problem.GOAL)\nnogoods ←an empty hash table\nfor tl = 0 to ∞do\nif goals all non-mutex in St of graph then\nsolution ←EXTRACT-SOLUTION(graph, goals, NUMLEVELS(graph), nogoods)\nif solution ̸= failure then return solution\nif graph and nogoods have both leveled off then return failure\ngraph ←EXPAND-GRAPH(graph, problem)\nFigure 10.9\nThe GRAPHPLAN algorithm. GRAPHPLAN calls EXPAND-GRAPH to add a\nlevel until either a solution is found by EXTRACT-SOLUTION, or no solution is possible.\nLet us now trace the operation of GRAPHPLAN on the spare tire problem from page 370.\nThe graph is shown in Figure 10.10. The first line of GRAPHPLAN initializes the planning\ngraph to a one-level (S0) graph representing the initial state. The positive fluents from the\nproblem description’s initial state are shown, as are the relevant negative fluents. Not shown\nare the unchanging positive literals (such as Tire(Spare)) and the irrelevant negative literals.\nThe goal At(Spare, Axle) is not present in S0, so we need not call EXTRACT-SOLUTION—\nwe are certain that there is no solution yet. Instead, EXPAND-GRAPH adds into A0 the three\nactions whose preconditions exist at level S0 (i.e., all the actions except PutOn(Spare, Axle)),\nalong with persistence actions for all the literals in S0. The effects of the actions are added at\nlevel S1. EXPAND-GRAPH then looks for mutex relations and adds them to the graph.\nAt(Spare, Axle) is still not present in S1, so again we do not call EXTRACT-SOLUTION.\nWe call EXPAND-GRAPH again, adding A1 and S1 and giving us the planning graph shown\nin Figure 10.10. Now that we have the full complement of actions, it is worthwhile to look at\nsome of the examples of mutex relations and their causes:\n• Inconsistent effects: Remove(Spare, Trunk) is mutex with LeaveOvernight because\none has the effect At(Spare, Ground) and the other has its negation.\n384\nChapter\n10.\nClassical Planning\nS0\nA1\nS2\nAt(Spare,Trunk)\nAt(Spare,Trunk)\nAt(Flat,Axle)\nAt(Flat,Axle)\nAt(Spare,Axle)\nAt(Flat,Ground)\nAt(Flat,Ground)\nAt(Spare,Ground)\nAt(Spare,Ground)\nAt(Spare,Trunk)\nAt(Spare,Trunk)\nAt(Flat,Axle)\nAt(Flat,Axle)\nAt(Spare,Axle)\nAt(Flat,Ground)\nAt(Flat,Ground)\nAt(Spare,Ground)\nAt(Spare,Ground)\nAt(Spare,Axle)\nAt(Spare,Trunk)\nAt(Flat,Axle)\nAt(Spare,Axle)\nAt(Flat,Ground)\nAt(Spare,Ground)\nPutOn(Spare,Axle)\nLeaveOvernight\nRemove(Flat,Axle)\nRemove(Spare,Trunk)\nRemove(Spare,Trunk)\nRemove(Flat,Axle)\nLeaveOvernight\n¬\n¬\n¬\n¬\n¬\n¬\n¬\n¬\n¬\n¬\n¬\n¬\n¬\nA0\nS1\nFigure 10.10\nThe planning graph for the spare tire problem after expansion to level S2.\nMutex links are shown as gray lines. Not all links are shown, because the graph would be too\ncluttered if we showed them all. The solution is indicated by bold lines and outlines.\n• Interference: Remove(Flat, Axle) is mutex with LeaveOvernight because one has the\nprecondition At(Flat, Axle) and the other has its negation as an effect.\n• Competing needs: PutOn(Spare, Axle) is mutex with Remove(Flat, Axle) because\none has At(Flat, Axle) as a precondition and the other has its negation.\n• Inconsistent support: At(Spare, Axle) is mutex with At(Flat, Axle) in S2 because the\nonly way of achieving At(Spare, Axle) is by PutOn(Spare, Axle), and that is mutex\nwith the persistence action that is the only way of achieving At(Flat, Axle). Thus, the\nmutex relations detect the immediate conflict that arises from trying to put two objects\nin the same place at the same time.\nThis time, when we go back to the start of the loop, all the literals from the goal are present\nin S2, and none of them is mutex with any other. That means that a solution might exist,\nand EXTRACT-SOLUTION will try to find it. We can formulate EXTRACT-SOLUTION as a\nBoolean constraint satisfaction problem (CSP) where the variables are the actions at each\nlevel, the values for each variable are in or out of the plan, and the constraints are the mutexes\nand the need to satisfy each goal and precondition.\nAlternatively, we can define EXTRACT-SOLUTION as a backward search problem, where\neach state in the search contains a pointer to a level in the planning graph and a set of unsat-\nisfied goals. We define this search problem as follows:\n• The initial state is the last level of the planning graph, Sn, along with the set of goals\nfrom the planning problem.\n• The actions available in a state at level Si are to select any conflict-free subset of the\nactions in Ai−1 whose effects cover the goals in the state. The resulting state has level\nSi−1 and has as its set of goals the preconditions for the selected set of actions. By\n“conflict free,” we mean a set of actions such that no two of them are mutex and no two\nof their preconditions are mutex.\nSection 10.3.\nPlanning Graphs\n385\n• The goal is to reach a state at level S0 such that all the goals are satisfied.\n• The cost of each action is 1.\nFor this particular problem, we start at S2 with the goal At(Spare, Axle). The only choice we\nhave for achieving the goal set is PutOn(Spare, Axle). That brings us to a search state at S1\nwith goals At(Spare, Ground) and ¬At(Flat, Axle). The former can be achieved only by\nRemove(Spare, Trunk), and the latter by either Remove(Flat, Axle) or LeaveOvernight.\nBut LeaveOvernight is mutex with Remove(Spare, Trunk), so the only solution is to choose\nRemove(Spare, Trunk) and Remove(Flat, Axle). That brings us to a search state at S0 with\nthe goals At(Spare, Trunk) and At(Flat, Axle). Both of these are present in the state, so\nwe have a solution: the actions Remove(Spare, Trunk) and Remove(Flat, Axle) in level\nA0, followed by PutOn(Spare, Axle) in A1.\nIn the case where EXTRACT-SOLUTION fails to find a solution for a set of goals at\na given level, we record the (level, goals) pair as a no-good, just as we did in constraint\nlearning for CSPs (page 220). Whenever EXTRACT-SOLUTION is called again with the same\nlevel and goals, we can find the recorded no-good and immediately return failure rather than\nsearching again. We see shortly that no-goods are also used in the termination test.\nWe know that planning is PSPACE-complete and that constructing the planning graph\ntakes polynomial time, so it must be the case that solution extraction is intractable in the worst\ncase. Therefore, we will need some heuristic guidance for choosing among actions during the\nbackward search. One approach that works well in practice is a greedy algorithm based on\nthe level cost of the literals. For any set of goals, we proceed in the following order:\n1. Pick first the literal with the highest level cost.\n2. To achieve that literal, prefer actions with easier preconditions. That is, choose an action\nsuch that the sum (or maximum) of the level costs of its preconditions is smallest.\n10.3.3\nTermination of GRAPHPLAN\nSo far, we have skated over the question of termination. Here we show that GRAPHPLAN will\nin fact terminate and return failure when there is no solution.\nThe first thing to understand is why we can’t stop expanding the graph as soon as it has\nleveled off. Consider an air cargo domain with one plane and n pieces of cargo at airport\nA, all of which have airport B as their destination. In this version of the problem, only one\npiece of cargo can fit in the plane at a time. The graph will level off at level 4, reflecting the\nfact that for any single piece of cargo, we can load it, fly it, and unload it at the destination in\nthree steps. But that does not mean that a solution can be extracted from the graph at level 4;\nin fact a solution will require 4n −1 steps: for each piece of cargo we load, fly, and unload,\nand for all but the last piece we need to fly back to airport A to get the next piece.\nHow long do we have to keep expanding after the graph has leveled off? If the function\nEXTRACT-SOLUTION fails to find a solution, then there must have been at least one set of\ngoals that were not achievable and were marked as a no-good. So if it is possible that there\nmight be fewer no-goods in the next level, then we should continue. As soon as the graph\nitself and the no-goods have both leveled off, with no solution found, we can terminate with\nfailure because there is no possibility of a subsequent change that could add a solution.\n386\nChapter\n10.\nClassical Planning\nNow all we have to do is prove that the graph and the no-goods will always level off. The\nkey to this proof is that certain properties of planning graphs are monotonically increasing or\ndecreasing. “X increases monotonically” means that the set of Xs at level i + 1 is a superset\n(not necessarily proper) of the set at level i. The properties are as follows:\n• Literals increase monotonically: Once a literal appears at a given level, it will appear\nat all subsequent levels. This is because of the persistence actions; once a literal shows\nup, persistence actions cause it to stay forever.\n• Actions increase monotonically: Once an action appears at a given level, it will appear\nat all subsequent levels. This is a consequence of the monotonic increase of literals; if\nthe preconditions of an action appear at one level, they will appear at subsequent levels,\nand thus so will the action.\n• Mutexes decrease monotonically: If two actions are mutex at a given level Ai, then they\nwill also be mutex for all previous levels at which they both appear. The same holds for\nmutexes between literals. It might not always appear that way in the figures, because\nthe figures have a simplification: they display neither literals that cannot hold at level\nSi nor actions that cannot be executed at level Ai. We can see that “mutexes decrease\nmonotonically” is true if you consider that these invisible literals and actions are mutex\nwith everything.\nThe proof can be handled by cases: if actions A and B are mutex at level Ai, it\nmust be because of one of the three types of mutex. The first two, inconsistent effects\nand interference, are properties of the actions themselves, so if the actions are mutex\nat Ai, they will be mutex at every level. The third case, competing needs, depends on\nconditions at level Si: that level must contain a precondition of A that is mutex with\na precondition of B. Now, these two preconditions can be mutex if they are negations\nof each other (in which case they would be mutex in every level) or if all actions for\nachieving one are mutex with all actions for achieving the other. But we already know\nthat the available actions are increasing monotonically, so, by induction, the mutexes\nmust be decreasing.\n• No-goods decrease monotonically: If a set of goals is not achievable at a given level,\nthen they are not achievable in any previous level. The proof is by contradiction: if they\nwere achievable at some previous level, then we could just add persistence actions to\nmake them achievable at a subsequent level.\nBecause the actions and literals increase monotonically and because there are only a finite\nnumber of actions and literals, there must come a level that has the same number of actions\nand literals as the previous level. Because mutexes and no-goods decrease, and because there\ncan never be fewer than zero mutexes or no-goods, there must come a level that has the\nsame number of mutexes and no-goods as the previous level. Once a graph has reached this\nstate, then if one of the goals is missing or is mutex with another goal, then we can stop the\nGRAPHPLAN algorithm and return failure. That concludes a sketch of the proof; for more\ndetails see Ghallab et al. (2004).\nSection 10.4.\nOther Classical Planning Approaches\n387\nYear\nTrack\nWinning Systems (approaches)\n2008\nOptimal\nGAMER (model checking, bidirectional search)\n2008\nSatisficing\nLAMA (fast downward search with FF heuristic)\n2006\nOptimal\nSATPLAN, MAXPLAN (Boolean satisfiability)\n2006\nSatisficing\nSGPLAN (forward search; partitions into independent subproblems)\n2004\nOptimal\nSATPLAN (Boolean satisfiability)\n2004\nSatisficing\nFAST DIAGONALLY DOWNWARD (forward search with causal graph)\n2002\nAutomated\nLPG (local search, planning graphs converted to CSPs)\n2002\nHand-coded\nTLPLAN (temporal action logic with control rules for forward search)\n2000\nAutomated\nFF (forward search)\n2000\nHand-coded\nTALPLANNER (temporal action logic with control rules for forward search)\n1998\nAutomated\nIPP (planning graphs); HSP (forward search)\nFigure 10.11\nSome of the top-performing systems in the International Planning Compe-\ntition. Each year there are various tracks: “Optimal” means the planners must produce the\nshortest possible plan, while “Satisficing” means nonoptimal solutions are accepted. “Hand-\ncoded” means domain-specific heuristics are allowed; “Automated” means they are not.\n10.4\nOTHER CLASSICAL PLANNING APPROACHES\nCurrently the most popular and effective approaches to fully automated planning are:\n• Translating to a Boolean satisfiability (SAT) problem\n• Forward state-space search with carefully crafted heuristics (Section 10.2)\n• Search using a planning graph (Section 10.3)\nThese three approaches are not the only ones tried in the 40-year history of automated plan-\nning. Figure 10.11 shows some of the top systems in the International Planning Competitions,\nwhich have been held every even year since 1998. In this section we first describe the transla-\ntion to a satisfiability problem and then describe three other influential approaches: planning\nas first-order logical deduction; as constraint satisfaction; and as plan refinement.\n10.4.1\nClassical planning as Boolean satisfiability\nIn Section 7.7.4 we saw how SATPLAN solves planning problems that are expressed in propo-\nsitional logic. Here we show how to translate a PDDL description into a form that can be\nprocessed by SATPLAN. The translation is a series of straightforward steps:\n• Propositionalize the actions: replace each action schema with a set of ground actions\nformed by substituting constants for each of the variables. These ground actions are not\npart of the translation, but will be used in subsequent steps.\n• Define the initial state: assert F 0 for every fluent F in the problem’s initial state, and\n¬F for every fluent not mentioned in the initial state.\n• Propositionalize the goal: for every variable in the goal, replace the literals that contain\nthe variable with a disjunction over constants. For example, the goal of having block A\n388\nChapter\n10.\nClassical Planning\non another block, On(A, x) ∧Block(x) in a world with objects A, B and C, would be\nreplaced by the goal\n(On(A, A) ∧Block(A)) ∨(On(A, B) ∧Block(B)) ∨(On(A, C) ∧Block(C)) .\n• Add successor-state axioms: For each fluent F, add an axiom of the form\nF t+1 ⇔ActionCausesF t ∨(F t ∧¬ActionCausesNotF t) ,\nwhere ActionCausesF is a disjunction of all the ground actions that have F in their\nadd list, and ActionCausesNotF is a disjunction of all the ground actions that have F\nin their delete list.\n• Add precondition axioms: For each ground action A, add the axiom At ⇒PRE(A)t,\nthat is, if an action is taken at time t, then the preconditions must have been true.\n• Add action exclusion axioms: say that every action is distinct from every other action.\nThe resulting translation is in the form that we can hand to SATPLAN to find a solution.\n10.4.2\nPlanning as first-order logical deduction: Situation calculus\nPDDL is a language that carefully balances the expressiveness of the language with the com-\nplexity of the algorithms that operate on it. But some problems remain difficult to express in\nPDDL. For example, we can’t express the goal “move all the cargo from A to B regardless\nof how many pieces of cargo there are” in PDDL, but we can do it in first-order logic, using a\nuniversal quantifier. Likewise, first-order logic can concisely express global constraints such\nas “no more than four robots can be in the same place at the same time.” PDDL can only say\nthis with repetitious preconditions on every possible action that involves a move.\nThe propositional logic representation of planning problems also has limitations, such\nas the fact that the notion of time is tied directly to fluents. For example, South2 means\n“the agent is facing south at time 2.” With that representation, there is no way to say “the\nagent would be facing south at time 2 if it executed a right turn at time 1; otherwise it would\nbe facing east.” First-order logic lets us get around this limitation by replacing the notion\nof linear time with a notion of branching situations, using a representation called situation\ncalculus that works like this:\nSITUATION\nCALCULUS\n• The initial state is called a situation.\nIf s is a situation and a is an action, then\nSITUATION\nRESULT(s, a) is also a situation. There are no other situations. Thus, a situation cor-\nresponds to a sequence, or history, of actions. You can also think of a situation as the\nresult of applying the actions, but note that two situations are the same only if their start\nand actions are the same: (RESULT(s, a) = RESULT(s′, a′)) ⇔(s = s′ ∧a = a′).\nSome examples of actions and situations are shown in Figure 10.12.\n• A function or relation that can vary from one situation to the next is a fluent. By conven-\ntion, the situation s is always the last argument to the fluent, for example At(x, l, s) is a\nrelational fluent that is true when object x is at location l in situation s, and Location is a\nfunctional fluent such that Location(x, s) = l holds in the same situations as At(x, l, s).\n• Each action’s preconditions are described with a possibility axiom that says when the\nPOSSIBILITY AXIOM\naction can be taken. It has the form Φ(s) ⇒Poss(a, s) where Φ(s) is some formula\nSection 10.4.\nOther Classical Planning Approaches\n389\nPIT\nPIT\nPIT\nGold\nPIT\nPIT\nPIT\nGold\nPIT\nPIT\nPIT\nGold\nS0\nForward\nResult(S0, Forward)\nResult(Result(S0, Forward),\nTurn(Right))\nTurn(Right)\nFigure 10.12\nSituations as the results of actions in the wumpus world.\ninvolving s that describes the preconditions. An example from the wumpus world says\nthat it is possible to shoot if the agent is alive and has an arrow:\nAlive(Agent, s) ∧Have(Agent, Arrow, s) ⇒Poss(Shoot, s)\n• Each fluent is described with a successor-state axiom that says what happens to the\nfluent, depending on what action is taken. This is similar to the approach we took for\npropositional logic. The axiom has the form\nAction is possible ⇒\n(Fluent is true in result state ⇔Action’s effect made it true\n∨It was true before and action left it alone) .\nFor example, the axiom for the relational fluent Holding says that the agent is holding\nsome gold g after executing a possible action if and only if the action was a Grab of g\nor if the agent was already holding g and the action was not releasing it:\nPoss(a, s) ⇒\n(Holding(Agent, g, Result(a, s)) ⇔\na = Grab(g) ∨(Holding(Agent, g, s) ∧a ̸= Release(g))) .\n• We need unique action axioms so that the agent can deduce that, for example, a ̸=\nUNIQUE ACTION\nAXIOMS\nRelease(g). For each distinct pair of action names Ai and Aj we have an axiom that\nsays the actions are different:\nAi(x, . . .) ̸= Aj(y, . . .)\n390\nChapter\n10.\nClassical Planning\nand for each action name Ai we have an axiom that says two uses of that action name\nare equal if and only if all their arguments are equal:\nAi(x1, . . . , xn) = Ai(y1, . . . , yn) ⇔x1 = y1 ∧. . . ∧xn = yn .\n• A solution is a situation (and hence a sequence of actions) that satisfies the goal.\nWork in situation calculus has done a lot to define the formal semantics of planning and to\nopen up new areas of investigation. But so far there have not been any practical large-scale\nplanning programs based on logical deduction over the situation calculus. This is in part\nbecause of the difficulty of doing efficient inference in FOL, but is mainly because the field\nhas not yet developed effective heuristics for planning with situation calculus.\n10.4.3\nPlanning as constraint satisfaction\nWe have seen that constraint satisfaction has a lot in common with Boolean satisfiability, and\nwe have seen that CSP techniques are effective for scheduling problems, so it is not surprising\nthat it is possible to encode a bounded planning problem (i.e., the problem of finding a plan of\nlength k) as a constraint satisfaction problem (CSP). The encoding is similar to the encoding\nto a SAT problem (Section 10.4.1), with one important simplification: at each time step we\nneed only a single variable, Actiont, whose domain is the set of possible actions. We no\nlonger need one variable for every action, and we don’t need the action exclusion axioms. It\nis also possible to encode a planning graph into a CSP. This is the approach taken by GP-CSP\n(Do and Kambhampati, 2003).\n10.4.4\nPlanning as refinement of partially ordered plans\nAll the approaches we have seen so far construct totally ordered plans consisting of a strictly\nlinear sequences of actions. This representation ignores the fact that many subproblems are\nindependent. A solution to an air cargo problem consists of a totally ordered sequence of\nactions, yet if 30 packages are being loaded onto one plane in one airport and 50 packages are\nbeing loaded onto another at another airport, it seems pointless to come up with a strict linear\nordering of 80 load actions; the two subsets of actions should be thought of independently.\nAn alternative is to represent plans as partially ordered structures: a plan is a set of\nactions and a set of constraints of the form Before(ai, aj) saying that one action occurs\nbefore another. In the bottom of Figure 10.13, we see a partially ordered plan that is a solution\nto the spare tire problem. Actions are boxes and ordering constraints are arrows. Note that\nRemove(Spare, Trunk) and Remove(Flat, Axle) can be done in either order as long as they\nare both completed before the PutOn(Spare, Axle) action.\nPartially ordered plans are created by a search through the space of plans rather than\nthrough the state space. We start with the empty plan consisting of just the initial state and\nthe goal, with no actions in between, as in the top of Figure 10.13. The search procedure then\nlooks for a flaw in the plan, and makes an addition to the plan to correct the flaw (or if no\nFLAW\ncorrection can be made, the search backtracks and tries something else). A flaw is anything\nthat keeps the partial plan from being a solution. For example, one flaw in the empty plan is\nthat no action achieves At(Spare, Axle). One way to correct the flaw is to insert into the plan\nSection 10.4.\nOther Classical Planning Approaches\n391\nFinish\nAt(Spare,Axle)\nStart\nAt(Flat,Axle)\nAt(Spare,Trunk)\n(a)\nRemove(Spare,Trunk)\nAt(Spare,Trunk)\nPutOn(Spare,Axle)\nAt(Spare,Ground)\nAt(Flat,Axle)\nFinish\nAt(Spare,Axle)\nStart\nAt(Flat,Axle)\nAt(Spare,Trunk)\n¬\n(b)\nStart\nRemove(Spare,Trunk)\nAt(Spare,Trunk)\nRemove(Flat,Axle)\nAt(Flat,Axle)\nPutOn(Spare,Axle)\nAt(Spare,Ground)\nAt(Flat,Axle)\nFinish\nAt(Spare,Axle)\nAt(Flat,Axle)\nAt(Spare,Trunk)\n¬\n(c)\nFigure 10.13\n(a) the tire problem expressed as an empty plan. (b) an incomplete partially\nordered plan for the tire problem. Boxes represent actions and arrows indicate that one action\nmust occur before another. (c) a complete partially-ordered solution.\nthe action PutOn(Spare, Axle). Of course that introduces some new flaws: the preconditions\nof the new action are not achieved. The search keeps adding to the plan (backtracking if\nnecessary) until all flaws are resolved, as in the bottom of Figure 10.13. At every step, we\nmake the least commitment possible to fix the flaw. For example, in adding the action\nLEAST COMMITMENT\nRemove(Spare, Trunk) we need to commit to having it occur before PutOn(Spare, Axle),\nbut we make no other commitment that places it before or after other actions. If there were a\nvariable in the action schema that could be left unbound, we would do so.\nIn the 1980s and 90s, partial-order planning was seen as the best way to handle plan-\nning problems with independent subproblems—after all, it was the only approach that ex-\nplicitly represents independent branches of a plan. On the other hand, it has the disadvantage\nof not having an explicit representation of states in the state-transition model. That makes\nsome computations cumbersome. By 2000, forward-search planners had developed excellent\nheuristics that allowed them to efficiently discover the independent subproblems that partial-\norder planning was designed for. As a result, partial-order planners are not competitive on\nfully automated classical planning problems.\nHowever, partial-order planning remains an important part of the field. For some spe-\ncific tasks, such as operations scheduling, partial-order planning with domain specific heuris-\ntics is the technology of choice. Many of these systems use libraries of high-level plans, as\ndescribed in Section 11.2. Partial-order planning is also often used in domains where it is im-\nportant for humans to understand the plans. Operational plans for spacecraft and Mars rovers\nare generated by partial-order planners and are then checked by human operators before being\nuploaded to the vehicles for execution. The plan refinement approach makes it easier for the\nhumans to understand what the planning algorithms are doing and verify that they are correct.\n392\nChapter\n10.\nClassical Planning\n10.5\nANALYSIS OF PLANNING APPROACHES\nPlanning combines the two major areas of AI we have covered so far: search and logic. A\nplanner can be seen either as a program that searches for a solution or as one that (construc-\ntively) proves the existence of a solution. The cross-fertilization of ideas from the two areas\nhas led both to improvements in performance amounting to several orders of magnitude in\nthe last decade and to an increased use of planners in industrial applications. Unfortunately,\nwe do not yet have a clear understanding of which techniques work best on which kinds of\nproblems. Quite possibly, new techniques will emerge that dominate existing methods.\nPlanning is foremost an exercise in controlling combinatorial explosion. If there are n\npropositions in a domain, then there are 2n states. As we have seen, planning is PSPACE-\nhard. Against such pessimism, the identification of independent subproblems can be a pow-\nerful weapon. In the best case—full decomposability of the problem—we get an exponential\nspeedup. Decomposability is destroyed, however, by negative interactions between actions.\nGRAPHPLAN records mutexes to point out where the difficult interactions are. SATPLAN rep-\nresents a similar range of mutex relations, but does so by using the general CNF form rather\nthan a specific data structure. Forward search addresses the problem heuristically by trying\nto find patterns (subsets of propositions) that cover the independent subproblems. Since this\napproach is heuristic, it can work even when the subproblems are not completely independent.\nSometimes it is possible to solve a problem efficiently by recognizing that negative\ninteractions can be ruled out. We say that a problem has serializable subgoals if there exists\nSERIALIZABLE\nSUBGOAL\nan order of subgoals such that the planner can achieve them in that order without having to\nundo any of the previously achieved subgoals. For example, in the blocks world, if the goal\nis to build a tower (e.g., A on B, which in turn is on C, which in turn is on the Table, as in\nFigure 10.4 on page 371), then the subgoals are serializable bottom to top: if we first achieve\nC on Table, we will never have to undo it while we are achieving the other subgoals. A\nplanner that uses the bottom-to-top trick can solve any problem in the blocks world without\nbacktracking (although it might not always find the shortest plan).\nAs a more complex example, for the Remote Agent planner that commanded NASA’s\nDeep Space One spacecraft, it was determined that the propositions involved in command-\ning a spacecraft are serializable. This is perhaps not too surprising, because a spacecraft is\ndesigned by its engineers to be as easy as possible to control (subject to other constraints).\nTaking advantage of the serialized ordering of goals, the Remote Agent planner was able to\neliminate most of the search. This meant that it was fast enough to control the spacecraft in\nreal time, something previously considered impossible.\nPlanners such as GRAPHPLAN, SATPLAN, and FF have moved the field of planning\nforward, by raising the level of performance of planning systems, by clarifying the repre-\nsentational and combinatorial issues involved, and by the development of useful heuristics.\nHowever, there is a question of how far these techniques will scale. It seems likely that further\nprogress on larger problems cannot rely only on factored and propositional representations,\nand will require some kind of synthesis of first-order and hierarchical representations with\nthe efficient heuristics currently in use.\nSection 10.6.\nSummary\n393\n10.6\nSUMMARY\nIn this chapter, we defined the problem of planning in deterministic, fully observable, static\nenvironments. We described the PDDL representation for planning problems and several\nalgorithmic approaches for solving them. The points to remember:\n• Planning systems are problem-solving algorithms that operate on explicit propositional\nor relational representations of states and actions. These representations make possi-\nble the derivation of effective heuristics and the development of powerful and flexible\nalgorithms for solving problems.\n• PDDL, the Planning Domain Definition Language, describes the initial and goal states\nas conjunctions of literals, and actions in terms of their preconditions and effects.\n• State-space search can operate in the forward direction (progression) or the backward\ndirection (regression). Effective heuristics can be derived by subgoal independence\nassumptions and by various relaxations of the planning problem.\n• A planning graph can be constructed incrementally, starting from the initial state. Each\nlayer contains a superset of all the literals or actions that could occur at that time step\nand encodes mutual exclusion (mutex) relations among literals or actions that cannot co-\noccur. Planning graphs yield useful heuristics for state-space and partial-order planners\nand can be used directly in the GRAPHPLAN algorithm.\n• Other approaches include first-order deduction over situation calculus axioms; encoding\na planning problem as a Boolean satisfiability problem or as a constraint satisfaction\nproblem; and explicitly searching through the space of partially ordered plans.\n• Each of the major approaches to planning has its adherents, and there is as yet no con-\nsensus on which is best. Competition and cross-fertilization among the approaches have\nresulted in significant gains in efficiency for planning systems.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nAI planning arose from investigations into state-space search, theorem proving, and control\ntheory and from the practical needs of robotics, scheduling, and other domains. STRIPS (Fikes\nand Nilsson, 1971), the first major planning system, illustrates the interaction of these influ-\nences. STRIPS was designed as the planning component of the software for the Shakey robot\nproject at SRI. Its overall control structure was modeled on that of GPS, the General Problem\nSolver (Newell and Simon, 1961), a state-space search system that used means–ends anal-\nysis. Bylander (1992) shows simple STRIPS planning to be PSPACE-complete. Fikes and\nNilsson (1993) give a historical retrospective on the STRIPS project and its relationship to\nmore recent planning efforts.\nThe representation language used by STRIPS has been far more influential than its al-\ngorithmic approach; what we call the “classical” language is close to what STRIPS used.\n394\nChapter\n10.\nClassical Planning\nThe Action Description Language, or ADL (Pednault, 1986), relaxed some of the STRIPS\nrestrictions and made it possible to encode more realistic problems. Nebel (2000) explores\nschemes for compiling ADL into STRIPS. The Problem Domain Description Language, or\nPDDL (Ghallab et al., 1998), was introduced as a computer-parsable, standardized syntax for\nrepresenting planning problems and has been used as the standard language for the Interna-\ntional Planning Competition since 1998. There have been several extensions; the most recent\nversion, PDDL 3.0, includes plan constraints and preferences (Gerevini and Long, 2005).\nPlanners in the early 1970s generally considered totally ordered action sequences. Prob-\nlem decomposition was achieved by computing a subplan for each subgoal and then stringing\nthe subplans together in some order. This approach, called linear planning by Sacerdoti\nLINEAR PLANNING\n(1975), was soon discovered to be incomplete. It cannot solve some very simple problems,\nsuch as the Sussman anomaly (see Exercise 10.7), found by Allen Brown during experimen-\ntation with the HACKER system (Sussman, 1975). A complete planner must allow for inter-\nleaving of actions from different subplans within a single sequence. The notion of serializable\nINTERLEAVING\nsubgoals (Korf, 1987) corresponds exactly to the set of problems for which noninterleaved\nplanners are complete.\nOne solution to the interleaving problem was goal-regression planning, a technique in\nwhich steps in a totally ordered plan are reordered so as to avoid conflict between subgoals.\nThis was introduced by Waldinger (1975) and also used by Warren’s (1974) WARPLAN.\nWARPLAN is also notable in that it was the first planner to be written in a logic program-\nming language (Prolog) and is one of the best examples of the remarkable economy that can\nsometimes be gained with logic programming: WARPLAN is only 100 lines of code, a small\nfraction of the size of comparable planners of the time.\nThe ideas underlying partial-order planning include the detection of conflicts (Tate,\n1975a) and the protection of achieved conditions from interference (Sussman, 1975). The\nconstruction of partially ordered plans (then called task networks) was pioneered by the\nNOAH planner (Sacerdoti, 1975, 1977) and by Tate’s (1975b, 1977) NONLIN system.\nPartial-order planning dominated the next 20 years of research, yet the first clear for-\nmal exposition was TWEAK (Chapman, 1987), a planner that was simple enough to allow\nproofs of completeness and intractability (NP-hardness and undecidability) of various plan-\nning problems. Chapman’s work led to a straightforward description of a complete partial-\norder planner (McAllester and Rosenblitt, 1991), then to the widely distributed implementa-\ntions SNLP (Soderland and Weld, 1991) and UCPOP (Penberthy and Weld, 1992). Partial-\norder planning fell out of favor in the late 1990s as faster methods emerged. Nguyen and\nKambhampati (2001) suggest that a reconsideration is merited: with accurate heuristics de-\nrived from a planning graph, their REPOP planner scales up much better than GRAPHPLAN\nin parallelizable domains and is competitive with the fastest state-space planners.\nThe resurgence of interest in state-space planning was pioneered by Drew McDer-\nmott’s UNPOP program (1996), which was the first to suggest the ignore-delete-list heuristic,\nThe name UNPOP was a reaction to the overwhelming concentration on partial-order plan-\nning at the time; McDermott suspected that other approaches were not getting the attention\nthey deserved. Bonet and Geffner’s Heuristic Search Planner (HSP) and its later deriva-\ntives (Bonet and Geffner, 1999; Haslum et al., 2005; Haslum, 2006) were the first to make\nBibliographical and Historical Notes\n395\nstate-space search practical for large planning problems. HSP searches in the forward di-\nrection while HSPR (Bonet and Geffner, 1999) searches backward. The most successful\nstate-space searcher to date is FF (Hoffmann, 2001; Hoffmann and Nebel, 2001; Hoffmann,\n2005), winner of the AIPS 2000 planning competition. FASTDOWNWARD (Helmert, 2006)\nis a forward state-space search planner that preprocesses the action schemas into an alter-\nnative representation which makes some of the constraints more explicit. FASTDOWNWARD\n(Helmert and Richter, 2004; Helmert, 2006) won the 2004 planning competition, and LAMA\n(Richter and Westphal, 2008), a planner based on FASTDOWNWARD with improved heuris-\ntics, won the 2008 competition.\nBylander (1994) and Ghallab et al. (2004) discuss the computational complexity of\nseveral variants of the planning problem. Helmert (2003) proves complexity bounds for many\nof the standard benchmark problems, and Hoffmann (2005) analyzes the search space of the\nignore-delete-list heuristic. Heuristics for the set-covering problem are discussed by Caprara\net al. (1995) for scheduling operations of the Italian railway. Edelkamp (2009) and Haslum\net al. (2007) describe how to construct pattern databases for planning heuristics. As we\nmentioned in Chapter 3, Felner et al. (2004) show encouraging results using pattern databases\nfor sliding blocks puzzles, which can be thought of as a planning domain, but Hoffmann et al.\n(2006) show some limitations of abstraction for classical planning problems.\nAvrim Blum and Merrick Furst (1995, 1997) revitalized the field of planning with their\nGRAPHPLAN system, which was orders of magnitude faster than the partial-order planners of\nthe time. Other graph-planning systems, such as IPP (Koehler et al., 1997), STAN (Fox and\nLong, 1998), and SGP (Weld et al., 1998), soon followed. A data structure closely resembling\nthe planning graph had been developed slightly earlier by Ghallab and Laruelle (1994), whose\nIXTET partial-order planner used it to derive accurate heuristics to guide searches. Nguyen\net al. (2001) thoroughly analyze heuristics derived from planning graphs. Our discussion of\nplanning graphs is based partly on this work and on lecture notes and articles by Subbarao\nKambhampati (Bryce and Kambhampati, 2007). As mentioned in the chapter, a planning\ngraph can be used in many different ways to guide the search for a solution. The winner\nof the 2002 AIPS planning competition, LPG (Gerevini and Serina, 2002, 2003), searched\nplanning graphs using a local search technique inspired by WALKSAT.\nThe situation calculus approach to planning was introduced by John McCarthy (1963).\nThe version we show here was proposed by Ray Reiter (1991, 2001).\nKautz et al. (1996) investigated various ways to propositionalize action schemas, find-\ning that the most compact forms did not necessarily lead to the fastest solution times. A\nsystematic analysis was carried out by Ernst et al. (1997), who also developed an auto-\nmatic “compiler” for generating propositional representations from PDDL problems. The\nBLACKBOX planner, which combines ideas from GRAPHPLAN and SATPLAN, was devel-\noped by Kautz and Selman (1998). CPLAN, a planner based on constraint satisfaction, was\ndescribed by van Beek and Chen (1999).\nMost recently, there has been interest in the representation of plans as binary decision\ndiagrams, compact data structures for Boolean expressions widely studied in the hardware\nBINARY DECISION\nDIAGRAM\nverification community (Clarke and Grumberg, 1987; McMillan, 1993). There are techniques\nfor proving properties of binary decision diagrams, including the property of being a solution\n396\nChapter\n10.\nClassical Planning\nto a planning problem. Cimatti et al. (1998) present a planner based on this approach. Other\nrepresentations have also been used; for example, Vossen et al. (2001) survey the use of\ninteger programming for planning.\nThe jury is still out, but there are now some interesting comparisons of the various\napproaches to planning. Helmert (2001) analyzes several classes of planning problems, and\nshows that constraint-based approaches such as GRAPHPLAN and SATPLAN are best for NP-\nhard domains, while search-based approaches do better in domains where feasible solutions\ncan be found without backtracking. GRAPHPLAN and SATPLAN have trouble in domains\nwith many objects because that means they must create many actions. In some cases the\nproblem can be delayed or avoided by generating the propositionalized actions dynamically,\nonly as needed, rather than instantiating them all before the search begins.\nReadings in Planning (Allen et al., 1990) is a comprehensive anthology of early work\nin the field. Weld (1994, 1999) provides two excellent surveys of planning algorithms of\nthe 1990s. It is interesting to see the change in the five years between the two surveys:\nthe first concentrates on partial-order planning, and the second introduces GRAPHPLAN and\nSATPLAN. Automated Planning (Ghallab et al., 2004) is an excellent textbook on all aspects\nof planning. LaValle’s text Planning Algorithms (2006) covers both classical and stochastic\nplanning, with extensive coverage of robot motion planning.\nPlanning research has been central to AI since its inception, and papers on planning are\na staple of mainstream AI journals and conferences. There are also specialized conferences\nsuch as the International Conference on AI Planning Systems, the International Workshop on\nPlanning and Scheduling for Space, and the European Conference on Planning.\nEXERCISES\n10.1\nDescribe the differences and similarities between problem solving and planning.\n10.2\nGiven the action schemas and initial state from Figure 10.1, what are all the applicable\nconcrete instances of Fly(p, from, to) in the state described by\nAt(P1, JFK) ∧At(P2, SFO) ∧Plane(P1) ∧Plane(P2)\n∧Airport(JFK ) ∧Airport(SFO) ?\n10.3\nThe monkey-and-bananas problem is faced by a monkey in a laboratory with some\nbananas hanging out of reach from the ceiling. A box is available that will enable the monkey\nto reach the bananas if he climbs on it. Initially, the monkey is at A, the bananas at B, and the\nbox at C. The monkey and box have height Low, but if the monkey climbs onto the box he\nwill have height High, the same as the bananas. The actions available to the monkey include\nGo from one place to another, Push an object from one place to another, ClimbUp onto or\nClimbDown from an object, and Grasp or Ungrasp an object. The result of a Grasp is that\nthe monkey holds the object if the monkey and object are in the same place at the same height.\na. Write down the initial state description.\nExercises\n397\nRoom 4\nRoom 3\nRoom 2\nRoom 1\nDoor 1\nDoor 2\nDoor 3\nDoor 4\nBox 1\nBox 2\nBox 3\nShakey\nSwitch 1\nSwitch 2\nSwitch 3\nSwitch 4\nBox 4\nCorridor\nFigure 10.14\nShakey’s world. Shakey can move between landmarks within a room, can\npass through the door between rooms, can climb climbable objects and push pushable objects,\nand can flip light switches.\nb. Write the six action schemas.\nc. Suppose the monkey wants to fool the scientists, who are off to tea, by grabbing the\nbananas, but leaving the box in its original place. Write this as a general goal (i.e., not\nassuming that the box is necessarily at C) in the language of situation calculus. Can this\ngoal be solved by a classical planning system?\nd. Your schema for pushing is probably incorrect, because if the object is too heavy, its\nposition will remain the same when the Push schema is applied. Fix your action schema\nto account for heavy objects.\n10.4\nThe original STRIPS planner was designed to control Shakey the robot. Figure 10.14\nshows a version of Shakey’s world consisting of four rooms lined up along a corridor, where\neach room has a door and a light switch. The actions in Shakey’s world include moving from\nplace to place, pushing movable objects (such as boxes), climbing onto and down from rigid\n398\nChapter\n10.\nClassical Planning\nobjects (such as boxes), and turning light switches on and off. The robot itself could not climb\non a box or toggle a switch, but the planner was capable of finding and printing out plans that\nwere beyond the robot’s abilities. Shakey’s six actions are the following:\n• Go(x, y, r), which requires that Shakey be At x and that x and y are locations In the\nsame room r. By convention a door between two rooms is in both of them.\n• Push a box b from location x to location y within the same room: Push(b, x, y, r). You\nwill need the predicate Box and constants for the boxes.\n• Climb onto a box from position x: ClimbUp(x, b); climb down from a box to position\nx: ClimbDown(b, x). We will need the predicate On and the constant Floor.\n• Turn a light switch on or off: TurnOn(s, b); TurnOff(s, b). To turn a light on or off,\nShakey must be on top of a box at the light switch’s location.\nWrite PDDL sentences for Shakey’s six actions and the initial state from Figure 10.14. Con-\nstruct a plan for Shakey to get Box 2 into Room2.\n10.5\nA finite Turing machine has a finite one-dimensional tape of cells, each cell containing\none of a finite number of symbols. One cell has a read and write head above it. There is a\nfinite set of states the machine can be in, one of which is the accept state. At each time step,\ndepending on the symbol on the cell under the head and the machine’s current state, there are\na set of actions we can choose from. Each action involves writing a symbol to the cell under\nthe head, transitioning the machine to a state, and optionally moving the head left or right.\nThe mapping that determines which actions are allowed is the Turing machine’s program.\nYour goal is to control the machine into the accept state.\nRepresent the Turing machine acceptance problem as a planning problem. If you can\ndo this, it demonstrates that determining whether a planning problem has a solution is at least\nas hard as the Turing acceptance problem, which is PSPACE-hard.\n10.6\nExplain why dropping negative effects from every action schema in a planning prob-\nlem results in a relaxed problem.\n10.7\nFigure 10.4 (page 371) shows a blocks-world problem that is known as the Sussman\nanomaly. The problem was considered anomalous because the noninterleaved planners of\nSUSSMAN ANOMALY\nthe early 1970s could not solve it. Write a definition of the problem and solve it, either by\nhand or with a planning program. A noninterleaved planner is a planner that, when given two\nsubgoals G1 and G2, produces either a plan for G1 concatenated with a plan for G2, or vice\nversa. Explain why a noninterleaved planner cannot solve this problem.\n10.8\nProve that backward search with PDDL problems is complete.\n10.9\nConstruct levels 0, 1, and 2 of the planning graph for the problem in Figure 10.1.\n10.10\nProve the following assertions about planning graphs:\na. A literal that does not appear in the final level of the graph cannot be achieved.\nExercises\n399\nb. The level cost of a literal in a serial graph is no greater than the actual cost of an optimal\nplan for achieving it.\n10.11\nThe set-level heuristic (see page 382) uses a planning graph to estimate the cost of\nachieving a conjunctive goal from the current state. What relaxed problem is the set-level\nheuristic the solution to?\n10.12\nExamine the definition of bidirectional search in Chapter 3.\na. Would bidirectional state-space search be a good idea for planning?\nb. What about bidirectional search in the space of partial-order plans?\nc. Devise a version of partial-order planning in which an action can be added to a plan if its\npreconditions can be achieved by the effects of actions already in the plan. Explain how\nto deal with conflicts and ordering constraints. Is the algorithm essentially identical to\nforward state-space search?\n10.13\nWe contrasted forward and backward state-space searchers with partial-order plan-\nners, saying that the latter is a plan-space searcher. Explain how forward and backward state-\nspace search can also be considered plan-space searchers, and say what the plan refinement\noperators are.\n10.14\nUp to now we have assumed that the plans we create always make sure that an action’s\npreconditions are satisfied. Let us now investigate what propositional successor-state axioms\nsuch as HaveArrow t+1\n⇔\n(HaveArrow t ∧¬Shoott) have to say about actions whose\npreconditions are not satisfied.\na. Show that the axioms predict that nothing will happen when an action is executed in a\nstate where its preconditions are not satisfied.\nb. Consider a plan p that contains the actions required to achieve a goal but also includes\nillegal actions. Is it the case that\ninitial state ∧successor-state axioms ∧p |= goal ?\nc. With first-order successor-state axioms in situation calculus, is it possible to prove that\na plan containing illegal actions will achieve the goal?\n10.15\nConsider how to translate a set of action schemas into the successor-state axioms of\nsituation calculus.\na. Consider the schema for Fly(p, from, to). Write a logical definition for the predicate\nPoss(Fly(p, from, to), s), which is true if the preconditions for Fly(p, from, to) are\nsatisfied in situation s.\nb. Next, assuming that Fly(p, from, to) is the only action schema available to the agent,\nwrite down a successor-state axiom for At(p, x, s) that captures the same information\nas the action schema.\n400\nChapter\n10.\nClassical Planning\nc. Now suppose there is an additional method of travel: Teleport(p, from, to). It has\nthe additional precondition ¬Warped(p) and the additional effect Warped(p). Explain\nhow the situation calculus knowledge base must be modified.\nd. Finally, develop a general and precisely specified procedure for carrying out the trans-\nlation from a set of action schemas to a set of successor-state axioms.\n10.16\nIn the SATPLAN algorithm in Figure 7.22 (page 272), each call to the satisfiabil-\nity algorithm asserts a goal gT , where T ranges from 0 to Tmax. Suppose instead that the\nsatisfiability algorithm is called only once, with the goal g0 ∨g1 ∨· · · ∨gTmax.\na. Will this always return a plan if one exists with length less than or equal to Tmax?\nb. Does this approach introduce any new spurious “solutions”?\nc. Discuss how one might modify a satisfiability algorithm such as WALKSAT so that it\nfinds short solutions (if they exist) when given a disjunctive goal of this form.\n" }, { "chapter": 11, "text": "11\nPLANNING AND ACTING\nIN THE REAL WORLD\nIn which we see how more expressive representations and more interactive agent\narchitectures lead to planners that are useful in the real world.\nThe previous chapter introduced the most basic concepts, representations, and algorithms for\nplanning. Planners that are are used in the real world for planning and scheduling the oper-\nations of spacecraft, factories, and military campaigns are more complex; they extend both\nthe representation language and the way the planner interacts with the environment. This\nchapter shows how. Section 11.1 extends the classical language for planning to talk about\nactions with durations and resource constraints. Section 11.2 describes methods for con-\nstructing plans that are organized hierarchically. This allows human experts to communicate\nto the planner what they know about how to solve the problem. Hierarchy also lends itself to\nefficient plan construction because the planner can solve a problem at an abstract level before\ndelving into details. Section 11.3 presents agent architectures that can handle uncertain envi-\nronments and interleave deliberation with execution, and gives some examples of real-world\nsystems. Section 11.4 shows how to plan when the environment contains other agents.\n11.1\nTIME, SCHEDULES, AND RESOURCES\nThe classical planning representation talks about what to do, and in what order, but the repre-\nsentation cannot talk about time: how long an action takes and when it occurs. For example,\nthe planners of Chapter 10 could produce a schedule for an airline that says which planes are\nassigned to which flights, but we really need to know departure and arrival times as well. This\nis the subject matter of scheduling. The real world also imposes many resource constraints;\nfor example, an airline has a limited number of staff—and staff who are on one flight cannot\nbe on another at the same time. This section covers methods for representing and solving\nplanning problems that include temporal and resource constraints.\nThe approach we take in this section is “plan first, schedule later”: that is, we divide\nthe overall problem into a planning phase in which actions are selected, with some ordering\nconstraints, to meet the goals of the problem, and a later scheduling phase, in which tempo-\nral information is added to the plan to ensure that it meets resource and deadline constraints.\n401\n402\nChapter\n11.\nPlanning and Acting in the Real World\nJobs({AddEngine1 ≺AddWheels1 ≺Inspect1 },\n{AddEngine2 ≺AddWheels2 ≺Inspect2})\nResources(EngineHoists(1), WheelStations(1), Inspectors(2), LugNuts(500))\nAction(AddEngine1, DURATION:30,\nUSE:EngineHoists(1))\nAction(AddEngine2, DURATION:60,\nUSE:EngineHoists(1))\nAction(AddWheels1 , DURATION:30,\nCONSUME:LugNuts(20), USE:WheelStations(1))\nAction(AddWheels2 , DURATION:15,\nCONSUME:LugNuts(20), USE:WheelStations(1))\nAction(Inspecti, DURATION:10,\nUSE:Inspectors(1))\nFigure 11.1\nA job-shop scheduling problem for assembling two cars, with resource con-\nstraints. The notation A ≺B means that action A must precede action B.\nThis approach is common in real-world manufacturing and logistical settings, where the plan-\nning phase is often performed by human experts. The automated methods of Chapter 10 can\nalso be used for the planning phase, provided that they produce plans with just the minimal\nordering constraints required for correctness. G RAPHPLAN (Section 10.3), SATPLAN (Sec-\ntion 10.4.1), and partial-order planners (Section 10.4.4) can do this; search-based methods\n(Section 10.2) produce totally ordered plans, but these can easily be converted to plans with\nminimal ordering constraints.\n11.1.1\nRepresenting temporal and resource constraints\nA typical job-shop scheduling problem, as first introduced in Section 6.1.2, consists of a\nset of jobs, each of which consists a collection of actions with ordering constraints among\nJOB\nthem. Each action has a duration and a set of resource constraints required by the action.\nDURATION\nEach constraint specifies a type of resource (e.g., bolts, wrenches, or pilots), the number\nof that resource required, and whether that resource is consumable (e.g., the bolts are no\nCONSUMABLE\nlonger available for use) or reusable (e.g., a pilot is occupied during a flight but is available\nREUSABLE\nagain when the flight is over). Resources can also be produced by actions with negative con-\nsumption, including manufacturing, growing, and resupply actions. A solution to a job-shop\nscheduling problem must specify the start times for each action and must satisfy all the tem-\nporal ordering constraints and resource constraints. As with search and planning problems,\nsolutions can be evaluated according to a cost function; this can be quite complicated, with\nnonlinear resource costs, time-dependent delay costs, and so on. For simplicity, we assume\nthat the cost function is just the total duration of the plan, which is called the makespan.\nMAKESPAN\nFigure 11.1 shows a simple example: a problem involving the assembly of two cars. The\nproblem consists of two jobs, each of the form [AddEngine, AddWheels, Inspect]. Then the\nSection 11.1.\nTime, Schedules, and Resources\n403\nResources statement declares that there are four types of resources, and gives the number\nof each type available at the start: 1 engine hoist, 1 wheel station, 2 inspectors, and 500 lug\nnuts. The action schemas give the duration and resource needs of each action. The lug nuts\nare consumed as wheels are added to the car, whereas the other resources are “borrowed” at\nthe start of an action and released at the action’s end.\nThe representation of resources as numerical quantities, such as Inspectors(2), rather\nthan as named entities, such as Inspector(I1) and Inspector(I2), is an example of a very\ngeneral technique called aggregation. The central idea of aggregation is to group individual\nAGGREGATION\nobjects into quantities when the objects are all indistinguishable with respect to the purpose\nat hand. In our assembly problem, it does not matter which inspector inspects the car, so there\nis no need to make the distinction. (The same idea works in the missionaries-and-cannibals\nproblem in Exercise 3.9.) Aggregation is essential for reducing complexity. Consider what\nhappens when a proposed schedule has 10 concurrent Inspect actions but only 9 inspectors\nare available. With inspectors represented as quantities, a failure is detected immediately and\nthe algorithm backtracks to try another schedule. With inspectors represented as individuals,\nthe algorithm backtracks to try all 10! ways of assigning inspectors to actions.\n11.1.2\nSolving scheduling problems\nWe begin by considering just the temporal scheduling problem, ignoring resource constraints.\nTo minimize makespan (plan duration), we must find the earliest start times for all the actions\nconsistent with the ordering constraints supplied with the problem. It is helpful to view these\nordering constraints as a directed graph relating the actions, as shown in Figure 11.2. We can\napply the critical path method (CPM) to this graph to determine the possible start and end\nCRITICAL PATH\nMETHOD\ntimes of each action. A path through a graph representing a partial-order plan is a linearly\nordered sequence of actions beginning with Start and ending with Finish. (For example,\nthere are two paths in the partial-order plan in Figure 11.2.)\nThe critical path is that path whose total duration is longest; the path is “critical”\nCRITICAL PATH\nbecause it determines the duration of the entire plan—shortening other paths doesn’t shorten\nthe plan as a whole, but delaying the start of any action on the critical path slows down the\nwhole plan. Actions that are off the critical path have a window of time in which they can be\nexecuted. The window is specified in terms of an earliest possible start time, ES, and a latest\npossible start time, LS. The quantity LS – ES is known as the slack of an action. We can\nSLACK\nsee in Figure 11.2 that the whole plan will take 85 minutes, that each action in the top job\nhas 15 minutes of slack, and that each action on the critical path has no slack (by definition).\nTogether the ES and LS times for all the actions constitute a schedule for the problem.\nSCHEDULE\nThe following formulas serve as a definition for ES and LS and also as the outline of a\ndynamic-programming algorithm to compute them. A and B are actions, and A ≺B means\nthat A comes before B:\nES(Start) = 0\nES(B) = maxA ≺B ES(A) + Duration(A)\nLS(Finish) = ES(Finish)\nLS(A) = minB ≻A LS(B) −Duration(A) .\n404\nChapter\n11.\nPlanning and Acting in the Real World\nStart\n [0,0]\nAddEngine1\n30\n [0,15]\nAddWheels1\n30\n [30,45]\n10\nInspect1\n [60,75]\nFinish\n [85,85]\n10\nInspect2\n [75,75]\n15\nAddWheels2\n [60,60]\n60\nAddEngine2\n [0,0]\nAddEngine1\nAddWheels1\nInspect1\nAddWheels2\nInspect2\nAddEngine2\n0\n10\n20\n30\n40\n50\n60\n70\n80\n90\nFigure 11.2\nTop: a representation of the temporal constraints for the job-shop scheduling\nproblem of Figure 11.1. The duration of each action is given at the bottom of each rectangle.\nIn solving the problem, we compute the earliest and latest start times as the pair [ES, LS],\ndisplayed in the upper left. The difference between these two numbers is the slack of an\naction; actions with zero slack are on the critical path, shown with bold arrows. Bottom: the\nsame solution shown as a timeline. Grey rectangles represent time intervals during which an\naction may be executed, provided that the ordering constraints are respected. The unoccupied\nportion of a gray rectangle indicates the slack.\nThe idea is that we start by assigning ES(Start) to be 0. Then, as soon as we get an action\nB such that all the actions that come immediately before B have ES values assigned, we\nset ES(B) to be the maximum of the earliest finish times of those immediately preceding\nactions, where the earliest finish time of an action is defined as the earliest start time plus the\nduration. This process repeats until every action has been assigned an ES value. The LS\nvalues are computed in a similar manner, working backward from the Finish action.\nThe complexity of the critical path algorithm is just O(Nb), where N is the number of\nactions and b is the maximum branching factor into or out of an action. (To see this, note that\nthe LS and ES computations are done once for each action, and each computation iterates\nover at most b other actions.) Therefore, finding a minimum-duration schedule, given a partial\nordering on the actions and no resource constraints, is quite easy.\nMathematically speaking, critical-path problems are easy to solve because they are de-\nfined as a conjunction of linear inequalities on the start and end times. When we introduce\nresource constraints, the resulting constraints on start and end times become more compli-\ncated. For example, the AddEngine actions, which begin at the same time in Figure 11.2,\nSection 11.1.\nTime, Schedules, and Resources\n405\nAddEngine1\nAddWheels1\nInspect1\nAddWheels2\nInspect2\nAddEngine2\n0\n10\n20\n30\n40\n50\n60\n70\n80\n90\n100\n110\n120\nEngineHoists(1)\nWheelStations(1)\nInspectors(2)\nFigure 11.3\nA solution to the job-shop scheduling problem from Figure 11.1, taking into\naccount resource constraints. The left-hand margin lists the three reusable resources, and\nactions are shown aligned horizontally with the resources they use. There are two possi-\nble schedules, depending on which assembly uses the engine hoist first; we’ve shown the\nshortest-duration solution, which takes 115 minutes.\nrequire the same EngineHoist and so cannot overlap. The “cannot overlap” constraint is a\ndisjunction of two linear inequalities, one for each possible ordering. The introduction of\ndisjunctions turns out to make scheduling with resource constraints NP-hard.\nFigure 11.3 shows the solution with the fastest completion time, 115 minutes. This is\n30 minutes longer than the 85 minutes required for a schedule without resource constraints.\nNotice that there is no time at which both inspectors are required, so we can immediately\nmove one of our two inspectors to a more productive position.\nThe complexity of scheduling with resource constraints is often seen in practice as\nwell as in theory. A challenge problem posed in 1963—to find the optimal schedule for a\nproblem involving just 10 machines and 10 jobs of 100 actions each—went unsolved for\n23 years (Lawler et al., 1993). Many approaches have been tried, including branch-and-\nbound, simulated annealing, tabu search, constraint satisfaction, and other techniques from\nChapters 3 and 4. One simple but popular heuristic is the minimum slack algorithm: on\nMINIMUM SLACK\neach iteration, schedule for the earliest possible start whichever unscheduled action has all\nits predecessors scheduled and has the least slack; then update the ES and LS times for each\naffected action and repeat. The heuristic resembles the minimum-remaining-values (MRV)\nheuristic in constraint satisfaction. It often works well in practice, but for our assembly\nproblem it yields a 130–minute solution, not the 115–minute solution of Figure 11.3.\nUp to this point, we have assumed that the set of actions and ordering constraints is\nfixed. Under these assumptions, every scheduling problem can be solved by a nonoverlapping\nsequence that avoids all resource conflicts, provided that each action is feasible by itself. If\na scheduling problem is proving very difficult, however, it may not be a good idea to solve\nit this way—it may be better to reconsider the actions and constraints, in case that leads to a\nmuch easier scheduling problem. Thus, it makes sense to integrate planning and scheduling\nby taking into account durations and overlaps during the construction of a partial-order plan.\nSeveral of the planning algorithms in Chapter 10 can be augmented to handle this information.\nFor example, partial-order planners can detect resource constraint violations in much the\nsame way they detect conflicts with causal links. Heuristics can be devised to estimate the\ntotal completion time of a plan. This is currently an active area of research.\n406\nChapter\n11.\nPlanning and Acting in the Real World\n11.2\nHIERARCHICAL PLANNING\nThe problem-solving and planning methods of the preceding chapters all operate with a fixed\nset of atomic actions. Actions can be strung together into sequences or branching networks;\nstate-of-the-art algorithms can generate solutions containing thousands of actions.\nFor plans executed by the human brain, atomic actions are muscle activations. In very\nround numbers, we have about 103 muscles to activate (639, by some counts, but many of\nthem have multiple subunits); we can modulate their activation perhaps 10 times per second;\nand we are alive and awake for about 109 seconds in all. Thus, a human life contains about\n1013 actions, give or take one or two orders of magnitude. Even if we restrict ourselves to\nplanning over much shorter time horizons—for example, a two-week vacation in Hawaii—a\ndetailed motor plan would contain around 1010 actions. This is a lot more than 1000.\nTo bridge this gap, AI systems will probably have to do what humans appear to do: plan\nat higher levels of abstraction. A reasonable plan for the Hawaii vacation might be “Go to\nSan Francisco airport; take Hawaiian Airlines flight 11 to Honolulu; do vacation stuff for two\nweeks; take Hawaiian Airlines flight 12 back to San Francisco; go home.” Given such a plan,\nthe action “Go to San Francisco airport” can be viewed as a planning task in itself, with a\nsolution such as “Drive to the long-term parking lot; park; take the shuttle to the terminal.”\nEach of these actions, in turn, can be decomposed further, until we reach the level of actions\nthat can be executed without deliberation to generate the required motor control sequences.\nIn this example, we see that planning can occur both before and during the execution\nof the plan; for example, one would probably defer the problem of planning a route from a\nparking spot in long-term parking to the shuttle bus stop until a particular parking spot has\nbeen found during execution. Thus, that particular action will remain at an abstract level\nprior to the execution phase. We defer discussion of this topic until Section 11.3. Here, we\nconcentrate on the aspect of hierarchical decomposition, an idea that pervades almost all\nHIERARCHICAL\nDECOMPOSITION\nattempts to manage complexity. For example, complex software is created from a hierarchy\nof subroutines or object classes; armies operate as a hierarchy of units; governments and cor-\nporations have hierarchies of departments, subsidiaries, and branch offices. The key benefit\nof hierarchical structure is that, at each level of the hierarchy, a computational task, military\nmission, or administrative function is reduced to a small number of activities at the next lower\nlevel, so the computational cost of finding the correct way to arrange those activities for the\ncurrent problem is small. Nonhierarchical methods, on the other hand, reduce a task to a\nlarge number of individual actions; for large-scale problems, this is completely impractical.\n11.2.1\nHigh-level actions\nThe basic formalism we adopt to understand hierarchical decomposition comes from the area\nof hierarchical task networks or HTN planning. As in classical planning (Chapter 10), we\nHIERARCHICAL TASK\nNETWORK\nassume full observability and determinism and the availability of a set of actions, now called\nprimitive actions, with standard precondition–effect schemas. The key additional concept is\nPRIMITIVE ACTION\nthe high-level action or HLA—for example, the action “Go to San Francisco airport” in the\nHIGH-LEVEL ACTION\nSection 11.2.\nHierarchical Planning\n407\nRefinement(Go(Home, SFO),\nSTEPS: [Drive(Home, SFOLongTermParking),\nShuttle(SFOLongTermParking, SFO)] )\nRefinement(Go(Home, SFO),\nSTEPS: [Taxi(Home, SFO)] )\nRefinement(Navigate([a, b], [x, y]),\nPRECOND: a = x ∧b = y\nSTEPS: [ ] )\nRefinement(Navigate([a, b], [x, y]),\nPRECOND:Connected([a, b], [a −1, b])\nSTEPS: [Left, Navigate([a −1, b], [x, y])] )\nRefinement(Navigate([a, b], [x, y]),\nPRECOND:Connected([a, b], [a + 1, b])\nSTEPS: [Right, Navigate([a + 1, b], [x, y])] )\n. . .\nFigure 11.4\nDefinitions of possible refinements for two high-level actions: going to San\nFrancisco airport and navigating in the vacuum world. In the latter case, note the recursive\nnature of the refinements and the use of preconditions.\nexample given earlier. Each HLA has one or more possible refinements, into a sequence1\nREFINEMENT\nof actions, each of which may be an HLA or a primitive action (which has no refinements\nby definition). For example, the action “Go to San Francisco airport,” represented formally\nas Go(Home, SFO), might have two possible refinements, as shown in Figure 11.4. The\nsame figure shows a recursive refinement for navigation in the vacuum world: to get to a\ndestination, take a step, and then go to the destination.\nThese examples show that high-level actions and their refinements embody knowledge\nabout how to do things. For instance, the refinements for Go(Home, SFO) say that to get to\nthe airport you can drive or take a taxi; buying milk, sitting down, and moving the knight to\ne4 are not to be considered.\nAn HLA refinement that contains only primitive actions is called an implementation\nIMPLEMENTATION\nof the HLA. For example, in the vacuum world, the sequences [Right, Right, Down] and\n[Down, Right, Right] both implement the HLA Navigate([1, 3], [3, 2]). An implementation\nof a high-level plan (a sequence of HLAs) is the concatenation of implementations of each\nHLA in the sequence. Given the precondition–effect definitions of each primitive action, it is\nstraightforward to determine whether any given implementation of a high-level plan achieves\nthe goal. We can say, then, that a high-level plan achieves the goal from a given state if at\nleast one of its implementations achieves the goal from that state. The “at least one” in this\ndefinition is crucial—not all implementations need to achieve the goal, because the agent gets\n1 HTN planners often allow refinement into partially ordered plans, and they allow the refinements of two\ndifferent HLAs in a plan to share actions. We omit these important complications in the interest of understanding\nthe basic concepts of hierarchical planning.\n408\nChapter\n11.\nPlanning and Acting in the Real World\nto decide which implementation it will execute. Thus, the set of possible implementations in\nHTN planning—each of which may have a different outcome—is not the same as the set of\npossible outcomes in nondeterministic planning. There, we required that a plan work for all\noutcomes because the agent doesn’t get to choose the outcome; nature does.\nThe simplest case is an HLA that has exactly one implementation. In that case, we\ncan compute the preconditions and effects of the HLA from those of the implementation\n(see Exercise 11.3) and then treat the HLA exactly as if it were a primitive action itself. It\ncan be shown that the right collection of HLAs can result in the time complexity of blind\nsearch dropping from exponential in the solution depth to linear in the solution depth, al-\nthough devising such a collection of HLAs may be a nontrivial task in itself. When HLAs\nhave multiple possible implementations, there are two options: one is to search among the\nimplementations for one that works, as in Section 11.2.2; the other is to reason directly about\nthe HLAs—despite the multiplicity of implementations—as explained in Section 11.2.3. The\nlatter method enables the derivation of provably correct abstract plans, without the need to\nconsider their implementations.\n11.2.2\nSearching for primitive solutions\nHTN planning is often formulated with a single “top level” action called Act, where the aim is\nto find an implementation of Act that achieves the goal. This approach is entirely general. For\nexample, classical planning problems can be defined as follows: for each primitive action ai,\nprovide one refinement of Act with steps [ai, Act]. That creates a recursive definition of Act\nthat lets us add actions. But we need some way to stop the recursion; we do that by providing\none more refinement for Act, one with an empty list of steps and with a precondition equal\nto the goal of the problem. This says that if the goal is already achieved, then the right\nimplementation is to do nothing.\nThe approach leads to a simple algorithm: repeatedly choose an HLA in the current\nplan and replace it with one of its refinements, until the plan achieves the goal. One possible\nimplementation based on breadth-first tree search is shown in Figure 11.5. Plans are consid-\nered in order of depth of nesting of the refinements, rather than number of primitive steps. It\nis straightforward to design a graph-search version of the algorithm as well as depth-first and\niterative deepening versions.\nIn essence, this form of hierarchical search explores the space of sequences that conform\nto the knowledge contained in the HLA library about how things are to be done. A great deal\nof knowledge can be encoded, not just in the action sequences specified in each refinement but\nalso in the preconditions for the refinements. For some domains, HTN planners have been\nable to generate huge plans with very little search. For example, O-PLAN (Bell and Tate,\n1985), which combines HTN planning with scheduling, has been used to develop production\nplans for Hitachi. A typical problem involves a product line of 350 different products, 35\nassembly machines, and over 2000 different operations. The planner generates a 30-day\nschedule with three 8-hour shifts a day, involving tens of millions of steps. Another important\naspect of HTN plans is that they are, by definition, hierarchically structured; usually this\nmakes them easy for humans to understand.\nSection 11.2.\nHierarchical Planning\n409\nfunction HIERARCHICAL-SEARCH(problem,hierarchy) returns a solution, or failure\nfrontier ←a FIFO queue with [Act] as the only element\nloop do\nif EMPTY?(frontier) then return failure\nplan ←POP(frontier) /* chooses the shallowest plan in frontier */\nhla ←the first HLA in plan, or null if none\nprefix,suffix ←the action subsequences before and after hla in plan\noutcome ←RESULT(problem.INITIAL-STATE, prefix)\nif hla is null then /* so plan is primitive and outcome is its result */\nif outcome satisfies problem.GOAL then return plan\nelse for each sequence in REFINEMENTS(hla,outcome,hierarchy) do\nfrontier ←INSERT(APPEND(prefix,sequence,suffix),frontier)\nFigure 11.5\nA breadth-first implementation of hierarchical forward planning search. The\ninitial plan supplied to the algorithm is [Act]. The REFINEMENTS function returns a set of\naction sequences, one for each refinement of the HLA whose preconditions are satisfied by\nthe specified state, outcome.\nThe computational benefits of hierarchical search can be seen by examining an ide-\nalized case. Suppose that a planning problem has a solution with d primitive actions. For\na nonhierarchical, forward state-space planner with b allowable actions at each state, the\ncost is O(bd), as explained in Chapter 3. For an HTN planner, let us suppose a very reg-\nular refinement structure: each nonprimitive action has r possible refinements, each into\nk actions at the next lower level. We want to know how many different refinement trees\nthere are with this structure. Now, if there are d actions at the primitive level, then the\nnumber of levels below the root is logk d, so the number of internal refinement nodes is\n1 + k + k2 + · · · + klogk d−1 = (d −1)/(k −1). Each internal node has r possible refine-\nments, so r(d−1)/(k−1) possible regular decomposition trees could be constructed. Examining\nthis formula, we see that keeping r small and k large can result in huge savings: essentially\nwe are taking the kth root of the nonhierarchical cost, if b and r are comparable. Small r and\nlarge k means a library of HLAs with a small number of refinements each yielding a long\naction sequence (that nonetheless allows us to solve any problem). This is not always pos-\nsible: long action sequences that are usable across a wide range of problems are extremely\nprecious.\nThe key to HTN planning, then, is the construction of a plan library containing known\nmethods for implementing complex, high-level actions. One method of constructing the li-\nbrary is to learn the methods from problem-solving experience. After the excruciating ex-\nperience of constructing a plan from scratch, the agent can save the plan in the library as a\nmethod for implementing the high-level action defined by the task. In this way, the agent can\nbecome more and more competent over time as new methods are built on top of old methods.\nOne important aspect of this learning process is the ability to generalize the methods that\nare constructed, eliminating detail that is specific to the problem instance (e.g., the name of\n410\nChapter\n11.\nPlanning and Acting in the Real World\nthe builder or the address of the plot of land) and keeping just the key elements of the plan.\nMethods for achieving this kind of generalization are described in Chapter 19. It seems to us\ninconceivable that humans could be as competent as they are without some such mechanism.\n11.2.3\nSearching for abstract solutions\nThe hierarchical search algorithm in the preceding section refines HLAs all the way to primi-\ntive action sequences to determine if a plan is workable. This contradicts common sense: one\nshould be able to determine that the two-HLA high-level plan\n[Drive(Home, SFOLongTermParking), Shuttle(SFOLongTermParking, SFO)]\ngets one to the airport without having to determine a precise route, choice of parking spot,\nand so on. The solution seems obvious: write precondition–effect descriptions of the HLAs,\njust as we write down what the primitive actions do. From the descriptions, it ought to be\neasy to prove that the high-level plan achieves the goal. This is the holy grail, so to speak, of\nhierarchical planning because if we derive a high-level plan that provably achieves the goal,\nworking in a small search space of high-level actions, then we can commit to that plan and\nwork on the problem of refining each step of the plan. This gives us the exponential reduction\nwe seek. For this to work, it has to be the case that every high-level plan that “claims” to\nachieve the goal (by virtue of the descriptions of its steps) does in fact achieve the goal in\nthe sense defined earlier: it must have at least one implementation that does achieve the goal.\nThis property has been called the downward refinement property for HLA descriptions.\nDOWNWARD\nREFINEMENT\nPROPERTY\nWriting HLA descriptions that satisfy the downward refinement property is, in princi-\nple, easy: as long as the descriptions are true, then any high-level plan that claims to achieve\nthe goal must in fact do so—otherwise, the descriptions are making some false claim about\nwhat the HLAs do. We have already seen how to write true descriptions for HLAs that have\nexactly one implementation (Exercise 11.3); a problem arises when the HLA has multiple\nimplementations. How can we describe the effects of an action that can be implemented in\nmany different ways?\nOne safe answer (at least for problems where all preconditions and goals are positive) is\nto include only the positive effects that are achieved by every implementation of the HLA and\nthe negative effects of any implementation. Then the downward refinement property would\nbe satisfied. Unfortunately, this semantics for HLAs is much too conservative. Consider again\nthe HLA Go(Home, SFO), which has two refinements, and suppose, for the sake of argu-\nment, a simple world in which one can always drive to the airport and park, but taking a taxi\nrequires Cash as a precondition. In that case, Go(Home, SFO) doesn’t always get you to\nthe airport. In particular, it fails if Cash is false, and so we cannot assert At(Agent, SFO) as\nan effect of the HLA. This makes no sense, however; if the agent didn’t have Cash, it would\ndrive itself. Requiring that an effect hold for every implementation is equivalent to assuming\nthat someone else—an adversary—will choose the implementation. It treats the HLA’s mul-\ntiple outcomes exactly as if the HLA were a nondeterministic action, as in Section 4.3. For\nour case, the agent itself will choose the implementation.\nThe programming languages community has coined the term demonic nondetermin-\nism for the case where an adversary makes the choices, contrasting this with angelic nonde-\nDEMONIC\nNONDETERMINISM\nSection 11.2.\nHierarchical Planning\n411\n(a)\n(b)\nFigure 11.6\nSchematic examples of reachable sets. The set of goal states is shaded. Black\nand gray arrows indicate possible implementations of h1 and h2, respectively. (a) The reach-\nable set of an HLA h1 in a state s. (b) The reachable set for the sequence [h1, h2]. Because\nthis intersects the goal set, the sequence achieves the goal.\nterminism, where the agent itself makes the choices. We borrow this term to define angelic\nANGELIC\nNONDETERMINISM\nsemantics for HLA descriptions. The basic concept required for understanding angelic se-\nANGELIC SEMANTICS\nmantics is the reachable set of an HLA: given a state s, the reachable set for an HLA h,\nREACHABLE SET\nwritten as REACH(s, h), is the set of states reachable by any of the HLA’s implementations.\nThe key idea is that the agent can choose which element of the reachable set it ends up in\nwhen it executes the HLA; thus, an HLA with multiple refinements is more “powerful” than\nthe same HLA with fewer refinements. We can also define the reachable set of a sequences of\nHLAs. For example, the reachable set of a sequence [h1, h2] is the union of all the reachable\nsets obtained by applying h2 in each state in the reachable set of h1:\nREACH(s, [h1, h2]) =\n\u000f\ns′∈REACH(s, h1)\nREACH(s′, h2) .\nGiven these definitions, a high-level plan—a sequence of HLAs—achieves the goal if its\nreachable set intersects the set of goal states. (Compare this to the much stronger condition\nfor demonic semantics, where every member of the reachable set has to be a goal state.)\nConversely, if the reachable set doesn’t intersect the goal, then the plan definitely doesn’t\nwork. Figure 11.6 illustrates these ideas.\nThe notion of reachable sets yields a straightforward algorithm: search among high-\nlevel plans, looking for one whose reachable set intersects the goal; once that happens, the\nalgorithm can commit to that abstract plan, knowing that it works, and focus on refining\nthe plan further. We will come back to the algorithmic issues later; first, we consider the\nquestion of how the effects of an HLA—the reachable set for each possible initial state—are\nrepresented. As with the classical action schemas of Chapter 10, we represent the changes\n412\nChapter\n11.\nPlanning and Acting in the Real World\nmade to each fluent. Think of a fluent as a state variable. A primitive action can add or delete\na variable or leave it unchanged. (With conditional effects (see Section 11.3.1) there is a\nfourth possibility: flipping a variable to its opposite.)\nAn HLA under angelic semantics can do more: it can control the value of a variable,\nsetting it to true or false depending on which implementation is chosen. In fact, an HLA can\nhave nine different effects on a variable: if the variable starts out true, it can always keep\nit true, always make it false, or have a choice; if the variable starts out false, it can always\nkeep it false, always make it true, or have a choice; and the three choices for each case can\nbe combined arbitrarily, making nine. Notationally, this is a bit challenging. We’ll use the \u0017\nsymbol to mean “possibly, if the agent so chooses.” Thus, an effect \u0017+A means “possibly add\nA,” that is, either leave A unchanged or make it true. Similarly, \u0017−A means “possibly delete\nA” and \u0017±A means “possibly add or delete A.” For example, the HLA Go(Home, SFO),\nwith the two refinements shown in Figure 11.4, possibly deletes Cash (if the agent decides to\ntake a taxi), so it should have the effect \u0017−Cash. Thus, we see that the descriptions of HLAs\nare derivable, in principle, from the descriptions of their refinements—in fact, this is required\nif we want true HLA descriptions, such that the downward refinement property holds. Now,\nsuppose we have the following schemas for the HLAs h1 and h2:\nAction(h1, PRECOND:¬A, EFFECT:A ∧\u0017−B) ,\nAction(h2, PRECOND:¬B, EFFECT: \u0017+A ∧\u0017±C) .\nThat is, h1 adds A and possible deletes B, while h2 possibly adds A and has full control over\nC. Now, if only B is true in the initial state and the goal is A ∧C then the sequence [h1, h2]\nachieves the goal: we choose an implementation of h1 that makes B false, then choose an\nimplementation of h2 that leaves A true and makes C true.\nThe preceding discussion assumes that the effects of an HLA—the reachable set for\nany given initial state—can be described exactly by describing the effect on each variable. It\nwould be nice if this were always true, but in many cases we can only approximate the ef-\nfects because an HLA may have infinitely many implementations and may produce arbitrarily\nwiggly reachable sets—rather like the wiggly-belief-state problem illustrated in Figure 7.21\non page 271. For example, we said that Go(Home, SFO) possibly deletes Cash; it also\npossibly adds At(Car, SFOLongTermParking); but it cannot do both—in fact, it must do\nexactly one. As with belief states, we may need to write approximate descriptions. We will\nuse two kinds of approximation: an optimistic description REACH+(s,h) of an HLA h may\nOPTIMISTIC\nDESCRIPTION\noverstate the reachable set, while a pessimistic description REACH−(s,h) may understate\nPESSIMISTIC\nDESCRIPTION\nthe reachable set. Thus, we have\nREACH−(s, h) ⊆REACH(s, h) ⊆REACH+(s, h) .\nFor example, an optimistic description of Go(Home, SFO) says that it possible deletes Cash\nand possibly adds At(Car, SFOLongTermParking). Another good example arises in the\n8-puzzle, half of whose states are unreachable from any given state (see Exercise 3.4 on\npage 113): the optimistic description of Act might well include the whole state space, since\nthe exact reachable set is quite wiggly.\nWith approximate descriptions, the test for whether a plan achieves the goal needs to\nbe modified slightly. If the optimistic reachable set for the plan doesn’t intersect the goal,\nSection 11.2.\nHierarchical Planning\n413\n(a)\n(b)\nFigure 11.7\nGoal achievement for high-level plans with approximate descriptions. The\nset of goal states is shaded. For each plan, the pessimistic (solid lines) and optimistic (dashed\nlines) reachable sets are shown. (a) The plan indicated by the black arrow definitely achieves\nthe goal, while the plan indicated by the gray arrow definitely doesn’t. (b) A plan that would\nneed to be refined further to determine if it really does achieve the goal.\nthen the plan doesn’t work; if the pessimistic reachable set intersects the goal, then the plan\ndoes work (Figure 11.7(a)). With exact descriptions, a plan either works or it doesn’t, but\nwith approximate descriptions, there is a middle ground: if the optimistic set intersects the\ngoal but the pessimistic set doesn’t, then we cannot tell if the plan works (Figure 11.7(b)).\nWhen this circumstance arises, the uncertainty can be resolved by refining the plan. This is\na very common situation in human reasoning. For example, in planning the aforementioned\ntwo-week Hawaii vacation, one might propose to spend two days on each of seven islands.\nPrudence would indicate that this ambitious plan needs to be refined by adding details of\ninter-island transportation.\nAn algorithm for hierarchical planning with approximate angelic descriptions is shown\nin Figure 11.8. For simplicity, we have kept to the same overall scheme used previously in\nFigure 11.5, that is, a breadth-first search in the space of refinements. As just explained, the\nalgorithm can detect plans that will and won’t work by checking the intersections of the opti-\nmistic and pessimistic reachable sets with the goal. (The details of how to compute the reach-\nable sets of a plan, given approximate descriptions of each step, are covered in Exercise 11.5.)\nWhen a workable abstract plan is found, the algorithm decomposes the original problem into\nsubproblems, one for each step of the plan. The initial state and goal for each subproblem\nare obtained by regressing a guaranteed-reachable goal state through the action schemas for\neach step of the plan. (See Section 10.2.2 for a discussion of how regression works.) Fig-\nure 11.6(b) illustrates the basic idea: the right-hand circled state is the guaranteed-reachable\ngoal state, and the left-hand circled state is the intermediate goal obtained by regressing the\n414\nChapter\n11.\nPlanning and Acting in the Real World\nfunction ANGELIC-SEARCH(problem,hierarchy,initialPlan) returns solution or fail\nfrontier ←a FIFO queue with initialPlan as the only element\nloop do\nif EMPTY?(frontier) then return fail\nplan ←POP(frontier) /* chooses the shallowest node in frontier */\nif REACH+(problem.INITIAL-STATE, plan) intersects problem.GOAL then\nif plan is primitive then return plan\n/* REACH+ is exact for primitive plans */\nguaranteed ←REACH−(problem.INITIAL-STATE,plan) ∩problem.GOAL\nif guaranteed̸={ } and MAKING-PROGRESS(plan, initialPlan) then\nfinalState ←any element of guaranteed\nreturn DECOMPOSE(hierarchy,problem.INITIAL-STATE,plan,finalState)\nhla ←some HLA in plan\nprefix,suffix ←the action subsequences before and after hla in plan\nfor each sequence in REFINEMENTS(hla,outcome,hierarchy) do\nfrontier ←INSERT(APPEND(prefix,sequence,suffix),frontier)\nfunction DECOMPOSE(hierarchy,s0,plan,sf ) returns a solution\nsolution ←an empty plan\nwhile plan is not empty do\naction ←REMOVE-LAST(plan)\nsi ←a state in REACH−(s0, plan) such that sf ∈REACH−(si,action)\nproblem ←a problem with INITIAL-STATE = si and GOAL = sf\nsolution ←APPEND(ANGELIC-SEARCH(problem,hierarchy,action),solution)\nsf ←si\nreturn solution\nFigure 11.8\nA hierarchical planning algorithm that uses angelic semantics to identify and\ncommit to high-level plans that work while avoiding high-level plans that don’t. The predi-\ncate MAKING-PROGRESS checks to make sure that we aren’t stuck in an infinite regression\nof refinements. At top level, call ANGELIC-SEARCH with [Act] as the initialPlan.\ngoal through the final action.\nThe ability to commit to or reject high-level plans can give ANGELIC-SEARCH a sig-\nnificant computational advantage over HIERARCHICAL-SEARCH, which in turn may have\na large advantage over plain old BREADTH-FIRST-SEARCH. Consider, for example, clean-\ning up a large vacuum world consisting of rectangular rooms connected by narrow corri-\ndors. It makes sense to have an HLA for Navigate (as shown in Figure 11.4) and one for\nCleanWholeRoom. (Cleaning the room could be implemented with the repeated application\nof another HLA to clean each row.) Since there are five actions in this domain, the cost\nfor BREADTH-FIRST-SEARCH grows as 5d, where d is the length of the shortest solution\n(roughly twice the total number of squares); the algorithm cannot manage even two 2 × 2\nrooms. HIERARCHICAL-SEARCH is more efficient, but still suffers from exponential growth\nbecause it tries all ways of cleaning that are consistent with the hierarchy. ANGELIC-SEARCH\nscales approximately linearly in the number of squares—it commits to a good high-level se-\nSection 11.3.\nPlanning and Acting in Nondeterministic Domains\n415\nquence and prunes away the other options. Notice that cleaning a set of rooms by cleaning\neach room in turn is hardly rocket science: it is easy for humans precisely because of the\nhierarchical structure of the task. When we consider how difficult humans find it to solve\nsmall puzzles such as the 8-puzzle, it seems likely that the human capacity for solving com-\nplex problems derives to a great extent from their skill in abstracting and decomposing the\nproblem to eliminate combinatorics.\nThe angelic approach can be extended to find least-cost solutions by generalizing the\nnotion of reachable set. Instead of a state being reachable or not, it has a cost for the most\nefficient way to get there. (The cost is ∞for unreachable states.) The optimistic and pes-\nsimistic descriptions bound these costs. In this way, angelic search can find provably optimal\nabstract plans without considering their implementations. The same approach can be used to\nobtain effective hierarchical lookahead algorithms for online search, in the style of LRTA∗\nHIERARCHICAL\nLOOKAHEAD\n(page 152). In some ways, such algorithms mirror aspects of human deliberation in tasks such\nas planning a vacation to Hawaii—consideration of alternatives is done initially at an abstract\nlevel over long time scales; some parts of the plan are left quite abstract until execution time,\nsuch as how to spend two lazy days on Molokai, while others parts are planned in detail, such\nas the flights to be taken and lodging to be reserved—without these refinements, there is no\nguarantee that the plan would be feasible.\n11.3\nPLANNING AND ACTING IN NONDETERMINISTIC DOMAINS\nIn this section we extend planning to handle partially observable, nondeterministic, and un-\nknown environments. Chapter 4 extended search in similar ways, and the methods here are\nalso similar: sensorless planning (also known as conformant planning) for environments\nwith no observations; contingency planning for partially observable and nondeterministic\nenvironments; and online planning and replanning for unknown environments.\nWhile the basic concepts are the same as in Chapter 4, there are also significant dif-\nferences. These arise because planners deal with factored representations rather than atomic\nrepresentations. This affects the way we represent the agent’s capability for action and obser-\nvation and the way we represent belief states—the sets of possible physical states the agent\nmight be in—for unobservable and partially observable environments. We can also take ad-\nvantage of many of the domain-independent methods given in Chapter 10 for calculating\nsearch heuristics.\nConsider this problem: given a chair and a table, the goal is to have them match—have\nthe same color. In the initial state we have two cans of paint, but the colors of the paint and\nthe furniture are unknown. Only the table is initially in the agent’s field of view:\nInit(Object(Table) ∧Object(Chair) ∧Can(C1) ∧Can(C2) ∧InView(Table))\nGoal(Color(Chair, c) ∧Color(Table, c))\nThere are two actions: removing the lid from a paint can and painting an object using the\npaint from an open can. The action schemas are straightforward, with one exception: we now\nallow preconditions and effects to contain variables that are not part of the action’s variable\n416\nChapter\n11.\nPlanning and Acting in the Real World\nlist. That is, Paint(x, can) does not mention the variable c, representing the color of the\npaint in the can. In the fully observable case, this is not allowed—we would have to name\nthe action Paint(x, can, c). But in the partially observable case, we might or might not\nknow what color is in the can. (The variable c is universally quantified, just like all the other\nvariables in an action schema.)\nAction(RemoveLid(can),\nPRECOND:Can(can)\nEFFECT:Open(can))\nAction(Paint(x, can),\nPRECOND:Object(x) ∧Can(can) ∧Color(can, c) ∧Open(can)\nEFFECT:Color(x, c))\nTo solve a partially observable problem, the agent will have to reason about the percepts it will\nobtain when it is executing the plan. The percept will be supplied by the agent’s sensors when\nit is actually acting, but when it is planning it will need a model of its sensors. In Chapter 4,\nthis model was given by a function, PERCEPT(s). For planning, we augment PDDL with a\nnew type of schema, the percept schema:\nPERCEPT SCHEMA\nPercept(Color(x, c),\nPRECOND:Object(x) ∧InView(x)\nPercept(Color(can, c),\nPRECOND:Can(can) ∧InView(can) ∧Open(can)\nThe first schema says that whenever an object is in view, the agent will perceive the color\nof the object (that is, for the object x, the agent will learn the truth value of Color(x, c) for\nall c). The second schema says that if an open can is in view, then the agent perceives the\ncolor of the paint in the can. Because there are no exogenous events in this world, the color\nof an object will remain the same, even if it is not being perceived, until the agent performs\nan action to change the object’s color. Of course, the agent will need an action that causes\nobjects (one at a time) to come into view:\nAction(LookAt(x),\nPRECOND:InView(y) ∧(x ̸= y)\nEFFECT:InView(x) ∧¬InView(y))\nFor a fully observable environment, we would have a Percept axiom with no preconditions\nfor each fluent. A sensorless agent, on the other hand, has no Percept axioms at all. Note\nthat even a sensorless agent can solve the painting problem. One solution is to open any can\nof paint and apply it to both chair and table, thus coercing them to be the same color (even\nthough the agent doesn’t know what the color is).\nA contingent planning agent with sensors can generate a better plan. First, look at the\ntable and chair to obtain their colors; if they are already the same then the plan is done. If\nnot, look at the paint cans; if the paint in a can is the same color as one piece of furniture,\nthen apply that paint to the other piece. Otherwise, paint both pieces with any color.\nFinally, an online planning agent might generate a contingent plan with fewer branches\nat first—perhaps ignoring the possibility that no cans match any of the furniture—and deal\nSection 11.3.\nPlanning and Acting in Nondeterministic Domains\n417\nwith problems when they arise by replanning. It could also deal with incorrectness of its\naction schemas. Whereas a contingent planner simply assumes that the effects of an action\nalways succeed—that painting the chair does the job—a replanning agent would check the\nresult and make an additional plan to fix any unexpected failure, such as an unpainted area or\nthe original color showing through.\nIn the real world, agents use a combination of approaches. Car manufacturers sell spare\ntires and air bags, which are physical embodiments of contingent plan branches designed\nto handle punctures or crashes. On the other hand, most car drivers never consider these\npossibilities; when a problem arises they respond as replanning agents. In general, agents\nplan only for contingencies that have important consequences and a nonnegligible chance\nof happening. Thus, a car driver contemplating a trip across the Sahara desert should make\nexplicit contingency plans for breakdowns, whereas a trip to the supermarket requires less\nadvance planning. We next look at each of the three approaches in more detail.\n11.3.1\nSensorless planning\nSection 4.4.1 (page 138) introduced the basic idea of searching in belief-state space to find\na solution for sensorless problems. Conversion of a sensorless planning problem to a belief-\nstate planning problem works much the same way as it did in Section 4.4.1; the main differ-\nences are that the underlying physical transition model is represented by a collection of action\nschemas and the belief state can be represented by a logical formula instead of an explicitly\nenumerated set of states. For simplicity, we assume that the underlying planning problem is\ndeterministic.\nThe initial belief state for the sensorless painting problem can ignore InView fluents\nbecause the agent has no sensors.\nFurthermore, we take as given the unchanging facts\nObject(Table) ∧Object(Chair) ∧Can(C1) ∧Can(C2) because these hold in every be-\nlief state. The agent doesn’t know the colors of the cans or the objects, or whether the cans\nare open or closed, but it does know that objects and cans have colors: ∀x ∃c Color(x, c).\nAfter Skolemizing, (see Section 9.5), we obtain the initial belief state:\nb0 = Color(x, C(x)) .\nIn classical planning, where the closed-world assumption is made, we would assume that\nany fluent not mentioned in a state is false, but in sensorless (and partially observable) plan-\nning we have to switch to an open-world assumption in which states contain both positive\nand negative fluents, and if a fluent does not appear, its value is unknown. Thus, the belief\nstate corresponds exactly to the set of possible worlds that satisfy the formula. Given this\ninitial belief state, the following action sequence is a solution:\n[RemoveLid(Can1), Paint(Chair, Can1), Paint(Table, Can1)] .\nWe now show how to progress the belief state through the action sequence to show that the\nfinal belief state satisfies the goal.\nFirst, note that in a given belief state b, the agent can consider any action whose pre-\nconditions are satisfied by b. (The other actions cannot be used because the transition model\ndoesn’t define the effects of actions whose preconditions might be unsatisfied.) According\n418\nChapter\n11.\nPlanning and Acting in the Real World\nto Equation (4.4) (page 139), the general formula for updating the belief state b given an\napplicable action a in a deterministic world is as follows:\nb′ = RESULT(b, a) = {s′ : s′ = RESULTP (s, a) and s ∈b}\nwhere RESULTP defines the physical transition model. For the time being, we assume that the\ninitial belief state is always a conjunction of literals, that is, a 1-CNF formula. To construct\nthe new belief state b′, we must consider what happens to each literal ℓin each physical state\ns in b when action a is applied. For literals whose truth value is already known in b, the truth\nvalue in b′ is computed from the current value and the add list and delete list of the action.\n(For example, if ℓis in the delete list of the action, then ¬ℓis added to b′.) What about a\nliteral whose truth value is unknown in b? There are three cases:\n1. If the action adds ℓ, then ℓwill be true in b′ regardless of its initial value.\n2. If the action deletes ℓ, then ℓwill be false in b′ regardless of its initial value.\n3. If the action does not affect ℓ, then ℓwill retain its initial value (which is unknown) and\nwill not appear in b′.\nHence, we see that the calculation of b′ is almost identical to the observable case, which was\nspecified by Equation (10.1) on page 368:\nb′ = RESULT(b, a) = (b −DEL(a)) ∪ADD(a) .\nWe cannot quite use the set semantics because (1) we must make sure that b′ does not con-\ntain both ℓand ¬ℓ, and (2) atoms may contain unbound variables. But it is still the case\nthat RESULT(b, a) is computed by starting with b, setting any atom that appears in DEL(a)\nto false, and setting any atom that appears in ADD(a) to true. For example, if we apply\nRemoveLid(Can1) to the initial belief state b0, we get\nb1 = Color(x, C(x)) ∧Open(Can1) .\nWhen we apply the action Paint(Chair, Can1), the precondition Color(Can1, c) is satisfied\nby the known literal Color(x, C(x)) with binding {x/Can1, c/C(Can1)} and the new belief\nstate is\nb2 = Color(x, C(x)) ∧Open(Can1) ∧Color(Chair, C(Can1)) .\nFinally, we apply the action Paint(Table, Can1) to obtain\nb3 = Color(x, C(x)) ∧Open(Can1) ∧Color(Chair, C(Can1))\n∧Color(Table, C(Can1)) .\nThe final belief state satisfies the goal, Color(Table, c) ∧Color(Chair, c), with the variable\nc bound to C(Can1).\nThe preceding analysis of the update rule has shown a very important fact: the family\nof belief states defined as conjunctions of literals is closed under updates defined by PDDL\naction schemas. That is, if the belief state starts as a conjunction of literals, then any update\nwill yield a conjunction of literals. That means that in a world with n fluents, any belief\nstate can be represented by a conjunction of size O(n). This is a very comforting result,\nconsidering that there are 2n states in the world. It says we can compactly represent all the\nsubsets of those 2n states that we will ever need. Moreover, the process of checking for belief\nSection 11.3.\nPlanning and Acting in Nondeterministic Domains\n419\nstates that are subsets or supersets of previously visited belief states is also easy, at least in\nthe propositional case.\nThe fly in the ointment of this pleasant picture is that it only works for action schemas\nthat have the same effects for all states in which their preconditions are satisfied. It is this\nproperty that enables the preservation of the 1-CNF belief-state representation. As soon as the\neffect can depend on the state, dependencies are introduced between fluents and the 1-CNF\nproperty is lost. Consider, for example, the simple vacuum world defined in Section 3.2.1.\nLet the fluents be AtL and AtR for the location of the robot and CleanL and CleanR for\nthe state of the squares. According to the definition of the problem, the Suck action has no\nprecondition—it can always be done. The difficulty is that its effect depends on the robot’s lo-\ncation: when the robot is AtL, the result is CleanL, but when it is AtR, the result is CleanR.\nFor such actions, our action schemas will need something new: a conditional effect. These\nCONDITIONAL\nEFFECT\nhave the syntax “when condition: effect,” where condition is a logical formula to be com-\npared against the current state, and effect is a formula describing the resulting state. For the\nvacuum world, we have\nAction(Suck,\nEFFECT:when AtL: CleanL ∧when AtR: CleanR) .\nWhen applied to the initial belief state True, the resulting belief state is (AtL ∧CleanL) ∨\n(AtR ∧CleanR), which is no longer in 1-CNF. (This transition can be seen in Figure 4.14\non page 141.) In general, conditional effects can induce arbitrary dependencies among the\nfluents in a belief state, leading to belief states of exponential size in the worst case.\nIt is important to understand the difference between preconditions and conditional ef-\nfects. All conditional effects whose conditions are satisfied have their effects applied to gener-\nate the resulting state; if none are satisfied, then the resulting state is unchanged. On the other\nhand, if a precondition is unsatisfied, then the action is inapplicable and the resulting state\nis undefined. From the point of view of sensorless planning, it is better to have conditional\neffects than an inapplicable action. For example, we could split Suck into two actions with\nunconditional effects as follows:\nAction(SuckL,\nPRECOND:AtL; EFFECT:CleanL)\nAction(SuckR,\nPRECOND:AtR; EFFECT:CleanR) .\nNow we have only unconditional schemas, so the belief states all remain in 1-CNF; unfortu-\nnately, we cannot determine the applicability of SuckL and SuckR in the initial belief state.\nIt seems inevitable, then, that nontrivial problems will involve wiggly belief states, just\nlike those encountered when we considered the problem of state estimation for the wumpus\nworld (see Figure 7.21 on page 271). The solution suggested then was to use a conservative\napproximation to the exact belief state; for example, the belief state can remain in 1-CNF\nif it contains all literals whose truth values can be determined and treats all other literals as\nunknown. While this approach is sound, in that it never generates an incorrect plan, it is\nincomplete because it may be unable to find solutions to problems that necessarily involve\ninteractions among literals. To give a trivial example, if the goal is for the robot to be on\n420\nChapter\n11.\nPlanning and Acting in the Real World\na clean square, then [Suck] is a solution but a sensorless agent that insists on 1-CNF belief\nstates will not find it.\nPerhaps a better solution is to look for action sequences that keep the belief state\nas simple as possible. For example, in the sensorless vacuum world, the action sequence\n[Right, Suck, Left, Suck] generates the following sequence of belief states:\nb0 = True\nb1 = AtR\nb2 = AtR ∧CleanR\nb3 = AtL ∧CleanR\nb4 = AtL ∧CleanR ∧CleanL\nThat is, the agent can solve the problem while retaining a 1-CNF belief state, even though\nsome sequences (e.g., those beginning with Suck) go outside 1-CNF. The general lesson is\nnot lost on humans: we are always performing little actions (checking the time, patting our\npockets to make sure we have the car keys, reading street signs as we navigate through a city)\nto eliminate uncertainty and keep our belief state manageable.\nThere is another, quite different approach to the problem of unmanageably wiggly be-\nlief states: don’t bother computing them at all. Suppose the initial belief state is b0 and we\nwould like to know the belief state resulting from the action sequence [a1, . . . , am]. Instead\nof computing it explicitly, just represent it as “b0 then [a1, . . . , am].” This is a lazy but un-\nambiguous representation of the belief state, and it’s quite concise—O(n + m) where n is\nthe size of the initial belief state (assumed to be in 1-CNF) and m is the maximum length\nof an action sequence. As a belief-state representation, it suffers from one drawback, how-\never: determining whether the goal is satisfied, or an action is applicable, may require a lot\nof computation.\nThe computation can be implemented as an entailment test: if Am represents the collec-\ntion of successor-state axioms required to define occurrences of the actions a1, . . . , am—as\nexplained for SATPLAN in Section 10.4.1—and Gm asserts that the goal is true after m steps,\nthen the plan achieves the goal if b0 ∧Am |= Gm, that is, if b0 ∧Am ∧¬Gm is unsatisfiable.\nGiven a modern SAT solver, it may be possible to do this much more quickly than computing\nthe full belief state. For example, if none of the actions in the sequence has a particular goal\nfluent in its add list, the solver will detect this immediately. It also helps if partial results\nabout the belief state—for example, fluents known to be true or false—are cached to simplify\nsubsequent computations.\nThe final piece of the sensorless planning puzzle is a heuristic function to guide the\nsearch. The meaning of the heuristic function is the same as for classical planning: an esti-\nmate (perhaps admissible) of the cost of achieving the goal from the given belief state. With\nbelief states, we have one additional fact: solving any subset of a belief state is necessarily\neasier than solving the belief state:\nif b1 ⊆b2 then h∗(b1) ≤h∗(b2) .\nHence, any admissible heuristic computed for a subset is admissible for the belief state itself.\nThe most obvious candidates are the singleton subsets, that is, individual physical states. We\nSection 11.3.\nPlanning and Acting in Nondeterministic Domains\n421\ncan take any random collection of states s1, . . . , sN that are in the belief state b, apply any\nadmissible heuristic h from Chapter 10, and return\nH(b) = max{h(s1), . . . , h(sN)}\nas the heuristic estimate for solving b. We could also use a planning graph directly on b itself:\nif it is a conjunction of literals (1-CNF), simply set those literals to be the initial state layer\nof the graph. If b is not in 1-CNF, it may be possible to find sets of literals that together entail\nb. For example, if b is in disjunctive normal form (DNF), each term of the DNF formula is\na conjunction of literals that entails b and can form the initial layer of a planning graph. As\nbefore, we can take the maximum of the heuristics obtained from each set of literals. We can\nalso use inadmissible heuristics such as the ignore-delete-lists heuristic (page 377), which\nseems to work quite well in practice.\n11.3.2\nContingent planning\nWe saw in Chapter 4 that contingent planning—the generation of plans with conditional\nbranching based on percepts—is appropriate for environments with partial observability, non-\ndeterminism, or both. For the partially observable painting problem with the percept axioms\ngiven earlier, one possible contingent solution is as follows:\n[LookAt(Table), LookAt(Chair),\nif Color(Table, c) ∧Color(Chair, c) then NoOp\nelse [RemoveLid(Can1), LookAt(Can1), RemoveLid(Can2), LookAt(Can2),\nif Color(Table, c) ∧Color(can, c) then Paint(Chair, can)\nelse if Color(Chair, c) ∧Color(can, c) then Paint(Table, can)\nelse [Paint(Chair, Can1), Paint(Table, Can1)]]]\nVariables in this plan should be considered existentially quantified; the second line says\nthat if there exists some color c that is the color of the table and the chair, then the agent\nneed not do anything to achieve the goal. When executing this plan, a contingent-planning\nagent can maintain its belief state as a logical formula and evaluate each branch condition\nby determining if the belief state entails the condition formula or its negation. (It is up to\nthe contingent-planning algorithm to make sure that the agent will never end up in a be-\nlief state where the condition formula’s truth value is unknown.) Note that with first-order\nconditions, the formula may be satisfied in more than one way; for example, the condition\nColor(Table, c) ∧Color(can, c) might be satisfied by {can/Can1} and by {can/Can2} if\nboth cans are the same color as the table. In that case, the agent can choose any satisfying\nsubstitution to apply to the rest of the plan.\nAs shown in Section 4.4.2, calculating the new belief state after an action and subse-\nquent percept is done in two stages. The first stage calculates the belief state after the action,\njust as for the sensorless agent:\nˆb = (b −DEL(a)) ∪ADD(a)\nwhere, as before, we have assumed a belief state represented as a conjunction of literals. The\nsecond stage is a little trickier. Suppose that percept literals p1, . . . , pk are received. One\nmight think that we simply need to add these into the belief state; in fact, we can also infer\n422\nChapter\n11.\nPlanning and Acting in the Real World\nthat the preconditions for sensing are satisfied. Now, if a percept p has exactly one percept\naxiom, Percept(p, PRECOND:c), where c is a conjunction of literals, then those literals can\nbe thrown into the belief state along with p. On the other hand, if p has more than one percept\naxiom whose preconditions might hold according to the predicted belief state ˆb, then we have\nto add in the disjunction of the preconditions. Obviously, this takes the belief state outside\n1-CNF and brings up the same complications as conditional effects, with much the same\nclasses of solutions.\nGiven a mechanism for computing exact or approximate belief states, we can generate\ncontingent plans with an extension of the AND–OR forward search over belief states used\nin Section 4.4. Actions with nondeterministic effects—which are defined simply by using a\ndisjunction in the EFFECT of the action schema—can be accommodated with minor changes\nto the belief-state update calculation and no change to the search algorithm.2 For the heuristic\nfunction, many of the methods suggested for sensorless planning are also applicable in the\npartially observable, nondeterministic case.\n11.3.3\nOnline replanning\nImagine watching a spot-welding robot in a car plant. The robot’s fast, accurate motions are\nrepeated over and over again as each car passes down the line. Although technically im-\npressive, the robot probably does not seem at all intelligent because the motion is a fixed,\npreprogrammed sequence; the robot obviously doesn’t “know what it’s doing” in any mean-\ningful sense. Now suppose that a poorly attached door falls off the car just as the robot is\nabout to apply a spot-weld. The robot quickly replaces its welding actuator with a gripper,\npicks up the door, checks it for scratches, reattaches it to the car, sends an email to the floor\nsupervisor, switches back to the welding actuator, and resumes its work. All of a sudden,\nthe robot’s behavior seems purposive rather than rote; we assume it results not from a vast,\nprecomputed contingent plan but from an online replanning process—which means that the\nrobot does need to know what it’s trying to do.\nReplanning presupposes some form of execution monitoring to determine the need for\nEXECUTION\nMONITORING\na new plan. One such need arises when a contingent planning agent gets tired of planning\nfor every little contingency, such as whether the sky might fall on its head.3 Some branches\nof a partially constructed contingent plan can simply say Replan; if such a branch is reached\nduring execution, the agent reverts to planning mode. As we mentioned earlier, the decision\nas to how much of the problem to solve in advance and how much to leave to replanning\nis one that involves tradeoffs among possible events with different costs and probabilities of\noccurring. Nobody wants to have their car break down in the middle of the Sahara desert and\nonly then think about having enough water.\n2 If cyclic solutions are required for a nondeterministic problem, AND–OR search must be generalized to a loopy\nversion such as LAO∗(Hansen and Zilberstein, 2001).\n3 In 1954, a Mrs. Hodges of Alabama was hit by meteorite that crashed through her roof. In 1992, a piece of\nthe Mbale meteorite hit a small boy on the head; fortunately, its descent was slowed by banana leaves (Jenniskens\net al., 1994). And in 2009, a German boy claimed to have been hit in the hand by a pea-sized meteorite. No serious\ninjuries resulted from any of these incidents, suggesting that the need for preplanning against such contingencies\nis sometimes overstated.\nSection 11.3.\nPlanning and Acting in Nondeterministic Domains\n423\nwhole plan\nplan\nrepair\nS\nP\nO\nE\nG\ncontinuation\nFigure 11.9\nBefore execution, the planner comes up with a plan, here called whole plan,\nto get from S to G. The agent executes steps of the plan until it expects to be in state E, but\nobserves it is actually in O. The agent then replans for the minimal repair plus continuation\nto reach G.\nReplanning may also be needed if the agent’s model of the world is incorrect. The model\nfor an action may have a missing precondition—for example, the agent may not know that\nMISSING\nPRECONDITION\nremoving the lid of a paint can often requires a screwdriver; the model may have a missing\neffect—for example, painting an object may get paint on the floor as well; or the model may\nMISSING EFFECT\nhave a missing state variable—for example, the model given earlier has no notion of the\nMISSING STATE\nVARIABLE\namount of paint in a can, of how its actions affect this amount, or of the need for the amount\nto be nonzero. The model may also lack provision for exogenous events such as someone\nEXOGENOUS EVENT\nknocking over the paint can. Exogenous events can also include changes in the goal, such\nas the addition of the requirement that the table and chair not be painted black. Without the\nability to monitor and replan, an agent’s behavior is likely to be extremely fragile if it relies\non absolute correctness of its model.\nThe online agent has a choice of how carefully to monitor the environment. We distin-\nguish three levels:\n• Action monitoring: before executing an action, the agent verifies that all the precondi-\nACTION MONITORING\ntions still hold.\n• Plan monitoring: before executing an action, the agent verifies that the remaining plan\nPLAN MONITORING\nwill still succeed.\n• Goal monitoring: before executing an action, the agent checks to see if there is a better\nGOAL MONITORING\nset of goals it could be trying to achieve.\nIn Figure 11.9 we see a schematic of action monitoring. The agent keeps track of both its\noriginal plan, wholeplan, and the part of the plan that has not been executed yet, which is\ndenoted by plan. After executing the first few steps of the plan, the agent expects to be in\nstate E. But the agent observes it is actually in state O. It then needs to repair the plan by\nfinding some point P on the original plan that it can get back to. (It may be that P is the goal\nstate, G.) The agent tries to minimize the total cost of the plan: the repair part (from O to P)\nplus the continuation (from P to G).\n424\nChapter\n11.\nPlanning and Acting in the Real World\nNow let’s return to the example problem of achieving a chair and table of matching\ncolor. Suppose the agent comes up with this plan:\n[LookAt(Table), LookAt(Chair),\nif Color(Table, c) ∧Color(Chair, c) then NoOp\nelse [RemoveLid(Can1), LookAt(Can1),\nif Color(Table, c) ∧Color(Can1, c) then Paint(Chair, Can1)\nelse REPLAN]] .\nNow the agent is ready to execute the plan. Suppose the agent observes that the table and\ncan of paint are white and the chair is black. It then executes Paint(Chair, Can1). At this\npoint a classical planner would declare victory; the plan has been executed. But an online\nexecution monitoring agent needs to check the preconditions of the remaining empty plan—\nthat the table and chair are the same color. Suppose the agent perceives that they do not\nhave the same color—in fact, the chair is now a mottled gray because the black paint is\nshowing through. The agent then needs to figure out a position in whole plan to aim for\nand a repair action sequence to get there. The agent notices that the current state is identical\nto the precondition before the Paint(Chair, Can1) action, so the agent chooses the empty\nsequence for repair and makes its plan be the same [Paint] sequence that it just attempted.\nWith this new plan in place, execution monitoring resumes, and the Paint action is retried.\nThis behavior will loop until the chair is perceived to be completely painted. But notice that\nthe loop is created by a process of plan–execute–replan, rather than by an explicit loop in a\nplan. Note also that the original plan need not cover every contingency. If the agent reaches\nthe step marked REPLAN, it can then generate a new plan (perhaps involving Can2).\nAction monitoring is a simple method of execution monitoring, but it can sometimes\nlead to less than intelligent behavior. For example, suppose there is no black or white paint,\nand the agent constructs a plan to solve the painting problem by painting both the chair and\ntable red. Suppose that there is only enough red paint for the chair. With action monitoring,\nthe agent would go ahead and paint the chair red, then notice that it is out of paint and cannot\npaint the table, at which point it would replan a repair—perhaps painting both chair and table\ngreen. A plan-monitoring agent can detect failure whenever the current state is such that the\nremaining plan no longer works. Thus, it would not waste time painting the chair red. Plan\nmonitoring achieves this by checking the preconditions for success of the entire remaining\nplan—that is, the preconditions of each step in the plan, except those preconditions that are\nachieved by another step in the remaining plan. Plan monitoring cuts off execution of a\ndoomed plan as soon as possible, rather than continuing until the failure actually occurs.4\nPlan monitoring also allows for serendipity—accidental success. If someone comes along\nand paints the table red at the same time that the agent is painting the chair red, then the final\nplan preconditions are satisfied (the goal has been achieved), and the agent can go home early.\nIt is straightforward to modify a planning algorithm so that each action in the plan\nis annotated with the action’s preconditions, thus enabling action monitoring. It is slightly\n4 Plan monitoring means that finally, after 424 pages, we have an agent that is smarter than a dung beetle (see\npage 39). A plan-monitoring agent would notice that the dung ball was missing from its grasp and would replan\nto get another ball and plug its hole.\nSection 11.4.\nMultiagent Planning\n425\nmore complex to enable plan monitoring. Partial-order and planning-graph planners have\nthe advantage that they have already built up structures that contain the relations necessary\nfor plan monitoring. Augmenting state-space planners with the necessary annotations can be\ndone by careful bookkeeping as the goal fluents are regressed through the plan.\nNow that we have described a method for monitoring and replanning, we need to ask,\n“Does it work?” This is a surprisingly tricky question. If we mean, “Can we guarantee that\nthe agent will always achieve the goal?” then the answer is no, because the agent could\ninadvertently arrive at a dead end from which there is no repair. For example, the vacuum\nagent might have a faulty model of itself and not know that its batteries can run out. Once\nthey do, it cannot repair any plans. If we rule out dead ends—assume that there exists a plan\nto reach the goal from any state in the environment—and assume that the environment is\nreally nondeterministic, in the sense that such a plan always has some chance of success on\nany given execution attempt, then the agent will eventually reach the goal.\nTrouble occurs when an action is actually not nondeterministic, but rather depends on\nsome precondition that the agent does not know about. For example, sometimes a paint\ncan may be empty, so painting from that can has no effect. No amount of retrying is going to\nchange this.5 One solution is to choose randomly from among the set of possible repair plans,\nrather than to try the same one each time. In this case, the repair plan of opening another can\nmight work. A better approach is to learn a better model. Every prediction failure is an\nopportunity for learning; an agent should be able to modify its model of the world to accord\nwith its percepts. From then on, the replanner will be able to come up with a repair that gets\nat the root problem, rather than relying on luck to choose a good repair. This kind of learning\nis described in Chapters 18 and 19.\n11.4\nMULTIAGENT PLANNING\nSo far, we have assumed that only one agent is doing the sensing, planning, and acting.\nWhen there are multiple agents in the environment, each agent faces a multiagent planning\nproblem in which it tries to achieve its own goals with the help or hindrance of others.\nMULTIAGENT\nPLANNING PROBLEM\nBetween the purely single-agent and truly multiagent cases is a wide spectrum of prob-\nlems that exhibit various degrees of decomposition of the monolithic agent. An agent with\nmultiple effectors that can operate concurrently—for example, a human who can type and\nspeak at the same time—needs to do multieffector planning to manage each effector while\nMULTIEFFECTOR\nPLANNING\nhandling positive and negative interactions among the effectors.\nWhen the effectors are\nphysically decoupled into detached units—as in a fleet of delivery robots in a factory—\nmultieffector planning becomes multibody planning. A multibody problem is still a “stan-\nMULTIBODY\nPLANNING\ndard” single-agent problem as long as the relevant sensor information collected by each body\ncan be pooled—either centrally or within each body—to form a common estimate of the\nworld state that then informs the execution of the overall plan; in this case, the multiple bod-\nies act as a single body. When communication constraints make this impossible, we have\n5 Futile repetition of a plan repair is exactly the behavior exhibited by the sphex wasp (page 39).\n426\nChapter\n11.\nPlanning and Acting in the Real World\nwhat is sometimes called a decentralized planning problem; this is perhaps a misnomer, be-\nDECENTRALIZED\nPLANNING\ncause the planning phase is centralized but the execution phase is at least partially decoupled.\nIn this case, the subplan constructed for each body may need to include explicit communica-\ntive actions with other bodies. For example, multiple reconnaissance robots covering a wide\narea may often be out of radio contact with each other and should share their findings during\ntimes when communication is feasible.\nWhen a single entity is doing the planning, there is really only one goal, which all the\nbodies necessarily share. When the bodies are distinct agents that do their own planning, they\nmay still share identical goals; for example, two human tennis players who form a doubles\nteam share the goal of winning the match. Even with shared goals, however, the multibody\nand multiagent cases are quite different. In a multibody robotic doubles team, a single plan\ndictates which body will go where on the court and which body will hit the ball. In a multi-\nagent doubles team, on the other hand, each agent decides what to do; without some method\nfor coordination, both agents may decide to cover the same part of the court and each may\nCOORDINATION\nleave the ball for the other to hit.\nThe clearest case of a multiagent problem, of course, is when the agents have different\ngoals. In tennis, the goals of two opposing teams are in direct conflict, leading to the zero-\nsum situation of Chapter 5. Spectators could be viewed as agents if their support or disdain\nis a significant factor and can be influenced by the players’ conduct; otherwise, they can be\ntreated as an aspect of nature—just like the weather—that is assumed to be indifferent to the\nplayers’ intentions.6\nFinally, some systems are a mixture of centralized and multiagent planning. For ex-\nample, a delivery company may do centralized, offline planning for the routes of its trucks\nand planes each day, but leave some aspects open for autonomous decisions by drivers and\npilots who can respond individually to traffic and weather situations. Also, the goals of the\ncompany and its employees are brought into alignment, to some extent, by the payment of\nincentives (salaries and bonuses)—a sure sign that this is a true multiagent system.\nINCENTIVE\nThe issues involved in multiagent planning can be divided roughly into two sets. The\nfirst, covered in Section 11.4.1, involves issues of representing and planning for multiple\nsimultaneous actions; these issues occur in all settings from multieffector to multiagent plan-\nning. The second, covered in Section 11.4.2, involves issues of cooperation, coordination,\nand competition arising in true multiagent settings.\n11.4.1\nPlanning with multiple simultaneous actions\nFor the time being, we will treat the multieffector, multibody, and multiagent settings in the\nsame way, labeling them generically as multiactor settings, using the generic term actor to\nMULTIACTOR\nACTOR\ncover effectors, bodies, and agents. The goal of this section is to work out how to define\ntransition models, correct plans, and efficient planning algorithms for the multiactor setting.\nA correct plan is one that, if executed by the actors, achieves the goal. (In the true multiagent\nsetting, of course, the agents may not agree to execute any particular plan, but at least they\n6 We apologize to residents of the United Kingdom, where the mere act of contemplating a game of tennis\nguarantees rain.\nSection 11.4.\nMultiagent Planning\n427\nActors(A, B)\nInit(At(A, LeftBaseline) ∧At(B, RightNet) ∧\nApproaching(Ball, RightBaseline)) ∧Partner(A, B) ∧Partner(B, A)\nGoal(Returned(Ball) ∧(At(a, RightNet) ∨At(a, LeftNet))\nAction(Hit(actor, Ball),\nPRECOND:Approaching(Ball, loc) ∧At(actor, loc)\nEFFECT:Returned(Ball))\nAction(Go(actor, to),\nPRECOND:At(actor,loc) ∧to ̸= loc,\nEFFECT:At(actor, to) ∧¬ At(actor, loc))\nFigure 11.10\nThe doubles tennis problem. Two actors A and B are playing together and\ncan be in one of four locations: LeftBaseline, RightBaseline, LeftNet, and RightNet. The\nball can be returned only if a player is in the right place. Note that each action must include\nthe actor as an argument.\nwill know what plans would work if they did agree to execute them.) For simplicity, we\nassume perfect synchronization: each action takes the same amount of time and actions at\nSYNCHRONIZATION\neach point in the joint plan are simultaneous.\nWe begin with the transition model; for the deterministic case, this is the function\nRESULT(s, a). In the single-agent setting, there might be b different choices for the action;\nb can be quite large, especially for first-order representations with many objects to act on,\nbut action schemas provide a concise representation nonetheless. In the multiactor setting\nwith n actors, the single action a is replaced by a joint action ⟨a1, . . . , an⟩, where ai is the\nJOINT ACTION\naction taken by the ith actor. Immediately, we see two problems: first, we have to describe\nthe transition model for bn different joint actions; second, we have a joint planning problem\nwith a branching factor of bn.\nHaving put the actors together into a multiactor system with a huge branching factor,\nthe principal focus of research on multiactor planning has been to decouple the actors to\nthe extent possible, so that the complexity of the problem grows linearly with n rather than\nexponentially. If the actors have no interaction with one another—for example, n actors each\nplaying a game of solitaire—then we can simply solve n separate problems. If the actors are\nloosely coupled, can we attain something close to this exponential improvement? This is, of\nLOOSELY COUPLED\ncourse, a central question in many areas of AI. We have seen it explicitly in the context of\nCSPs, where “tree like” constraint graphs yielded efficient solution methods (see page 225),\nas well as in the context of disjoint pattern databases (page 106) and additive heuristics for\nplanning (page 378).\nThe standard approach to loosely coupled problems is to pretend the problems are com-\npletely decoupled and then fix up the interactions. For the transition model, this means writing\naction schemas as if the actors acted independently. Let’s see how this works for the doubles\ntennis problem. Let’s suppose that at one point in the game, the team has the goal of returning\nthe ball that has been hit to them and ensuring that at least one of them is covering the net.\n428\nChapter\n11.\nPlanning and Acting in the Real World\nA first pass at a multiactor definition might look like Figure 11.10. With this definition, it is\neasy to see that the following joint plan plan works:\nJOINT PLAN\nPLAN 1:\nA : [Go(A, RightBaseline), Hit(A, Ball)]\nB : [NoOp(B), NoOp(B)] .\nProblems arise, however, when a plan has both agents hitting the ball at the same time. In the\nreal world, this won’t work, but the action schema for Hit says that the ball will be returned\nsuccessfully. Technically, the difficulty is that preconditions constrain the state in which an\naction can be executed successfully, but do not constrain other actions that might mess it up.\nWe solve this by augmenting action schemas with one new feature: a concurrent action list\nCONCURRENT\nACTION LIST\nstating which actions must or must not be executed concurrently. For example, the Hit action\ncould be described as follows:\nAction(Hit(a, Ball),\nCONCURRENT:b ̸= a ⇒¬Hit(b, Ball)\nPRECOND:Approaching(Ball, loc) ∧At(a, loc)\nEFFECT:Returned(Ball)) .\nIn other words, the Hit action has its stated effect only if no other Hit action by another\nagent occurs at the same time. (In the SATPLAN approach, this would be handled by a\npartial action exclusion axiom.) For some actions, the desired effect is achieved only when\nanother action occurs concurrently. For example, two agents are needed to carry a cooler full\nof beverages to the tennis court:\nAction(Carry(a, cooler, here, there),\nCONCURRENT:b ̸= a ∧Carry(b, cooler, here, there)\nPRECOND:At(a, here) ∧At(cooler, here) ∧Cooler(cooler)\nEFFECT:At(a, there) ∧At(cooler, there) ∧¬At(a, here) ∧¬At(cooler, here)).\nWith these kinds of action schemas, any of the planning algorithms described in Chapter 10\ncan be adapted with only minor modifications to generate multiactor plans. To the extent that\nthe coupling among subplans is loose—meaning that concurrency constraints come into play\nonly rarely during plan search—one would expect the various heuristics derived for single-\nagent planning to also be effective in the multiactor context. We could extend this approach\nwith the refinements of the last two chapters—HTNs, partial observability, conditionals, exe-\ncution monitoring, and replanning—but that is beyond the scope of this book.\n11.4.2\nPlanning with multiple agents: Cooperation and coordination\nNow let us consider the true multiagent setting in which each agent makes its own plan. To\nstart with, let us assume that the goals and knowledge base are shared. One might think\nthat this reduces to the multibody case—each agent simply computes the joint solution and\nexecutes its own part of that solution. Alas, the “the” in “the joint solution” is misleading.\nFor our doubles team, more than one joint solution exists:\nPLAN 2:\nA : [Go(A, LeftNet), NoOp(A)]\nB : [Go(B, RightBaseline), Hit(B, Ball)] .\nSection 11.4.\nMultiagent Planning\n429\nIf both agents can agree on either plan 1 or plan 2, the goal will be achieved. But if A chooses\nplan 2 and B chooses plan 1, then nobody will return the ball. Conversely, if A chooses 1 and\nB chooses 2, then they will both try to hit the ball. The agents may realize this, but how can\nthey coordinate to make sure they agree on the plan?\nOne option is to adopt a convention before engaging in joint activity. A convention is\nCONVENTION\nany constraint on the selection of joint plans. For example, the convention “stick to your side\nof the court” would rule out plan 1, causing the doubles partners to select plan 2. Drivers on\na road face the problem of not colliding with each other; this is (partially) solved by adopting\nthe convention “stay on the right side of the road” in most countries; the alternative, “stay\non the left side,” works equally well as long as all agents in an environment agree. Similar\nconsiderations apply to the development of human language, where the important thing is not\nwhich language each individual should speak, but the fact that a community all speaks the\nsame language. When conventions are widespread, they are called social laws.\nSOCIAL LAWS\nIn the absence of a convention, agents can use communication to achieve common\nknowledge of a feasible joint plan. For example, a tennis player could shout “Mine!” or\n“Yours!” to indicate a preferred joint plan. We cover mechanisms for communication in more\ndepth in Chapter 22, where we observe that communication does not necessarily involve a\nverbal exchange. For example, one player can communicate a preferred joint plan to the other\nsimply by executing the first part of it. If agent A heads for the net, then agent B is obliged\nto go back to the baseline to hit the ball, because plan 2 is the only joint plan that begins with\nA’s heading for the net. This approach to coordination, sometimes called plan recognition,\nPLAN RECOGNITION\nworks when a single action (or short sequence of actions) is enough to determine a joint plan\nunambiguously. Note that communication can work as well with competitive agents as with\ncooperative ones.\nConventions can also arise through evolutionary processes. For example, seed-eating\nharvester ants are social creatures that evolved from the less social wasps. Colonies of ants\nexecute very elaborate joint plans without any centralized control—the queen’s job is to re-\nproduce, not to do centralized planning—and with very limited computation, communica-\ntion, and memory capabilities in each ant (Gordon, 2000, 2007). The colony has many roles,\nincluding interior workers, patrollers, and foragers. Each ant chooses to perform a role ac-\ncording to the local conditions it observes. For example, foragers travel away from the nest,\nsearch for a seed, and when they find one, bring it back immediately. Thus, the rate at which\nforagers return to the nest is an approximation of the availability of food today. When the\nrate is high, other ants abandon their current role and take on the role of scavenger. The ants\nappear to have a convention on the importance of roles—foraging is the most important—and\nants will easily switch into the more important roles, but not into the less important. There is\nsome learning mechanism: a colony learns to make more successful and prudent actions over\nthe course of its decades-long life, even though individual ants live only about a year.\nOne final example of cooperative multiagent behavior appears in the flocking behavior\nof birds. We can obtain a reasonable simulation of a flock if each bird agent (sometimes\ncalled a boid) observes the positions of its nearest neighbors and then chooses the heading\nBOID\nand acceleration that maximizes the weighted sum of these three components:\n430\nChapter\n11.\nPlanning and Acting in the Real World\n(a)\n(b)\n(c)\nFigure 11.11\n(a) A simulated flock of birds, using Reynold’s boids model. Image courtesy\nGiuseppe Randazzo, novastructura.net. (b) An actual flock of starlings. Image by Eduardo\n(pastaboy sleeps on flickr). (c) Two competitive teams of agents attempting to capture the\ntowers in the NERO game. Image courtesy Risto Miikkulainen.\n1. Cohesion: a positive score for getting closer to the average position of the neighbors\n2. Separation: a negative score for getting too close to any one neighbor\n3. Alignment: a positive score for getting closer to the average heading of the neighbors\nIf all the boids execute this policy, the flock exhibits the emergent behavior of flying as a\nEMERGENT\nBEHAVIOR\npseudorigid body with roughly constant density that does not disperse over time, and that\noccasionally makes sudden swooping motions. You can see a still images in Figure 11.11(a)\nand compare it to an actual flock in (b). As with ants, there is no need for each agent to\npossess a joint plan that models the actions of other agents.\nThe most difficult multiagent problems involve both cooperation with members of one’s\nown team and competition against members of opposing teams, all without centralized con-\ntrol. We see this in games such as robotic soccer or the NERO game shown in Figure 11.11(c),\nin which two teams of software agents compete to capture the control towers. As yet, meth-\nods for efficient planning in these kinds of environments—for example, taking advantage of\nloose coupling—are in their infancy.\n11.5\nSUMMARY\nThis chapter has addressed some of the complications of planning and acting in the real world.\nThe main points:\n• Many actions consume resources, such as money, gas, or raw materials. It is convenient\nto treat these resources as numeric measures in a pool rather than try to reason about,\nsay, each individual coin and bill in the world. Actions can generate and consume\nresources, and it is usually cheap and effective to check partial plans for satisfaction of\nresource constraints before attempting further refinements.\n• Time is one of the most important resources. It can be handled by specialized schedul-\ning algorithms, or scheduling can be integrated with planning.\nBibliographical and Historical Notes\n431\n• Hierarchical task network (HTN) planning allows the agent to take advice from the\ndomain designer in the form of high-level actions (HLAs) that can be implemented in\nvarious ways by lower-level action sequences. The effects of HLAs can be defined with\nangelic semantics, allowing provably correct high-level plans to be derived without\nconsideration of lower-level implementations. HTN methods can create the very large\nplans required by many real-world applications.\n• Standard planning algorithms assume complete and correct information and determin-\nistic, fully observable environments. Many domains violate this assumption.\n• Contingent plans allow the agent to sense the world during execution to decide what\nbranch of the plan to follow. In some cases, sensorless or conformant planning can be\nused to construct a plan that works without the need for perception. Both conformant\nand contingent plans can be constructed by search in the space of belief states. Efficient\nrepresentation or computation of belief states is a key problem.\n• An online planning agent uses execution monitoring and splices in repairs as needed\nto recover from unexpected situations, which can be due to nondeterministic actions,\nexogenous events, or incorrect models of the environment.\n• Multiagent planning is necessary when there are other agents in the environment with\nwhich to cooperate or compete. Joint plans can be constructed, but must be augmented\nwith some form of coordination if two agents are to agree on which joint plan to execute.\n• This chapter extends classic planning to cover nondeterministic environments (where\noutcomes of actions are uncertain), but it is not the last word on planning. Chapter 17\ndescribes techniques for stochastic environments (in which outcomes of actions have\nprobabilities associated with them): Markov decision processes, partially observable\nMarkov decision processes, and game theory. In Chapter 21 we show that reinforcement\nlearning allows an agent to learn how to behave from past successes and failures.\nBIBLIOGRAPHICAL AND HISTORICAL NOTES\nPlanning with time constraints was first dealt with by DEVISER (Vere, 1983). The repre-\nsentation of time in plans was addressed by Allen (1984) and by Dean et al. (1990) in the\nFORBIN system. NONLIN+ (Tate and Whiter, 1984) and SIPE (Wilkins, 1988, 1990) could\nreason about the allocation of limited resources to various plan steps. O-PLAN (Bell and\nTate, 1985), an HTN planner, had a uniform, general representation for constraints on time\nand resources. In addition to the Hitachi application mentioned in the text, O-PLAN has\nbeen applied to software procurement planning at Price Waterhouse and back-axle assembly\nplanning at Jaguar Cars.\nThe two planners SAPA (Do and Kambhampati, 2001) and T4 (Haslum and Geffner,\n2001) both used forward state-space search with sophisticated heuristics to handle actions\nwith durations and resources. An alternative is to use very expressive action languages, but\nguide them by human-written domain-specific heuristics, as is done by ASPEN (Fukunaga\net al., 1997), HSTS (Jonsson et al., 2000), and IxTeT (Ghallab and Laruelle, 1994).\n432\nChapter\n11.\nPlanning and Acting in the Real World\nA number of hybrid planning-and-scheduling systems have been deployed: ISIS (Fox\net al., 1982; Fox, 1990) has been used for job shop scheduling at Westinghouse, GARI (De-\nscotte and Latombe, 1985) planned the machining and construction of mechanical parts,\nFORBIN was used for factory control, and NONLIN+ was used for naval logistics planning.\nWe chose to present planning and scheduling as two separate problems; (Cushing et al., 2007)\nshow that this can lead to incompleteness on certain problems. There is a long history of\nscheduling in aerospace. T-SCHED (Drabble, 1990) was used to schedule mission-command\nsequences for the UOSAT-II satellite. OPTIMUM-AIV (Aarup et al., 1994) and PLAN-ERS1\n(Fuchs et al., 1990), both based on O-PLAN, were used for spacecraft assembly and obser-\nvation planning, respectively, at the European Space Agency. SPIKE (Johnston and Adorf,\n1992) was used for observation planning at NASA for the Hubble Space Telescope, while\nthe Space Shuttle Ground Processing Scheduling System (Deale et al., 1994) does job-shop\nscheduling of up to 16,000 worker-shifts. Remote Agent (Muscettola et al., 1998) became\nthe first autonomous planner–scheduler to control a spacecraft when it flew onboard the Deep\nSpace One probe in 1999. Space applications have driven the development of algorithms for\nresource allocations; see Laborie (2003) and Muscettola (2002). The literature on scheduling\nis presented in a classic survey article (Lawler et al., 1993), a recent book (Pinedo, 2008),\nand an edited handbook (Blazewicz et al., 2007).\nThe facility in the STRIPS program for learning macrops—“macro-operators” consist-\nMACROPS\ning of a sequence of primitive steps—could be considered the first mechanism for hierarchi-\ncal planning (Fikes et al., 1972). Hierarchy was also used in the LAWALY system (Siklossy\nand Dreussi, 1973). The ABSTRIPS system (Sacerdoti, 1974) introduced the idea of an ab-\nstraction hierarchy, whereby planning at higher levels was permitted to ignore lower-level\nABSTRACTION\nHIERARCHY\npreconditions of actions in order to derive the general structure of a working plan. Austin\nTate’s Ph.D. thesis (1975b) and work by Earl Sacerdoti (1977) developed the basic ideas of\nHTN planning in its modern form. Many practical planners, including O-PLAN and SIPE,\nare HTN planners. Yang (1990) discusses properties of actions that make HTN planning ef-\nficient. Erol, Hendler, and Nau (1994, 1996) present a complete hierarchical decomposition\nplanner as well as a range of complexity results for pure HTN planners. Our presentation of\nHLAs and angelic semantics is due to Marthi et al. (2007, 2008). Kambhampati et al. (1998)\nhave proposed an approach in which decompositions are just another form of plan refinement,\nsimilar to the refinements for non-hierarchical partial-order planning.\nBeginning with the work on macro-operators in STRIPS, one of the goals of hierarchical\nplanning has been the reuse of previous planning experience in the form of generalized plans.\nThe technique of explanation-based learning, described in depth in Chapter 19, has been\napplied in several systems as a means of generalizing previously computed plans, including\nSOAR (Laird et al., 1986) and PRODIGY (Carbonell et al., 1989). An alternative approach is\nto store previously computed plans in their original form and then reuse them to solve new,\nsimilar problems by analogy to the original problem. This is the approach taken by the field\ncalled case-based planning (Carbonell, 1983; Alterman, 1988; Hammond, 1989). Kamb-\nCASE-BASED\nPLANNING\nhampati (1994) argues that case-based planning should be analyzed as a form of refinement\nplanning and provides a formal foundation for case-based partial-order planning.\nBibliographical and Historical Notes\n433\nEarly planners lacked conditionals and loops, but some could use coercion to form\nconformant plans. Sacerdoti’s NOAH solved the “keys and boxes” problem, a planning chal-\nlenge problem in which the planner knows little about the initial state, using coercion. Ma-\nson (1993) argued that sensing often can and should be dispensed with in robotic planning,\nand described a sensorless plan that can move a tool into a specific position on a table by a\nsequence of tilting actions, regardless of the initial position.\nGoldman and Boddy (1996) introduced the term conformant planning, noting that sen-\nsorless plans are often effective even if the agent has sensors. The first moderately efficient\nconformant planner was Smith and Weld’s (1998) Conformant Graphplan or CGP. Ferraris\nand Giunchiglia (2000) and Rintanen (1999) independently developed SATPLAN-based con-\nformant planners. Bonet and Geffner (2000) describe a conformant planner based on heuristic\nsearch in the space of belief states, drawing on ideas first developed in the 1960s for partially\nobservable Markov decision processes, or POMDPs (see Chapter 17).\nCurrently, there are three main approaches to conformant planning. The first two use\nheuristic search in belief-state space: HSCP (Bertoli et al., 2001a) uses binary decision\ndiagrams (BDDs) to represent belief states, whereas Hoffmann and Brafman (2006) adopt\nthe lazy approach of computing precondition and goal tests on demand using a SAT solver.\nThe third approach, championed primarily by Jussi Rintanen (2007), formulates the entire\nsensorless planning problem as a quantified Boolean formula (QBF) and solves it using a\ngeneral-purpose QBF solver. Current conformant planners are five orders of magnitude faster\nthan CGP. The winner of the 2006 conformant-planning track at the International Planning\nCompetition was T0 (Palacios and Geffner, 2007), which uses heuristic search in belief-state\nspace while keeping the belief-state representation simple by defining derived literals that\ncover conditional effects. Bryce and Kambhampati (2007) discuss how a planning graph can\nbe generalized to generate good heuristics for conformant and contingent planning.\nThere has been some confusion in the literature between the terms “conditional” and\n“contingent” planning. Following Majercik and Littman (2003), we use “conditional” to\nmean a plan (or action) that has different effects depending on the actual state of the world,\nand “contingent” to mean a plan in which the agent can choose different actions depending\non the results of sensing. The problem of contingent planning received more attention after\nthe publication of Drew McDermott’s (1978a) influential article, Planning and Acting.\nThe contingent-planning approach described in the chapter is based on Hoffmann and\nBrafman (2005), and was influenced by the efficient search algorithms for cyclic AND–OR\ngraphs developed by Jimenez and Torras (2000) and Hansen and Zilberstein (2001). Bertoli\net al. (2001b) describe MBP (Model-Based Planner), which uses binary decision diagrams\nto do conformant and contingent planning.\nIn retrospect, it is now possible to see how the major classical planning algorithms led\nto extended versions for uncertain domains. Fast-forward heuristic search through state space\nled to forward search in belief space (Bonet and Geffner, 2000; Hoffmann and Brafman,\n2005); SATPLAN led to stochastic SATPLAN (Majercik and Littman, 2003) and to planning\nwith quantified Boolean logic (Rintanen, 2007); partial order planning led to UWL (Etzioni\net al., 1992) and CNLP (Peot and Smith, 1992); GRAPHPLAN led to Sensory Graphplan or\nSGP (Weld et al., 1998).\n434\nChapter\n11.\nPlanning and Acting in the Real World\nThe first online planner with execution monitoring was PLANEX (Fikes et al., 1972),\nwhich worked with the STRIPS planner to control the robot Shakey. The NASL planner\n(McDermott, 1978a) treated a planning problem simply as a specification for carrying out a\ncomplex action, so that execution and planning were completely unified. SIPE (System for\nInteractive Planning and Execution monitoring) (Wilkins, 1988, 1990) was the first planner\nto deal systematically with the problem of replanning. It has been used in demonstration\nprojects in several domains, including planning operations on the flight deck of an aircraft\ncarrier, job-shop scheduling for an Australian beer factory, and planning the construction of\nmultistory buildings (Kartam and Levitt, 1990).\nIn the mid-1980s, pessimism about the slow run times of planning systems led to the\nproposal of reflex agents called reactive planning systems (Brooks, 1986; Agre and Chap-\nREACTIVE PLANNING\nman, 1987). PENGI (Agre and Chapman, 1987) could play a (fully observable) video game\nby using Boolean circuits combined with a “visual” representation of current goals and the\nagent’s internal state. “Universal plans” (Schoppers, 1987, 1989) were developed as a lookup-\ntable method for reactive planning, but turned out to be a rediscovery of the idea of policies\nPOLICY\nthat had long been used in Markov decision processes (see Chapter 17). A universal plan (or\na policy) contains a mapping from any state to the action that should be taken in that state.\nKoenig (2001) surveys online planning techniques, under the name Agent-Centered Search.\nMultiagent planning has leaped in popularity in recent years, although it does have\na long history. Konolige (1982) formalizes multiagent planning in first-order logic, while\nPednault (1986) gives a STRIPS-style description. The notion of joint intention, which is es-\nsential if agents are to execute a joint plan, comes from work on communicative acts (Cohen\nand Levesque, 1990; Cohen et al., 1990). Boutilier and Brafman (2001) show how to adapt\npartial-order planning to a multiactor setting. Brafman and Domshlak (2008) devise a mul-\ntiactor planning algorithm whose complexity grows only linearly with the number of actors,\nprovided that the degree of coupling (measured partly by the tree width of the graph of inter-\nactions among agents) is bounded. Petrik and Zilberstein (2009) show that an approach based\non bilinear programming outperforms the cover-set approach we outlined in the chapter.\nWe have barely skimmed the surface of work on negotiation in multiagent planning.\nDurfee and Lesser (1989) discuss how tasks can be shared out among agents by negotiation.\nKraus et al. (1991) describe a system for playing Diplomacy, a board game requiring negoti-\nation, coalition formation, and dishonesty. Stone (2000) shows how agents can cooperate as\nteammates in the competitive, dynamic, partially observable environment of robotic soccer. In\na later article, Stone (2003) analyzes two competitive multiagent environments—RoboCup,\na robotic soccer competition, and TAC, the auction-based Trading Agents Competition—\nand finds that the computational intractability of our current theoretically well-founded ap-\nproaches has led to many multiagent systems being designed by ad hoc methods.\nIn his highly influential Society of Mind theory, Marvin Minsky (1986, 2007) proposes\nthat human minds are constructed from an ensemble of agents. Livnat and Pippenger (2006)\nprove that, for the problem of optimal path-finding, and given a limitation on the total amount\nof computing resources, the best architecture for an agent is an ensemble of subagents, each\nof which tries to optimize its own objective, and all of which are in conflict with one another.\nExercises\n435\nThe boid model on page 429 is due to Reynolds (1987), who won an Academy Award\nfor its application to swarms of penguins in Batman Returns. The NERO game and the meth-\nods for learning strategies are described by Bryant and Miikkulainen (2007).\nRecent book on multiagent systems include those by Weiss (2000a), Young (2004),\nVlassis (2008), and Shoham and Leyton-Brown (2009). There is an annual conference on\nautonomous agents and multiagent systems (AAMAS).\nEXERCISES\n11.1\nThe goals we have considered so far all ask the planner to make the world satisfy the\ngoal at just one time step. Not all goals can be expressed this way: you do not achieve the\ngoal of suspending a chandelier above the ground by throwing it in the air. More seriously,\nyou wouldn’t want your spacecraft life-support system to supply oxygen one day but not\nthe next. A maintenance goal is achieved when the agent’s plan causes a condition to hold\ncontinuously from a given state onward. Describe how to extend the formalism of this chapter\nto support maintenance goals.\n11.2\nYou have a number of trucks with which to deliver a set of packages. Each package\nstarts at some location on a grid map, and has a destination somewhere else. Each truck is di-\nrectly controlled by moving forward and turning. Construct a hierarchy of high-level actions\nfor this problem. What knowledge about the solution does your hierarchy encode?\n11.3\nSuppose that a high-level action has exactly one implementation as a sequence of\nprimitive actions. Give an algorithm for computing its preconditions and effects, given the\ncomplete refinement hierarchy and schemas for the primitive actions.\n11.4\nSuppose that the optimistic reachable set of a high-level plan is a superset of the goal\nset; can anything be concluded about whether the plan achieves the goal? What if the pes-\nsimistic reachable set doesn’t intersect the goal set? Explain.\n11.5\nWrite an algorithm that takes an initial state (specified by a set of propositional literals)\nand a sequence of HLAs (each defined by preconditions and angelic specifications of opti-\nmistic and pessimistic reachable sets) and computes optimistic and pessimistic descriptions\nof the reachable set of the sequence.\n11.6\nIn Figure 11.2 we showed how to describe actions in a scheduling problem by using\nseparate fields for DURATION, USE, and CONSUME. Now suppose we wanted to combine\nscheduling with nondeterministic planning, which requires nondeterministic and conditional\neffects. Consider each of the three fields and explain if they should remain separate fields, or\nif they should become effects of the action. Give an example for each of the three.\n11.7\nSome of the operations in standard programming languages can be modeled as actions\nthat change the state of the world. For example, the assignment operation changes the con-\ntents of a memory location, and the print operation changes the state of the output stream. A\nprogram consisting of these operations can also be considered as a plan, whose goal is given\n" }, { "chapter": 12, "text": "12\nKNOWLEDGE\nREPRESENTATION\nIn which we show how to use first-order logic to represent the most important\naspects of the real world, such as action, space, time, thoughts, and shopping.\nThe previous chapters described the technology for knowledge-based agents: the syntax,\nsemantics, and proof theory of propositional and first-order logic, and the implementation of\nagents that use these logics. In this chapter we address the question of what content to put\ninto such an agent’s knowledge base—how to represent facts about the world.\nSection 12.1 introduces the idea of a general ontology, which organizes everything in\nthe world into a hierarchy of categories. Section 12.2 covers the basic categories of objects,\nsubstances, and measures; Section 12.3 covers events, and Section 12.4 discusses knowledge\nabout beliefs. We then return to consider the technology for reasoning with this content:\nSection 12.5 discusses reasoning systems designed for efficient inference with categories,\nand Section 12.6 discusses reasoning with default information. Section 12.7 brings all the\nknowledge together in the context of an Internet shopping environment.\n12.1\nONTOLOGICAL ENGINEERING\nIn “toy” domains, the choice of representation is not that important; many choices will work.\nComplex domains such as shopping on the Internet or driving a car in traffic require more\ngeneral and flexible representations. This chapter shows how to create these representations,\nconcentrating on general concepts—such as Events, Time, Physical Objects, and Beliefs—\nthat occur in many different domains. Representing these abstract concepts is sometimes\ncalled ontological engineering.\nONTOLOGICAL\nENGINEERING\nThe prospect of representing everything in the world is daunting. Of course, we won’t\nactually write a complete description of everything—that would be far too much for even a\n1000-page textbook—but we will leave placeholders where new knowledge for any domain\ncan fit in. For example, we will define what it means to be a physical object, and the details of\ndifferent types of objects—robots, televisions, books, or whatever—can be filled in later. This\nis analogous to the way that designers of an object-oriented programming framework (such as\nthe Java Swing graphical framework) define general concepts like Window, expecting users to\n437\n438\nChapter\n12.\nKnowledge Representation\nAnything\nAbstractObjects\nSets\nNumbers RepresentationalObjects\nInterval\nPlaces\nProcesses\nPhysicalObjects\nHumans\nCategories\nSentences Measurements\nMoments\nThings\nStuff\nTimes\nWeights\nAnimals Agents\nSolid Liquid Gas\nGeneralizedEvents\nFigure 12.1\nThe upper ontology of the world, showing the topics to be covered later in\nthe chapter. Each link indicates that the lower concept is a specialization of the upper one.\nSpecializations are not necessarily disjoint; a human is both an animal and an agent, for\nexample. We will see in Section 12.3.3 why physical objects come under generalized events.\nuse these to define more specific concepts like SpreadsheetWindow. The general framework\nof concepts is called an upper ontology because of the convention of drawing graphs with\nUPPER ONTOLOGY\nthe general concepts at the top and the more specific concepts below them, as in Figure 12.1.\nBefore considering the ontology further, we should state one important caveat. We\nhave elected to use first-order logic to discuss the content and organization of knowledge,\nalthough certain aspects of the real world are hard to capture in FOL. The principal difficulty\nis that most generalizations have exceptions or hold only to a degree. For example, although\n“tomatoes are red” is a useful rule, some tomatoes are green, yellow, or orange. Similar\nexceptions can be found to almost all the rules in this chapter. The ability to handle exceptions\nand uncertainty is extremely important, but is orthogonal to the task of understanding the\ngeneral ontology. For this reason, we delay the discussion of exceptions until Section 12.5 of\nthis chapter, and the more general topic of reasoning with uncertainty until Chapter 13.\nOf what use is an upper ontology? Consider the ontology for circuits in Section 8.4.2.\nIt makes many simplifying assumptions: time is omitted completely; signals are fixed and do\nnot propagate; the structure of the circuit remains constant. A more general ontology would\nconsider signals at particular times, and would include the wire lengths and propagation de-\nlays. This would allow us to simulate the timing properties of the circuit, and indeed such\nsimulations are often carried out by circuit designers. We could also introduce more inter-\nesting classes of gates, for example, by describing the technology (TTL, CMOS, and so on)\nas well as the input–output specification. If we wanted to discuss reliability or diagnosis, we\nwould include the possibility that the structure of the circuit or the properties of the gates\nmight change spontaneously. To account for stray capacitances, we would need to represent\nwhere the wires are on the board.\nSection 12.1.\nOntological Engineering\n439\nIf we look at the wumpus world, similar considerations apply. Although we do represent\ntime, it has a simple structure: Nothing happens except when the agent acts, and all changes\nare instantaneous. A more general ontology, better suited for the real world, would allow for\nsimultaneous changes extended over time. We also used a Pit predicate to say which squares\nhave pits. We could have allowed for different kinds of pits by having several individuals\nbelonging to the class of pits, each having different properties. Similarly, we might want to\nallow for other animals besides wumpuses. It might not be possible to pin down the exact\nspecies from the available percepts, so we would need to build up a biological taxonomy to\nhelp the agent predict the behavior of cave-dwellers from scanty clues.\nFor any special-purpose ontology, it is possible to make changes like these to move\ntoward greater generality. An obvious question then arises: do all these ontologies converge\non a general-purpose ontology? After centuries of philosophical and computational inves-\ntigation, the answer is “Maybe.” In this section, we present one general-purpose ontology\nthat synthesizes ideas from those centuries. Two major characteristics of general-purpose\nontologies distinguish them from collections of special-purpose ontologies:\n• A general-purpose ontology should be applicable in more or less any special-purpose\ndomain (with the addition of domain-specific axioms). This means that no representa-\ntional issue can be finessed or brushed under the carpet.\n• In any sufficiently demanding domain, different areas of knowledge must be unified,\nbecause reasoning and problem solving could involve several areas simultaneously. A\nrobot circuit-repair system, for instance, needs to reason about circuits in terms of elec-\ntrical connectivity and physical layout, and about time, both for circuit timing analysis\nand estimating labor costs. The sentences describing time therefore must be capable\nof being combined with those describing spatial layout and must work equally well for\nnanoseconds and minutes and for angstroms and meters.\nWe should say up front that the enterprise of general ontological engineering has so far had\nonly limited success. None of the top AI applications (as listed in Chapter 1) make use\nof a shared ontology—they all use special-purpose knowledge engineering. Social/political\nconsiderations can make it difficult for competing parties to agree on an ontology. As Tom\nGruber (2004) says, “Every ontology is a treaty—a social agreement—among people with\nsome common motive in sharing.” When competing concerns outweigh the motivation for\nsharing, there can be no common ontology. Those ontologies that do exist have been created\nalong four routes:\n1. By a team of trained ontologist/logicians, who architect the ontology and write axioms.\nThe CYC system was mostly built this way (Lenat and Guha, 1990).\n2. By importing categories, attributes, and values from an existing database or databases.\nDBPEDIA was built by importing structured facts from Wikipedia (Bizer et al., 2007).\n3. By parsing text documents and extracting information from them. TEXTRUNNER was\nbuilt by reading a large corpus of Web pages (Banko and Etzioni, 2008).\n4. By enticing unskilled amateurs to enter commonsense knowledge. The OPENMIND\nsystem was built by volunteers who proposed facts in English (Singh et al., 2002;\nChklovski and Gil, 2005).\n440\nChapter\n12.\nKnowledge Representation\n12.2\nCATEGORIES AND OBJECTS\nThe organization of objects into categories is a vital part of knowledge representation. Al-\nCATEGORY\nthough interaction with the world takes place at the level of individual objects, much reason-\ning takes place at the level of categories. For example, a shopper would normally have the\ngoal of buying a basketball, rather than a particular basketball such as BB9. Categories also\nserve to make predictions about objects once they are classified. One infers the presence of\ncertain objects from perceptual input, infers category membership from the perceived proper-\nties of the objects, and then uses category information to make predictions about the objects.\nFor example, from its green and yellow mottled skin, one-foot diameter, ovoid shape, red\nflesh, black seeds, and presence in the fruit aisle, one can infer that an object is a watermelon;\nfrom this, one infers that it would be useful for fruit salad.\nThere are two choices for representing categories in first-order logic: predicates and\nobjects. That is, we can use the predicate Basketball(b), or we can reify1 the category as\nREIFICATION\nan object, Basketballs. We could then say Member(b, Basketballs), which we will abbre-\nviate as b ∈Basketballs, to say that b is a member of the category of basketballs. We say\nSubset(Basketballs, Balls), abbreviated as Basketballs ⊂Balls, to say that Basketballs is\na subcategory of Balls. We will use subcategory, subclass, and subset interchangeably.\nSUBCATEGORY\nCategories serve to organize and simplify the knowledge base through inheritance. If\nINHERITANCE\nwe say that all instances of the category Food are edible, and if we assert that Fruit is a\nsubclass of Food and Apples is a subclass of Fruit, then we can infer that every apple is\nedible. We say that the individual apples inherit the property of edibility, in this case from\ntheir membership in the Food category.\nSubclass relations organize categories into a taxonomy, or taxonomic hierarchy. Tax-\nTAXONOMY\nonomies have been used explicitly for centuries in technical fields. The largest such taxonomy\norganizes about 10 million living and extinct species, many of them beetles,2 into a single hi-\nerarchy; library science has developed a taxonomy of all fields of knowledge, encoded as the\nDewey Decimal system; and tax authorities and other government departments have devel-\noped extensive taxonomies of occupations and commercial products. Taxonomies are also an\nimportant aspect of general commonsense knowledge.\nFirst-order logic makes it easy to state facts about categories, either by relating ob-\njects to categories or by quantifying over their members. Here are some types of facts, with\nexamples of each:\n• An object is a member of a category.\nBB9 ∈Basketballs\n• A category is a subclass of another category.\nBasketballs ⊂Balls\n• All members of a category have some properties.\n(x ∈Basketballs) ⇒Spherical(x)\n1 Turning a proposition into an object is called reification, from the Latin word res, or thing. John McCarthy\nproposed the term “thingification,” but it never caught on.\n2 The famous biologist J. B. S. Haldane deduced “An inordinate fondness for beetles” on the part of the Creator.\nSection 12.2.\nCategories and Objects\n441\n• Members of a category can be recognized by some properties.\nOrange(x) ∧Round(x) ∧Diameter(x) = 9.5′′ ∧x ∈Balls ⇒x ∈Basketballs\n• A category as a whole has some properties.\nDogs ∈DomesticatedSpecies\nNotice that because Dogs is a category and is a member of DomesticatedSpecies, the latter\nmust be a category of categories. Of course there are exceptions to many of the above rules\n(punctured basketballs are not spherical); we deal with these exceptions later.\nAlthough subclass and member relations are the most important ones for categories,\nwe also want to be able to state relations between categories that are not subclasses of each\nother. For example, if we just say that Males and Females are subclasses of Animals, then\nwe have not said that a male cannot be a female. We say that two or more categories are\ndisjoint if they have no members in common. And even if we know that males and females\nDISJOINT\nare disjoint, we will not know that an animal that is not a male must be a female, unless\nwe say that males and females constitute an exhaustive decomposition of the animals. A\nEXHAUSTIVE\nDECOMPOSITION\ndisjoint exhaustive decomposition is known as a partition. The following examples illustrate\nPARTITION\nthese three concepts:\nDisjoint({Animals, Vegetables})\nExhaustiveDecomposition({Americans, Canadians, Mexicans},\nNorthAmericans)\nPartition({Males, Females}, Animals) .\n(Note that the ExhaustiveDecomposition of NorthAmericans is not a Partition, because\nsome people have dual citizenship.) The three predicates are defined as follows:\nDisjoint(s) ⇔(∀c1, c2 c1 ∈s ∧c2 ∈s ∧c1 ̸= c2 ⇒Intersection(c1, c2) = { })\nExhaustiveDecomposition(s, c) ⇔(∀i i ∈c ⇔∃c2 c2 ∈s ∧i ∈c2)\nPartition(s, c) ⇔Disjoint(s) ∧ExhaustiveDecomposition(s, c) .\nCategories can also be defined by providing necessary and sufficient conditions for\nmembership. For example, a bachelor is an unmarried adult male:\nx ∈Bachelors ⇔Unmarried(x) ∧x ∈Adults ∧x ∈Males .\nAs we discuss in the sidebar on natural kinds on page 443, strict logical definitions for cate-\ngories are neither always possible nor always necessary.\n12.2.1\nPhysical composition\nThe idea that one object can be part of another is a familiar one. One’s nose is part of one’s\nhead, Romania is part of Europe, and this chapter is part of this book. We use the general\nPartOf relation to say that one thing is part of another. Objects can be grouped into PartOf\nhierarchies, reminiscent of the Subset hierarchy:\nPartOf (Bucharest, Romania)\nPartOf (Romania, EasternEurope)\nPartOf (EasternEurope, Europe)\nPartOf (Europe, Earth) .\n442\nChapter\n12.\nKnowledge Representation\nThe PartOf relation is transitive and reflexive; that is,\nPartOf (x, y) ∧PartOf (y, z) ⇒PartOf (x, z) .\nPartOf (x, x) .\nTherefore, we can conclude PartOf (Bucharest, Earth).\nCategories of composite objects are often characterized by structural relations among\nCOMPOSITE OBJECT\nparts. For example, a biped has two legs attached to a body:\nBiped(a)\n⇒\n∃l1, l2, b Leg(l1) ∧Leg(l2) ∧Body(b) ∧\nPartOf (l1, a) ∧PartOf (l2, a) ∧PartOf (b, a) ∧\nAttached(l1, b) ∧Attached(l2, b) ∧\nl1 ̸= l2 ∧[∀l3 Leg(l3) ∧PartOf (l3, a) ⇒(l3 = l1 ∨l3 = l2)] .\nThe notation for “exactly two” is a little awkward; we are forced to say that there are two\nlegs, that they are not the same, and that if anyone proposes a third leg, it must be the same\nas one of the other two. In Section 12.5.2, we describe a formalism called description logic\nmakes it easier to represent constraints like “exactly two.”\nWe can define a PartPartition relation analogous to the Partition relation for cate-\ngories. (See Exercise 12.8.) An object is composed of the parts in its PartPartition and can\nbe viewed as deriving some properties from those parts. For example, the mass of a compos-\nite object is the sum of the masses of the parts. Notice that this is not the case with categories,\nwhich have no mass, even though their elements might.\nIt is also useful to define composite objects with definite parts but no particular struc-\nture. For example, we might want to say “The apples in this bag weigh two pounds.” The\ntemptation would be to ascribe this weight to the set of apples in the bag, but this would be\na mistake because the set is an abstract mathematical concept that has elements but does not\nhave weight. Instead, we need a new concept, which we will call a bunch. For example, if\nBUNCH\nthe apples are Apple1, Apple2, and Apple3, then\nBunchOf ({Apple1, Apple2, Apple3})\ndenotes the composite object with the three apples as parts (not elements). We can then use the\nbunch as a normal, albeit unstructured, object. Notice that BunchOf ({x}) = x. Furthermore,\nBunchOf (Apples) is the composite object consisting of all apples—not to be confused with\nApples, the category or set of all apples.\nWe can define BunchOf in terms of the PartOf relation. Obviously, each element of\ns is part of BunchOf (s):\n∀x x ∈s ⇒PartOf (x, BunchOf (s)) .\nFurthermore, BunchOf (s) is the smallest object satisfying this condition. In other words,\nBunchOf (s) must be part of any object that has all the elements of s as parts:\n∀y [∀x x ∈s ⇒PartOf (x, y)] ⇒PartOf (BunchOf (s), y) .\nThese axioms are an example of a general technique called logical minimization, which\nLOGICAL\nMINIMIZATION\nmeans defining an object as the smallest one satisfying certain conditions.\nSection 12.2.\nCategories and Objects\n443\nNATURAL KINDS\nSome categories have strict definitions: an object is a triangle if and only if it is\na polygon with three sides. On the other hand, most categories in the real world\nhave no clear-cut definition; these are called natural kind categories. For example,\ntomatoes tend to be a dull scarlet; roughly spherical; with an indentation at the top\nwhere the stem was; about two to four inches in diameter; with a thin but tough\nskin; and with flesh, seeds, and juice inside. There is, however, variation: some\ntomatoes are yellow or orange, unripe tomatoes are green, some are smaller or\nlarger than average, and cherry tomatoes are uniformly small. Rather than having\na complete definition of tomatoes, we have a set of features that serves to identify\nobjects that are clearly typical tomatoes, but might not be able to decide for other\nobjects. (Could there be a tomato that is fuzzy like a peach?)\nThis poses a problem for a logical agent. The agent cannot be sure that an\nobject it has perceived is a tomato, and even if it were sure, it could not be cer-\ntain which of the properties of typical tomatoes this one has. This problem is an\ninevitable consequence of operating in partially observable environments.\nOne useful approach is to separate what is true of all instances of a cate-\ngory from what is true only of typical instances. So in addition to the category\nTomatoes, we will also have the category Typical(Tomatoes). Here, the Typical\nfunction maps a category to the subclass that contains only typical instances:\nTypical(c) ⊆c .\nMost knowledge about natural kinds will actually be about their typical instances:\nx ∈Typical(Tomatoes) ⇒Red(x) ∧Round(x) .\nThus, we can write down useful facts about categories without exact defini-\ntions. The difficulty of providing exact definitions for most natural categories was\nexplained in depth by Wittgenstein (1953). He used the example of games to show\nthat members of a category shared “family resemblances” rather than necessary\nand sufficient characteristics: what strict definition encompasses chess, tag, soli-\ntaire, and dodgeball?\nThe utility of the notion of strict definition was also challenged by\nQuine (1953). He pointed out that even the definition of “bachelor” as an un-\nmarried adult male is suspect; one might, for example, question a statement such\nas “the Pope is a bachelor.” While not strictly false, this usage is certainly infe-\nlicitous because it induces unintended inferences on the part of the listener. The\ntension could perhaps be resolved by distinguishing between logical definitions\nsuitable for internal knowledge representation and the more nuanced criteria for\nfelicitous linguistic usage. The latter may be achieved by “filtering” the assertions\nderived from the former. It is also possible that failures of linguistic usage serve as\nfeedback for modifying internal definitions, so that filtering becomes unnecessary.\n444\nChapter\n12.\nKnowledge Representation\n12.2.2\nMeasurements\nIn both scientific and commonsense theories of the world, objects have height, mass, cost,\nand so on.\nThe values that we assign for these properties are called measures.\nOrdi-\nMEASURE\nnary quantitative measures are quite easy to represent. We imagine that the universe in-\ncludes abstract “measure objects,” such as the length that is the length of this line seg-\nment:\n. We can call this length 1.5 inches or 3.81 centimeters. Thus,\nthe same length has different names in our language.We represent the length with a units\nfunction that takes a number as argument.\n(An alternative scheme is explored in Exer-\nUNITS FUNCTION\ncise 12.9.) If the line segment is called L1, we can write\nLength(L1) = Inches(1.5) = Centimeters(3.81) .\nConversion between units is done by equating multiples of one unit to another:\nCentimeters(2.54 × d) = Inches(d) .\nSimilar axioms can be written for pounds and kilograms, seconds and days, and dollars and\ncents. Measures can be used to describe objects as follows:\nDiameter(Basketball 12) = Inches(9.5) .\nListPrice(Basketball 12) = $(19) .\nd ∈Days ⇒Duration(d) = Hours(24) .\nNote that $(1) is not a dollar bill! One can have two dollar bills, but there is only one object\nnamed $(1). Note also that, while Inches(0) and Centimeters(0) refer to the same zero\nlength, they are not identical to other zero measures, such as Seconds(0).\nSimple, quantitative measures are easy to represent. Other measures present more of a\nproblem, because they have no agreed scale of values. Exercises have difficulty, desserts have\ndeliciousness, and poems have beauty, yet numbers cannot be assigned to these qualities. One\nmight, in a moment of pure accountancy, dismiss such properties as useless for the purpose of\nlogical reasoning; or, still worse, attempt to impose a numerical scale on beauty. This would\nbe a grave mistake, because it is unnecessary. The most important aspect of measures is not\nthe particular numerical values, but the fact that measures can be ordered.\nAlthough measures are not numbers, we can still compare them, using an ordering\nsymbol such as >. For example, we might well believe that Norvig’s exercises are tougher\nthan Russell’s, and that one scores less on tougher exercises:\ne1 ∈Exercises ∧e2 ∈Exercises ∧Wrote(Norvig, e1) ∧Wrote(Russell, e2) ⇒\nDifficulty(e1) > Difficulty(e2) .\ne1 ∈Exercises ∧e2 ∈Exercises ∧Difficulty(e1) > Difficulty(e2) ⇒\nExpectedScore(e1) < ExpectedScore(e2) .\nThis is enough to allow one to decide which exercises to do, even though no numerical values\nfor difficulty were ever used. (One does, however, have to discover who wrote which exer-\ncises.) These sorts of monotonic relationships among measures form the basis for the field of\nqualitative physics, a subfield of AI that investigates how to reason about physical systems\nwithout plunging into detailed equations and numerical simulations. Qualitative physics is\ndiscussed in the historical notes section.\nSection 12.2.\nCategories and Objects\n445\n12.2.3\nObjects: Things and stuff\nThe real world can be seen as consisting of primitive objects (e.g., atomic particles) and\ncomposite objects built from them. By reasoning at the level of large objects such as apples\nand cars, we can overcome the complexity involved in dealing with vast numbers of primitive\nobjects individually. There is, however, a significant portion of reality that seems to defy any\nobvious individuation—division into distinct objects. We give this portion the generic name\nINDIVIDUATION\nstuff. For example, suppose I have some butter and an aardvark in front of me. I can say\nSTUFF\nthere is one aardvark, but there is no obvious number of “butter-objects,” because any part of\na butter-object is also a butter-object, at least until we get to very small parts indeed. This is\nthe major distinction between stuff and things. If we cut an aardvark in half, we do not get\ntwo aardvarks (unfortunately).\nThe English language distinguishes clearly between stuff and things. We say “an aard-\nvark,” but, except in pretentious California restaurants, one cannot say “a butter.” Linguists\ndistinguish between count nouns, such as aardvarks, holes, and theorems, and mass nouns,\nCOUNT NOUNS\nMASS NOUN\nsuch as butter, water, and energy. Several competing ontologies claim to handle this distinc-\ntion. Here we describe just one; the others are covered in the historical notes section.\nTo represent stuff properly, we begin with the obvious. We need to have as objects in\nour ontology at least the gross “lumps” of stuff we interact with. For example, we might\nrecognize a lump of butter as the one left on the table the night before; we might pick it up,\nweigh it, sell it, or whatever. In these senses, it is an object just like the aardvark. Let us\ncall it Butter3. We also define the category Butter. Informally, its elements will be all those\nthings of which one might say “It’s butter,” including Butter3. With some caveats about very\nsmall parts that we w omit for now, any part of a butter-object is also a butter-object:\nb ∈Butter ∧PartOf (p, b) ⇒p ∈Butter .\nWe can now say that butter melts at around 30 degrees centigrade:\nb ∈Butter ⇒MeltingPoint(b, Centigrade(30)) .\nWe could go on to say that butter is yellow, is less dense than water, is soft at room tempera-\nture, has a high fat content, and so on. On the other hand, butter has no particular size, shape,\nor weight. We can define more specialized categories of butter such as UnsaltedButter,\nwhich is also a kind of stuff. Note that the category PoundOfButter, which includes as\nmembers all butter-objects weighing one pound, is not a kind of stuff. If we cut a pound of\nbutter in half, we do not, alas, get two pounds of butter.\nWhat is actually going on is this: some properties are intrinsic: they belong to the very\nINTRINSIC\nsubstance of the object, rather than to the object as a whole. When you cut an instance of\nstuff in half, the two pieces retain the intrinsic properties—things like density, boiling point,\nflavor, color, ownership, and so on. On the other hand, their extrinsic properties—weight,\nEXTRINSIC\nlength, shape, and so on—are not retained under subdivision. A category of objects that\nincludes in its definition only intrinsic properties is then a substance, or mass noun; a class\nthat includes any extrinsic properties in its definition is a count noun. The category Stuffis\nthe most general substance category, specifying no intrinsic properties. The category Thing\nis the most general discrete object category, specifying no extrinsic properties.\n446\nChapter\n12.\nKnowledge Representation\n12.3\nEVENTS\nIn Section 10.4.2, we showed how situation calculus represents actions and their effects.\nSituation calculus is limited in its applicability: it was designed to describe a world in which\nactions are discrete, instantaneous, and happen one at a time. Consider a continuous action,\nsuch as filling a bathtub. Situation calculus can say that the tub is empty before the action and\nfull when the action is done, but it can’t talk about what happens during the action. It also\ncan’t describe two actions happening at the same time—such as brushing one’s teeth while\nwaiting for the tub to fill. To handle such cases we introduce an alternative formalism known\nas event calculus, which is based on points of time rather than on situations.3\nEVENT CALCULUS\nEvent calculus reifies fluents and events. The fluent At(Shankar, Berkeley) is an ob-\nject that refers to the fact of Shankar being in Berkeley, but does not by itself say anything\nabout whether it is true. To assert that a fluent is actually true at some point in time we use\nthe predicate T, as in T(At(Shankar, Berkeley), t).\nEvents are described as instances of event categories.4 The event E1 of Shankar flying\nfrom San Francisco to Washington, D.C. is described as\nE1 ∈Flyings ∧Flyer(E1, Shankar) ∧Origin(E1, SF) ∧Destination(E1, DC ) .\nIf this is too verbose, we can define an alternative three-argument version of the category of\nflying events and say\nE1 ∈Flyings(Shankar, SF, DC) .\nWe then use Happens(E1, i) to say that the event E1 took place over the time interval i, and\nwe say the same thing in functional form with Extent(E1) = i. We represent time intervals\nby a (start, end) pair of times; that is, i = (t1, t2) is the time interval that starts at t1 and ends\nat t2. The complete set of predicates for one version of the event calculus is\nT(f, t)\nFluent f is true at time t\nHappens(e, i)\nEvent e happens over the time interval i\nInitiates(e, f, t)\nEvent e causes fluent f to start to hold at time t\nTerminates(e, f, t)\nEvent e causes fluent f to cease to hold at time t\nClipped(f, i)\nFluent f ceases to be true at some point during time interval i\nRestored(f, i)\nFluent f becomes true sometime during time interval i\nWe assume a distinguished event, Start, that describes the initial state by saying which fluents\nare initiated or terminated at the start time. We define T by saying that a fluent holds at a point\nin time if the fluent was initiated by an event at some time in the past and was not made false\n(clipped) by an intervening event. A fluent does not hold if it was terminated by an event and\n3 The terms “event” and “action” may be used interchangeably. Informally, “action” connotes an agent while\n“event” connotes the possibility of agentless actions.\n4 Some versions of event calculus do not distinguish event categories from instances of the categories.\nSection 12.3.\nEvents\n447\nnot made true (restored) by another event. Formally, the axioms are:\nHappens(e, (t1, t2)) ∧Initiates(e, f, t1) ∧¬Clipped(f, (t1, t)) ∧t1 < t ⇒\nT(f, t)\nHappens(e, (t1, t2)) ∧Terminates(e, f, t1) ∧¬Restored(f, (t1, t)) ∧t1 < t ⇒\n¬T(f, t)\nwhere Clipped and Restored are defined by\nClipped(f, (t1, t2)) ⇔\n∃e, t, t3 Happens(e, (t, t3)) ∧t1 ≤t < t2 ∧Terminates(e, f, t)\nRestored(f, (t1, t2)) ⇔\n∃e, t, t3 Happens(e, (t, t3)) ∧t1 ≤t < t2 ∧Initiates(e, f, t)\nIt is convenient to extend T to work over intervals as well as time points; a fluent holds over\nan interval if it holds on every point within the interval:\nT(f, (t1, t2)) ⇔[∀t (t1 ≤t < t2) ⇒T(f, t)]\nFluents and actions are defined with domain-specific axioms that are similar to successor-\nstate axioms. For example, we can say that the only way a wumpus-world agent gets an\narrow is at the start, and the only way to use up an arrow is to shoot it:\nInitiates(e, HaveArrow(a), t) ⇔e = Start\nTerminates(e, HaveArrow(a), t) ⇔e ∈Shootings(a)\nBy reifying events we make it possible to add any amount of arbitrary information about\nthem. For example, we can say that Shankar’s flight was bumpy with Bumpy(E1). In an\nontology where events are n-ary predicates, there would be no way to add extra information\nlike this; moving to an n + 1-ary predicate isn’t a scalable solution.\nWe can extend event calculus to make it possible to represent simultaneous events (such\nas two people being necessary to ride a seesaw), exogenous events (such as the wind blowing\nand changing the location of an object), continuous events (such as the level of water in the\nbathtub continuously rising) and other complications.\n12.3.1\nProcesses\nThe events we have seen so far are what we call discrete events—they have a definite struc-\nDISCRETE EVENTS\nture. Shankar’s trip has a beginning, middle, and end. If interrupted halfway, the event would\nbe something different—it would not be a trip from San Francisco to Washington, but instead\na trip from San Francisco to somewhere over Kansas. On the other hand, the category of\nevents denoted by Flyings has a different quality. If we take a small interval of Shankar’s\nflight, say, the third 20-minute segment (while he waits anxiously for a bag of peanuts), that\nevent is still a member of Flyings. In fact, this is true for any subinterval.\nCategories of events with this property are called process categories or liquid event\nPROCESS\nLIQUID EVENT\ncategories. Any process e that happens over an interval also happens over any subinterval:\n(e ∈Processes) ∧Happens(e, (t1, t4)) ∧(t1 < t2 < t3 < t4) ⇒Happens(e, (t2, t3)) .\nThe distinction between liquid and nonliquid events is exactly analogous to the difference\nbetween substances, or stuff, and individual objects, or things. In fact, some have called\nliquid events temporal substances, whereas substances like butter are spatial substances.\nTEMPORAL\nSUBSTANCE\nSPATIAL SUBSTANCE\n448\nChapter\n12.\nKnowledge Representation\n12.3.2\nTime intervals\nEvent calculus opens us up to the possibility of talking about time, and time intervals. We\nwill consider two kinds of time intervals: moments and extended intervals. The distinction is\nthat only moments have zero duration:\nPartition({Moments, ExtendedIntervals}, Intervals)\ni ∈Moments ⇔Duration(i) = Seconds(0) .\nNext we invent a time scale and associate points on that scale with moments, giving us ab-\nsolute times. The time scale is arbitrary; we measure it in seconds and say that the moment\nat midnight (GMT) on January 1, 1900, has time 0. The functions Begin and End pick out\nthe earliest and latest moments in an interval, and the function Time delivers the point on the\ntime scale for a moment. The function Duration gives the difference between the end time\nand the start time.\nInterval(i) ⇒Duration(i) = (Time(End(i)) −Time(Begin(i))) .\nTime(Begin(AD1900)) = Seconds(0) .\nTime(Begin(AD2001)) = Seconds(3187324800) .\nTime(End(AD2001)) = Seconds(3218860800) .\nDuration(AD2001) = Seconds(31536000) .\nTo make these numbers easier to read, we also introduce a function Date, which takes six\narguments (hours, minutes, seconds, day, month, and year) and returns a time point:\nTime(Begin(AD2001)) = Date(0, 0, 0, 1, Jan, 2001)\nDate(0, 20, 21, 24, 1, 1995) = Seconds(3000000000) .\nTwo intervals Meet if the end time of the first equals the start time of the second. The com-\nplete set of interval relations, as proposed by Allen (1983), is shown graphically in Figure 12.2\nand logically below:\nMeet(i, j)\n⇔\nEnd(i) = Begin(j)\nBefore(i, j)\n⇔\nEnd(i) < Begin(j)\nAfter(j, i)\n⇔\nBefore(i, j)\nDuring(i, j)\n⇔\nBegin(j) < Begin(i) < End(i) < End(j)\nOverlap(i, j)\n⇔\nBegin(i) < Begin(j) < End(i) < End(j)\nBegins(i, j)\n⇔\nBegin(i) = Begin(j)\nFinishes(i, j)\n⇔\nEnd(i) = End(j)\nEquals(i, j)\n⇔\nBegin(i) = Begin(j) ∧End(i) = End(j)\nThese all have their intuitive meaning, with the exception of Overlap: we tend to think of\noverlap as symmetric (if i overlaps j then j overlaps i), but in this definition, Overlap(i, j)\nonly holds if i begins before j. To say that the reign of Elizabeth II immediately followed that\nof George VI, and the reign of Elvis overlapped with the 1950s, we can write the following:\nMeets(ReignOf (GeorgeVI ), ReignOf (ElizabethII )) .\nOverlap(Fifties, ReignOf (Elvis)) .\nBegin(Fifties) = Begin(AD1950) .\nEnd(Fifties) = End(AD1959) .\nSection 12.3.\nEvents\n449\nFigure 12.2\nPredicates on time intervals.\ntime\n1801\n1797\n1789\nWashington\nAdams\nJefferson\nFigure 12.3\nA schematic view of the object President(USA) for the first 15 years of its\nexistence.\n12.3.3\nFluents and objects\nPhysical objects can be viewed as generalized events, in the sense that a physical object is\na chunk of space–time. For example, USA can be thought of as an event that began in,\nsay, 1776 as a union of 13 states and is still in progress today as a union of 50. We can\ndescribe the changing properties of USA using state fluents, such as Population(USA). A\nproperty of the USA that changes every four or eight years, barring mishaps, is its president.\nOne might propose that President(USA) is a logical term that denotes a different object\nat different times. Unfortunately, this is not possible, because a term denotes exactly one\nobject in a given model structure. (The term President(USA, t) can denote different objects,\ndepending on the value of t, but our ontology keeps time indices separate from fluents.) The\n450\nChapter\n12.\nKnowledge Representation\nonly possibility is that President(USA) denotes a single object that consists of different\npeople at different times. It is the object that is George Washington from 1789 to 1797, John\nAdams from 1797 to 1801, and so on, as in Figure 12.3. To say that George Washington was\npresident throughout 1790, we can write\nT(Equals(President(USA), GeorgeWashington), AD1790) .\nWe use the function symbol Equals rather than the standard logical predicate =, because\nwe cannot have a predicate as an argument to T, and because the interpretation is not that\nGeorgeWashington and President(USA) are logically identical in 1790; logical identity is\nnot something that can change over time. The identity is between the subevents of each object\nthat are defined by the period 1790.\n12.4\nMENTAL EVENTS AND MENTAL OBJECTS\nThe agents we have constructed so far have beliefs and can deduce new beliefs. Yet none\nof them has any knowledge about beliefs or about deduction. Knowledge about one’s own\nknowledge and reasoning processes is useful for controlling inference. For example, suppose\nAlice asks “what is the square root of 1764” and Bob replies “I don’t know.” If Alice insists\n“think harder,” Bob should realize that with some more thought, this question can in fact\nbe answered. On the other hand, if the question were “Is your mother sitting down right\nnow?” then Bob should realize that thinking harder is unlikely to help. Knowledge about\nthe knowledge of other agents is also important; Bob should realize that his mother knows\nwhether she is sitting or not, and that asking her would be a way to find out.\nWhat we need is a model of the mental objects that are in someone’s head (or some-\nthing’s knowledge base) and of the mental processes that manipulate those mental objects.\nThe model does not have to be detailed. We do not have to be able to predict how many\nmilliseconds it will take for a particular agent to make a deduction. We will be happy just to\nbe able to conclude that mother knows whether or not she is sitting.\nWe begin with the propositional attitudes that an agent can have toward mental ob-\nPROPOSITIONAL\nATTITUDE\njects: attitudes such as Believes, Knows, Wants, Intends, and Informs. The difficulty is\nthat these attitudes do not behave like “normal” predicates. For example, suppose we try to\nassert that Lois knows that Superman can fly:\nKnows(Lois, CanFly(Superman)) .\nOne minor issue with this is that we normally think of CanFly(Superman) as a sentence, but\nhere it appears as a term. That issue can be patched up just be reifying CanFly(Superman);\nmaking it a fluent. A more serious problem is that, if it is true that Superman is Clark Kent,\nthen we must conclude that Lois knows that Clark can fly:\n(Superman = Clark) ∧Knows(Lois, CanFly(Superman))\n|= Knows(Lois, CanFly(Clark)) .\nThis is a consequence of the fact that equality reasoning is built into logic. Normally that is\na good thing; if our agent knows that 2 + 2 = 4 and 4 < 5, then we want our agent to know\nSection 12.4.\nMental Events and Mental Objects\n451\nthat 2 + 2 < 5. This property is called referential transparency—it doesn’t matter what\nREFERENTIAL\nTRANSPARENCY\nterm a logic uses to refer to an object, what matters is the object that the term names. But for\npropositional attitudes like believes and knows, we would like to have referential opacity—the\nterms used do matter, because not all agents know which terms are co-referential.\nModal logic is designed to address this problem. Regular logic is concerned with a sin-\nMODAL LOGIC\ngle modality, the modality of truth, allowing us to express “P is true.” Modal logic includes\nspecial modal operators that take sentences (rather than terms) as arguments. For example,\n“A knows P” is represented with the notation KAP, where K is the modal operator for knowl-\nedge. It takes two arguments, an agent (written as the subscript) and a sentence. The syntax\nof modal logic is the same as first-order logic, except that sentences can also be formed with\nmodal operators.\nThe semantics of modal logic is more complicated. In first-order logic a model con-\ntains a set of objects and an interpretation that maps each name to the appropriate object,\nrelation, or function. In modal logic we want to be able to consider both the possibility that\nSuperman’s secret identity is Clark and that it isn’t. Therefore, we will need a more com-\nplicated model, one that consists of a collection of possible worlds rather than just one true\nPOSSIBLE WORLD\nworld. The worlds are connected in a graph by accessibility relations, one relation for each\nACCESSIBILITY\nRELATIONS\nmodal operator. We say that world w1 is accessible from world w0 with respect to the modal\noperator KA if everything in w1 is consistent with what A knows in w0, and we write this\nas Acc(KA, w0, w1). In diagrams such as Figure 12.4 we show accessibility as an arrow be-\ntween possible worlds. As an example, in the real world, Bucharest is the capital of Romania,\nbut for an agent that did not know that, other possible worlds are accessible, including ones\nwhere the capital of Romania is Sibiu or Sofia. Presumably a world where 2 + 2 = 5 would\nnot be accessible to any agent.\nIn general, a knowledge atom KAP is true in world w if and only if P is true in every\nworld accessible from w. The truth of more complex sentences is derived by recursive appli-\ncation of this rule and the normal rules of first-order logic. That means that modal logic can\nbe used to reason about nested knowledge sentences: what one agent knows about another\nagent’s knowledge. For example, we can say that, even though Lois doesn’t know whether\nSuperman’s secret identity is Clark Kent, she does know that Clark knows:\nKLois[KClarkIdentity(Superman, Clark) ∨KClark¬Identity(Superman, Clark)]\nFigure 12.4 shows some possible worlds for this domain, with accessibility relations for Lois\nand Superman.\nIn the TOP-LEFT diagram, it is common knowledge that Superman knows his own iden-\ntity, and neither he nor Lois has seen the weather report. So in w0 the worlds w0 and w2 are\naccessible to Superman; maybe rain is predicted, maybe not. For Lois all four worlds are ac-\ncessible from each other; she doesn’t know anything about the report or if Clark is Superman.\nBut she does know that Superman knows whether he is Clark, because in every world that is\naccessible to Lois, either Superman knows I, or he knows ¬I. Lois does not know which is\nthe case, but either way she knows Superman knows.\nIn the TOP-RIGHT diagram it is common knowledge that Lois has seen the weather\nreport. So in w4 she knows rain is predicted and in w6 she knows rain is not predicted.\n452\nChapter\n12.\nKnowledge Representation\n(a)\n(b)\n(c)\nw0: I,R\nw2: I,¬R\nw3: ¬I,¬R\nw1: ¬I,R\nw4: I,R\nw6: I,¬R\nw7: ¬I,¬R\nw5: ¬I,R\nw0: I,R\nw2: I,¬R\nw3: ¬I,¬R\nw1: ¬I,R\nw4: I,R\nw5: ¬I,R\nw6: I,¬R\nw7: ¬I,¬R\nFigure 12.4\nPossible worlds with accessibility relations KSuperman (solid arrows) and\nKLois (dotted arrows). The proposition R means “the weather report for tomorrow is rain”\nand I means “Superman’s secret identity is Clark Kent.” All worlds are accessible to them-\nselves; the arrows from a world to itself are not shown.\nSuperman does not know the report, but he knows that Lois knows, because in every world\nthat is accessible to him, either she knows R or she knows ¬R.\nIn the BOTTOM diagram we represent the scenario where it is common knowledge that\nSuperman knows his identity, and Lois might or might not have seen the weather report. We\nrepresent this by combining the two top scenarios, and adding arrows to show that Superman\ndoes not know which scenario actually holds. Lois does know, so we don’t need to add any\narrows for her. In w0 Superman still knows I but not R, and now he does not know whether\nLois knows R. From what Superman knows, he might be in w0 or w2, in which case Lois\ndoes not know whether R is true, or he could be in w4, in which case she knows R, or w6, in\nwhich case she knows ¬R.\nThere are an infinite number of possible worlds, so the trick is to introduce just the ones\nyou need to represent what you are trying to model. A new possible world is needed to talk\nabout different possible facts (e.g., rain is predicted or not), or to talk about different states\nof knowledge (e.g., does Lois know that rain is predicted). That means two possible worlds,\nsuch as w4 and w0 in Figure 12.4, might have the same base facts about the world, but differ\nin their accessibility relations, and therefore in facts about knowledge.\nModal logic solves some tricky issues with the interplay of quantifiers and knowledge.\nThe English sentence “Bond knows that someone is a spy” is ambiguous. The first reading is\nSection 12.5.\nReasoning Systems for Categories\n453\nthat there is a particular someone who Bond knows is a spy; we can write this as\n∃x KBondSpy(x) ,\nwhich in modal logic means that there is an x that, in all accessible worlds, Bond knows to\nbe a spy. The second reading is that Bond just knows that there is at least one spy:\nKBond∃x Spy(x) .\nThe modal logic interpretation is that in each accessible world there is an x that is a spy, but\nit need not be the same x in each world.\nNow that we have a modal operator for knowledge, we can write axioms for it. First,\nwe can say that agents are able to draw deductions; if an agent knows P and knows that P\nimplies Q, then the agent knows Q:\n(KaP ∧Ka(P ⇒Q)) ⇒KaQ .\nFrom this (and a few other rules about logical identities) we can establish that KA(P ∨¬P)\nis a tautology; every agent knows every proposition P is either true or false. On the other\nhand, (KAP) ∨(KA¬P) is not a tautology; in general, there will be lots of propositions that\nan agent does not know to be true and does not know to be false.\nIt is said (going back to Plato) that knowledge is justified true belief. That is, if it is\ntrue, if you believe it, and if you have an unassailably good reason, then you know it. That\nmeans that if you know something, it must be true, and we have the axiom:\nKaP ⇒P .\nFurthermore, logical agents should be able to introspect on their own knowledge. If they\nknow something, then they know that they know it:\nKaP ⇒Ka(KaP) .\nWe can define similar axioms for belief (often denoted by B) and other modalities. However,\none problem with the modal logic approach is that it assumes logical omniscience on the\nLOGICAL\nOMNISCIENCE\npart of agents. That is, if an agent knows a set of axioms, then it knows all consequences of\nthose axioms. This is on shaky ground even for the somewhat abstract notion of knowledge,\nbut it seems even worse for belief, because belief has more connotation of referring to things\nthat are physically represented in the agent, not just potentially derivable. There have been\nattempts to define a form of limited rationality for agents; to say that agents believe those\nassertions that can be derived with the application of no more than k reasoning steps, or no\nmore than s seconds of computation. These attempts have been generally unsatisfactory.\n12.5\nREASONING SYSTEMS FOR CATEGORIES\nCategories are the primary building blocks of large-scale knowledge representation schemes.\nThis section describes systems specially designed for organizing and reasoning with cate-\ngories. There are two closely related families of systems: semantic networks provide graph-\nical aids for visualizing a knowledge base and efficient algorithms for inferring properties\n454\nChapter\n12.\nKnowledge Representation\nof an object on the basis of its category membership; and description logics provide a for-\nmal language for constructing and combining category definitions and efficient algorithms\nfor deciding subset and superset relationships between categories.\n12.5.1\nSemantic networks\nIn 1909, Charles S. Peirce proposed a graphical notation of nodes and edges called existential\ngraphs that he called “the logic of the future.” Thus began a long-running debate between\nEXISTENTIAL\nGRAPHS\nadvocates of “logic” and advocates of “semantic networks.” Unfortunately, the debate ob-\nscured the fact that semantics networks—at least those with well-defined semantics—are a\nform of logic. The notation that semantic networks provide for certain kinds of sentences\nis often more convenient, but if we strip away the “human interface” issues, the underlying\nconcepts—objects, relations, quantification, and so on—are the same.\nThere are many variants of semantic networks, but all are capable of representing in-\ndividual objects, categories of objects, and relations among objects. A typical graphical no-\ntation displays object or category names in ovals or boxes, and connects them with labeled\nlinks. For example, Figure 12.5 has a MemberOf link between Mary and FemalePersons,\ncorresponding to the logical assertion Mary ∈FemalePersons; similarly, the SisterOf link\nbetween Mary and John corresponds to the assertion SisterOf (Mary, John). We can con-\nnect categories using SubsetOf links, and so on. It is such fun drawing bubbles and arrows\nthat one can get carried away. For example, we know that persons have female persons as\nmothers, so can we draw a HasMother link from Persons to FemalePersons? The answer\nis no, because HasMother is a relation between a person and his or her mother, and categories\ndo not have mothers.5\nFor this reason, we have used a special notation—the double-boxed link—in Figure 12.5.\nThis link asserts that\n∀x x ∈Persons ⇒[∀y HasMother(x, y) ⇒y ∈FemalePersons] .\nWe might also want to assert that persons have two legs—that is,\n∀x x ∈Persons ⇒Legs(x, 2) .\nAs before, we need to be careful not to assert that a category has legs; the single-boxed link\nin Figure 12.5 is used to assert properties of every member of a category.\nThe semantic network notation makes it convenient to perform inheritance reasoning\nof the kind introduced in Section 12.2. For example, by virtue of being a person, Mary inherits\nthe property of having two legs. Thus, to find out how many legs Mary has, the inheritance\nalgorithm follows the MemberOf link from Mary to the category she belongs to, and then\nfollows SubsetOf links up the hierarchy until it finds a category for which there is a boxed\nLegs link—in this case, the Persons category. The simplicity and efficiency of this inference\n5 Several early systems failed to distinguish between properties of members of a category and properties of the\ncategory as a whole. This can lead directly to inconsistencies, as pointed out by Drew McDermott (1976) in his\narticle “Artificial Intelligence Meets Natural Stupidity.” Another common problem was the use of IsA links for\nboth subset and membership relations, in correspondence with English usage: “a cat is a mammal” and “Fifiis a\ncat.” See Exercise 12.22 for more on these issues.\nSection 12.5.\nReasoning Systems for Categories\n455\nMammals\nJohn\nMary\nPersons\nMale\nPersons\nFemale\nPersons\n1\n2\nSubsetOf\nSubsetOf\nSubsetOf\nMemberOf\nMemberOf\nSisterOf\nLegs\nLegs\nHasMother\nFigure 12.5\nA semantic network with four objects (John, Mary, 1, and 2) and four cate-\ngories. Relations are denoted by labeled links.\nMemberOf\nFlyEvents\nFly17\nShankar\nNewYork\nNewDelhi\nYesterday\nAgent\nOrigin\nDestination\nDuring\nFigure 12.6\nA fragment of a semantic network showing the representation of the logical\nassertion Fly(Shankar, NewYork, NewDelhi, Yesterday).\nmechanism, compared with logical theorem proving, has been one of the main attractions of\nsemantic networks.\nInheritance becomes complicated when an object can belong to more than one category\nor when a category can be a subset of more than one other category; this is called multiple in-\nheritance. In such cases, the inheritance algorithm might find two or more conflicting values\nMULTIPLE\nINHERITANCE\nanswering the query. For this reason, multiple inheritance is banned in some object-oriented\nprogramming (OOP) languages, such as Java, that use inheritance in a class hierarchy. It is\nusually allowed in semantic networks, but we defer discussion of that until Section 12.6.\nThe reader might have noticed an obvious drawback of semantic network notation, com-\npared to first-order logic: the fact that links between bubbles represent only binary relations.\nFor example, the sentence Fly(Shankar, NewYork, NewDelhi, Yesterday) cannot be as-\nserted directly in a semantic network. Nonetheless, we can obtain the effect of n-ary asser-\ntions by reifying the proposition itself as an event belonging to an appropriate event category.\nFigure 12.6 shows the semantic network structure for this particular event. Notice that the\nrestriction to binary relations forces the creation of a rich ontology of reified concepts.\nReification of propositions makes it possible to represent every ground, function-free\natomic sentence of first-order logic in the semantic network notation. Certain kinds of univer-\n456\nChapter\n12.\nKnowledge Representation\nsally quantified sentences can be asserted using inverse links and the singly boxed and doubly\nboxed arrows applied to categories, but that still leaves us a long way short of full first-order\nlogic. Negation, disjunction, nested function symbols, and existential quantification are all\nmissing. Now it is possible to extend the notation to make it equivalent to first-order logic—as\nin Peirce’s existential graphs—but doing so negates one of the main advantages of semantic\nnetworks, which is the simplicity and transparency of the inference processes. Designers can\nbuild a large network and still have a good idea about what queries will be efficient, because\n(a) it is easy to visualize the steps that the inference procedure will go through and (b) in some\ncases the query language is so simple that difficult queries cannot be posed. In cases where\nthe expressive power proves to be too limiting, many semantic network systems provide for\nprocedural attachment to fill in the gaps. Procedural attachment is a technique whereby\na query about (or sometimes an assertion of) a certain relation results in a call to a special\nprocedure designed for that relation rather than a general inference algorithm.\nOne of the most important aspects of semantic networks is their ability to represent\ndefault values for categories. Examining Figure 12.5 carefully, one notices that John has one\nDEFAULT VALUE\nleg, despite the fact that he is a person and all persons have two legs. In a strictly logical KB,\nthis would be a contradiction, but in a semantic network, the assertion that all persons have\ntwo legs has only default status; that is, a person is assumed to have two legs unless this is\ncontradicted by more specific information. The default semantics is enforced naturally by the\ninheritance algorithm, because it follows links upwards from the object itself (John in this\ncase) and stops as soon as it finds a value. We say that the default is overridden by the more\nOVERRIDING\nspecific value. Notice that we could also override the default number of legs by creating a\ncategory of OneLeggedPersons, a subset of Persons of which John is a member.\nWe can retain a strictly logical semantics for the network if we say that the Legs asser-\ntion for Persons includes an exception for John:\n∀x x ∈Persons ∧x ̸= John ⇒Legs(x, 2) .\nFor a fixed network, this is semantically adequate but will be much less concise than the\nnetwork notation itself if there are lots of exceptions. For a network that will be updated with\nmore assertions, however, such an approach fails—we really want to say that any persons as\nyet unknown with one leg are exceptions too. Section 12.6 goes into more depth on this issue\nand on default reasoning in general.\n12.5.2\nDescription logics\nThe syntax of first-order logic is designed to make it easy to say things about objects. De-\nscription logics are notations that are designed to make it easier to describe definitions and\nDESCRIPTION LOGIC\nproperties of categories. Description logic systems evolved from semantic networks in re-\nsponse to pressure to formalize what the networks mean while retaining the emphasis on\ntaxonomic structure as an organizing principle.\nThe principal inference tasks for description logics are subsumption (checking if one\nSUBSUMPTION\ncategory is a subset of another by comparing their definitions) and classification (checking\nCLASSIFICATION\nwhether an object belongs to a category).. Some systems also include consistency of a cate-\ngory definition—whether the membership criteria are logically satisfiable.\nSection 12.5.\nReasoning Systems for Categories\n457\nConcept\n→\nThing | ConceptName\n|\nAnd(Concept, . . .)\n|\nAll(RoleName, Concept)\n|\nAtLeast(Integer, RoleName)\n|\nAtMost(Integer, RoleName)\n|\nFills(RoleName, IndividualName, . . .)\n|\nSameAs(Path, Path)\n|\nOneOf(IndividualName, . . .)\nPath\n→\n[RoleName, . . .]\nFigure 12.7\nThe syntax of descriptions in a subset of the CLASSIC language.\nThe CLASSIC language (Borgida et al., 1989) is a typical description logic. The syntax\nof CLASSIC descriptions is shown in Figure 12.7.6 For example, to say that bachelors are\nunmarried adult males we would write\nBachelor = And(Unmarried, Adult, Male) .\nThe equivalent in first-order logic would be\nBachelor(x) ⇔Unmarried(x) ∧Adult(x) ∧Male(x) .\nNotice that the description logic has an an algebra of operations on predicates, which of\ncourse we can’t do in first-order logic. Any description in CLASSIC can be translated into an\nequivalent first-order sentence, but some descriptions are more straightforward in CLASSIC.\nFor example, to describe the set of men with at least three sons who are all unemployed\nand married to doctors, and at most two daughters who are all professors in physics or math\ndepartments, we would use\nAnd(Man, AtLeast(3, Son), AtMost(2, Daughter),\nAll(Son, And(Unemployed, Married, All(Spouse, Doctor))),\nAll(Daughter, And(Professor, Fills(Department, Physics, Math)))) .\nWe leave it as an exercise to translate this into first-order logic.\nPerhaps the most important aspect of description logics is their emphasis on tractability\nof inference. A problem instance is solved by describing it and then asking if it is subsumed\nby one of several possible solution categories. In standard first-order logic systems, predicting\nthe solution time is often impossible. It is frequently left to the user to engineer the represen-\ntation to detour around sets of sentences that seem to be causing the system to take several\nweeks to solve a problem. The thrust in description logics, on the other hand, is to ensure that\nsubsumption-testing can be solved in time polynomial in the size of the descriptions.7\n6 Notice that the language does not allow one to simply state that one concept, or category, is a subset of\nanother. This is a deliberate policy: subsumption between categories must be derivable from some aspects of the\ndescriptions of the categories. If not, then something is missing from the descriptions.\n7\nCLASSIC provides efficient subsumption testing in practice, but the worst-case run time is exponential.\n458\nChapter\n12.\nKnowledge Representation\nThis sounds wonderful in principle, until one realizes that it can only have one of two\nconsequences: either hard problems cannot be stated at all, or they require exponentially\nlarge descriptions! However, the tractability results do shed light on what sorts of constructs\ncause problems and thus help the user to understand how different representations behave.\nFor example, description logics usually lack negation and disjunction. Each forces first-\norder logical systems to go through a potentially exponential case analysis in order to ensure\ncompleteness. CLASSIC allows only a limited form of disjunction in the Fills and OneOf\nconstructs, which permit disjunction over explicitly enumerated individuals but not over de-\nscriptions. With disjunctive descriptions, nested definitions can lead easily to an exponential\nnumber of alternative routes by which one category can subsume another.\n12.6\nREASONING WITH DEFAULT INFORMATION\nIn the preceding section, we saw a simple example of an assertion with default status: people\nhave two legs. This default can be overridden by more specific information, such as that\nLong John Silver has one leg. We saw that the inheritance mechanism in semantic networks\nimplements the overriding of defaults in a simple and natural way. In this section, we study\ndefaults more generally, with a view toward understanding the semantics of defaults rather\nthan just providing a procedural mechanism.\n12.6.1\nCircumscription and default logic\nWe have seen two examples of reasoning processes that violate the monotonicity property of\nlogic that was proved in Chapter 7.8 In this chapter we saw that a property inherited by all\nmembers of a category in a semantic network could be overridden by more specific informa-\ntion for a subcategory. In Section 9.4.5, we saw that under the closed-world assumption, if a\nproposition α is not mentioned in KB then KB |= ¬α, but KB ∧α |= α.\nSimple introspection suggests that these failures of monotonicity are widespread in\ncommonsense reasoning. It seems that humans often “jump to conclusions.” For example,\nwhen one sees a car parked on the street, one is normally willing to believe that it has four\nwheels even though only three are visible. Now, probability theory can certainly provide a\nconclusion that the fourth wheel exists with high probability, yet, for most people, the possi-\nbility of the car’s not having four wheels does not arise unless some new evidence presents\nitself. Thus, it seems that the four-wheel conclusion is reached by default, in the absence of\nany reason to doubt it. If new evidence arrives—for example, if one sees the owner carrying\na wheel and notices that the car is jacked up—then the conclusion can be retracted. This kind\nof reasoning is said to exhibit nonmonotonicity, because the set of beliefs does not grow\nNONMONOTONICITY\nmonotonically over time as new evidence arrives. Nonmonotonic logics have been devised\nNONMONOTONIC\nLOGIC\nwith modified notions of truth and entailment in order to capture such behavior. We will look\nat two such logics that have been studied extensively: circumscription and default logic.\n8 Recall that monotonicity requires all entailed sentences to remain entailed after new sentences are added to the\nKB. That is, if KB |= α then KB ∧β |= α.\nSection 12.6.\nReasoning with Default Information\n459\nCircumscription can be seen as a more powerful and precise version of the closed-\nCIRCUMSCRIPTION\nworld assumption. The idea is to specify particular predicates that are assumed to be “as false\nas possible”—that is, false for every object except those for which they are known to be true.\nFor example, suppose we want to assert the default rule that birds fly. We would introduce a\npredicate, say Abnormal1(x), and write\nBird(x) ∧¬Abnormal1(x) ⇒Flies(x) .\nIf we say that Abnormal 1 is to be circumscribed, a circumscriptive reasoner is entitled to\nassume ¬Abnormal1(x) unless Abnormal1(x) is known to be true. This allows the con-\nclusion Flies(Tweety) to be drawn from the premise Bird(Tweety), but the conclusion no\nlonger holds if Abnormal1(Tweety) is asserted.\nCircumscription can be viewed as an example of a model preference logic. In such\nMODEL\nPREFERENCE\nlogics, a sentence is entailed (with default status) if it is true in all preferred models of the KB,\nas opposed to the requirement of truth in all models in classical logic. For circumscription,\none model is preferred to another if it has fewer abnormal objects.9 Let us see how this idea\nworks in the context of multiple inheritance in semantic networks. The standard example for\nwhich multiple inheritance is problematic is called the “Nixon diamond.” It arises from the\nobservation that Richard Nixon was both a Quaker (and hence by default a pacifist) and a\nRepublican (and hence by default not a pacifist). We can write this as follows:\nRepublican(Nixon) ∧Quaker(Nixon) .\nRepublican(x) ∧¬Abnormal2(x) ⇒¬Pacifist(x) .\nQuaker(x) ∧¬Abnormal3(x) ⇒Pacifist(x) .\nIf we circumscribe Abnormal2 and Abnormal3, there are two preferred models: one in\nwhich Abnormal 2(Nixon) and Pacifist(Nixon) hold and one in which Abnormal 3(Nixon)\nand ¬Pacifist(Nixon) hold. Thus, the circumscriptive reasoner remains properly agnostic as\nto whether Nixon was a pacifist. If we wish, in addition, to assert that religious beliefs take\nprecedence over political beliefs, we can use a formalism called prioritized circumscription\nPRIORITIZED\nCIRCUMSCRIPTION\nto give preference to models where Abnormal3 is minimized.\nDefault logic is a formalism in which default rules can be written to generate contin-\nDEFAULT LOGIC\nDEFAULT RULES\ngent, nonmonotonic conclusions. A default rule looks like this:\nBird(x) : Flies(x)/Flies(x) .\nThis rule means that if Bird(x) is true, and if Flies(x) is consistent with the knowledge base,\nthen Flies(x) may be concluded by default. In general, a default rule has the form\nP : J1, . . . , Jn/C\nwhere P is called the prerequisite, C is the conclusion, and Ji are the justifications—if any\none of them can be proven false, then the conclusion cannot be drawn. Any variable that\n9 For the closed-world assumption, one model is preferred to another if it has fewer true atoms—that is, preferred\nmodels are minimal models. There is a natural connection between the closed-world assumption and definite-\nclause KBs, because the fixed point reached by forward chaining on definite-clause KBs is the unique minimal\nmodel. See page 258 for more on this point.\n460\nChapter\n12.\nKnowledge Representation\nappears in Ji or C must also appear in P. The Nixon-diamond example can be represented\nin default logic with one fact and two default rules:\nRepublican(Nixon) ∧Quaker(Nixon) .\nRepublican(x) : ¬Pacifist(x)/¬Pacifist(x) .\nQuaker(x) : Pacifist(x)/Pacifist(x) .\nTo interpret what the default rules mean, we define the notion of an extension of a default\nEXTENSION\ntheory to be a maximal set of consequences of the theory. That is, an extension S consists\nof the original known facts and a set of conclusions from the default rules, such that no\nadditional conclusions can be drawn from S and the justifications of every default conclusion\nin S are consistent with S. As in the case of the preferred models in circumscription, we have\ntwo possible extensions for the Nixon diamond: one wherein he is a pacifist and one wherein\nhe is not. Prioritized schemes exist in which some default rules can be given precedence over\nothers, allowing some ambiguities to be resolved.\nSince 1980, when nonmonotonic logics were first proposed, a great deal of progress\nhas been made in understanding their mathematical properties. There are still unresolved\nquestions, however. For example, if “Cars have four wheels” is false, what does it mean\nto have it in one’s knowledge base? What is a good set of default rules to have? If we\ncannot decide, for each rule separately, whether it belongs in our knowledge base, then we\nhave a serious problem of nonmodularity. Finally, how can beliefs that have default status be\nused to make decisions? This is probably the hardest issue for default reasoning. Decisions\noften involve tradeoffs, and one therefore needs to compare the strengths of belief in the\noutcomes of different actions, and the costs of making a wrong decision. In cases where the\nsame kinds of decisions are being made repeatedly, it is possible to interpret default rules\nas “threshold probability” statements. For example, the default rule “My brakes are always\nOK” really means “The probability that my brakes are OK, given no other information, is\nsufficiently high that the optimal decision is for me to drive without checking them.” When\nthe decision context changes—for example, when one is driving a heavily laden truck down a\nsteep mountain road—the default rule suddenly becomes inappropriate, even though there is\nno new evidence of faulty brakes. These considerations have led some researchers to consider\nhow to embed default reasoning within probability theory or utility theory.\n12.6.2\nTruth maintenance systems\nWe have seen that many of the inferences drawn by a knowledge representation system will\nhave only default status, rather than being absolutely certain. Inevitably, some of these in-\nferred facts will turn out to be wrong and will have to be retracted in the face of new informa-\ntion. This process is called belief revision.10 Suppose that a knowledge base KB contains\nBELIEF REVISION\na sentence P—perhaps a default conclusion recorded by a forward-chaining algorithm, or\nperhaps just an incorrect assertion—and we want to execute TELL(KB, ¬P). To avoid cre-\nating a contradiction, we must first execute RETRACT(KB, P). This sounds easy enough.\n10 Belief revision is often contrasted with belief update, which occurs when a knowledge base is revised to reflect\na change in the world rather than new information about a fixed world. Belief update combines belief revision\nwith reasoning about time and change; it is also related to the process of filtering described in Chapter 15.\nSection 12.6.\nReasoning with Default Information\n461\nProblems arise, however, if any additional sentences were inferred from P and asserted in\nthe KB. For example, the implication P\n⇒Q might have been used to add Q. The obvious\n“solution”—retracting all sentences inferred from P—fails because such sentences may have\nother justifications besides P. For example, if R and R ⇒Q are also in the KB, then Q\ndoes not have to be removed after all. Truth maintenance systems, or TMSs, are designed\nTRUTH\nMAINTENANCE\nSYSTEM\nto handle exactly these kinds of complications.\nOne simple approach to truth maintenance is to keep track of the order in which sen-\ntences are told to the knowledge base by numbering them from P1 to Pn. When the call\nRETRACT(KB, Pi) is made, the system reverts to the state just before Pi was added, thereby\nremoving both Pi and any inferences that were derived from Pi. The sentences Pi+1 through\nPn can then be added again. This is simple, and it guarantees that the knowledge base will\nbe consistent, but retracting Pi requires retracting and reasserting n −i sentences as well as\nundoing and redoing all the inferences drawn from those sentences. For systems to which\nmany facts are being added—such as large commercial databases—this is impractical.\nA more efficient approach is the justification-based truth maintenance system, or JTMS.\nJTMS\nIn a JTMS, each sentence in the knowledge base is annotated with a justification consisting\nJUSTIFICATION\nof the set of sentences from which it was inferred. For example, if the knowledge base\nalready contains P\n⇒Q, then TELL(P) will cause Q to be added with the justification\n{P, P\n⇒Q}. In general, a sentence can have any number of justifications. Justifica-\ntions make retraction efficient. Given the call RETRACT(P), the JTMS will delete exactly\nthose sentences for which P is a member of every justification. So, if a sentence Q had\nthe single justification {P, P ⇒Q}, it would be removed; if it had the additional justi-\nfication {P, P ∨R ⇒Q}, it would still be removed; but if it also had the justification\n{R, P ∨R ⇒Q}, then it would be spared. In this way, the time required for retraction of P\ndepends only on the number of sentences derived from P rather than on the number of other\nsentences added since P entered the knowledge base.\nThe JTMS assumes that sentences that are considered once will probably be considered\nagain, so rather than deleting a sentence from the knowledge base entirely when it loses\nall justifications, we merely mark the sentence as being out of the knowledge base. If a\nsubsequent assertion restores one of the justifications, then we mark the sentence as being\nback in. In this way, the JTMS retains all the inference chains that it uses and need not\nrederive sentences when a justification becomes valid again.\nIn addition to handling the retraction of incorrect information, TMSs can be used to\nspeed up the analysis of multiple hypothetical situations. Suppose, for example, that the\nRomanian Olympic Committee is choosing sites for the swimming, athletics, and eques-\ntrian events at the 2048 Games to be held in Romania. For example, let the first hypothe-\nsis be Site(Swimming, Pitesti), Site(Athletics, Bucharest), and Site(Equestrian, Arad).\nA great deal of reasoning must then be done to work out the logistical consequences and\nhence the desirability of this selection. If we want to consider Site(Athletics, Sibiu) in-\nstead, the TMS avoids the need to start again from scratch.\nInstead, we simply retract\nSite(Athletics, Bucharest) and assert Site(Athletics, Sibiu) and the TMS takes care of the\nnecessary revisions. Inference chains generated from the choice of Bucharest can be reused\nwith Sibiu, provided that the conclusions are the same.\n462\nChapter\n12.\nKnowledge Representation\nAn assumption-based truth maintenance system, or ATMS, makes this type of context-\nATMS\nswitching between hypothetical worlds particularly efficient. In a JTMS, the maintenance of\njustifications allows you to move quickly from one state to another by making a few retrac-\ntions and assertions, but at any time only one state is represented. An ATMS represents all the\nstates that have ever been considered at the same time. Whereas a JTMS simply labels each\nsentence as being in or out, an ATMS keeps track, for each sentence, of which assumptions\nwould cause the sentence to be true. In other words, each sentence has a label that consists of\na set of assumption sets. The sentence holds just in those cases in which all the assumptions\nin one of the assumption sets hold.\nTruth maintenance systems also provide a mechanism for generating explanations.\nEXPLANATION\nTechnically, an explanation of a sentence P is a set of sentences E such that E entails P.\nIf the sentences in E are already known to be true, then E simply provides a sufficient ba-\nsis for proving that P must be the case. But explanations can also include assumptions—\nASSUMPTION\nsentences that are not known to be true, but would suffice to prove P if they were true. For\nexample, one might not have enough information to prove that one’s car won’t start, but a\nreasonable explanation might include the assumption that the battery is dead. This, combined\nwith knowledge of how cars operate, explains the observed nonbehavior. In most cases, we\nwill prefer an explanation E that is minimal, meaning that there is no proper subset of E that\nis also an explanation. An ATMS can generate explanations for the “car won’t start” problem\nby making assumptions (such as “gas in car” or “battery dead”) in any order we like, even if\nsome assumptions are contradictory. Then we look at the label for the sentence “car won’t\nstart” to read off the sets of assumptions that would justify the sentence.\nThe exact algorithms used to implement truth maintenance systems are a little compli-\ncated, and we do not cover them here. The computational complexity of the truth maintenance\nproblem is at least as great as that of propositional inference—that is, NP-hard. Therefore,\nyou should not expect truth maintenance to be a panacea. When used carefully, however, a\nTMS can provide a substantial increase in the ability of a logical system to handle complex\nenvironments and hypotheses.\n12.7\nTHE INTERNET SHOPPING WORLD\nIn this final section we put together all we have learned to encode knowledge for a shopping\nresearch agent that helps a buyer find product offers on the Internet. The shopping agent is\ngiven a product description by the buyer and has the task of producing a list of Web pages\nthat offer such a product for sale, and ranking which offers are best. In some cases the\nbuyer’s product description will be precise, as in Canon Rebel XTi digital camera, and the\ntask is then to find the store(s) with the best offer. In other cases the description will be only\npartially specified, as in digital camera for under $300, and the agent will have to compare\ndifferent products.\nThe shopping agent’s environment is the entire World Wide Web in its full complexity—\nnot a toy simulated environment. The agent’s percepts are Web pages, but whereas a human\nSection 12.7.\nThe Internet Shopping World\n463\nExample Online Store\nSelect from our fine line of products:\n• Computers\n• Cameras\n• Books\n• Videos\n• Music\n