Complexity and Emergence : Chaos and Complexity ( Part One )
Michael is Research Associate at Henley Management College, U.K, and a Senior Advisor, Tripod Inc. He is also Director, Organization Science Related Programs, at the New England Complex Systems Institute.
Somehow Michael finds time to be Editor-in-Chief of "Emergence: A Journal of Complexity Issues In Organizations and Management".
Michael can be contacted at email@example.com or at lissack.com
The last few years have seen an extraordinary growth of interest in the study of complex systems. From ecology to economics, from particle physics to parallel computing, a new vocabulary is emerging to describe discoveries about wide-ranging and fundamental phenomena. Many of the terms have already become familiar: artificial life, biocomplexity, cellular automata, chaos, criticality, fractals, learning systems, neural networks, non-linear dynamics, parallel computation, percolation, self-organization, and many more.
As the general public has become more familiar with the vocabulary of complexity, the ascribed claims for "complexity theory" have also increased. Critical rebuke came in the form of a Scientific American article last June entitled, "From Complexity to Perplexity: Can science achieve a unified theory of complex systems ? Even at the Santa Fe Institute, some researchers have their doubts.''
The reality is that complexity theory is in practical use every day by companies across the globe. The uses, however, are far more modest than a "theory of everything." Complexity theory research has allowed for new insights into many phenomena and for the development of a new language. This paper argues that a shared language based on the vocabulary of complexity can have an important role in a management context. Simulated annealing, for example, translates into ascribing a creative value to "noise" and seeking to make use of that value -in one example, by bringing outside perspectives into focus groups at critical moments when making decisions. The notion of "fitness landscapes" and their "correlation length" to describe technological competition can similarly provide new direction to the management of research efforts and marketing.
Two case studies - one of a high tech bio-engineering firm, the other of an Internet content provider - are presented to illustrate such practical applications. In addition, the results of a series of focus group meetings from Santa Fe are presented to further illuminate such concepts as the use of "NK models, patches, tau theory, and flocking."
This paper also argues that rather than viewing management control as impossible, a different kind of "control" is needed - one that influences without force and without guarantee of detail, but with confidence in patterns. The use of the vocabulary of complexity theory to develop a shared language among decision makers and important actors can be used to provide that confidence.
Complexity theory deals with systems that show complex structures in time or space, often hiding simple deterministic rules. This theory contends that once these rules are found, it will be possible to make effective predictions and even to effectuate control of the apparent complexity. The appeal of "making the complex simple" and of extending spheres of control has led to many misunderstandings of what complexity theory ( if there is a complexity "theory" ) is and what relevance it may have to the world in which we live.
Corporate managers can be seduced by the popular vocabulary associated with complexity. Such a seduction can lead to new ways of understanding existing problems or it can lead to a general dissatisfaction with the entire concept. "Complex," "chaotic" and "complicated" seem to be used with little distinction in common language. While each of these terms refers to an inability to easily make sense of a situation, within the scientific community these words have very different meanings. If one does not understand the conceptual framework which supports the language of complexity, one cannot understand the vocabulary. The confusion between the common and the scientific can easily lead to disillusionment.
Perhaps the most prominent example of the disillusioned is to be found in Horgan, who questions the desire for and the potential achievement of a "unified theory." Horgan quotes John Holland: ``Many of our most troubling long-range problems - trade balances, unsustainability, AIDS, genetic defects, mental health, computer viruses - center on certain systems of extraordinary complexity. The systems that host these problems - economies, ecologies, immune systems, embryos, nervous systems, computer networks - appear to be as diverse as the problems. Despite appearances, however, the systems do share significant characteristics, so much so that we group them under a single classification . . . This is more than terminology. It signals our intuition that there are general principles that govern all [such] behavior, principles that point to ways of solving the attendant problems.''
At Williams College the main stairs are inscribed "Climb high, climb far, your goal the sky, your aim the star." Too often, it seems, in the popular world of complexity the aim has been confused for the goal. If we define an organization as "systems of coordinated actions among individuals and groups whose preferences, information, interests and knowledge differ" ( March and Simon, 2 ), then the central task of an organizational manager is "the delicate conversion of conflict into cooperation." Such a conversion can be greatly assisted when the participants speak a common language and share access to somewhat similar views of the world they inhabit. The better the cooperation the better the results.
Complexity research and complexity theory have demonstrable applications which corporate managers can avail themselves of today. We do not need to wait for a "unified theory," instead we need to exploit some of the research results along the way. The aim of this paper is to illuminate some of those steps.
Two case studies - one of a bio-tech firm, another of an Internet content provider - are presented along with summary insights gained from a series of focus groups held under the auspices of the Santa Fe Center for Management Strategy. The central idea gleaned from these investigations is that the complexity metaphor can be a powerful tool for organizing a corporate response to the competitive environment.
Complexity research is not at the point of describing an underlying theory of everything. But its descriptive powers are at a point where they can help to shape the world around us. Meanings and metaphors matter. ( Lakoff ) The meanings that we give to ourselves, our products, our competitors, our customers, and all the relevant others in our world determine the space of our possible actions - and, to a large extent, how we act. ( Lane and Maxfield )
Because the scientific and management communities seem to embody the very words used in complexity research with a variety of meanings, it is essential to begin with a set of definitions. M ike McMasters has suggested the following as "operational definitions for management and organizational purposes:
- chaotic refers to a state where patterns cannot be made nor details understood,
- complicated refers to a state where patterns cannot be made but details, parts and subsystems can be understood, and
- complex refers to a state where the details cannot be understood but the whole ( or general result ) can be understood by the ability to make patterns."
I will be using these distinctions throughout with apologies to members of the scientific community who will note that I have made no reference to sensitive dependence on initial conditions, Hurst ratios, Lyapunov exponents, Lorentz Lattice Gas Cellular Automata, etc. It is my observation that such meanings have little relevancy to the practicing manager.
Phelan notes: "Chaos theory has the potential to contribute valuable insights into the nature of complex systems in the business world. However, care must be taken when reading popular accounts of chaos in the management literature. As is often the case with the introduction of a new management metaphor, "chaos" tends to be suddenly seen in almost all managerial systems ( Levy, 1994 )."
Yet what is it that these management experts now see ? A lack of ability to control. "It would seem that increased turbulence in the business world and the widely-reported accelerating rate of change is sufficient to label the system as chaotic. Despite the limited evidence, bold claims are being made about chaos theory being the `next major breakthrough in management'." ( Phelan ) Leading authors agree that " . . . if one accepts the premise that the dynamic of success is chaotic . . . all forms of long-term planning are completely ineffective" ( Stacey, 188 ).
Returning to Phelan: "In the absence of any ability to plan or control the future, managers are urged to develop an adapting stance and a preparedness to react to unexpected and unanticipated events. The term, organizational learning, is often used to describe the process whereby groups and individuals within the organization challenge existing mental models of behavior and learn to rapidly and creatively adapt to a changing environment. In the absence of foresight, it is clear that a competitive advantage may be gained by effectively adapting to novel and unpredictable situations faster than the competition."
So far so good, but the question is how ? Remember March and Simon - "the central task of an organizational manager is the delicate conversion of conflict into cooperation." The descriptive metaphor that everything is changing and thus the organization must be poised to adapt to change says nothing about what to do next or about how to convert conflict into cooperation. Indeed the material may be useful as a sales pitch for organizational change, but the product - what to change, how, when and why - is not connected to the management experts' observation regarding loss of control. Something more is needed.
Let us turn to other metaphors found in complexity research. First, look at the concept of a "fitness landscape." This term, first used by Wright in the 1930's in the field of evolutionary biology, has been greatly expanded by complexity researchers. A fitness landscape is a "mountainous terrain showing the locations of the global maximum ( highest peak ) and global minimum ( lowest valley ) [and] the height of a feature is a measure of its fitness." ( Coveney and Highland, 108 ) Competition can be said to occur on a fitness terrain. That terrain itself is not fixed but changes and deforms as the actors within its sphere act and change and as the general environment changes. Kauffman states, "Real fitness landscapes in evolution and economies are not fixed, but continually deforming. Such deformations occur because the outside world alters, because existing players and technologies change and impact one another, and because new players, species, technologies, or organizational innovations, enter the playing field. ( Kauffman and Macready ) Fitness landscapes change because the environment changes. And the fitness landscape of one species changes because the other species that form its niche themselves adapt on their own fitness landscapes . . . "( 208 ) We can construct such a landscape for any system of connected interactions ( such as a firm and its environment ), and it is the presence of "conflicting constraints that makes the landscape rugged and multi peaked. Because so many constraints are in conflict, there are a large number of rather modest compromise solutions rather than an obvious superb solution. ( 173 )"
Second, look at the term "attractor." An attractor is a model representation of the behavioral results of a system. The attractor is not a force of attraction or a goal-oriented presence in the system, but simply depicts where the system is headed based on the rules of motion in the system. ( Coveney and Highland, Cohen and Stewart ) I.e. the term is descriptive, but the word itself seems to imply a prescriptive force. For the practicing manager, one more word must be added - "passive." The attractor just is, it is a passive being not an active force. But, being passive it means that the actors can drift from one attractor to another. Again the key question is how.
The language of complexity and the metaphors it creates can change the definition of management
Lakoff and Johnson ( 14 ) argue that "many of our activities ( arguing, solving problems, budgeting time, etc. ) are metaphorical in nature. The metaphorical concepts that characterize those activities structure our present reality. New metaphors have the power to create a new reality." Corporate managers tend to view their companies as being in a race - be it for success, market share, revenues, or survival. That metaphor influences the way they see the world and the way they manage their companies.
In the race metaphor, the landscape is fixed even if the course is not. One has an identified goal and a set of competitors. In the fitness landscape metaphor, the landscape itself is always changing. One's goals, course, and competitors are but factors which can and do affect the shape of the landscape itself. The objective is to climb to a non-local peak and your peaks may be very different from your competitors.
Similarly, marketing literature and economic textbooks each purport to be able to describe the behaviors of consumers. Economic man has behaviors which can be described by patterns or in the words of complexity by "attractors." The metaphorical difference is huge.
In the metaphor of patterns, decisions fall into a set of predictable behaviors which need reinforcement. In the metaphor of attractors, the actions are self sustaining. Once in the basin of an attractor an actor will behave much as the attractor outlines. The problem becomes one of classification - which basin of attraction is the client, target, customer, whomever in ? The basin defines the action. Because the attractor is passive, reinforcement of old messages, to promote an active choice, is but a waste of time and money.
In the race metaphor, information and data can be confused. Too much data lead to a loss of vision, a potential diversion from the goal, and the risk of overload. In the complexity metaphor, data is merely unused potential information. Information changes the landscape and data becomes information when it is ascribed value ( whether correctly or not ). Noise is a risk and a diversion in the race metaphor and a source of new understanding and potential information in the complexity metaphor.
To quote Lakoff and Johnson ( 179 ), "We understand a statement as being true in a given situation when our understanding of the statement fits our understanding of the situation closely enough for our purposes." ( 46 ) "...metaphors partially structure our everyday experience and this structure is reflected in our literal language." Managers who can make use of the metaphors of complexity see their companies in a different light than those who don't, and in a sense, are competing in a different world.
One of the key roles of a manager is to lay out strategy for his or her organization. But what is a strategy ? Lane and Maxfield comment as follows:
"What is a strategy ? Once upon a time, everybody knew the answer to this question. A strategy specified a precommitment to a particular course of action. Moreover, choosing a strategy meant optimizing among a set of specified alternatives, on the basis of an evaluation of the value and the probability of their possible consequences. Optimizing precommitment makes sense when a firm knows enough about its world to specify alternative courses of actions and to foresee the consequences that will likely follow from each of them. But of course foresight horizons are not always so clear. The world in which you must act does not sit passively out there waiting to yield up its secrets. Instead, your world is under active construction, you are part of the construction crew - and there isn't any blueprint. Not only are the identities of the agents and artifacts that currently make up your world undergoing rapid change, but so also are the interpretations that all the agents make about who and what these things are.
Strategy lies between directedness and execution. It lays down 'lines of action" that the firm intends to initiate and that are supposed to bring about desired outcomes. Since outcomes depend on the interactions with and between many other agents ( inside and outside the firm's boundaries ), strategy really represents an attempt to control a process of interactions, with the firm's own intended "lines of action" as control parameters.
When the foresight horizon is clear, it may be possible to anticipate all the consequences of any possible course of action, including the responses of all other relevant agents, and to chart out a best course that takes account of all possible contingencies. As foresight horizons become more complicated, the strategist can no longer foresee enough to map out courses of action that guarantee desired outcomes. Strategy then must include provisions for actively monitoring the world to discover unexpected consequences, as well as mechanisms for adjusting projected action plans in response to what turns up.
In contexts like this, the relation between strategy and control is very different from the classical conception. It is just not meaningful to interpret strategy as a plan to assert control. Rather, strategy must be seen as a process to understand control: where it resides, and how it has been exercised within each of its loci. With the insights gleaned from these processes, each agent can orient itself in agent/artifact space - and figure out how it might reorient itself there, in the future."
Thus in a complex world, strategy is a set of processes for monitoring the behaviors of both the world and of the agents of the organization, observing where attractors are and attempting to supply resources and incentives for future moves. Command and control are impossible ( at least in the absolute and in the aggregate ), but the manager does retain the ability to influence the shape of the fitness landscape.
As Kauffman phrases it, "Adaptive organizations need to develop flexible internal structures that optimize learning. That flexibility can be achieved, in part, by structures, internal boundaries, and incentives that allow some of the constraints to be ignored some of the time. Properly done, such flexibility may help achieve higher peaks on fixed landscapes, and optimize tracking on a deforming landscape."
We then come back to "How ?"
The key variables which complexity research seems to suggest are important in answering "How ?" are :-
- defining the size and number of business units
- defining the nature of interactions among these units
- arriving at a language or other mechanism for information to be passed among and processed by these units
- defining a strategy for "search" - the hunt for improvements out amongst the fitness landscape
- making room for and use of "noise".
Kauffman carried out a number of studies of search on rugged landscapes which demonstrate that, when fitness is average, search is best carried out far away across the space of possibilities. But, as fitness increases, the fittest variants are found ever closer to the current location in the space of possibilities.
On complex surfaces ( i.e., rugged fitness landscapes with many hills and valleys ) systems can become trapped on poor local optima ( the wrong hill ). Kauffman's research has developed a variety of approaches to "simulated annealing" to assist in getting organizations away from these local optima and moving toward a more global optimum. "Simulated annealing is an optimization procedure based on using an analogue of temperature, which is gradually lowered so that the system nearly equilibrates at each temperature and is gradually trapped into deep energy wells. The general concept lying behind simulated annealing is that at a finite temperature the system sometimes "ignores" some of the constraints and takes a step "the wrong" way, hence increases energy temporarily. Ignoring constraints in a judicious way can help avoid being trapped on poor local optima. " ( Kauffman )
One such procedure he calls "patches" - "partitioning a system on a complex, rugged fitness landscape into independent departments, or patches, each of which thereafter optimizes selfishly. Because the departments are independent and selfish, actions by one department to improve itself can move the entire system "the wrong way," hence [those independent actions] can [allow the entirety to] avoid bad local minima."
Kauffman suggests "flatter, decentralized organizations . . . business, political, and otherwise might actually be more flexible and carry an overall competitive advantage." By flatness, Kauffman means an organization designed around a relatively flat fitness landscape - one without many jagged peaks and valleys, but not a piece of slate either. By decentralized, Kauffman means an organization broken "into 'patches' where each party attempts to optimize for its own selfish benefit, even if that is harmful to the whole." Such a structure "can lead, as if by an invisible hand, to the welfare of the whole organization."
This clearly is a "how." Break up the organization into patches. Kauffman emphasizes that these patches must interact. Thus, this advice is clearly different from the old management standby of the independent self-sufficient business unit. It is in the nature and quantity of the interactions that Kauffman finds that the organization as a whole can be moved toward a global optimum even though each patch is acting selfishly. Interactions require language or some other mechanism of fairly continual communication. He stresses that the patches must be coupled. In management lingo, the pieces must communicate, and not just at quarterly review sessions.
"The basic idea of patch procedure is simple: take a hard, conflict-laden task in which many parts interact, and divide it into a quilt of non-overlapping patches. Try to optimize within each patch. As this occurs, the couplings between parts in two patches across patch boundaries will mean that finding a "'good" solution in one patch will change the problem to be solved by the parts in adjacent patches. Since changes in each patch will alter the problems confronted by neighboring patches, and the adaptive moves by those patches in turn will alter the problem faced by yet other patches, the system is just like our model coevolving ecosystems."
Kauffman further finds that as coupling increases and thus drives the system towards chaos, an ameliorating effect is derived by slicing the system up into smaller patches. In other words, "hard problems with many linked variables and loads of conflicting constraints can be well solved by breaking the entire problem into non-overlapping domains. Further . . . as the conflicting constraints become worse, patches become ever more helpful."
Kauffman's other two procedural suggestions are to ignore some of the inputs coming into the organization ( the theory seems to be that accommodating all inputs leads to freezing and that the necessary degrees of freedom to better find optima can only be accomplished by deliberately ignoring some of the inputs ), and to recognize that too much data cease to be information ( that which informs the agent or actor ) but instead acts like a brake on the system. This latter concept he calls "tau" for his measure of how many simultaneous changes an interacting system can tolerate before freezing up.
"While simulated annealing is an important mathematical optimization procedure, it involves agents making errors "on purpose" at a controlled diminishing frequency. People and organizations do not appear to behave this way. On the other hand, human agents in organizations almost certainly respond to the stresses of conflicting demands by ignoring some of the constraints some of the time. Such apparent irresponsibility can, as we have seen, work to the overall benefit of the organization." Ignoring some of the information being generated allows for a new set of priorities to emerge, and at least some of the time, a significant improvement can result.
Kauffman describes the role of error and of ignoring some constraints. The annealing process can also be looked at as one of deliberately introducing noise into a system to see what happens.Guastello ( ch. 4 ) refers to this as " . . . the chaotic controller. Chaotic controllers are based on the law of requisite variety, which is an engineering principle that posits that the controller of a system needs to be at least as complex as the system it intends to control. Chaotic systems obviously need something special. Chaotic control works counter intuitively by first adding a small amount of low-dimensional noise into the system. The reasoning is that the amount of sensitivity to initial conditions is not uniform throughout the attractor's space; sensitivity is less in the basin of the attractor and least in its center . . . Adding noise to the system allows the attractor to expand to its fullest range."
This is a very different concept of noise than the traditional. Where traditional managers may have wished to delete the extraneous, the complexity research educated manager may be attempting to cause the deliberate addition of noise at various places along the way.
Of course, noise can still be noise, and search strategies must be able to separate wheat from chaff if the enterprise is to be at all successful.
What we find when we search is a function of how we look.
Timmermans' study found "that the selection, evaluation and integration of information are clearly affected by the complexity of the problem. Results concerning information selection show that, although the total amount of information processed increased with the number of alternatives, the average number of attributes used per alternative decreased. Another finding is that the alternative which is finally chosen is evaluated more extensively on more attributes than the non-chosen alternatives. In addition, more references were made to the selected alternative as compared to the remaining alternatives."
He went on to indicate something many of us have observed - namely that managers make decisions from gut feel after whatever holistic input they feel appropriate. "Further analysis of the decision process revealed that the evaluation of the information was most often absolute ( i.e., no comparisons were made between alternatives ) and usually referred to a specific aspect of an alternative and not to the alternative as a whole. Less than 15% of the judgments compared two or more alternatives on a specific aspect. Increasing the number of alternatives reduced the tendency to compare alternatives on a specific aspect ( comparative dimensional judgments ) and increased the tendency to evaluate alternatives as a whole ( absolute holistic judgments )."
If search strategy on a fitness landscape is important, and if judgments are being made holistically, it becomes all the more important to have searches "out past the correlation length" - which incorporate "noise" - which can better inform the holistic judgment. The more powerful the searching device, the more creative its input, the better the ability to find an uncorrelated peak for further exploration.
Brian Arthur notes: "What counts to some degree, but only to some degree, is technical expertise, deep pockets, will, and courage. The rewards go, above all, to the players that are first to make sense out of the new games looming out of the technological tog, to see their shape, to cognize them. The much-discussed Bill Gates is not a wizard of technology. He is a wizard of precognition, of sussing out the next game . . . You cannot optimize in the casino of ill-defined games. You can be smart. You can be cunning. You can position. You can observe. Adaptation, in the pro-active sense, means watching for the next wave that is coming, figuring out how it will work, and setting the company up to take advantage of it."
Lane and Maxfield suggest, "To succeed, even survive, in the face of rapid structural change, it is essential to make sense out of what is happening and to act on the basis of that understanding. . . . But of course making sense isn't enough. Agents act - and they act by interacting with other agents. In complex foresight horizons, opportunities arise unexpectedly, and they do so in the context of generative relationships. By "generative relationship," we mean a relationship that can induce changes in the way the participants see their world and act in it and even give rise to new entities, like agents, artifacts, even institutions. In this context, the most important actions that agents can take are those that enhance the generative potential of the relationships into which they enter. As a result, agents must monitor relationships for generativeness, and they must learn to take actions that foster the relationships with most generative potential. Then, when new opportunities emerge from these relationships, agents must learn to set aside prior expectations and plans and follow where the relationships lead."
"Two kinds of strategic practices are particularly important when foresight horizons are complex. Through the first, agents seek to construct a representation of the structure of their world that can serve them as a kind of road map on which to locate the effects of their actions. Through the second, agents try to secure positions from which distributed control processes can work to their benefit." Constructing a representation of the world - I believe these are words for developing a language, vocabulary, and set of metaphors with which to communicate with each other and with others. Exhibit I illustrates some of the metaphors and their practical applications.
With this as background we now turn to our two case studies.