Chaos, Order, and Software Development

(Originally posted 2003.Mar.31 Mon; links may have expired.)

Kevin Kelly published Out Of Control: The New Biology of Machines, Social Systems, and the Economic World in 1994.

Jim Highsmith (James A. Highsmith III), wrote Adaptive Software Development: A Collaborative Approach to Managing Complex Systems in published in 2000, before he read Extreme Programming Explained: Embrace Change by Kent Beck, published in 1999.

The book on Scrum was published in 2001, and Highsmith's Agile Software Development Ecosystems was published in 2002.

What do these books have in common? Order (or the semblance of purpose) emerging from independent agents in situation that one might expected to be purely chaotic.

Kevin Kelly's book covers the most ground, of course, from bee hives and ant colonies, boot-strapping ecosystems, competing/cooperating agents within our brains, distributed control within robots, evolution and genetics, genetic algorithms, and so on. The theme is that order can arise "by itself" (it emerges, rather being designed up-front.)

The agile software authors are saying that a good software product can arise with a minimum of planning up-front. However, creating this order is not 'random', it arises from the constant thinking and re-thinking of the people involved through the life of the project.

Some people have objected to a lack of up-front planning or designing, making the analogy to a "hill-climbing" algorithm that gets stuck on a local maxima (getting stuck on a small hill, when the goal is a larger hill, a valley away). The difference of course, is that the hill-climbing algorithm is stupid, whereas many people are smart. People can see the big picture, and can do a little planning of refactoring to get from the current design to the desired design, even though the refactorings may temporarily go through a poor design along the way (but with all tests still passing!)

In a recent web-search, I came upon a paper written by a member of a group researching independent software agents. The paper was about how Extreme Programming is helping them write software successfully, allowing new programmers to become productive members of the team quickly, and how the process allows them to increase software re-use. That paper is here: Using Extreme Programming for Knowledge Transfer. Using XP to continue research like those described by Kevin Kelly... I don't have my copy of XPExplained or Adaptive Software Development handy, but I would expect Kelly's book, or books he references, to be referenced in the bibliography of Kent's and Highsmith's books. I enjoy feedback loops like that.