Stigmergy and the World-Wide Web

or how we all became ants building a nest we can't even see.

Weblogs, Neighborhoods, and Google

Weblogs, Neighborhoods, and Google are all phenomena of the World-Wide Web. All of these are fairly new and they are all very powerful. Weblogs are successfully taking on large publishers on their fact checking. A minor shift in Googles ranking algorithms creates huge ripples. Warchalking swept through the web and onto street corners in a matter of days. Is there any connection between them? I want to convince you that they are all intertwingled, all the result of the same phenomenon, and that we have much more interesting and powerful phenomenon on the horizon.

Weblogs

I have a weblog. You are reading it. Just a slim page among the billions on the internet. I stop by here several times a week and write up recent ideas, observations or just point to plain old weird stuff I have found on the internet. Stop and think about that for a minute. If this weren't on the web, what would be the closest analogy for what I am doing? Writing my thoughts and ideas down on a chalkboard on a street corner is the closest I can come. Imaging a street with chalkboards mounted at random points along the thoroughfare. I write my stuff on my chalkboard. Other people have their own chalkboards on the same street. Sometimes I walk by their boards, read what they have to say and if I find it interesting then either I scribble a little note on their board (if their weblog has a comment system) otherwise I run back to my board and scribble a note about what I read 'over there' and what I think about it. If I find something interesting happening on the street I go back to my board and write about it. We never speak directly. We only communicate through the boards. Writing on our own boards, or scribbling even smaller notes on other peoples boards is the only way we converse.

Now if you saw this happy little analogy playing out on the street you would do what any sane person would do; have everyone involved committed to a psychiatric ward. For pete's sake, you'd exclaim, just talk to each other or use a phone. But this is exactly what we're doing when we use our weblogs, communicating that is... not getting committed. If we want to talk so badly why not use email or pick up the phone? The weblog has got to be the single most inefficient mechanism for communication that has even been invented. Webloggers should be committed en masse. There's only one problem: It works.

Neighborhoods

I have a weblog. You are reading it. Just a slim page among the billions on the internet, oh wait, I already said that. But anyway look at the people in my neighborhood, as defined by Google. Or maybe the neighborhood found by Mark Pilgrim's Recommended Reading. Google and Mark's tool do a very good job of picking out similar sites with the vast majority of their recommendations already in my news aggregator. I can check them several times a day with my news aggregator while I still have to visit a few throwbacks manually. They write about stuff that is interesting to me, I hopefully do the same for them, and we often link among ourselves or a group of us will all link to the same story we found on the internet. There is no geographic grouping that supports this grouping, these people are from all over the planet. Billions of pages, millions of users and yet here is a tight clustering of people with similar interests, a grouping of people that know about each other.

Google

Google is the premier search engine in the world. Here is their description of their basic PageRank algorithm:

PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page's value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. But, Google looks at more than the sheer volume of votes, or links a page receives; it also analyzes the page that casts the vote. Votes cast by pages that are themselves "important" weigh more heavily and help to make other pages "important." [Google Technology]

Memes

A meme is a unit of intellectual or cultural information that survives long enough to be recognized as such, and which can pass from mind to mind. They can be carried by word of mouth, dead trees, e-mail, or the web. On the web, in particular on weblogs, memes are tracked by by links to particular sites or stories, thus the rise of blogdex, daypop and popdex. Memes just in the realm of newspapers are now tracked by news.google.com which does an excellent jobs of summarizing the news the of the day automatically, no human internvention required, by collectively scanning thousands of news websites and pointing to the stories that are being talked about the most.

In a similar vein periodical bursts of activity will swirl through the weblogs. Examples of such bursts of activity include the road to RSS 2.0, RSD [More on RSD], neighborhood explorers, TrackBack, XHTML Syndication, and StructuredText (aka Textile aka WikiML). Is this all just serendipity?

Why?

Why do memes spread so easily on the web? Why is Google's PageRank algorithm so successful? Why do Neighborhoods form? Why does communicating through a weblog work at all? What is stigmergy?

What is stigmergy?

Stigmergy, a term coined by French biologist Pierre-Paul Grasse[2] is interaction through the environment.

Self-Organization in social insects often requires interactions among insects: such interactions can be direct or indirect. Direct interactions are the "obvious" interactions: antennation, trophallaxis (food or liquid exchange), mandibular contact, visual contact, chemical contact (the odor of nearby nestmates), etc. Indirect interactions are more subtle: two individuals interact indirectly when one of then modifies the environment and the other responds to the new environment at a later time. Such an interaction is an example of stigmergy. [1, p.14]

Examples from nature

Ant Trails

Ants are the premier example of organisms that use stigmergy. Did you ever drop a bit of food on the ground outside and return to later find it swarming with ants, all following a single narrow trail back to their nest? Why do they settle on just one path? How do they find the food and get everyone to use that same path? The answer is stigmery. When an ant finds food it heads back to the nest leaving a pheremone trail, a trail of scent that will evaporate over time. Other ants that come across the trail will follow it and if they find food will also head back to the nest also leaving a trail, re-enforcing the scent. The more food, the more ants, the stronger the scent on the trail. Once the food is gone the trail evaporates. Not only do ants use stigmeric communciation to find the food, they also end up finding the shortest route to the food. If multiple trails are found then the shortest route will dominate because the shortcut will result in more trips per ant per unit of time and thus the scent on that trail will be stronger and draw more ants.

Termite nest-building

Termites in Africa can build incredible nests that are up to 30 feet tall that contain many tunnels and chambers. These tall complex structures are used to regulate temperature, humidity and oxygen in the nest which is all carefully controlled to promote the growth of the fungus which the ants eat. The construction of the termite mound is controlled through stigmergy, in particular the construction of columns and arches is controlled by the evaporation of pheromones. The ants place pheremones on the balls of mud they use for construction. The new balls are placed near existing pheromone concentrations. As construction proceeds and the columns get taller the pheromones near the bottom evaporate. If two columns are built near each other then the concentration of pheremones at the top of the columns will cause the two to be joined into an arch.

Ant corpse-gathering

Ants are dying all the time and their little corpses are gathered into a single burial mound. This may seem symbolic but it is functional and easily coordinated through stigmergy, in this case the stigmeric communication is not through pheromones but the corpses themselves. Ants pick up their fallen brethren and put them in piles, with a preference for larger piles over smaller piles. Ants can also pick a corpse from an already formed pile and move it to another pile, again preferring to drop corpses in larger piles over smaller piles. Many ants will perform these actions over time and over time all the piles end up merged into a single large pile.

Web = Stigmergy

And now finally we get the point. The whole point of this is the connection between the web and stigmergy. The World-Wide Web is the first stimeric communication medium for humans.

The telephone and email don't count as stigmeric communication since they are only readable by the people on either end of the phone call, or the e-mail. In order for an environment to support stigmeric communciation the messages must be readable by everyone. Radio and TV don't count since they are a read-only medium as far as most people are concerned. In order for an environment to support sitgmery everyone has to be able to not only read it but to be able to write into it also.

Oh sure, we have had books and newspapers, but for the vast majority of people the only avenue they have to 'write-back' into that environment is in the 'letter-to-the-editors' department. Now we have groups.yahoo.com, K5, Slashdot, and weblogs. All avenues for anyone to enter into the conversation. Avenues for anyone to place a mub ball here or an ant corpse there. Yes, I know, not everyone has access to the internet, and it certainly isn't free for most folks, but the point is that it is the most accessible system yet and prices are falling and more people are getting on everyday.

Note that some other internet based communciation forms have moved onto the internet. Many mailing lists now have web accessible archives. Groups.yahoo.com takes it even further and allows posting from the web to the list as well. Another notable migration has been of usenet newsgroups onto the web via the much lauded groups.google.com. In both of these cases the mailing list and newsgroups were enhanced from being able to be searched and linked to from the web.

Now that we know web is a stigmeric communciation medium and that we've seen some of the power that nature has gotten out of stigmergy the answers to our earlier questions become rather easy.

Weblogs

Why does communicating through a weblog work? Stigmergy.

Using a weblog is communcicating through stigmergy. Just like an ant, as I blog I leave a trail of information and links to other information I find interesting.

Google

Why is Google's PageRank algorithm so good? It is just following the Ant Trails.

If links represent a dropping of pheremone then Google is just following the trails laid down to the tastiest morsels.

Neighborhoods

Why do Neighborhoods form? Ant Corpse Piles.

Just like Ant Corpse Piles, if I link to you and you link to me that brings our weblogs closer together. The more we talk about similar stuff the more likely we are to cross link to each other. The more links to each other and the more links from us to similar material on the web the closely Google thinks we are related. The habit of 'welcoming' new bloggers with simlar interests by linking to their site with a welcome message only grows the pile.

Memes

Why do memes spread so effectively on the web? Stigmergy.

Because they are travelling through a stigmeric medium. They can live on the internet where anyone can find them either intentionally, by using Google to follow the trail, or serendipitiously by the idea moving into a receptive neighborhood.

Summary

The World-Wide Web is human stigmergy. The web and it's ability to let anyone read anything and also to write back to that environment allows stigmeric communication between humans. Some of the most powerful forces on the web today, Google and weblogs are fundamentally driven by stigmeric communication and their behaviour follows similar natural systems like Ant Trails and Nest Building that are accomplished using stigmergy. The web is new. In the context of written human history is barely a blink of an eye. Yet as new as the web is, it is already showing it's ability to support complex human interactions that mimic natural systems use of stigmergy. And were just getting started...

Further Reading

Some of the things I have described follow from fairly simple interactions. Here are some of the things Ants can accomplish using stigmergy:
A Superorganism's Fuzzy Boundaries
Structure of Macrotermes mounds

The classic paper on why the web is successful is Clay Shirky's In Praise of Evolvable Systems.

Learning to view the world in a decentralized way is hard. Here's some reasons why and how to break out of the habit: Learning About Life

Bibliography - Dead Tree References