About Serendipity

The art of making unsought findings

Foreword

Back in 2007 I was studying for my Computer Science Bachelor at the University of Bari. I was really excited about Search Engines and Recommender Systems, but I was quite critic about them, to no surprise of anyone who knew me at least a little. During one of Prof. Semeraro's lessons, when he was explaining the content based approach for recommending new item to users, I asked him "this way the algorithm will recommend only similar items. How do you solve this problem?". He answered "Have you ever heard the word serendipity?". It open up my mind and made it the topic of my thesis. My goal was to find a way out what in 2009, two years before the idea was popularized by Eli Pariser in his "The Filter Bubble", I called "mind cages" parapharasing Sting's Soul Cages: information search and filtering algorithms can lead to exposure to a small subset of all possible items (news, books, media, points of view) and this, as described by Ethan Zuckerman, can lead to isolation and extremization of opinions, or, more trivially, the impossibility to discover something new. The way out of it was a simple algorithm that using analogies tried to suggest potentially serendipitous items. The first of incarnation of this study is, to date, my most cited paper, even if the methodology was really unripe and now I'm a bit ashamed of it. The second incarnation was much more sophisticated and was described in my thesis, but was never published in english.
Since then I moved on to work on a variety of different other topics (question answering, machine learning, semanics, deep learning), but the topic of serendipity re-emerges from time to time in my mind. The last time it re-emerged was during the summer of 2016. I just joined Geometric Intelligence and got hooked by the book written by two colleagues, Joel Lehman and Kenneth Stanley, titled Why Greatness Cannot Be Planned. It become my favourite book from 2016. It touched several topics all tied together by the idea that novelty is the way unexpected discoveries happen, in contrast with the blind following of predetermined goals. This resonated really well with me and has a strong connection with my research on serendipity. So I decided to translate the part of my thesis that summarizes the relationship between serendipity, information seeking and creativity so that Joel and Ken could read it and we could talk about it. Maria Monno helped me in this time consuming task of translating the complex and baroque Italian prose of my inexperienced 20 years old self in a more straightforward essay in English.
After sharing it with Joel and Ken they encouraged me to publish it in the informal way of a blog post, so here we are. I hope that this reading would be interesting and that you will forgive the naivety and ingenuity of my younger self writing and appreciate the content and the connections I was trying to draw.
This article was also published on Medium.

About Serendipity

The origins of Serendipity

"Serendipity is the art of making unsought findings." - Pek van Andel

"Serendipity. Look for something, find something else, and realize that what you've found is more suited to your needs than what you thought you were looking for." - Lawrence Block

"Serendipity is looking in a haystack for a needle and discovering a farmer's daughter." - Julius Comroe Jr.

“It's a bizarre but wonderful feeling, to arrive dead center of a target you didn't even know you were aiming for.” - Lois McMaster Bujold

The word "serendipity" was coined in 1754 by Horace Walpole and means "to make discoveries thanks to chance and sagacity of things you were not looking for". It derives from the Persian tale Walpole describes in his letter to Horace Mann on 28th January 1754, where the word was used for the first time:

"It was once when I read a silly fairy tale, called The Three Princes of Serendip: as their highnesses travelled, they were always making discoveries, by accidents and sagacity, of things which they were not in quest of: for instance, one of them discovered that a mule blind of the right eye had travelled the same road lately, because the grass was eaten only on the left side, where it was worse than on the right - now do you understand serendipity? One of the most remarkable instances of this accidental sagacity (for you must observe that no discovery of a thing you are looking for, comes under this description) was of my Lord Shaftsbury, who happening to dine at Lord Chancellor Clarendon's, found out the marriage of the Duke of York and Mrs. Hyde, by the respect with which her mother treated her at table."

The origins of the tale are convoluted: in 1557 in Venice, Michele Tramezzino published a book in Italian named "Peregrinaggio di tre figluoli del re di Serendippo" (Pilgrimage of the three sons of the King of Serendip), which was translated into French first and then from French into English, the version that Walpole read. Tramezzino declared he adapted a story dating back to 1302 contained in the first book of the "Hasht Bihisht" (Note 1) by Amir Khusro, an Indian scholar, poet and musician of Persian origin, translated into Italian by Cristoforo Armeno. The protagonist of the story is Giaffer, the king of Serendip, modern-day Sri-Lanka, and his three sons. This imaginary character is based on Bahram V (Bahram Gur), emperor of the Sasanian Empire from 420 to 440 b.C.. The story describes the king as concerned about the education of his three sons: he wanted them to inherit not only his power and domain, but also his virtues. For this to happen, he decided to let them be tutored by the best teachers, who constantly had to report to the king about the progress of the three princes. However, the king decided to test his children and proposed to each of them to succeed him on the throne of Serendip. All the three of them declined the proposal, acknowledging the greater wisdom of their father. The king, pretending to be enraged by their refusal, decided to send them away to visit foreign countries. During their travels, before coming back to their father, the three princes encountered a series of challenging situations that tested their knowledge against real-world circumstances and showed their sagacity. The three princes returned home to their father, grown up and morally enriched far beyond their father’s expectations.
Based on Walpole's definition, it is easy to note that many of the most important discoveries and inventions in the history of human kind happened thanks to serendipitous events. Examples range from Christopher Columbus’ (re-)discovery of America, the invention of dynamite by Alfred Nobel, the discovery of the properties of penicillin by Alexander Fleming, the discovery of the psychedelic effects of LSD by Albert Hofmann, the discovery of the structure of benzene by Friedric August Kekule and several others. These are all examples of the importance of serendipity in the progress of humankind.

Serendipity and Scientific Discoveries

"The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!', but 'That's funny…'" - Isaac Asimov

M. K. Stoskopf (Foster 2003) was one of the first scholars to understand the importance of serendipity in science. He asserts that serendipitous discoveries have a valuable role in the history of scientific progress and sometimes they have pushed intellectual strides considerably.
In (Roberts 1989), the focus of the analysis was on how much serendipitous discoveries depend on the information seeker, in this case the researcher. In particular, to get to this kind of discoveries, an open-minded attitude is needed, which prevents the researcher from discarding a-priori inputs coming from varied sources. It takes curiosity towards what might not seem directly useful for the purpose of the research, and it takes a wide culture. Such features enable the acknowledgment of serendipity when it manifests. The following parts of this dissertation will be focused on this particular aspect.

Serendipity and Information Seeking

“With a library it is easier to hope for serendipity than to look for a precise answer.” ― Lemony Snicket, When Did You See Her Last?

In (Erdelez 1997), the authors interviewed among regular library attendees. It emerged that these individuals were a lot more inclined to serendipity compared to the rest of the population, to the extent that they were defined as “super-encounters”. The interviewees also asserted to feel “the pressure of the amount of information waiting to be discovered”.

In a study by (Ross 1999), 194 regular readers were interviewed. Here, it emerged that many of them could find solutions to their problems mainly in leisure reading; solutions that they were able to find “without actually looking for them”. A further study carried out by (Williamson 1998) investigated the approach to information seeking in 202 Australian adults by registering their conversations. It revealed that the acquisition of new information among them happens mostly accidentally.
Even though these studies do not involve digital information systems, they provide evidence of the role of serendipity in the process of information acquisition. This must be certainly taken into account when designing information retrieval and filtering systems.
The leading thread of all these studies is the strong link between serendipitous discoveries and the subject’s implicit or explicit need for information. It is evident that serendipity is strongly personal and that personality plays a fundamental role in the ability to recognize serendipity. This makes any attempt to conceptualize and factually analyze this phenomenon very difficult.
(Foster 2003) already described this difficulty of studying serendipity. By nature, serendipity escapes any systematic control or possibility of prediction, which are fundamental elements if one wants fostering it. Nonetheless, and despite all the complications originating from such an fuzzy concept, serendipity must be considered as a fundamental element in the complex phenomenon of information seeking.
A systematic work about the different activities in information search and their relation to serendipity was carried out by (Toms 2000). He identifies three main activities:

The author claims that, whereas serendipitous encounters are useless in the first two activities, they are extremely important in the third. However, it is important to note that the first two categories are voluntary acts of information research, while the third can occur without a specific intention and can happen while performing the first two activities. Serendipity can play a fundamental role to accomplish the first two tasks even if it's not necessarily needed.
Moreover, there is a certain intensity of emotional response associated to serendipitous discoveries. It is difficult to grasp with a strict mathematical evaluation system, but it enhances greatly the satisfaction embedded in the discovery. Suggesting to the information seeker something new, never seen before, but at the same time interesting, he can be encouraged to make serendipitous discoveries.
In the deep enquiry by (van Andel 1994), the author suggests that the occurrence of serendipity cannot be programmed. Therefore, since it is undeniable that serendipity cannot be predicted, the purpose of a modern information seeking system should be not to program serendipity, but to program for serendipity. Which means, trying to create the conditions for serendipity to happen.

Formalizations of Serendipity

So far, I talked about how blurred the concept of serendipity is, but as our goal is to favor its occurrence, we need to formalize it in order to recognize it. (De Figuereido 2001) suggests four useful equations for the purpose. These equations distinguish creativity from different kinds of serendipity.

Definitions:

“Conventional creativity” is an occurrence of this kind:

P1 ∈ (KP1), M ∈ (KM) ⇒ S1 ∈ (KP1, KM, NK)

Which means that, given a certain problem P1 in the knowledge domain KP1 and an inspiring metaphor M in the knowledge domain KM, a solution S1 to the problem P1 is generated, belonging both to KP1 and to KM. In this way, new knowledge KN is produced. An example of this occurrence is Archimedes’ solution to the problem posed by the king of Syracuse Hiero II. The inspiring metaphor in this case is the push up Archimedes perceived when stepping into his bath, a very different domain from metallurgy. The new knowledge this creative process produced is the physical law known today as “Archimedes’ Principle” (Note 2).

Real "Serendipity" is instead described as an occurrence of this kind:

P1 ∈ (KP1), M ∈ (KM) ⇒ P2 ∈ (KP2), S2 ∈ (KP2, KM, NK)

Meaning: given a P1 problem in the knowledge domain KP1 and an inspiring metaphor M in the knowledge domain KM, a problem P2 in KP2 and a solution S2 are produced. This solution, belonging both KP2 and KM, produces new knowledge NK. The main difference between serendipity and conventional creativity is that the latter achieves the solution to a problem we already had from the beginning and to which we were just looking for a solution. On the contrary, serendipity enables to find a solution to a problem we had not posed or been posed before. A solution we were not looking for.
A practical example of this is having a problem regarding a too acidic pH of soil and, by studying different subjects, discovering a new cure to a rare disease. In this case, we were not looking for the cure to the disease, and the problem’s, metaphor’s and second problem’s domains differ.

There's also another variant of serendipity, described as “serendipity without inspiration”:

P1 ∈ (KP1) ⇒ P2 ∈ (KP2), S2 ∈ (KP2, NK)

It is an occurrence of this kind: given a P1 problem and its knowledge domain KP1 and without an inspiring metaphor, both a new problem P2 in knowledge domain KP2 and its solution S2 using knowledge from KP2 are produced. This generates knew knowledge NK. The main difference compared to serendipity is the absence of the bridging inspiration.

The last formalization is a further variation of serendipity called “serendipity from incorrect knowledge”. It is an occurrence described in the following equation:

P1 ∈ (KP1, IKP1) ⇒ P2 ∈ (KP2), S2 ∈ (KP2, NK)

Meaning: given a problem P1 in its knowledge domain KP1 and an incorrect knowledge regarding P1 as IKP1, a new problem P2 in the knowledge domain KP2 is posed and its solution S2 is achieved through KP2, creating new knowledge NK. An example for this is Columbus’ discovery of the Americas. In this case, motivated by the incorrect knowledge that, by sailing westward, he would reach India, Columbus discovered the New World.
Hence, a new problem was generated, i.e. how to reach the new continent, and also its solution, together with the new knowledge of its existence – even if Columbus was still convinced of having reached India.

Serendipity, Creativity, Randomness and Diversity

"In the field of observation, chance favors only the prepared mind." - Louis Pasteur

"Luck is being ready." - Brian Eno

We discussed how the concepts of serendipity and creativity are so tightly interwoven that formalizations are needed in order to distinguish one from the other. But actually, the characteristics boosting the occurrence of serendipity are exactly the characteristics that we associate with creative people Creative people have the ability, the open-mindedness, the competence needed to discover analogies and associations between apparently random elements. It is important to underline that this randomness is only apparent. Randomness plays a very small role compared to creativity.
A good example to understand the role played by randomness in the creative process comes from a architecture (Gladwell 2000): in the 60s, Jane Jacobs was the only urban planner who criticized the US urban renewal program pointing out that replacing organic districts - which produced a mixed flux of people and goods - with a more rigidly organized structure was a terrible idea. According to Jacobs, non-structured districts encouraged creativity, as they favored random and non-threatening encounters. Though unheard in the field of urban planning, her critique is the foundation of today’s conception of working spaces: divided sectors and rigid structures have been supplanted by open spaces, for the sake of collaboration, exchange of ideas, casual conversations and open conversations among different teams.
To let being creative and consequently allow serendipity to happen, randomness plays the role of opening up unexpected possibilities and favoring the contact with different people, contexts and experiences. The psychologist Carl Rogers, in his book On Becoming A Person (Rogers 1961), reinforces this concept as follows:
“It was discovered that the more a person is "open" to all experiences [...] the more sure we can be that his creativity will be personally and socially constructive. In a person who is open to experience, each stimulus is freely relayed through the nervous system, without being distorted by any process of defensiveness. [...] Whether the stimulus originates in the environment, in the impact of form, color, or sound on the sensory nerves, or whether it originates in the viscera or as a memory trace in the central nervous system, it is available to awareness. [...] This last suggests another way of describing openness to experience. It means lack of rigidity and permeability of boundaries in concepts, beliefs, perceptions, and hypotheses. It means a tolerance for ambiguity where ambiguity exists. It means the ability to receive much conflicting information without forcing closure upon the situation. [...] This complete openness of awareness to what exists at this moment is, I believe, an important condition of constructive creativity.”
Gregory Bateson, anthropologist, sociologist, language scientist and one of the fathers of the cybernetic revolution, published a further study about creativity in his "Steps to an Ecology of Mind" (Bateson 1972), describing it as the result of knowledge acquisition based on diversity. He distinguishes among different levels of learning: proto-learning is the acquisition of factual data, specific knowledge and contents; deutero-learning is the set of cognitive habits or models that everyone uses to frame knowledge domains. For example, learning a new language is not only knowing words and structures, but also learning some habits of mind on how to treat linguistic issues. This creates learning models or styles that will be particularly useful for the treatment of epistemologically analog problems, a mindset that can help in similar contexts. This kind of learning prevents the acquired notions from making our minds dull on fixed structures as they are provided. However, the model itself can become a fixed structure, if the person creating that model is not conscious of it. A third level of learning, a form of meta-learning, is thus defined, which contains a very high creativity rate. It consists in the ability to question the own cognitive models and let them evolve, sometimes even to demolish and reconstruct them from scratch, or join them with models taken from other domains. This “plasticity” (Possenti 2008) prevents the cognitive models from becoming rigid models and improves the person’s ability to adapt to any condition. From a pedagogic point of view, the necessity for transcending a purely sciolistic approach is fundamental to allow a second-level type of learning. For a third-level kind, an increased variety of cognitive models has to be presented to the pupils, teaching them to “doubt”, exposing them to as many experiences and knowledge domains as possible and educating them to dialectics and critical thinking. It is once more evident how tight the link between creativity and diversity is, and how all this opens the path to that mental competence needed for serendipity.
To support this concept, a study published in (Florida 2002) puts into relation indices regarding the presence of innovative high-tech companies in the US with the amount of artists, writers, creatives and LGBT communities in the major US cities. It turned out that the cities with the highest presence of social, cultural and sexual diversity are also the places with the greatest amount of creative people and innovative enterprises.
To conclude, creativity is surely indispensable for the acknowledgment of serendipity; but in order to trigger the creative process, diversity of skills, experiences and know-hows is essential. Diversity, in turn, is favored by being exposed to random events, which, being unpredictable and different, improve the creative possibilities of a person. The bottom line is that people exposed to systematic diversity can increase their creative capacities and hence their possibility to encounter serendipity.

Lateral Thnking

Something that connects perfectly to what has been illustrated so far and that can produce interesting suggestions is the concept of “lateral thinking”, developed by the psychologist Edward de Bono in his book Lateral Thinking: Creativity Step By Step (de Bono 1970). Here, he outlines two methods for solving problems of the human mind and compares them: “vertical thinking” is the classic problem solving modality based on addressing the problem directly. It focuses on obvious initial considerations and, through a logical, consequential process, it reaches a complete solution; “lateral thinking” is the use of a more indirect approach to the problem, observing it from various alternative standpoints. The first approach leads to a selective and sequential process, while the second tends to accept accidental aspects that might seem irrelevant at first glance or that simply had never been paid attention to.
According to De Bono, the application of lateral thinking is possible because every problem has more than one solution. But some solutions can be found only by moving away from the apparently only possible solution and by starting to look at elements, ideas, intuitions, inspirations outside of the problem’s knowledge domain, of that rigid logic chain that might seem to be the only way for solving the problem. The author also suggests a series of instruments to support this mental process of creating original solution hypotheses, to be used alongside our knowledge of the problem in order to achieve the set goal. Among them, there are conceptual maps, solution maps and the six hats technique, that consists in analyzing the problem under six different perspectives represented by six hats of different color.
Therefore, it is easy to realize that a lateral approach is very suitable to favor serendipity and its manifestation. It has a lot in common with Bateson’s theory, because lateral thinking can be viewed as the manifestation of the elasticity and mental flexibility needed for Bateson’s third level of learning. The idea of an alternative, “creative” way to solve a problem is common to all the studies in the field.

Notes

  1. Literally: Eight Paradises, a famous poem by Amir Khusro. It is a story is based on a previous epic poem, the "Shahnameh", written by the Persian poet Ferdowsi around 1010 and its following adaptation, the "Haft Paykar" by Nizami (Persian poet and prose writer), written around 1197. The "Shahnameh" is a long poem narrating different epochs of Persian history. The "Hasht Bihisht" narrates just a little part of the life of Bahram Gur and, compared to the original, this version is more fictional and less true to historical events. Khurso seems to be the first author to add the characters of the three princes of Serendip and the history of the stolen camel and its recovery, alongside the more traditional stories about Bahram Gur. (Ben-Amos 2006)
  2. According to the legend narrated in De Architectura by Marcus Vitruvius Pollio, one day, when immersing himself into his bath, Archimedes felt the hydrostatic push up of the water and understood its cause. This led him to the solution of the problem that the king of his city, Hiero II, posed him. The king wanted to verify an unpleasant suspicion: to celebrate his success, he commissioned a golden crown, providing the goldsmith with a certain quantity of gold. The finished crown had exactly the same weight of the gold he gave, but Hiero suspected that the gold had been replaced with a lesser precious metal having identical weight (copper or silver). Based on his intuition, Archimedes understood that the two metals, though having the same weight, will have two different volumes (like a kg of iron and a kg of wood), so their buoyancy will be different when immersed in water. Archimedes thus put the crown on one plate of the scale and a gold bar on the other. The scale was of course in equilibrium. Then he immersed the two objects in water, still on the plates of the scale. The crown, being the same weight but of different volume, had a different buoyancy. The scale inclined in favor of the gold bar and the fraud was discovered.

Afterword

Today I'm stil interested on this issues, and they are always in the back of my mind. If I had to start over working on serendipity and information seeking today, after studying Reinforcement Learning, I would probably approach the problem trying to find a balance between an hypotethical exploitation phase and an hypothetical exploration phase, in which the algorithm alternates suggesting new things and following the usual approach. The problem anyway is still the same: "Serendipity cannot be planned" (paraphrasing Joel and Ken's book title), and setting an accuracy metric as the goal of the algorithm can be misleading convince us we are doing progress when we are only aggravating the problem.

Even if a lot of reasearchers tried to address the issues I was talking about in my thesis in the following years, almost nothing changed in how the algorithms of the dominant publishers of information (Google and Facebook, even they don't want to be called this way, as being a publisher involves responsability) work.
The problems derived from the mind cage that information filtering and personalization create around us can now be seen more clearly and if combined with the constant flooding of fake news and bot generated content become even more scary for the effect they can have on our decisions. We are in the end a function of our past history and what we are exposed to, and the consequence is that if someone has the power to manipulate what we are exposed to, it practically manipulates also the outcome, our actions. For this reason studying serendipity as a possible way to break this circle is still as relevant today as it was 10 years ago.

Bibliography

Foster, A. a. F., N. (2003). "Serendipity and information seeking: an empirical study." Journal of Documentation 59(3): 321-340.
Roberts, R. M. (1989). Serendipity: Accidental Discoveries in Science, John Wiley & Sons, Inc.
Erdelez, S. (1997). Information encountering: a conceptual framework for accidental information discovery. Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Difference Contexts, Tampere, Finland, Taylor Graham: 412-421.
Ross, C. S. (1999). "Finding without seeking: the information encounter in the context of reading for pleasure." Information Processing and Management 35: 783-799.
Williamson, K. (1998). "Discovered by chance: the role of incidental information acquisition in an ecological model of information use." Library and Information Science Research 20(1): 23-40.
Toms, E. (2000). "Serendipitous Information Retrieval." Proc. of DELOS Workshop: Information Seeking, Searching and Querying in Digital Libraries.
van Andel, P. (1994). "Anatomy of the Unsought Finding. Serendipity: Orgin, History, Domains, Traditions, Appearances, Patterns and Programmability." British Journal Philosophy Science 45(2): 631-648.
de Figueiredo, Antonio Dias and Campos, José, The Serendipity Equations (2001). Navy Center for Applied Research in Artificial Intelligence, Naval Research Laboratory, Washington, DC, Technical Note AIC-01-003.
Gladwell, Malcolm "Design for working" The New Yorker , December 11, 2000, 60-70
Rogers, C. (1961). On becoming a person.
Bateson, G. (1972). Steps to an Ecology of Mind: Collected Essays in Anthropology, Psychiatry, Evolution, and Epistemology, University Of Chicago Press.
Possenti, I. (2008). Flessibilità ed istruzione: una replica a Bauman. MicroMega: Un'onda vi seppellirà.
Florida, R. (2002). The Rise of the Creative Class. And How It's Transforming Work, Leisure and Everyday Life, Basic Books.
de Bono, E. (1970). Lateral Thinking: Creativity Step by Step, Harper & Row.