The secret to creative breakthroughs, hot streaks and success

The trouble is, the insight is only half-true and therefore in need of an update. Whether you're heading for college, starting out in your career, stuck in a mid-life crisis or entering retirement, you shouldn't ignore Satchidananda's exhortation — but you should add this nuance. First spend some time flitting around and yes, staying shallow. Then, and only then, go for the deep dive.

That's the upshot of research published in Nature Communications by Lu Liu, Nima Dehmamy, Jillian Chown, C. Lee Giles and Dashun Wang, variously at Northwestern University and Pennsylvania State University. They were interested, as their title says, in "Understanding the onset of hot streaks across artistic, cultural and scientific careers."

A belief in hot streaks, or the hot hand, is widespread among gamblers, stock pickers and basketball players, among others. It's a hunch that success leads to more success — that sometimes you're just on a roll. Psychologists tend to think it's bunk. Others aren't so sure. In basketball, for example, players statistically do sometimes have a hot hand, according to a new analysis — provided they're shooting from the same spot rather than moving around.

But that's just hoops. Crunching data from the arts, culture and science is more interesting, because patterns you find there might extend to all fields of endeavour. And patterns there are.

The researchers definitely found hot streaks, defined as "bursts of high-impact works clustered in close succession." For example, Jackson Pollock made his biggest splashes (literally) in the late 1940s, during his famous "drip period," when he began pouring or hurling paint onto canvases spread out on his floor as he danced all over them.

In film, director Peter Jackson made history with his Lord of the Rings trilogy between 2000 and 2003, bringing Tolkien's epic to the big screen. In science, John Fenn made breakthroughs in how we understand biomolecules like proteins when he pioneered a technique called electrospray ionization, for which he later shared the Nobel Prize in chemistry.

These hot streaks and many others had something in common, the researchers found. In each case, the individual had previously spent a lot of time dabbling in things, trying stuff out, searching but not quite finding a groove. This is called exploration in social science. For example, Pollock experimented with murals, Jackson with everything from horror comedies to mockumentaries, Fenn with chemistry too obscure to grasp for mere mortals.

But at some point — Fenn was already in his 70s, when it arrived for him — each espied something they found promising and then concentrated for extended periods on just that. This is called exploitation, when an individual accumulates expertise and digs deep. It's in the transition phase from exploration to exploitation when hot streaks are ignited.

Tomes have been written about "How the human brain manages the trade-off between exploitation and exploration." But that needn't worry you unduly if you're just thinking about your life and what to do with it. The main point is that you need both and the ability to toggle.

Youth, for example, is a great time for exploration. That's why I'm such a fan of liberal-arts education, which exists in its full-fledged form only in North America. In most other cultures, students already choose career paths after high school. The best advice I received when I first showed up at my alma mater was to study anything at all — except what I was planning to specialise in.

The other end of life is oddly similar. Retirement, in this sense, can be a transition from an exploitation phase to a new bout of exploration. Sounds rejuvenating, doesn't it?

While you're exploring — as when you're searching for water in a desert — you have to be patient and to persevere. Something will turn up. Your job is to recognise that moment and trust the depth of the well. It could take the form of a PhD thesis, a startup, a book, or whatever. This is when, as Satchidananda put it, you should "use dynamite and keep going down. If you leave that to dig another well, all the first effort is wasted."

The good news is that the cycles of exploration and exploitation don't have to end. Potentially, they can recur throughout life. When you're feeling midlife burnout, for example, it may simply mean that you've been exploiting too long and have drained one particular well. So get out that dowsing rod, but bring the spade. Scratch a bit here and there. And eventually — but not too soon — start digging again.

Andreas Kluth is a columnist for Bloomberg Opinion. He was previously editor in chief of Handelsblatt Global and a writer for the Economist. He is the author of 'Hannibal and Me.'

Disclaimer: This article first appeared on Bloomberg and is published by special syndication arrangement.