You may have a brilliant new business strategy, but unless employees are stoked about the idea, it won’t succeed, says leadership guru John P. Kotter, a Harvard Business School professor and author of several books on organizational change.” So begins an interview with Professor John Kotter, widely regarded as an authority on change. Kotter goes on to promote a counterintuitive way to drive adoption of an idea: by encouraging employees to criticize it.

Change in an organization can be categorized, in one of the ways, from its breadth of impact. A change of software development from traditional (waterfall) to agile (say, Scrum) is a radical change. For a software organization, it touches not only engineering or development, but also sales, marketing, finance, even HR. The same is for Lean transformation from batch mode. A crucial element change leaders often ignore or underestimate is psychology.

Studies after studies in heath care tell us that 90% percent of the people who belong in high risk group like who have undergone coronary-artery bypass surgery do not change their lifestyle. In the words of Dr. Edward Miller, the dean of the medical school and CEO of the hospital at Johns Hopkins University..”…Even though they know they have a very bad disease and they know they should change their lifestyle, for whatever reason, they can’t…“.

The same article goes onto reflect the view of John Kotter: “Changing the behavior of people isn’t just the biggest challenge in health care. It’s the most important challenge for businesses trying to compete in a turbulent world, says John Kotter, a Harvard Business School professor who has studied dozens of organizations in the midst of upheaval: “The central issue is never strategy, structure, culture, or systems…. And as individuals, we may want to change our own styles of work — how we mentor subordinates, for example, or how we react to criticism. Yet more often than not, we can’t.

Having seen through a fair share of lean and agile transitions, I can relate. The predominant emotions to be dealt with are fear, confusion, ego, greed (of power). In many organizations, years of traditional/waterfall with phase gate approach, command-and-control has defined a whole different culture, even provided a platform of survival to many. For them, it is now too important to let go.

I recall of an IT organization, global and big in all respects, for which the biggest chunks of revenue lay in client accounts using traditional way of running projects. Agile for it was more of a sales gimmick. They go agile only when clients insist on it. More than agile it’s a facade of agile: a series of short waterfalls. Obviously, people working in such projects suffered conflicts wondering what really the beast called agile is.

The same Fast Company article goes on: Unfortunately, that kind of emotional persuasion isn’t taught in business schools, and it doesn’t come naturally to the technocrats who run things — … who pride themselves on disciplined, analytical thinking. There’s compelling science behind the psychology of change — it draws on discoveries from emerging fields such as cognitive science, linguistics, and neuroscience — but its insights and techniques often seem paradoxical or irrational.

Dr. Dean Ornish, a professor of medicine at the University of California at San Francisco and founder of the Preventative Medicine Research Institute, in Sausalito, California, could bring the change where John Hopkins and Global Medical Forum couldn’t. Ornish helped 333 seemingly incorrigible patients quit smoking, follow diet, exercise; 77% of whom stuck with the lifestyle changes after 3 years.

Why does the Ornish program succeed while the conventional approach has failed? For starters, Ornish recasts the reasons for change. Doctors had been trying to motivate patients mainly with the fear of death, he says, and that simply wasn’t working. For a few weeks after a heart attack, patients were scared enough to do whatever their doctors said. But death was just too frightening to think about, so their denial would return, and they’d go back to their old ways.

So instead of trying to motivate them with the “fear of dying,” Ornish reframes the issue. He inspires a new vision of the “joy of living” — convincing them they can feel better, not just live longer….

“Joy is a more powerful motivator than fear ” he says.

A McKinsey articleThe Psychology of Change Management” prescribes four conditions for changing mindsets: a purpose to believe in, reinforcement systems, skills required, and consistent roles models. And this S&B article tells us “Change is, at its core, a people process, and people are creatures of habit, hardwired to resist adopting new mind-sets, practices, and behaviors” and suggests “Build an emotional and rational case for change” as one of the key success factors.

Chip Heath, who along with his brother Dan Heath authored Switch, shows how managers can catalyze change more effectively by drawing on an enormous body of research from psychologists on how the brain works .In this interview with McKinsey Quaterly, Chip Heath posits : Pay attention to creating an emotional case for change, not just an analytical one.

Linda Rising and Mary Lynn Manns, in their excellent “Fearless Change: Patterns for Introducing New Ideassays:  “I think the most important message behind the patterns is that we are not logical decision-makers. We act based on emotion, primarily fear. Most of us are afraid of change, even if it seems exciting. As a result, you need to appeal to emotion and relieve the fear.”

I have seen, unfortunately, a few organizations that couldn’t overcome the fear and couldn’t sustain the change; and then fell back upon old traditional way of building software. For high risk heart patients who are given the option to change or die, the odds are nine to one that people really changed themselves.

Like a Russian prime minister, Viktor Chernomyrdin, once saidWhatever organisation we try to create, it always ends up looking like the Communist Party.