Innovation and Optimization at War

One Takeaway: The skills and tools that help with optimization often prevent companies from innovating. This isn’t a management failure. It’s an economic reality. Understanding what creative destruction is explains why this happens and how to manage both, together.

Throughout this series, we’ve explored why coordination is expensive, how knowledge resists centralization, why measurement can kill innovation, and how structure enables or prevents complex work. But there’s an underlying tension beneath each of these seperate ideas that none of them fully resolves.

The systems that make operators and refiners effective (standardized processes, clear metrics, hierarchy) are the same systems that kill creator work. And the freedom that creators need threatens the reliability that operators and refiners depend on. This isn’t a tension you can end or assume away. It’s one you have to learn to use rather than be constrained by.

Two Companies, One Shift

When cloud computing emerged as an alternative to on-site data centers, two enterprise software companies with similar market positions faced the same choice.

Company A (optimization wins). Their on-site business was highly profitable. $500M in annual revenue, 40% margins, customers on predictable multi-year contracts. Moving to cloud would cannibalize this revenue stream. The entire organization was optimized for on-premise success.

When product development proposed a cloud initiative, every function had rational objections. Sales: cloud deals are smaller and compensation will drop. Operations: we don’t have cloud capabilities and building them will distract from our profitable core. Customer success: our existing customers are happy with on-premise and don’t want to migrate. Finance: cloud will hurt current profitability because we’ll recognize revenue slower.

Everyone was right from their individual perspective. The cloud initiative got delayed, scaled back, and eventually killed. Company A continued optimizing their on-premise business. They focused on better processes, enhanced features, more efficient operations.

They optimized themselves right into irrelevance. As the market shifted to cloud over five years, revenue declined 60%. They were eventually acquired at a fraction of their peak valuation.

Company B (managed creative destruction). Same initial position. Same profitable on-premise business that cloud would cannibalize. But the executive team recognized the core dilemma: if we don’t destroy our own business, someone else will.

They created a separate cloud division with fundamentally different economics. Different success metrics—customer acquisition and platform stability, not profitability. A five-year runway to reach profitability while the on-premise business funded the transition. A separate budget that didn’t compete with on-premise optimization. And a dedication to bridging the two divisions. They needed a way to ensure the cloud team learned from on-site customer relationships without on-site blocking cloud’s progress.

The result: Company B managed the transition while maintaining the core business. Cloud revenue eventually exceeded on-site. They maintained market leadership while competitors optimized themselves into decline.

The difference. Both companies understood cloud was important. Both had the talent. Both had customer relationships to leverage. The difference was managing the economic conflict between optimization and innovation. Company A let optimization win because that’s where the immediate rewards were for every individual team. Company B restructured the economics so both were viable at the same time.

Why Innovation Creates Value by Destroying It

Joseph Schumpeter, a highly influential economist from the early 20th Century, had a fundamental insight called creative destruction¹. It wasn’t just that innovation and optimization conflict. It was that innovation creates value precisely by destroying existing value.

A new technology makes existing skills irrelevant. A new business model undermines existing competitive advantages. A new product makes existing products less valuable. The value creation happens through the displacement itself. Digital cameras didn’t add to film photography. They destroyed it. Cloud computing didn’t supplement on-site infrastructure. It displaced it.

Inside organizations, this creates an inherent conflict. Your operators and refiners have optimized their current processes, relationships, skills, systems. These represent real economic value built through years of accumulated learning. Your creators identify opportunities that likely will make those valuable creations less valuable or even obsolete.

The same organization must both protect value it’s created and destroy value it’s created. These aren’t just different activities. They’re opposing forces where one’s gain feels like the other’s loss. It only feels this way, though, if we approach our work with an expectation that it should never change.

Most organizations resolve this by choosing preservation over destruction. They protect existing value. They default to what is certain and measurable at the expense of creating new value, which is uncertain and difficult to measure. This is rational for every individual and team inside the system. The problem is it’s often catastrophic for the organization as a whole.

Why Capabilities Become Constraints

Economists Richard Nelson and Sidney Winter showed that organizational skills can become barriers to change.² Organizations function through routines. They create established patterns for how teams qualify leads. They create consistent approached to designing products. They have a predictable way to handle exceptions, divide resources, and coordinate across teams.

These routines are genuine economic assets. They encode valuable learning from experience. They enable coordination without constant negotiation. They create the consistency that customers depend on. They allow the specialization that operators and refiners need to do their jobs well.

But the same routines that enable efficient execution create barriers to innovation. The better you get at your current approach, the harder it becomes to adopt a different one. Company A was exceptional at on-site software because they’d refined those routines over a decade. That same expertise made them terrible at cloud. It wasn’t because their people were incapable. They needed different routines than they’d developed. Economists call this a “competency trap.” Capabilities become constraints.

People whose expertise is embedded in specific routines resist innovations that would require different routines. This isn’t stubbornness. It’s rational economic self-interest. If your value comes from mastering certain routines, innovations that make those routines obsolete can be threatening.

When a creator identifies an opportunity, they’re implicitly saying: the routines you’ve spent years developing need to change or become obsolete. Even if the creator is right, they’re threatening value that operators and refiners have created. So creator insights get rejected. They aren’t turned down for being wrong. They’re turned down because they threaten existing organizational capital.

The Organizational Immune System

The resistance to innovation that most organizations experience isn’t about closed-mindedness. It emerges from individually rational economic behavior that produces collectively destructive outcomes.

The operator’s logic: “My job is to maintain stability. Innovation creates disruption that threatens the metrics I’m evaluated on. My bonus depends on uptime, and changes can reduce it.”

The refiner’s logic: “Resources spent on innovation are resources not spent on the optimization I’m measured on. My promotion depends on efficiency results.”

The middle manager’s logic: “Innovation might destroy my division’s revenue. My career depends on my quarterly performance.”

The executive’s logic: “R&D investments reduce near-term earnings and stock price. I may not be here when they pay off.”

Every one of these ideas is individually rational. Taken together, they create an organization that rejects beneficial innovation. Think of it like a biological immune system. An immune system doesn’t evaluate whether a foreign body is helpful or harmful. It identifies anything unfamiliar and attacks it. That’s exactly what’s happening here. Innovation is unfamiliar work that disrupts established routines. It threatens existing metrics and redirects resources away from proven activities. The organization’s incentive structure identifies it as a threat and mobilizes against it. The same self-preserving logic that makes immune systems effective in the first place ends up stopping beneficial, new work.

This is why alignment speeches don’t work. Telling people “we need to innovate” doesn’t change their economic incentives. They understand that innovation matters. It’s intuitive. But it’s still irrational for them as individuals given how they’re actually measured and rewarded. The CEO’s speech changes nothing about the economic logic they face every day.

Company A’s leaders were all individually rational. Sales protecting their commission structure. Operations protecting their expertise. Finance protecting current profitability. They were all protecting their trees, while the forest slowly died.

Why Portfolio Thinking Matters

The economic problem isn’t that organizations evaluate innovation projects badly. It’s that they evaluate them using individual project logic when portfolio logic applies.

Individual project evaluation works for optimization. Each initiative either generates positive returns or doesn’t. You can measure results within quarters. Kill what doesn’t work. Expand what does.

Innovation requires portfolio evaluation. Most individual experiments fail. But the portfolio succeeds if you maintain enough experiments to capture the rare dramatic successes while learning from everything else. If a creator team runs ten experiments and eight fail, one produces modest results, and one opens a $50M market that’s an extraordinarily successful year. But individual measurement reports an 80% failure rate.

You can kill innovation by using individual project logic on work that only makes sense as a portfolio. Each “failed” experiment becomes evidence to cut the program. But the “failures” were generating learning that made the successes possible.

This connects to the measurement problem from “Why Metrics Kill Innovation.” It’s easy to measure individual project failure. It’s hard to measure portfolio learning and option value. Organizations tend to underweight innovation because the costs are immediate and visible while the benefits are distant and hard to quantify.

Financial economists understand that some investments create “option value.” This can be thought of as the ability but not obligation to go after future opportunities. Innovation investments work the same way. Company B’s cloud experiments had three types of value.

1. Direct revenue from cloud products

2. Learning that improved their business overall

3. The option to pursue new opportunities as markets evolved.

Traditional ROI analysis captures only the first. Portfolio and option value thinking captures all three. This is why creator work is essential even when it doesn’t generate immediate measurable returns. Creator work creates flexibility that the organization can turn to when conditions change. Organizations that underinvest in creator work are destroying option value. They sacrifice future adaptability and resilience for current optimization.

The Exploration-Exploitation Tradeoff

Organizational theorist James March formalized this tension.³

• Exploitation (what we’ve been calling optimization) refines existing capabilities within known parameters. Returns are quick, certain, and measurable.

• Exploration (what we’ve been calling innovation) searches for new capabilities in unknown domains. Returns are slow, uncertain, and distant.

The bias toward exploitation is structural, not cultural. Managers get evaluated on time horizons shorter than exploration payoffs. Annual performance reviews punish exploration “failures” before successes can emerge. Boards get nervous about exploration spending without visible returns. Investors pressure for near-term results.

But organizations need to optimize over their existence, which, hopefully, is decades or longer. The people making resource use decisions have 3-5 year time horizons. Exploration investments that would pay off in year seven often may not benefit the manager who made them. Rational managers favor exploitation. The result is underinvestment in exploration even when more investment would be optimal for the long-term.

March had a key insight. The right balance depends on how quickly your environment changes. Stable environments need less exploration. Unstable environments need much more. Most organizations get this backwards. They explore aggressively when young and small, then reduce exploration as they mature and have more the resources to fund it. Companies gain capacity for investment exactly when their systems evolve to prevent it.

How Harmonizer Thinking Changes the Economics

The solution isn’t to convince people to act against their economic interests. It’s to restructure the economics so that supporting innovation becomes individually rational.

This is the same principle we’ve seen before in this series applied to the innovation-optimization tension.

• In “The Cost of Working Together,” harmonizer thinking meant building new rules and structures that made coordination the rational choice.

• In “What Headquarters Can’t See,” it meant brokering knowledge between people who had it and people who needed it.

• In “Why Metrics Can Kill Innovation,” it meant protecting valuable work from measurement systems that would destroy it.

• In “Why Structure Determines Strategy,” it meant bridging separated functions so organizations got specialization without fragmentation.

Here, it means doing all of these together to manage creative destruction.

In Company B someone thinking this way might have:

• Created shared success metrics. The cloud team’s progress contributed to everyone’s evaluation. This made cloud support individually rewarding rather than individually threatening.

• Maintained separate budgets. so cloud “failure” on profitability metrics didn’t hurt on-premise teams.

• Made sure the cloud team learned from on-site customer relationships.

• Made sure the on-site team didn’t block experimentation.

• Translated between different evaluation frameworks so leadership could assess both divisions on appropriate terms.

When the individual economic logic aligns with organizational needs, the immune system becomes an adaptive rather than a barrier. People still act in their economic self-interest. But their self-interest now includes supporting innovation rather than blocking it. You don’t change human nature. You change the economics that guides rational human behavior.

The Bottom Line

Schumpeter’s creative destruction explains why the success patterns from Growth Isn’t One Sided are so difficult to sustain. Innovation creates value by destroying existing value. Inside organizations, this means the same company must both protect what it’s built and destroy what it’s built. These are opposing forces.

Organizations understandably default to optimization. Those returns are near-term and measurable. Individuals are evaluated on shorter time horizons. But the company needs longer term thinking. Routines make companies good at what they already do. But, they make them bad at doing things differently. Incentive structures almost always reward exploitation over exploration.

Understanding this as an economics problem enables systematic solutions. Portfolio evaluation that judges innovation on collective learning rather than individual project success. Structural protection that prevents optimization from crowding out innovation. Different measurement approaches for different types of work. And harmonization that aligns individual incentives with organizational needs rather than trying to convince people to act against their interests.

The competitive advantage goes to organizations that can manage creative destruction internally rather than waiting for external pressure to force their hand. They optimize what works today while discovering what will work tomorrow. They exploit and explore simultaneously.

Doing this requires every principle this series has covered. You have to match coordination efforts to the type of work being done. You have to make sure knowledge reaches the right decisions. You have to measure what matters without destroying what you can’t measure. You have to design structures that enable different types of work to coexist. And you have to manage the tension between preserving value and creating it. Not by choosing one over the other, but by building systems where both can happen at the same time.

None of this is easy. But it’s a lot harder if you don’t recognize the problem in the first place. Most organizations experiencing creative destruction don’t know that’s what’s happening. They think they have a motivation problem, or a culture problem, or a leadership problem. They have an economics problem. And economics problems have economics solutions.

1. https://www.econlib.org/library/Enc/CreativeDestruction.html

2. https://www.jstor.org/stable/3114818?seq=1

3. https://www.jstor.org/stable/2634940