Volume 9: Fall 2012 Leading Qualitative and Quantitative Analysis of Ideas

The Rocky Peak Leadership Center “News from the Peak” newsletter is a seasonal newsletter aimed to educate, encourage and inform leaders. Its practical down-to-earth approach to present timely leadership information can stimulate and inspire others to engage in constructive dialogue and intends to help move organizations forward in a positive direction.

Leading Qualitative and Quantitative Analysis of Ideas

“Ideas are useless unless used” according to Ted Levitt former editor at Harvard Business Review. Leading Innovation and the creation of new ideas are considered by most organizations as a requirement to succeed today.

Two big problems exist for Leaders which create overwhelming challenges. The first is a situation where you have problems or issues and no clear solutions. You “Stay the Course” with a product that is not meeting all of your customer’s needs or a process that has known inefficiencies because it’s not completely broken. That same organization does not have a formal “Innovation” process and thinks that saying they are innovative to everyone is enough.

This method of Status Quo Idea Leadership is passive and based on taking no actions that can really help. The second poor leadership method is when you’re overwhelmed with a constant stream of new ideas and you don’t know what to do with them. In this situation, you don’t have strategies, goals or tools to help you sort, organize or utilize ideas…. so you do nothing. This is called Wait and See Idea Leadership and can be just as destructive because it does not take advantage of the value stream created by having new ideas and solutions.

Modern organizations are ultimately required to create new ideas to survive. Without them they will become inefficient in their products, services or processes and be outperformed by their competitors. Unfortunately, many of these same organizations make mistakes in their idea review processes that result in rejecting the most potentially innovative ideas in favor of less innovative ones.  Even harder is understanding if an idea meets the qualitative measurements required for being considered to move to the next phase of idea development or implementation.  Very seldom is there a combined quantitative and qualitative measurement that provides an accurate or valid ability to differentiate the value of one idea versus another idea.

So if your answer to the question of what is a good idea?....that “you will know it when you see it” the odds are that you won’t be able to. Since everyone evaluates ideas differently, based on their individual perceptions and perspectives this becomes a problem. The problem becomes more problematic when agreement between different parties is necessary. How can two people determine the best idea? What tool or system do they use? Sometimes a basic pass-fail criterion is used and anything that does not seem to “fit” is discarded. 

Some organizations have embraced quantitative analysis or grading of their ideas but find that they don’t know what the measurement criteria should be to qualify an idea to move forward. Either option does not blend a combined analysis because they are thinking modes that are opposites of each other. One is subjective using interpretation, your experiences and feelings about something while the other is objective requiring facts and logic based truths.

You can't know whether the idea in front of you is good or bad or anywhere in-between because you haven't given proper consideration about what you’re looking for. So what’s required is an “Idea Evaluation Tool” that allows you to rank, rate, prioritize and determine accurate idea fits to your organizational goals and strategies. Additionally, it needs to be able to allow you to understand what is “Low Hanging Fruit” (implemented immediately) against what will take considerable effort to implement because it is a “Concept”.  These concepts sound good on paper but can be impossible to realistically develop.

So let’s discuss a process and toolset that allows idea evaluation based on both, quantitative and qualitative measurements that are integrated into the organization’s strategic goals which contain:

Great ideas are constantly characterized by the simplicity of their conceptual description. Einstein’s theory of relativity is elegant when described as E=mc2.  The Apple IPOD was originally described as “1000 songs in your pocket”. Some people are told their ideas should fit on the back of a business card…while others are told to create elaborately biased and convoluted stories.

When Dave Goodman was tasked with redesigning Disneyland’s fireworks show, he sold the entire concept with a picture of a man with his small son on his shoulders, tongue out, catching snowflakes. His idea statement was that the fireworks were going to make it snow throughout the entire park, every night, and would make people feel the magic of this picture. What he was able to combine was a mixture of feelings, a vision and an idea that was amazingly clear and simple.

  Qualitative Idea Analysis
The thing about people is that we are all different when it comes to creating and understanding a subjective view of ideas. Some may like one idea where others may not. This makes the process of idea grading both exciting and frustrating. You must find a way to move everyone involved into a feeling of collaboration, not compromise. The British retail giant Tesco uses the following criteria for selecting their new ideas:

This is a very easy context to understand and apply. Any idea that is better, simpler and cheaper for Tesco is bound to be considered a very good idea. A possible drawback to using this type of criteria is that when ideas are controversial, people can be inhibited or influenced by the opinion of others. The qualitative idea process is based upon opinion and the subjective evaluation of participants.  Almost always the idea of “Risk” is part of that evaluation. Ideas that are easy to implement or have little risk are typically called “Low Hanging Fruit”. This perception is that this type of idea takes almost no change or modification before it can be used. When you find these you should just go do it!

Additionally, there are typically two other qualitative subjective based idea separations on a continuum. At the other extreme is an idea which is only a “Concept”. This type of idea will take considerable effort to implement. It may require many different design iterations or steps to be able to make it feasible. The elements of cost and schedule may not support it and you may simply not have the resources necessary to bring it into fruition. 

The majority of ideas will fall into the middle category of the beginning of an idea. This type of idea is incomplete and needs further refinement and clarity. It needs some level of “tweaking” to make it fit or adjustment to make it realistic. These ideas are sometimes used as inspiration for another level of idea generation so that they can spark additional ideas or concepts.

Qualitative analysis of ideas alone can be incomplete and deceptive. If I were to ask you a question which was “what color is snow?”…..almost everyone would answer with “White”. To most of us this answer would seem accurate and satisfactory, but to an Eskimo it would seem to be a joke. Eskimos distinguish between many different varieties of “White” because they need to differentiate between the different weather conditions of snow and ice. So we can easily see that someone’s perception or point-of-view has a big influence on what they may think is a Good idea, a Great idea or a Bad idea.

Edward de Bono’s Lateral Thinking tools can be used to assist in the deliberate, systemic and comprehensive collection of ideas. You would not be impressed by a farmer who spent a lot of time and effort to plant & sow his crops but when harvest time came only collected a quarter of the crop and threw away the rest. This is what happens many times in idea creating sessions. When collecting the ideas make sure to collect all of them, not just specific ideas that seem to be immediately practical and valuable.
With the use of additional Lateral Thinking tools, the results of your creative thinking can double the output from your creative skills.

Utilizing these tools as a team can provide the opportunity to gain consensus of the idea to understand what should be accepted and what should be rejected. This evaluation of ideas is acceptable for simple product, process and services problems, issues or concerns. But this process of idea evaluation is incomplete with interconnected, muti-dimensional competing ideas. When you have multiple ideas that are based on the same idea category, theme or flavor something stronger and more comprehensive is required.

In the end people make these decisions and people are human. They can have biases, mental models and paradigm paralysis that allow them to accept or reject and idea without a clear analytical or objective evaluation of the merit of the idea but more to do with the people and politics. This is why a quantitative and qualitative idea evaluation is required.

  Quantitative Idea Analysis
While “Risk” based assessments and evaluation of ideas are great ways to review your internal capacity to realistically implement an idea, to truly rank, rate and analytically evaluate ideas requires a different mindset. This mindset is objective and leads to a quantitative interpretation or “Grading” of ideas based on specific weighted paired comparison(s).

This means that each idea has a specific weighted ranking that is compared against the other ideas to create a pallet of potential ideas along with a clear identification of the “Best” idea based upon an “Idea Criteria”. In other words, if we used cost as criteria, a Volkswagen would score higher than a KIA, and both of them would score below a Cadillac. This simple illustration is easy to see when you only have simple ideas but when you have many ideas and multiple idea criteria(s) identifying the best ideas can be very complex. The figure below is taken from an idea grading and evaluation tool called SCOR∑ ©  (www.scoreidea.com). This naming translation is an acronym which stands for:

Each SCOR∑ © process step has predefined, specific fields for you to fill out to fully describe, define, rank, rate, sort, organize and analyze multiple ideas for quantification.

The SCOR∑ © tool contains an “Idea Criteria” column which could be cost, schedule or a technical parameter such as weight, size, performance or functionality. This criteria column is weighted for importance from 1-10. If multiple columns are created, then you have multiple criteria that the idea is ranked and rated by. Each criteria used is averaged and summed together to create a quantitative score which can be sorted by hitting an “Analyze” button.

In the figure above, we have a series of ideas based upon the availability of “Water” for everyone, everywhere at any time. A multitude of ideas were imported and ranked and rated against pre-determined criteria. This ranking and rating process was accomplished by a group of Subject Matter Experts (SME’s) who have a valid background to analytically describe each idea. In the figure below, the original imported information was analyzed and the tool has created a hierarchical weighted comparison that is both Qualitative (ABC) and Quantitative (123). “A” represents low hanging fruit, “B” represents the beginning of a concept and “C” represents a concept. Numerically a higher score represents a more positive position in the idea criteria(s).

You are left with a weighted and paired cascading hierarchical listing of the ideas. It is now very easy to understand which ideas are the easiest to implement and which also meet the highest grading criteria. Ultimately, the “Best” ideas are at the top.

Austria authors Dr. Miroslav Rebernik & Barbara Bradač have published idea evaluation methods and techniques which describe over 29 different unique tools and methods. A partial listing is:

Clearly, there is no lack of tools and techniques to analytically evaluate ideas. Most organizations blend an evaluation tool with their current mental models and allow key leaders using their gut feel to influence what new ideas should be explored. Many organizations do not understand “What” ideas to implement and “How” to implement them.

  Idea Implementation
Once organizations have ideas that have been analytically ranked, rated and evaluated, the next step becomes just as hard as any of the previous steps. They must now determine what and how to implement that great idea.  Implementation is the process of moving an idea from concept to reality.  T. Levitt continues to warn us that the ability to come up with brilliantly novel ideas-can actually be destructive to businesses. By failing to take into account practical matters of implementation, people end up talking about it, but never do it. Expending time and resources to create ideas that are not acted upon is financially wasteful and sends a message to those people or stakeholders that their thinking is not valuable.

Leadership is critical with idea implementation because it exposes the gaps that may exist in what we will call an Idea Enterprise.  Ideas without a home are no more valuable than a box of toys that is hidden in a closet.  The implementation of that great idea must be linked to a strategy. Leaders know that their ideas should have been generated to support a “need” (product, process service problem, issue or concern).
This implementation of that great Strategic Idea is going to require four things as a minimum to make it possible:

If you do not have each of these idea support structures as a minimum, your great idea(s) do not have the opportunity to succeed.  We must put these ideas to work in an action-producing sense. Before you can implement an idea, it must be described in clear and understandable detail. It needs to illustrate what makes the idea special or what is its unique selling point.  Another point to be made is that just because you have a great idea does not mean it will be successful.

The opportunity for an idea to succeed is also connected to a philosophy called “Diffusion”.  E. Rogers defines Diffusion as the spontaneous unplanned process in which new ideas are communicated through certain channels & people networks over time among the members of a social system.  In other words the “Buzz” about a new idea may impact its acceptance and success. Clearly over the last 15 years, Apple Corporation used CEOs like Steve Jobs to pitch new product announcements and proposed next generation technologies at “Apple Events”. Look at the subtle marketing announcements that are currently underway for the iPhone 5 with hidden text in image shadows and pre-released function descriptions about what “might” be included in the next product release.

Take the QWERTY de-facto keyboard in which keys are arranged to avoid mechanical jams by purposely slowing down typing on the first generation of economically successful typewriters. Problems with it were that many common letter combinations required awkward finger motions. This required a finger to jump over the home row and type with one hand. About 16% of QWERTY typing is done on the lower row, 52% on the top row and only 32% on the home row. Compare this to the DVORK keyboard.
The DVORK keyboard was an idea patented in 1936 designed to use less finger motion, increase typing rate, and reduce errors compared to the standard QWERTY keyboard.

This reduction in finger distance movement was originally purported to permit faster rates of typing, and also in later years, it was purported to reduce repetitive strain injuries. So here is an example of a new great idea that was initially seen as superior to its predecessor, but to this day has still not been embraced as a replacement to the previous keyboard operating system. What was not considered in this scenario was the popularity of the QWERTY operating system and the psychological inertia to “not change” away from a product, brand name or idea that we became emotionally invested in. At times a smaller incremental change in an idea is easier to propose than a radical new idea.

There are new problems that can occur when ideas are implemented based solely upon the type and utility of the new idea. The worst thing that can happen is replacing one problem or issue with another or bigger problem. Innovation and ideas are elements of a “system” and the best way to understand if an idea is the best fit is to look at system ideality or an “ideal system”. In an ideal system there is a ratio between a system's useful functions and its harmful functions. You would never implement a new idea that contains more harmful system functions or elements that the original idea or solution it was replacing. This can be illustrated with the equation for System Ideality where the sum equals the useful functions over the harmful functions plus the new costs :

Any new idea that is implemented should be reviewed against this “Ideality” criterion. In other words never implement a new idea that is worse than what was there previously. Always ask three fundamental questions of a new idea.

First, what are the useful functions that this product serves?

Next, what are the harmful functions (anything which is not useful, but users or stakeholders must put up with)?

Finally, what factors determine the cost after it has been implemented?

An example of a failure of system ideality was an implementation of a road spray containing the hazardous waste Dioxin. As I was growing up in St. Louis Missouri County we watched the removal of the local community of Times Beach which was once home to more than two thousand people. The town had to be completely evacuated early in 1983 due to a dioxin contamination that made national headlines. It was the largest civilian exposure to dioxin in the country's history. Plagued with a dust problem in the early 1970’s due to its dirt roads and lack of pavement funds, the city of Times Beach hired waste hauler Russell Bliss to oil the roads in and around the town in 1971. Independent Petrochemical Corporation (ICP) was being paid $3,000 per load to remove toxic waste from Northeastern Pharmaceutical and Chemical Company (NEPACCO) and in turn paid Bliss $125 dollars to eliminate the waste.

From 1972 to 1976, Bliss was contracted by the town of Times Beach and sprayed waste oil containing the waste dioxin on the roads at a cost of six cents per gallon used. After a flood of the Meramec River in Dec.1982 approx. 95% of the town was flooded and the EPA announced it had identified dangerous levels of Dioxin and PCB’s in the ground soil. The test results showed dioxin levels 100 times higher than the one part per billion generally considered to be hazardous to humans. The town was “Removed” from existence with the soil being incinerated and the cleanup cost to the government estimated at a total of $110 million.

This is just one example of an “Idea” that never took into account the costs associated with its harmful effects. Imagine the impact of just 10 ideas like this to your organization’s efficiency, effectiveness and long range sustainable performance.

One key point needs to be made which is very often confused by organizations. The process of innovation and the process of idea implementation are two separate things. They are interconnected but they are distinctly different processes, requiring different types of people, and thinking modes. Idea implementation includes everything required to realize sustainable value from an idea. Many organizations have embraced an Idea Management System (IMS) to help manage this idea implementation process. Before doing that, you should ask yourself the following questions:

Until you have a clear understanding of the answers to these, you cannot create clear idea performance metrics that can illustrate your accurate position. There are many reasons why good and great ideas do not get implemented. The majority of these reasons have to do with the perceived risks.  Clarity of the risks versus the benefits is essential.  Any anticipated constraints to idea implementation must be identified and known to move forward. You need to know “Who” is needed for the idea implementation and their ability to support it.  Russell Ackoff states in his book Idealized Design that “a major reason that most plans are not fully implemented is that those people responsible for implementing it have no ownership of it”.

Additionally, it is essential for an implementation action plan to estimate cost, schedule and resource constraints to accurately create a measurable set of implementation idea metrics. Idea implementation is ultimately a social-political process requiring constant communication of idea metrics from concept to creation.

  Idea Metrics
Let’s make sure we understand what idea metrics are and why we need to create them. Idea metrics, as a minimum, represent the ROI for the organization based on a “return on investment” formula. Typically this is simply calculated as the amount of financial gain over the amount of total investment. A more complex financial and accounting formula adds the parameters of cost and benefits in addition to measuring the baseline performance. It is understandable that this type of calculation is confusing when we are measuring ideas. What makes it even more challenging is when the term “R.O.I.” has multiple meaning such as:

We are now potentially even more confused because the metrics language we have used previously, which describes financial situations seems to be illogical. The standard metric logic we use needs to be modified to make it adapt to a subjective “idea” mindset and philosophy. If we use a process similar to the Infinity Diagram identified below we can illustrate three different “Value” positions in the “idea” cycle. This can allow you to see the difference between a predicted value and a demonstrated value, both of which are critical for understanding a portion of your idea metrics.

Infinity Diagram

The figure above was presented by Professor Robert Shaw at a Direct Marketing Association (DMA) meeting to illustrate a lack of accountability in the 21st century for proper marketing valuation of ideas.

But what else should we be measuring and why…?

Let’s start with some basics that everyone seems to agree with. Most organizations agree that starting with a lot of ideas and filtering them down by evaluation to a select group for implementation is proper. But in your organization these steps may be completed informally and without records. Start by creating “Idea” metrics for:

When you have calculated this data as a minimum you can determine your total cost for each idea based upon the complexity of the idea and the process cost and time for implementation. Additionally, it will give you cycle time metrics for the time it takes for you to generate an “idea” from start to finish. This list is a recommended baseline but each organization needs to determine what additional data and metrics they want to capture. Remember that metrics are used to measure your efficiency or inefficiencies and what the cost of that is for every dollar you spend.

Some organizations want to know how many patents they generate from their ideas while others want to know what level of learning is being generated. It depends on your organization and what you feel is critical and beneficial to measure, but the option of measuring nothing is simply wrong. You need to find measurements and metrics that are useful not easy. Adopting another organizations metrics is like adopting another company’s strategy. Your idea metrics should be tied to your organizational goals and should be clearly visible as to what the performance expectations are for everyone in the “Value Chain” connected to your ideas.

This means you will need to partner with interconnected organizations and communicate what you’re doing and why. Getting started is easy…..but working efficiently with all of the stakeholders means your leaders need to speak a new language. They need to become idea enablers who can look for opportunities in addition to reducing risks. As a leader, you should ask yourself “did I  help to create a positive impact on the creation of a marketable idea today”? If you can answer “YES!” to this every day you’re helping to create a culture of change, opportunity and ideas for the future which will improve your financial bottom line.

Dale S. Deardorff

The “News from the Peak” Newsletter is FREE. And it's delivered to your email box at the transition of the quarterly season. If you have any comments, or questions or to learn more about our leadership and Innovation programs call us at 818-730-8203 or email: dale@rockypeaklc.com.

We want to hear from you. To subscribe or unsubscribe, visit our website: Feel free to forward this newsletter. Permission is granted to reproduce, copy, or distribute as long as copyright and full author contact information is included. Thank you.

Copyright Dale S. Deardorff, 2012. All rights reserved.