Prioritization is a frequent problem. Whether it’s running a business, preparing for a show, or organizing a busy day-to-day life, deciding where to expend effort and time is of paramount importance. Alone or in groups, it’s not always evident where to start, where to finish, or how much will be accomplished. While small tasks and endeavors can survive a lack of formalized planning and prioritization, larger efforts can grind to a disastrous halt, leave efforts half-finished, or have urgent priorities left till the end, often without sufficient time to resolve them. It is imperative in these cases to have both a formalized language or process for decision (even if it’s simple), and to ensure that you are making decisions with as much data as possible.
The book and movie “Moneyball” tell the story of the Oakland Athletics and a radically disruptive approach to decision-making. While frequently the limiting factor in accomplishment is time, for the A’s it was money. As the main character, general manager Billy Beane notes (roughly paraphrased), “There are well-paid teams. Then there are poorly paid teams. Then there is dogshit. Then there is us.” In the world of professional baseball, leveling measures such as salary caps weren’t present at the time. The A’s were trying to figure out how to make a team competitive enough to garner fans without in advance having the money that said fans would provide. Their novel solution was to make recruitment about data (known as “sabermetrics”).
Prior to this innovation, there were qualitative, emotional, and anecdotal reasons for taking one candidate over another. “Big hitter” was a frequent justification for a given player, with most of the money allocated to a few big stars and/or personalities and the remainder on ever-less-valuable players. The supposed “quality” of a players’ girlfriend was even brought to bear, reasoning that it resulted in a more confident player. The disruptive change, brought by a wall-street oriented numbers guy, was that the best predictor of “value” in a baseball player wasn’t the swing, or their appearance, or the appearance of their partner. It wasn’t even their batting average. It was just the likelihood that upon coming up to bat, they would be on base. This was true even if they didn’t hit the ball; in fact, they could be unlucky enough to never even take a hittable pitch. They just needed to get on base.
This philosophy led the team to take what at the time seemed unthinkable moves. Hiring supposed “has-beens” like David Justice. Eschewing power hitters and stardom for predictable, even boring, behavior. The results were anything but boring. In 2006, using the sabermetrics strategy, the Oakland A’s had the 5th best performance while being the 24th-lowest paid team (out of 30).
Most of us, and certainly myself, are not focused on baseball. We’re maybe not even focused on recruiting, or the salary limitations that come with it. But whether we are or not, the lessons of Moneyball, and of deciding with data, are critically valuable. Without understanding where “value” is in a potential list of things to do/accomplish, we will naturally fall back to oversimplified anecdote, hyperbole, and emotion. But with data, the way we make decisions is enhanced. It focuses our efforts on why a thing is valuable, and notably NOT why we want to do it. This is a disruptive premise for teams. Don’t tell me what you want to do. I don’t care. Tell me where the value is. And we should all objectively agree that we want to do things that are valuable.
A frequent exercise in my career, whether in professional or nonprofit work, is to be provided a list of potential work (the “backlog”), and have one or more leads or team members collectively decide what to focus on. The most common approach to this is “thumbs” or “stars”. Each person is given a certain number of tokens or votes, and asked to weigh in on the list of work. This is typically done when a basic understanding of the “What” is present, but rarely the “Why”, “When”, “Who”, or “How”. And that’s mostly okay, as it also is an inefficient use of time to drill into all of those details on work that you’re ultimately not going to decide to do. Mostly okay, but not “all” okay, because the “why” is being omitted. And not only should “why” be understood, it needs to be codified and quantified, or it becomes the parent’s last-resort justification of authority: because I said so. Or the child’s: because I want it.
Valuation is an exercise in estimation, and of relative comparison. There are many different tools and strategies for expressing value; it is not necessary to invent your own, and often not desirable. An example of a simple, one-dimensional metric is pain score. This attempts to estimate the relative pain or friction being experienced by a user or audience, and can be effectively applied to bugs or problem reports and used to target user experience and usability improvements. While a single metric may still be gameable (one can express personal opinions and bias in the pain score), it still focuses the “why” to a specific dimension (user pain). Hence, “I like cornflower blue”, for example, will likely not bubble to the top, no matter how much a fictional mid-level executive may prefer it.
Valuation gets more informative as more dimensions are considered and encoded. There is a falloff point, to be certain, where you are overanalyzing and/or over-encoding things you might know. The sweet spot, as with so many things, is “the simplest thing that works and adds value”. Two-axis valuations can be far more valuable, as they’re able to encode and quantify competing considerations. Many years ago while working for a nonprofit, we applied a strategy of estimating importance and urgency. Importance was specifically NOT temporal. It set time aside, and only focused on how overall necessary a thing was to have. Urgency was entirely time-focused, and related to how soon action would need to be taken. This helped disambiguate, and via a simple scatter plot, visualize where immediate action needed to be taken (such as disaster recovery). It highlighted important efforts that couldn’t be discarded, but also weren’t time sensitive and just needed to stay “on the radar”. And more importantly, it shifted the discussion in the room between what people cared about and wanted (spoiler alert: everything) to what the relative importance and urgency of work was. In the end, it was clear and unambiguous why certain work would (or wouldn’t) be prioritized.
While importance/urgency is a useful strategy, and certainly more valuable than thumb-in-the-wind voting and emotional sentiment, it misses some key considerations that may be critical to your decision-making process. In particular, it ignores the effort behind work. And while we don’t want to design-in-advance too early on, it is important to delve a bit into the “How” to determine value. In particular, if you are in a resource-constrained environment (time or money), then it is critically important to understand the distinctions between high- and low- effort work. If we imagine our Moneyball example, this is the first strategy that could be applied to recruitment. Pain score or importance/urgency may work well for tasks, but they don’t work well for big groups of projects, especially when you don’t expect to get to everything. Instead, you can analyze and encode impact and effort, which can be far more useful. Impact looks at the effectiveness of the work, and Effort looks at the expenditure to accomplish it. In our baseball example, impact would be the likelihood that a given player gets on base, and effort would be the salary they expect. Once quantified, it is a simple matter of dividing impact by effort, and using the resulting score as a guide to the most valuable players (the biggest bang for the buck).
There are a couple of important principles to keep in mind when performing quantitative estimation. The first is to be “relatively” true and focus on estimates. Clothing sizing can be a useful metric at the early stage (XS/S/M/L/XL), where the extents of the metrics (extra-small and extra large) represent the biggest and smallest things in your possible list of work. You are not looking for absolute truth (ie: is this a 40 hr. job?), but a relative one (is the effort of Project A greater than Project B?). The second principle is human-determinism. Even before the rise of AI and automated decision-making, I have always been skeptical of letting formulas make decisions, especially imperfect ones. The valuation score that your strategy provides should be a guide, not a rule, on how you prioritize your work. In fact, when providing these strategies, I always leave a free-text area to explain why given work still might need to be done “first”. In my experience, these have been as simple as, “a vice-president said so”, and “we won’t actually get the contract if we don’t do this”. Rather than infinitely game an impact or effort score to get a desired result, you should let those relative estimates stand, and just write down your rationale for future reference and justification.
Over the evolution of my career in business and nonprofits, the scale, complexity, and number of resources available to accomplish work grew significantly. The goal of the valuation exercise is to sufficiently disambiguate work so that the “value” is relatively clear. To scale, you can look at different dimensions (evolving from importance/urgency to impact/effort), or you can add additional dimensions to further quantify important factors in your valuation. The RICE methodology adds two dimensions to impact/effort: reach and confidence. Reach quantifies the relative audience size, and adds important considerations to your value: will this work provide a small benefit to a large audience, or a large benefit to a smal audience? Is the work we’re doing targeting all of our expected audiences? This pushes a bit beyond the pure “Why” and begins to encroach on the “Who”.
Additionally, the RICE strategy employs confidence, a measure of how certain you are about your estimates. This approach has been successful and found adoption in engineering organizations, particularly those with large corpuses of historical data to fall back on for their estimates such as aerospace and pharmaceuticals. Frequently, however, this data is not available. This is especially true of smaller businesses, rapidly changing ones, and community nonprofits, all of which for various reasons rarely have a battery of historical quantitative research and trending to draw on. This makes the metric difficult to estimate, and a bit oversimplified, as it doesn’t disambiguate confidence in impact (ie: market research) from confidence in effort (ie: engineering maturity). And especially for new teams, effort estimation is frequently fraught with peril. As we’re established earlier, metrics that are vague and oversimplified are ripe with opportunity to be made emotional metrics. I can use confidence to encode a gut feeling, and wildly affect the valuation based on it. Considering that the goal of this endeavor is to quantify our decisions to data, and remove emotionality as much as possible, this is highly counter-productive.
All change is disruptive; even beneficial change. And everything you’re planning to do is almost certainly going to come with a non-zero amount of disruption, whether you are changing a behavior, stopping it, or starting it. And a given audience - whether it’s end-users and customers, or your own employees and engineers - has a certain tolerance for disruption, beyond which satisfaction and effectiveness plummet. To this end, I suggest an evolution and modification of RICE to RIDE – Reach, Impact, Disruption, and Effort. Similar to Effort’s inherent capacity limits (be they time, money, or something else), your audience has a tolerance for a certain level of disruption. Pack too many disruptive changes (even highly valuable ones) and you may do more damage than improvement.
There is a lot of flexibility in the valuation strategy you can use. From single-metric analysis like Pain Score to robust dimensions like RICE and RIDE, what is important is to find the strategy that encodes what matters to your effort – if you only have one audience segment, and all of your work affects them equally, including Reach is probably a waste of time. But find the right set of dimensions and your team, group, or even yourself alone will start reasoning, discussing, and deciding on priorities based on value, not emotion. Let emotion guide you to your value, not dictate it.
Did this message resonate with you? Have you used these (or other) valuation strategies in the past? How did they work out? I’d love to hear feedback, reflections, and recommendations on other ways to move beyond emotion and “gut”, and to start deciding with data.
Comments