A lot of my blog posts so far have been about the need to understand the relevance of political processes, values and the importance of qualitative judgments in decision-making. I’m certainly not the sort of guy you’d accuse of being against quantitative analysis—I’m a huge fan—however, it doesn’t explain everything and thinking it does can be incredibly dangerous. In this post, I’ll make the case for taking the time to learn quantitative skills, but also talk about how and why they can be a double-edged sword.
Quantitative skills are really important and valuable
While both quantitative and qualitative analytical skills are really important, the reality is that in the field of policy and government generally, you find a lot of people that have always operated at a pretty high level of abstraction. They might be masters of qualitative analysis, but few of them actually have direct experience undertaking really solid, robust quantitative analysis. This scarcity means that people that can do and understand quantitative analysis at a deep level are extremely valuable.
One of the most common pieces of advice I give to people in the early years of their careers in my field is to get up to their elbows in quantitative analysis as quickly and as often as they can. This is especially important early in your career because applying these techniques to the real world takes a lot of trial and error (and there are a lot of errors). When you get more senior (and more expensive), people are less willing to have you sit around with a model for days at a time trying to figure out why you can’t get your balance sheet to balance.
I like to say you’re not a real modeller until you have been standing in front of a room full of important people presenting your findings and someone asks a question that makes you realise you have made a mistake in your calculations. If you do enough modelling, this will happen. Better that it happens earlier in your career than later.
If you’ve never done the analysis yourself, you won’t have the same feel for how outcomes change. Certain questions that you might think are impossible, are, in fact, easy to answer. Conversely, certain questions that seem trivial, are really, really hard. Knowing the difference can save you lots of time and money.
This comic is from one of my favourite webcomics, Xkcd. Click through for the alt-text.
How big an impact will the exchange rate have on this project? Can we save money if we make the project smaller? Will giving the industry an allowance for accelerated depreciation help them expand more quickly? Why do we need to index this portion of the payment to the coal price? These are questions that get asked by important people when they are trying to make important decisions. If you have done a whole bunch of number crunching in the relevant sector, you might be able to answer these sorts of questions in seconds, or at least promise an answer within a reasonable timeframe. If no one in the room understands the numbers, most likely the questions will get answered incorrectly, or forgotten about and you increase the chance that you’ll get a bad outcome.
Quantitative analysis is not all that matters
In the real world, there's almost no such thing as a purely quantitative problem. Even if your client has asked you to build a simple financial model, given you all of the inputs, as a good modeller, you should make sure they understand the implications of their choices of inputs. How might the outputs vary if you choose different inputs? Are there hidden assumptions or logical flaws in some aspect of the model?
Your mathematical calculations may be perfect, but you need to be able to take a step back and think about how your numbers relate to the real world. For example, someone might ask you to run a scenario in your public transport model with high ticket prices and high ridership. Your model can certainly do the mathematical calculation, but is it likely that you will be able to get high ridership at the high prices assumed? What happens if you assume this and you’re wrong? Who bears this risk? What are the societal impacts of charging those higher ticket prices?
As I said earlier, there are relatively few people that can do sophisticated quantitative analysis, but among that population, those that really understand the real world implications of the models they make are rarer still.
Comic from Saturday Morning Breakfast Cereal, one of my other favourite webcomics. Click through for the alt-text. You could definitely replace "engineers" in the comic above with "finance professionals", or "economists"...
There’s nothing more dangerous than a genius with a model. When they’re wrong, which will happen, it will be hard to find out. Partly it’s hard because they may have used complicated techniques that are difficult to follow, but it’s also hard because they tend to be certain that they are right, and might scoff at the idea that mere mortals could identify a problem with their analysis.
When I think about my most valuable employees, colleagues and bosses, the best ones are those that have strong quantitative and qualitative skills. Following them, I'll take strong qualitative skills alone before I'll take strong quantitative skills.
The Nusantara Economics matrix of employee value
If I’m forming a team of Bs and Cs, following the Nusantara Economics matrix of employee value above, I’ll definitely put a B in charge, because they are the ones that do real-world sense checks, and can communicate the analysis to the client or counterpart.
Striking the balance
You can cause problems by either over-weighting quantitative or qualitative concerns in decision-making. The sub-prime mortgage crisis is a pretty galling example of the former, while Indonesia's insistence on being self-sufficient in beef is an example of the latter.
When you look at a problem, you'll need to determine what the appropriate balance of quantitative and qualitative factors will be. For something like how determining the alignment of a power transmission line, maybe its 80/20 quantitative/qualitative. For something like designing standard operating procedures for how child protection services engage with families of abused children, it's probably 10/90. The right balance will be different for every problem, and will even change for the same problem over time or in different contexts.
If you recognise this, and actively seek to find the right balance of quantitative and qualitative factors in your analysis, then I think you'll have the best chance of delivering great analysis that helps people make the best decisions in truly difficult circumstances.