We’re living in a brave new world of analytics, machine learning, and artificial intelligence—and even the lumbering U.S. military is catching on. For example, the Army has developed an “Enterprise Data Analytics Strategy,” the Navy’s Fleet Forces Command is embarking on an initiative to use analytics to support Fleet readiness, and apparently the Department of Defense has an Office of People Analytics.
Motivating the military’s move to analytics is the argument that they will make assessments more sound and less subjective . . . because numbers are truth. But, a numerical assessment is at best incomplete, and blind confidence in it can be dangerous. Numbers can be misconstrued, can lack context, and can be, dare we say it, subjective.
Numbers can be misconstrued or misrepresented. There are a million variations of the quote about lies, damn lies, and statistics, or that 90 percent of statistics are made up. The way you cut the data, the assumptions you make, and the outliers you reject can dictate what the numbers ultimately say more than the numbers themselves . . . and those decisions are largely hidden behind a graph and unappreciated by its consumer. In attempts to paint a rosier than accurate picture of progress in developing the Afghan National Police, Stephen Downes-Martin noted that a statistic describing their ranks as “100 percent filled” failed to capture the fact that they were “filled” with a disproportionate and problematic number of lower ranked patrolmen, and not nearly enough leadership. The British government has reported that nearly half of all terrorism arrests are of people aged 30 and over, but divides the 18-to-29-year old population into four (uneven) segments and groups everyone 30 and over together. That is to say, 18 to 29-year-olds also represent half of the arrests…but because they are divided into 18 to 20, 21 to 24, and 25 to 29, the 30 and older group looks by far the largest.
Numbers can lack context. Military training requirements are often described in terms of tasks, conditions, and standards. Certainly, the completion of a particular task can be captured numerically (1 or 0, yes or no), and success can be defined in terms of things like number of bombs on target or percent of submarines identified. However, the corresponding conditions are much harder. How do you describe the weather that day, the visibility, the quality of the adversary emulation? These all can vary from event to event, making analysis across training performance data sets they may have been collected under different conditions suspect. Numbers without context also can yield confusing results. For instance, the military was faced with a vexing paradox of increasing reports of sexual assault despite its concerted prevention training. This trend makes more sense when paired with the knowledge that the military was also educating its service members on reporting resources and mechanisms, which may have increased the proportion of assault victims making reports.
Numbers can be subjective. Over my decade plus time as an analyst supporting the military, I’ve seen many attempts by well-meaning staffs to assign numbers to the red-yellow-green stoplight charts the military regular employs. The problem with assigning quantitative (numerical) values to qualitative (red/yellow/green) assessments is that it can imbue the data with relationships that they don’t have. For instance—if you assigned red a value of 1 and green a value of 3, that implies that the green thing is 3 times better than the red. If you assigned that 1 and 10, respectively, now that green thing is TEN times better. Similar observations have been made by my colleagues, including one particularly pointed critique of operational assessments by Jonathan Schroden. In that case, he pointed out that the Army’s doctrinal example of an assessment framework involved comparing numerical metrics with different units, and assigning them subjective weights—“thereby undermining its ‘mathematical rigor.’” Making qualitative data quantitative can thus create subjective quantitative data.
Conversely, not all qualitative data are necessarily subjective. Consider the simple example of hair color. For the most part, you can easily—and with little variation from one observer to the next—describe someone’s hair color in words—that is, qualitatively. But how would you quantify that? Should brown be worth more or less than blond, and how much? Returning to the weather conditions in our training example—although sea state may be quantifiable, what about precipitation? How would you describe, numerically, the difference between fog, drizzling, light rain, and a downpour? Is someone tracking that with a rain gauge in every training event?
Yes, quantitative data and analyses—particularly those enabled by big data and machines learning—hold great promise for our military to better manage readiness and improve decision-making. But they do not obviate the role of qualitative data, and relying solely on them will not save our military from “subjective assessments.”
What can save it, then? First, accepting that blind faith in numbers is risky and unfounded. And then, ensuring that people trained in understanding the merits and challenges of numerical analyses are allowed into the circle of trust and involved in the decisions to employ them. These personnel will dig into the details and look for subjectivity red flags, such as:
- Quantitative data are measuring things that cannot be counted or do not have logical units of measure
- Quantitative data are not captured under reasonably consistent conditions
- Quantitative analyses do not come with explicit description of assumptions and other analytic choices.
Decision-makers and action officers would do well to leverage these experts when considering the merits of new computational options to answer their questions. The sooner we can all appreciate that subjective and qualitative are not equivalent, the sooner our military may best avail itself of the benefits that quantitative analysis can provide.
