Numerical Rhetoric, Complexity as Deception, and the "Hindu Shuffle"
Pick a Card
Card tricks are a staple of stage magic. When performing a card trick, the magician will typically ask someone to "pick a card, any card", do something with it, then put it back. They will then say something (usually something amusing) while shuffling the cards over and over, commenting about how long they are shuffling and how mixed up the cards are getting. They will then reveal the card the person picked in some dramatic and exciting way, and receive applause.
But the key to many of these tricks is a method of shuffling known as the "Hindu Shuffle". This involves taking a stack of cards off the bottom of the deck, moving them out from under it, then bringing them back under the deck and taking another stack of cards off the bottom of the deck. One does this over and over until the entire deck has been "shuffled". In motion, it looks convincing -- the cards are moving, the entire deck goes through the process, and it truly appears that they are being mixed up, even if you are watching closely.
But if you think about how the cards are actually moving, it quickly becomes clear that the magician isn't shuffling the cards *at all*. They are just transferring the cards in their current order from one hand to the other and telling you they are mixing them up. It is basically just a more elaborate version of picking up a deck of cards by the sides and letting the stack fall back down.
Here is what it looks like in motion and how it allows all kinds of tricks and control.
There are additional ways to add even more flourish. You can do false cuts, where you cut the deck in half, move the top half underneath the bottom half, then move it out the other side of your hand and put it back on top. You can break it out into multiple stacks, move the stacks around, and then re-stack them in the same order but from different positions. And so on.
But the end result is that you aren't mixing the cards up at all. You are just moving them around.
The reason the Hindu Shuffle fools people is two fold:
1. There is the appearance of complex activity
2. The person doing it is confidently assuring you that something meaningful is happening
Keep this in mind as we move the main focus of our discussion: security and risk ratings/grades, scoring systems, and scientific management.
Security Vulnerabilities and Risk
When you start working in cybersecurity you will begin to encounter various versions of the following concepts (which are often misused and misunderstood in a way that only adds to the confusion).
- Vulnerability: a flaw or weakness in an asset’s design, implementation, or operation and management that could be exploited by a threat. "How could harm occur?"
- Threat: a potential for a threat agent to exploit a vulnerability. "Who or what could cause harm?"
- Risk: the potential for loss when the threat happens. Risk = (Probability that a threat occurs) * (Cost to the asset owner)
The ultimate business purpose of cybersecurity (and security in general) is to reduce Risk by either reducing the probability a threat will occur (ie patching vulnerabilities, making it harder to hack things, etc) and/or reducing the cost to the asset owner (ie redundancy, backups, ease of recovery, etc).
This is all fairly easy to follow on a conceptual level. The presence of an equation in there adds a flavor of empirical rigor to the idea, and it seems to greatly simplify what initially seems like something very complex.
But then you start building out lists of these things, and it gets complicated again. For instance, if an organization has 1,238 software vulnerabilities spread across 300 systems, what does that mean? Are you going to go through each vulnerability, list out all the ways that vulnerability could be exploited, do a bunch of research to calculate the probability of that occurring on various time scales, and...then what?
You probably want to deal with the highest risks first, so you identify the system with the most vulnerabilities and ask that all these vulnerabilities be patched.
You quickly hear back with an answer: "not a chance".
Because the system you picked is your core production system. It costs the organization $1.2 million per minute it is down, and patching all the vulnerabilities you identified will take 12 hours. So the downtime alone will cost about $14 million...and the maximum fine the government can impose if the entire system is breached is $10 million. So the Risk of patching this system is greater than the Risk of getting hacked.
So you bring up the fact that getting hacked costs more than just the fine. For one, there will be legally imposed downtime following a hack. But beyond that, there is the damage to the organization's reputation. There is the cost of lawsuits. And so on.
You are told to run the numbers for all of those and present a full report.
And around this time is when you start to see how impossible it is to actually apply this simply sort of framework to a real world organization. There are so many Vulnerabilities, Threats, and so much Risk that it is impossible to even list them, let alone prioritize them. And the longer you work on it, the more of a Threat *that* becomes -- while you are working, nothing is getting done, and the chances of an unknown unknown being exploited increase.
This is when people start to turn to severity ratings and scoring systems.
Severity Ratings and Scoring Systems
When you start working in cybersecurity, you also begin to encounter various versions of Critical, High, Medium, Low, and Informational Vulnerabilities. These ratings attempt to simplify the challenge we encountered earlier -- rather than forcing you to analyze each Vulnerability in detail just to understand the Threats involved and the Risk it adds, we simply adopt a quick and dirty method of approximating the urgency of a Vulnerability based on its likely affect on Risk and accept that there will be some errors and inefficiencies but assume that these will be outweighed by the speed and ease with which you can work on them.
But this prompts an obvious question: how are we deciding whether a Vulnerability is "Critical", "High", or whichever?
The answer to that is to use some vulnerability scoring system. The most commonly used one seems to be CVSS (there are multiple versions at this point), but there are others. It consists of a bunch of criteria that describe what the Vulnerability allows and how tricky it is to exploit and what happens to the affected asset if it is exploited --for instance, can the Vulnerability be exploited from over a network, or does it require local access first? Does the Vulnerability require Low or High technical sophistication to exploit? Does it impact Confidentiality, Integrity, and/or Availability of the Vulnerable asset, and if so is it Low or High impact? And so on.
These also include a bunch of mathematical weighting that shifts things around depending on which combinations of factors have been selected.
So the idea is that, when you discover a Vulnerability, you go through all these criteria questionnaire-style and make your selections, and the end result is a score 0.0 - 10.0. Critical Vulnerabilities are those that score 9.0 or higher. High Vulnerabilities are those that score 7.0 to 8.9. Medium Vulnerabilities are 4.0 to 6.9. Low Vulnerabilities are 3.9 or below.
This is a little arbitrary, but at a glance it seems reasonable. You're still dealing with thousands of vulnerabilities, so it generally isn't feasible for you to go through them all individually. Fortunately, pretty much all Vulnerability scanners will automatically categorize Vulnerabilities this way, so you can just work off of their ratings.
However, this just moves the problem -- you can now say that the production system has 20 Critical Vulnerabilities, 34 High Vulnerabilities, 120 Medium Vulnerabilities, and 459 Low Vulnerabilities, but if fixing even just the Criticals results in unacceptable downtime, the answer will still be "no".
This is when people might seek to move to an Overall Security grade.
Overall Security Grade
Many penetration test results and security consultants will include some sort of "Security Grade" on their reports. This may be a A, B, C, D, or F, like in grade school. Or it might be an actual score out of 100.
The way these grades are calculated will be very complex, and honestly I have never read through it all. Typically, it will involve assigning a number to Vulnerability Severities from above and adding them up, including certain additional weighting based on the significance of the system (is the affected system Critical, Important, or Peripheral, etc), and then converting that to some percentage and multiplying by 100 (and then possibly assigning a letter grade based on that).
The way these are typically used, the report will include the overall grade, a bunch of graphs and analysis, and then a recommendation that, if the org fixes some assortment of Vulnerabilities, it will improve their Overall Security Grade by some amount that makes it seem more "acceptable".
Then, if history is any guide, the org will fix those Vulnerabilities, feel good about it, and eventually get breached because of some Vulnerability they didn't even know about because it was present on a system the Vulnerability scanner wasn't scanning.
"Numerical Rhetoric"
So if we look at all of this, we see that it is basically an attempt to answer the question "what do I need to do to be safe?"
But in my experience, the answer to this is often quite straightforward and doesn't require all these calculations. Experienced security staff can often tell intuitively which systems and which Vulnerabilities are most likely to present a problem and should therefore be fixed first.
The problem is that this is rarely persuasive to upper management, especially if it involves expensive downtime. They are often hesitant to do this based solely on the opinion of their security personnel. So they will ask for additional analysis. But upper management tends not to like to read large amounts of text, so security teams are put into an awkward position: they need to convince upper management to listen to their recommendation without making them read very much.
This is where folks tend to resort to what I call "numerical rhetoric" -- that is, using numbers to convince people to do what you say.
Numbers carry a weird sort of authority because it makes whatever they are attached to seem like math, and math is the language of science, and managers like to feel like they are being scientific in their decisions. You can also fit them into small amounts of space and use them to make things look simple. And generally if you can show a number going up or down, it makes it seem like something is happening.
The problem with all this is that it is an illusion. It is the Hindu Shuffle. It is simply taking an opinion and repackaging it.
For instance, take Vulnerability Severity Scoring. The actual rating criteria for these is highly subjective and mutable. For instance, what is the difference between "Low" privilege and "High" privilege? Obviously, an Admin user would be "High" privilege...but there may be multiple intermediate levels that don't obviously fit in there. Also, this can be highly situational -- a Vulnerability that requires "High" privilege will be rated as less severe than one that only requires "Low" privilege...but if your organization is running so that all regular users are local admins on their workstations, then basically *everybody* is operating at "High" privilege level from the beginning. So in that case *all* affected Vulnerabilities of this type should be adjusted to require only "Low" privilege...but depending on what a person wants to do, they could easily do this or refrain from doing this. And this can make the difference between, for instance, a Medium and a High severity Vulnerability.
The problem is ultimately that organizations are relying on the same people to jump through all these hoops in order to justify their recommendation. So if you don't believe me that a Vulnerability is sufficiently bad that it justifies downtime to fix it because of my judgement and experience, why would you believe me if I tell you that this Vulnerability is Critical Severity based on CVSSv3 and is single-handedly reducing our Security Grade by 24%?
Because I can make the numbers say that. These scoring systems are almost designed to facilitate this. And unless you want to dig through all of the text about how these scoring systems work and what my specific selections were (and you don't), you aren't going to be in a position to question this.
Complexity as Deception
Another way these sorts of "quantitative" security schemes are like the Hindu Shuffle is that they introduce complexity without adding any new information or perspective.
For instance, me scoring a Vulnerability using the CVSSv3 method is still just me and my knowledge and experience and opinion. The scoring system itself isn't adding any additional insight -- if anything, it may be *subtracting* insight because it is generalized standard made by people who made assumptions that may be different than we are making. So it is simply converting my opinion from one form into another...just like a magician isn't mixing up the cards but rather simply moving them from one hand to another via a complex series of steps.
So there is no rational reason this should add confidence or authority to anything. But nevertheless, that is *exactly* what it does.
For example, let's say that two programmers are offering you a solution to some problem and are demonstrating their code. One programmer has 10 lines of code that, when run, immediately returns some calculated result based on a given dataset.
The other programmer shows you a huge file containing hundreds of lines of code, jumping from section to section to highlight and call out certain portions of the script, and then, after explaining it, runs it...and after a little over a minute of intensive processing, it returns a screen full of numbers, does another round of processing, and finally returns the calculated value you are interested in.
Which programmer do you have more confidence in?
The only correct answer here is that you shouldn't have *any* opinion on that question, because I haven't given you any relevant information towards deciding that! The amount of code has absolutely nothing to do with whether it is providing a good result. You are only going to know anything about the quality of code if you have some basis for evaluating the result, or if you actually go through and analyze the code. More code takes longer to analyze, and so it tends to dissuade people from looking too closely...but it also makes it look like that person spent more time and effort on it, which leads you to assume they were more thorough.
I've seen and participated in several instances like this, and I have yet to see the person with the shorter code win out. It makes no sense...but that is *absolutely* how it works in practice. The appearance of greater complexity imparts additional authority on whatever it accompanies.
Note: this is basically the entire idea behind religion...that is a conversation for another day, but I wanted to mention it, because this is an *incredibly* broad tendency of humans.
Conclusion
The reason I bring this is up is because these sorts of questions consume a *ton* of resources in cybersecurity. Many people have devoted their entire lives to coming up with these methods and standards, and many companies have paid people millions of dollars to sort through this stuff.
But at the end of the day, none of it really means anything, sad to say.
At the end of the day, all we're doing is taking an opinion based on experience, judgement, and ultimately vibes, and running it through a process to trick people into taking us seriously...and I don't think this is a good way to approach security.
In my view, the foundation of cybersecurity is Truth. It is about taking complex systems, stripping away all expectations and assumptions, and finding out what they are capable of so that it can be faced and dealt with. It is basically the Hacker's Ethos -- using systems to achieve objectives other than those for which they were intended to achieve.
Obscuring this with numerical rhetoric, deceptive complexity, and by doing the Hindu Shuffle is counter-productive. At best, it perpetuates misinformation about cybersecurity by fooling ignorant people rather than educating them until they can make intelligent decisions. But more problematically, it disconnects us and everyone else from reality...which is the *opposite* of science.
I am a big believer in science. I think it has improved our lives more than just about any other factor. And I understand the desire to apply scientific methodology to cybersecurity, because the two have a lot of similarities -- science *also* seeks the Truth, after all.
But you can't fake it till you make it with science. The simple fact that we are cloaking our opinions in the aesthetics of scientific management and numbers does not make it any more scientific. And I think it ultimately discredits science and us to resort to such methods.