While this point arose from a discussion on the ISO-27001 mailing list, in other InfoSec/Audit forums I’m known as a strong proponent of Failure Mode Effect Analysis (FMEA), to the point where people naturally associated it with me. However it clear in my 20+ years in InfoSec in a number of countries (including the UK, where ‘7799 and 2700x come from) that the accepted FMEA approach is not a normal method of ‘risk assessment‘ and is not taught (or examined for) as part of InfoSec, for example as part of the CISSP.
I learnt FMEA (and other techniques we now bundle under 6-sigma, ITIL and so forth, before they were categorized and labelled) in engineering – physical, electrical and aviation. In general, I’d say that InfoSec has a lot to learn from other engineering professions about managing threats, vulnerabilities and failures, and what actually constitutes “risk”. For a start, we have too much of a techie-geek outlook and we are not well educated in statistical methods.
Just giving numbers is meaningless. I will not trust any ‘graph’ that does not have ‘error bars’ and which does not document the sample size vs the total population and show the variance, for example against a random population. These test are easy to perform and I’m disappointed that so many numerically justified ‘risk analysis’ models are really just a pile of spread-sheet mumbo jumbo with no discipline of process behind them.
Estimating “on a scale of 1-5” the components of the risk equation for many factors then multiplying out and averaging is pretty meaningless, yet I’ve seen it done by consultants from TLA companies and accepted by managers. At the very least it ignores cross interactions, and is essentially just a “your guess is a good as mine” approach.
I have nothing against opinions about risks, what I do object to is trying to make opinions into solid numbers. That “estimate on a scale of 1-5” will have an error bar of size >= 5 !
Lets face it, most people don’t understand statistics. All to often I’ve seen managers ignore the “once in 100 years” MTBF (and ignore any MTTR – see FMEA) because they only plan to be with the company for five years, or ignore it because it happened 25 years ago so they think they have a 75 year breathing space. Yes, I know it sounds apocryphal, but I’ve met it too often.
To my mind FMEA is not only easier than TRA, but it focuses the mind on two key issues – survival and recovery (see MTTR) – that TRA doesn’t.
