Software Risk Analysis

[Symbology]

So where are we going with all this?

Lets say a perfect software risk algorithm is created and we can assign nice neat scores to each module. If we discover high risk modules, what will be done with them?

• Will they be assigned additional scrutiny and/or testing?
• Do they need to have backup code created and run in parallel like the shuttles 5ish side by side decision makers?

Knowing what will be done about it can help reverse engineer and tune what is trying to be scored and how.

[LaurieAG]

Originally Posted by Symbology

Lets say a perfect software risk algorithm is created and we can assign nice neat scores to each module. If we discover high risk modules, what will be done with them?

Knowing what will be done about it can help reverse engineer and tune what is trying to be scored and how.

Hi Symbology,

At the moment the shuttle has been grounded because of a faulty fuel guage that shows empty when it's nearly full. NASA knows that the fuel tank is nearly full because they have put so much fuel in without it leaking out but the guage disagrees. The temperature environment in the tank changed when the fuel started to be added. This probably has something to do with the problem.

This type of problem goes beyond software risk analysis and the solution is a holistic combination of all the factors that directly contribute to the problem, before the problem develops (i.e. environment, function, operation etc).

I think this is what Pyro is after, a generic early warning system for human intervention if the software stuffs up by going to or past its boundaries, because if it knew it was stuffing up it could fix the problem itself without any intervention from anybody (which is ideal but not realistic due to the nature of the problem(s)).

[alexander]

Hi, back from a mini-leaveation ( leavation - a leave in a vacation sense, only where you didnt really go anywhere, and were working the entire time)

It's not even about FINDING bugs or flaws.

Right, that is what software vulnerability assessment is for by many various means....

Ok, so thoughts as to how you can do Software Risk Analysis...

Firstly i need to tell you that it is not something that can have concrete mathematics, nothing like probablility, although some math may be needed. In all reality this one of the topics i could write a thesis on, and although i dont know it very extremely well, i already have lots and lots of thoughts as to what factors should contribute...

so thoughs:

First of all, just how do you quantify software risk? It is a highly complex set of factors, and many a things need to be looked at before you can begin... Remember some of these pertain to an organization, so if you are auditing some software for somebody else, you may or may not apply some of the points below.

Application - what the apllication for the software is, what kinds of data it holds, what kinds of settings there are, permissions, so forth... (on a scale of pointless data to high risk stuff)

People - Who is responsible for the software, who has the ability to change data, who has the ability to view data, who has the ability to add data, within all those factors a person using this software may be the biggest determining factor for software risk... (more people, more chance of a mishap)

Importance - related to application, importance is more towards the line of, what will happen if a part or parts of this software stop functioning, how damaging would it be to a business. For example, if the software stops displaying data, would it hinder the performance of a business using it, if so what amount of hinderance are we talking here? remember consider short-term and long-term cases... what if there is a miscalcualtion in the output... all that stuff. (a combination of numbers and verbal assessment is here)

Software security - yes you need to look at this, if you are a high class pro, you should assess how secure the software is, if you are not, look at the regularity of fixes administered, try to find patterns in them. Best yet try to find patterns in fixes, and do your own assessment, various ways to do so, it's best to try a few, like fuzzing... (on a level of not secure to secure)

Costs - for every question under importance, what would be the worst-case cost of anything happening in that category, factor in the code and software security sections. Also remember that time is also a cost, possibility of a lawsuit is another one. (you can get solid numbers out of this)

Data Security - determine where the data is stored, how it is stored, if its backed up, people that have access to those places, and determine the probability of data loss, including hardware security, including natural processes, fire, thunderstorms, etc... (for some of this, you can do hard statistical analysis with complex math )


look at this too Software Metrics Program for Risk Assessment

and lastly this Software Risk Assessment

both of the links have some interesting info, some of it will repeat me, some of it will be new...

i hope some of my thoughts dont sound too crazy

[Pyrotex]

No, it doesn't sound crazy at all. However, not all of it applies to my situation. Risk from hackers/crackers & vandalism is not an issue here. Analyzing the risk (probability) that a piece of software will "fail" in some way, leading to disaster ("crash!") is a very difficult problem that has been addressed only in limited cases.

One way of doing this is by Bayesian Updates. This can be summarized by the question, "Given that the probability of failure was initially F1, what is the probability of failure now (F2), now that the software has been used X times without a failure?"

--optionally--

"...now that the software has been used X times and it has only failed Z times?"

In certain very constrained cases, where there is a mammoth body of real-world experience with the software, this is a successful approach. You start off by showing that after integration testing, 5 lethal bugs were found, and we estimate (loosely) from this that the risk of disaster is 1 / 200.

After 1,000 actual runs of the software, we have 1 disaster. We do the math, and this yields a new risk of disaster is now 1 / 465.

After another 1,000 actual runs, with no disasters, the math yields an updated risk of 1 / 970. And so on, and so on. Presumably, the longer this goes on, the more accurate the updated risk probability is.

But with avionics software, say, like the stuff in the Shuttle, you just cannot run it often enough to generate the kind of disaster data you need.

Real-world execution of software in a challenging environment (guiding a jet with terrain surveillance radar, or guiding a re-entry vehicle into the atmosphere, for example--where "disaster" means Loss of Vehicle) operates in rapid interface with a highly dynamic environment, which cannot be totally predicted. The volume of input parameters attempts to model the external environment and vehicle state, but never can completely. What if the software is asked to handle 2 boundary problems ("edge of the model") at the same time? How rare an occurrence is that?

Testing HW essentially performs a massively parallel set of tests on every atom, molecule and component simultaneously--until one of them breaks. Testing SW is essentially a sequential process, testing each one of the millions of paths one at a time. A small compiler or command-and-control module could potentially have 10^15 paths. Or 10^20. Combinatorics make exhaustive testing impossible.

What if there is a pure math approach similar to Bayesian Update or Weibull Analysis that can take a body of code, analyze it for complexity (path density), and then use historic information to calculate the probable number of bugs still in there undetected?

Hmmmmmm, he said.

[alexander]

Risk from hackers/crackers & vandalism is not an issue here.

Pyro, it ALWAYS is an issue, no matter how much you try to deny it, always is and will be for still quite some time!

[Symbology]

Originally Posted by alexander

Pyro, it ALWAYS is an issue, no matter how much you try to deny it, always is and will be for still quite some time!

Only if you are connected to the Internet, and only if you have no physical security. If you are not connected to the internet then its purely up to physical security. Which I bet is pretty good in this case.

But it can never hurt to be too careful. It only costs you time and resources.

[Symbology]

Originally Posted by Pyrotex

Testing HW essentially performs a massively parallel set of tests on every atom, molecule and component simultaneously--until one of them breaks. Testing SW is essentially a sequential process, testing each one of the millions of paths one at a time. A small compiler or command-and-control module could potentially have 10^15 paths. Or 10^20. Combinatorics make exhaustive testing impossible.

Combinatorial Explosion is usually handled very well by an inference engine.

Using something like a Rete Network, the system can know which specific rules are eligible to fire (R1, R2, R3 etc) when an alpha variable is changed by a rule or a beta variable is changed by an external source such as an interrupt or I/O, instead of having to reevaluate the entire system. The trick here would be how to treat modules of code like rules in an inference engine.

Basically if there is a mapping between what variables a module can modify vs which variables are inputs to a module, then you know the dependencies.

[LaurieAG]

Originally Posted by Pyrotex

What if there is a pure math approach similar to Bayesian Update or Weibull Analysis that can take a body of code, analyze it for complexity (path density), and then use historic information to calculate the probable number of bugs still in there undetected?

Hmmmmmm, and you could use the new analysis program to determine how many undetected bugs you had in the new analysis program.

Originally Posted by Pyrotex

Real-world execution of software in a challenging environment (guiding a jet with terrain surveillance radar, or guiding a re-entry vehicle into the atmosphere, for example--where "disaster" means Loss of Vehicle) operates in rapid interface with a highly dynamic environment, which cannot be totally predicted. The volume of input parameters attempts to model the external environment and vehicle state, but never can completely. What if the software is asked to handle 2 boundary problems ("edge of the model") at the same time? How rare an occurrence is that?

What's wrong with testing all combinations of boundary conditions beforehand to identify the real nasty combinations that should be tracked in real time? Also, otherwise benign (combinations of) boundary conditions may actually become critical at/after a certain time in a mission, i.e. after certain points previously benign factors can become critical and potentially critical ones can become benign.

[CraigD]

As this thread still seems open and brainstorm-y, and I’ve not seen mention of it yet, I throw out a concept: provable correctness.

This was a largely hypothetical approach to software popular, as best I can tell, mostly in the 1980s. Basically, it required that language compilers be what one book I read called “higher order”, so that one could prove that all the object/executable code they produced could function only within a formally well-defined domain, corresponding roughly to “bug free”

A common acryonym-oid for such software was/is ProCoS. A “popular” languages of this kind was called AXIS. Here is as basically random web-search result finding reference to both.

Along with a couple of management folk, I tried to stir interest in ProCoS within my org in the early 1990s, with little success, encountering daunting “cultural” obstacles. To this day, I often find it difficult to successfully argue much weaker applications of such an approach, such as the idea that very formally defined unit testing can substantially reduce the need for end-to-end integration and regression testing. In some projects in which I’ve had personal stake, this what-I-perceive-as cultural barrier has, I’m convinced, incurred million of unnecessary cost, and in the worst case, actually made resources (including me) unavailable for unit testing, resulting in significantly more undetected defects in deployed software.

While I suspect the culture at the JSC to be a bit … well, better than in my medical IT world – that discussions like these are even coming out of it appears to confirm my suspicion – the basic ProCoS concept – that special language and compiler design can effectively eliminate specific categories of errors, and to a great extent the need for testing – is, I suspect, not much more comfortably received in any IT community than in my own.

[Pyrotex]

Originally Posted by alexander

...Firstly i need to tell you that it is not something that can have concrete mathematics, nothing like probablility, although some math may be needed. In all reality this one of the topics i could write a thesis on, and although i dont know it very extremely well, i already have lots and lots of thoughts as to what factors should contribute...

Write that thesis! Can you have it finished and in my hands by April?
I also have given this thought. I have made a list (exhaustive?) of all transformations that are sources of data-risk, and those that mitigate or even fix data-risk. What this produces is a complex "network" of data-risk-flow, that yields at the end, the probability that there will still be flaws or bugs in the Flight Load (the final step in my chart, so far). I still have to address what the probabilities are that 1) the bug will actually be executed, and 2) the executed bug will actually cause loss of mission.

We already have real-world examples of bugs in Flight Loads that would have caused a crash, but the circumstances that would have triggered that code never came about. (phew!!!)

...Application - what the apllication for the software is, what kinds of data it holds, what kinds of settings there are, permissions, so forth...

Dirt simple. Input parameters in, calculations on the parameters, outputs (commands) out -- to attitude controls, explosive bolts, gizmos and other hardware subsystems. No loops to speak of. No construction of data structures, no saving of history.

...People - Who is responsible for the software

parallel teams of professionals all located at the sw facility. No internet. No external networks. No modems or floppies or electronic contact with the world.

...Importance - related to application, importance is more towards the line of, what will happen if a part or parts of this software stop functioning

CRASH!!!! BOOM!!!! ARRRGGGGGHHHHHHHHHHH!!!!!

...Software security - yes you need to look at this

No internet. No external networks. No modems or floppies or electronic contact with the world. Guards at doors. Lots of "stinkin' badges".

...Data Security...

Multiple backups, top-line Config Mngt, off-site copies, data dictionaries. The whole garbanzo.

What I have concluded is this: 1) The probability of software "failing" is essentially an analysis of HUMAN ERROR in all the various stages and transformations of the software.

2) The probability of a (given) software bug causing a CRASH!!!! BOOM!!!! ARRRGGGGGHHHHHHHHHHH!!!!! is essentially dependent on anomalous environmental conditions and events. "Edge of the Model" circumstances. Rare events or rare combinations of events.