Software Risk Analysis

[Symbology]

Originally Posted by freeztar

Excellent question Pyro, but I'm having trouble understanding the differentiation of SPRA from the collective others. I'm sure the answer is in the last paragraph I quoted above, but it's a bit elusive still. Can you give a practical example using, perhaps, the "Operations of human beings" dimension?

I sent this to Pyro yesterday.

These men have just finished placing solid steel pillars in concrete to stop vehicles from parking on the pavement outside a downtown sports bar. They are cleaning up at the end of the day. And..........


Pyro responsed with the following:

Originally Posted by Pyrotex

You see, the mission failed!! Loss of Vehicle (LOV). Beautiful
But the truck did not fail.
The barricades did not fail.
The operations performed by the people were correct.

And yet! LOV. This is the problem we face.

[Symbology]

Originally Posted by CraigD

If this is not the case, what is to prevent the probability of failure due to software from being made effective zero by not having any software, even though such a vehicle would almost certainly have much lower performance and much greater probability of failure due to non-software causes?

Beautiful point.

I'll reference this related quote again:

The question of whether computers can think is like the question of whether submarines can swim
- Edsger Wybe Dijkstra

And to my experience defining a limit like this is very useful. Though it annoys the hell out of the guys that live by their "credibility" factor. In conversation it's something like asking "ok well let's consider homicide in this case..." It does a great job of showing there really is a limit even when other thought that there was not one. It's just not one that falls within the "common sense" boundary.

Common Sense is not so common
~Voltaire

And really I bet it is those "beyond common sense" items that Pyro is trying to detect. The truth is we are all very ignorant relative to those that will live 60 years into the future. In a similar way to those guys putting up the steel posts were ignorant relative to the guy with the camera 60 feet up. I can guarantee that our two fellows will never make that mistake a 2nd time. But the challenge seems to be in figuring out what we don't know so that we can avoid that 1st mistake in real life.

It is very helpful to have a mentor that can advise us (much as Hypography offers, or Google Sets) that can give us that view from 60 feet up. "Have you considered ____" ex "Have you considered how you will get the truck out when you are done?"

It is also helpful to think similarly to chess "several moves ahead".
1) Bring up vehicle
2) Get vehicle into good work position out of traffic so it wont get hit
3) Mix cement
4) Install poles
5) Pack up gear
6) Drive home.... oh wait a minute

A classic flaw for some of us is not to "think it all the way through". And it can be especially difficult when you have a few million decision points to "think all the way through".

It is an accomplishment to make something foolproof,
because fools are so ingenious.
- N. Kohn

[Symbology]

Originally Posted by Buffy

I know this from nothing...

But if you want to gain some legitimacy, it may be worthwhile to look at the "accepted" approach for hardware, to wit: the basic idea on the hardware side is you have a countable number of components and a mean-time-before-failure and a system analysis that takes into account criticality and redundancy to determine time before system failure (ignoring whether its mission or crew for the moment: that's an implementation detail!).

It would appear that "software is completely different," but I'll argue that its not: "components" in software are "program modules" and while its hard to assess specific "MTBF" to a specific module since its one of a kind, what you can do is pull out one of those weird computer science concepts that not many people pay attention to any more: the good ol' "function point."

While its been a long time since I've gone near them, I'm pretty sure you can scare up statistics on "MTBF-per-function point" with appropriate analysis of the types of FPs and so on and end up using a methodology that's exactly like what's done on the hardware side.

Now if I were actually an astronaut, I'd be scared to death of this kind of "analysis," ("That's a bunch of crap," is what Gus Grissom would probably say) but its probably just as justifiable as the hardware stuff is...

You pulled it out of where?
Buffy

Excellent excellent point. I had talked to Pyro about looking across the code in mass for things like IF, LOOP, I/O etc but had not realized that really each of those things are Function Points.

I went looking on the RUP site for a list of what coding objects are Function Points but couldn't find the list. I see your link points to a paper for $19. Does it include that list? If so (and its the only place that lists them) I will pony up the dough

[Buffy]

Symby:

Congratulations on getting me to fall out of my chair from laughing at your photo! I now have to get my margarita and go out to the hot tub to sooth my poor tail bone!

You will find that Function Point Analysis is one of those black arts of computer science, where "those who tell don't know, and those who know don't tell," and they all want you to cough up a ton of money for consulting. Its been so long since I've thought about them, that I can't really point you to a good source, but good luck! I do hope you guys have some R&D budget to spend on this!

Onngh Yanng,
Buffy

[Pyrotex]

Well actually, except for my salary (and Symbology's should he be hired), there is uh... NO money for R&D at all.

If this job were easy, somebody else would be doing it.

[Symbology]

You might find Predictive Dynamix slide show of interest - especially slides 11-21 on the different kind of results that can be obtained with various forms of analysis. In talking to their lead programmer, I was realizing though that really what you are trying to figure out which dots are Y and which dots are N or more accurately what their score is. Once they have been scored then these other forms of analysis can be put on top to find a predictive model of the behavior.

But I am thinking that if some forms of negative condition can be found then the predictive model might be able to estimate what other conditions might also create a negative outcome.

From what I have learned from guys coding them, neural nets for process control are run it in 2 directions. First they create a simulator for the existing environment. For these X in puts you get Y outputs.

Then you turn it around and train a reverse model... for Y outputs you need X inputs.

Then you toss the model an optimal value of Y that you want and the reverse model gives you the inputs you need to get there.

In your case you are looking for the conditions that will get you a negative result. Then test your code on those input values and debug for them.

As Buffy said... you pulled that out from where?

[Symbology]

Originally Posted by Buffy

Symby:

Congratulations on getting me to fall out of my chair from laughing at your photo! I now have to get my margarita and go out to the hot tub to sooth my poor tail bone!

You will find that Function Point Analysis is one of those black arts of computer science, where "those who tell don't know, and those who know don't tell," and they all want you to cough up a ton of money for consulting. Its been so long since I've thought about them, that I can't really point you to a good source, but good luck! I do hope you guys have some R&D budget to spend on this!

Onngh Yanng,
Buffy

Sounds like building a good chess program. The key is what the developer identifies as a good move vs a bad move - more technically how he scores the move. And you have identified one of the best ways I have seen through the years... identify the metadata - the abstraction. That magical set of spectacles that lets one see patterns in the structure.

[Symbology]

I found the following Function Point Calculator.

And also this better one.

[LaurieAG]

Originally Posted by Pyrotex

NOTE: The software doesn't have to FAIL, necessarily. It could be doing exactly what it was designed to do. Remember, there are other dimensions: the Environment and the Operations of human beings.

Hi Pyrotex,

I've always had the philosophy in software development that if the user absolutely positively doesn't have to provide input, then don't let them.

(1) The Environment varies over time.
(2) Functions are constant over time or vary depending on environment? (not clear)
(3) In certain environments machines cannot operate.
(4) In certain environments humans cannot operate.
(5) Human Operation varies with environment over time.

So the first step is to create a machine (or a set of rules for one) that will be able to transport goods from A to B that will not damage the computer(s) controlling all of the above. i.e. only points (1), (2) and (3) at the start.

Once this model is 'working' you can add all the other bits (perishable goods) and also have a good set of override rules for those pesky humans.

Also, the calculation of your risk number could either be on a fixed time basis or a fixed event basis (i.e. a combination of variant factors trigger the risk calculation).

[Symbology]

Relative to Buffy's Black Arts reference...

Alternative Metaphors:

1) If I was out to assess whether any given tree was going to fall over or not, one good assessment would be to know its root structure. I can't necessarily know from the surface what a live tree's root structure looks like. I don't have x-ray glasses to look at its roots. But I might know from experience of digging up other trees that when a tree looks like an oak that its root structure generally mimics its upper branches and is very solid.

By contrast if it looks like a palm tree then I know its root structure will likely be a ball of roots that is not all that deep.

By assessing enough tree types and figuring out identifying markers then I can use those markers to make a best guess as to what an unknown tree is like and therefore what its chances of root failure are.

Basically I am assessing the character of the tree.

If a tree falls in a forest and no one is around to hear it,
do the other trees still make fun of it?

2) A similar metaphor is that of assessing whether a house will develop structural cracks or not. In this case assessing its foundation and the type of ground it was built on will go a long ways towards assessing the character of the house. Was it built on rock, clay, or sand? Did they use rebar? What do samples of the cement say it was made from?

3) People can instantly size up the character of a person by their shoes they are wearing and their fingernails. Some other characteristics like hair style, clothing, how they carry themselves etc also factor in.

4) The Meyer Briggs can do a good job of assessing the archetype of a person - which is designed to be neutral in terms of the expected success or failure of the person. But assessing whether a person has an addiction tends to be a fairly reliable indicator of their... well... reliability.

5) If I hear a person say "Well Number 1 I think ___ and B I think ____" I instantly begin to doubt their reliability and consistency. (I know that Buffy has had a few similar reasons to doubt my own reliability )

5) I can look at a piece of source code and in about 5 seconds tell you the character of the programmer behind it.

• Was he consistent
• did she follow typical coding conventions
• did he use variables like vstrq or vector_strength_quotient
• Did she use buffers, pointers, meta data tables, encryption, hashing
• Is it sophisticated or simplistic
• Does it look "tight" (like a well practiced music band)

If we would give it a score of "well written" then its reliability could be expected to be higher. If we would give it a score of "poorly written" then its reliability could be expected to be lower. Granted these can be difficult to automatically asses. I'm just putting this out for the total concept of assessing the character of things.

Now considering we should expect professional code to all be well written, and tested, we could still be looking for subtle characteristics.


To reel all this back in, I think that recognizing basic syntax (If, Loop, Pointer reference) can help assess the character of a code module. Also, as Buffy said, the Function Points of how many inputs, outputs, files used etc help assess the character of the code.

So if these various syntax elements can be individually walked and assigned a "not risky" to "highly risky" score, then some level of summation of the risk for the module should be usable as a general assessment.