Should non-function requirements testing be done only for peak load?

There are some who argue that testing of non-function requirements (NFRs)  should only be performed at peak loads; positing that testing them at less than peak loads is a waste of time.  The argument goes that if the solution cannot meet the peak load requirements, then it is unsuitable anyway, so what’s the point of testing at less than peak loads? Implicit in the argument is that if it performs as required at peak load, then it will perform as required at normal load.

First, let’s investigate the assertion that if the NFRs do not meet their peak load specifications, then the solution is not of use to the organisation.

What does peak load mean?

In order to do this, we need to define what we mean by ‘peak load’. A peak, by definition, is the summit, the highest point, a singularity.  In the stock market, the price of a share fluctuates. One of those prices is the highest for that day. That is the peak (the ‘high’ for that day).  There are also highs for the current year, high on a year-on-year basis, and an all-time high. 

Do we mean that testing should be performed only for the very highest possible transaction? If so, what is that very highest possible transaction?  Is it the maximum that the solution is rated for?  This could well be something that occurs as a 1 in 25 year event.

In addition, does peak load refer to the peak volume that the environment is expected to provide or the peak volume that the solution is specified to handle. Suppose we are talking about concurrent users of a website. If the organisation has 20 million customers, a theoretical peak load might be all those customers simulatenously accessing the website.  Another theoretical peak might be the website being subjected to a denial of service attack.

The other peak load is the volume that the system is rated to meet. Suppose we decide that the maximum is 3 million concurrent users.  Do we then test the system for 3 million concurrent users or 20 million concurrent users?

Peak loads can occur once every 10 years

Depending on what definition of peak load was used, its occurence may happen so rarely and only for such a short time that it doesn’t matter.  Suppose the peak load happens only for half an hour on the 1st of January each year. If we tested only for that, then we are not testing the behaviour of the system 99.99% of the its time.

Perhaps by peak load, rather than a single point we want to use a percentile.  Say, the top 5% of the expected load.

What about the specifications for average loads?

Most requirements specifications dictate the performance required from the system at average loads, and also at heavier loads. Without testing for average loads, how would we verify that the solution has met the specification?  If we will not test at average loads, what’s the point of specifying performance for that? The solution may be performing as sluggishly at average loads as with peak loads – we wouldn’t know.

NFRs is not just about volume

Many  NFRs have nothing to do with volume transaction but will also have their analog to a peak concept.  Consider usability. Usability might cater to extremes, like being usable to a colour-blind person, or someone who cannot use a mouse.  Shall we test only those extremes with the argument that if it works well there, it works well for the average case? Of course not.  Think also about the extremes for maintainability, accessibility, availability, security, portability, and so on.

Conclusion

NFRs should be tested against all specification. If we specified performance at average levels, the requirements should be verified at average levels. 

Book Review “The Thinker’s Toolkit”

The Thinker’s Toolkit: 14 Powerful Techniques for Problem Solving, Revised and Updated. Author: Morgan D. Jones

Review

Forthcoming…

 

 

Books referenced in this title.

Those with *** I have read and recommend.

Andriole, Stephen J. Handbook of Decision Support Systems. TAB Professional and Reference Books, 1989.

Balachandran, Sarojini. Decision Making: An Information Sourcebook. Oryx Press, 1987.

Baron, Jonathan. Thinking and Deciding. Cambridge University Press, 1988.

Bolles, Edmund Blair. A Second Way of Knowing: The Riddle of Human Perception. New York: Prentice Hall Press, 1991.

Campbell, Jeremy. The Improbable Machine: What the Upheavals in Artificial Intelligence Research Reveal about How the Mind Really Works. New York: Simon and Schuster, 1989.

Daniel, Wayne W. Decision Trees for Management Decision Making: An Annotated Bibliography. Monticello, Ill.: Vance Bibliographies, 1979.

De Bono, Edward.New Think; the Use of Lateral Thinking in the Generation of New Ideas. Basic Books, 1968.

Fishburn, Peter C. The Foundations of Expected Utility. Dordrecht, Holland: D. Reidel Pub. ;, 1982.

Gilovich, Thomas. How We Know What Isn't So: The Fallibility of Human Reason in Everyday Life. New York, N.Y.: Free Press, 1991.

Hicks, Michael J. Problem Solving in Business and Management. London: Chapman & Hall, 1991.

Hunt, Morton M. The Universe Within: A New Science Explores the Human Mind. Simon and Schuster, 1982.

Laplace, Pierre Simon, and Frederick Wilson Truscott. A Philosophical Essay on Probabilities. Dover, 1951.

Margolis, Howard. Patterns, Thinking, and Cognition: A Theory of Judgment. University of Chicago Press, 1987.

McKim, Robert H. Thinking Visually: A Strategy Manual for Problem Solving. Lifetime Learning Publications, 1980.

Mockler, Robert J. Computer Software to Support Strategic Management Decision Making. Macmillan Pub, 1992.

*** Neustadt, Richard E., and Ernest R. May. Thinking in Time: The Uses of History for Decision-makers. Free Press, 1986.

Nisbett, Richard E., and Lee Ross. Human Inference: Strategies and Shortcomings of Social Judgment. Prentice-Hall, 1980.

*** Nutt, Paul C. Making Tough Decisions: Tactics for Improving Managerial Decision Making. Jossey-Bass Publishers, 1989.

Ornstein, Robert E. The Psychology of Consciousness. 2d ed. Harcourt Brace Jovanovich, 1977.

Restak, Richard M. The Brain Has a Mind of Its Own: Insights from a Practicing Neurologist. Harmony Books, 1991.

Rivett, Patrick. The Craft of Decision Modelling. Wiley, 1994.

Robinson, Roxana. Georgia O'Keeffe: A Life. Harper & Row, 1989.

Williamson, David L. Group Power: How to Develop, Lead, and Help Groups Achieve Goals. Prentice-Hall, 1982.

Wisniewski, Mik. Quantitative Methods for Decision Makers. Pitman, 1994.

Book Review ‘Capability Cases’

Capability Cases: A Solution Envisioning Approach
Irene Polikoff, Robeert Coyne, Ralph Hodgson

Introduces the concep t of ‘Capability Cases’ – a concept made up one or more functions that collectively provide a business value.

 

 

 

Titles referenced in the book:

Title	Author	Remarks
Towards an Architecture Handbook	Bruce Anderson
The Timeless Way of Building	Christopher Alexander
Real Options:Managing Strategic Investment in an Uncertain World	Martha Amram and Nalin Kalatilaka
Creativity – The Magic Synthesis	Silvano Arieti
Information Systems Development	D. E. Avison and G. Fitzgerald
Learnings and Inquiry Based Reuse Adoption	S. Bailing, M. Simos, L. Levine, and R. Creps
Web Services and Service-Oriented Architectures	Douglas K Barry
Extreme Programming Explained	Kent Beck
Designing Hard Software – The Essential Tasks	Dougla Bennett
Multiple Criteria Decision Analysis: An Integrated Approach	Val Belton and Theodoer J. Stewart
Software Configuration Management Patterns: Effective Teamwork, Practical Integration	Stephen Berczuk, Brad Appleton
The Creative Mind: Myths and Mechanisms	Margaret A. Boden
The Mythical Man-Month and Other Essays on Software Engineering	Frederick P. Bookrs
The Social Life of Information	John Seely Brown and Paul Duguild
Web Services, E-Business, and the Semantics Web	C. Bussler, R. Hull, S.A. McIlraith, M.E.Orlowska, B. Pernici, and J. Yang
The Mind Map Book	Tony Buzan
Scenario-Based Design	J.M. Carroll
Enterprise Service Bus	David A. Chappell
To be continued…

Product Development for the Lean Enterprise

Book Review of Michael N. Kennedy’s ‘Product Development for the Lean Enterprise: Why Toyota’s system is four times more productive and how you can implement it.’

A book that presents a ‘lean’ approach to product development. The word Toyota appears in the book, but it’s not clear whether the book claims that what it discusses is the method used by Toyota.  Kennedy’s background included 30 years at Texas Instruments. No mention of Toyota, so he apparently does not have any insider experience.

 

Using a short novel approach popularised (to my knowledge) by Eliyahu Goldratt, the book reveals one by one the various aspects of the principles involved in lean product development.

 

It’s not easy to write an engaging story. Goldratt pulled it off in his hard-to-put-down ‘The Goal’ but then did not quite achieve the same impact with his other books, notably ‘Critical Chain’. 

 

Unfortunately I didn’t find Kennedy’s storytelling to be interesting, which makes going through the book a little challenging.  Both the narrator and the characters speak in an unnatural manner, and wouldn’t be out of place in ‘The Brady Bunch’.   The characters were pretty much cardboard characters, indistinguishable from each other.  Add to that, the ratio of story-telling words to the actual ‘meat’ of the discussion is pretty high.

 

I haven’t finished reading it, but will continue and will update this review.

The World's (Mis)Leading Expert

Experts are often introduced as ‘the world’s leading expert on …’. Given that experts are often wrong, some of them should be introduced as ‘the world’s misleading expert…’

Lean Bibliography

Deming, W. Edwards. Out of the Crisis. Cambridge, Mass.: MIT Press, 2000.

Among the greatest causes of wastes are variation, poor utilisation of human potential, and managing the organisation without an appreciation of it as a system. Deming’s ideas is widely acknowledged as a major contributor to Japan’s transition to a producer of quality products. (Link to Amazon).

Fujimoto, Takahiro. The Evolution of a Manufacturing System at Toyota. New York: Oxford University Press, 1999.

The word ‘Lean’ is a Western conception, but the idea originated from observing how Toyota’s production system operated. (Link to Amazon).

Goldratt, Eliyahu M., and Jeff Cox. The Goal. Abridged; 3rd Rev. Ed., 20th Anniversary ed. Minneapolis, MN: HighBridge, 2006.

Lean is about removing flow bottlenecks to help smooth the flow of value. There’s no better introduction to the phenomenon of bottlenecks than this novel by the late Goldratt. (Link to Amazon).

Liker, Jeffrey K. The Toyota Way: 14 Management Principles from the World's Greatest Manufacturer. New York: McGraw-Hill, 2004.

Definitive description of Toyota’s principles by a Western author. (Link to Amazon).

Liker, Jeffrey K., and Gary L. Convis. The Toyota Way to Lean Leadership: Achieving and Sustaining Excellence through Leadership Development. New York: McGraw-Hill, 2011.

(Link to Amazon).

Modig, Niklas, and Pa M. This Is Lean: Resolving the Efficiency Paradox. Stockholm: Rheologica, 2012.

Short and engaging introduction to Lean. The opening example of a 500% more efficient health care is memorable. Focused on the tensions between resource availability, resource utilisation, and flow throughput, and how they can be addressed. (Link to Amazon).

Ohno, Taiichi. Toyota Production System: Beyond Large-scale Production. Cambridge, Mass.: Productivity Press, 1988.

(Link to Amazon).

Seddon, John. Freedom from Command & Control: Rethinking Management for Lean Service. New York: Productivity Press, 2005.

Recommended by the authors of “This is Lean”. (Link to Amazon).

Taylor, F. (1911). The principles of scientific management. Mineola, N.Y.: Dover Publications.

Though not directly about Lean, Taylor’s work began the science of scientific management – how to maximise work efficiency by maximising utilisation of resources. Depending on how you interpret maximising resource utilisation, you either end up as lean or as a heavy organisation. (Link to Amazon).

Womack, James P., and Daniel T. Jones. Lean Solutions: How Companies and Customers Can Create Value and Wealth Together. New York, NY: Free Press, 2005.

Lean production is focused on producing goods with a minimum of waste. But what happens after you’ve produced the products?  There is a lot of waste delivering the product’s value to the customer. This book addresses that question. (Link to Amazon).

Womack, James P., and Daniel T. Jones. Lean Thinking: Banish Waste and Create Wealth in Your Corporation. New York: Free Press, 2003.

Popular introduction to the concepts of Lean production. (Link to Amazon).

Womack, J., & Jones, D. (2007). The machine that changed the world: The story of lean production--Toyota's secret weapon in the global car wars that is revolutionizing world industry. New York: Free Press.

The book that introduced the Toyota Production System to the popular business literature. (Link to Amazon).

Woollard, F., & Emiliani, B. (2009). Principles of mass and flow production: Including some notes on British methods of continuous production 1925.

Recommended by the authors of “This is Lean”. (Link to Amazon).

The Keys to Success

Some thoughts on success.

There is not one key to success; there are many.  There are many doors to success.  A problem we have is that neither the doors nor the keys are labelled. We don’t know which keys opens which door. While some keys can open several of the doors, some of the doors require multiple keys. We have to try the keys we have against the doors we find.  If we're lucky, we might open one of the doors.

The key to success is remembering these three rules:

1. Have an iron determination - never be swayed.
2. Be adaptable.
3. Know which of the first two keys is applies to a situation

Success consists of first defining what you mean by success.

If at first you don't succeed, try redefining what success means

Very often, success and happiness do not go hand in hand. The same way that success and good looks don't necessarily come hand in hand. If you achieve success, you achieve success; you don’t necessarily achieve happiness.