Agile Performance Testing?

Neill,

You touched two very interesting points here. I have formulated them in a different way for myself, but probably it is the same (if I got it correctly).

The first point, as I got it, perhaps can be named “start performance testing early”. Everybody talking about it, but not much done. Mainly, the only thing that a performance tester (as it is usually defined now) can do until some functionality will work is to collect requirements/use-cases/scenarios.

Looks like the role of performance/load tester should be re-defined, I’d prefer to named it “performance engineer” (accidentally it is my current title, although so far it is rather a title than what I mean here). Your point about monitoring early/unit tests looks very valuable for me. Usually developers don’t monitor what is going on. The problem here is that monitoring one-user load can be useful only for really pathological cases. You need multi-user load to make it meaningful.

Probably we need to talk here about early unit performance/load testing (using some kinds of test harnesses). Mainly done by developers with help of a performance engineer.

I’d say that it will create a bridge between “performance analysis of design” (Software Performance Engineering (SPE) approach by Dr. Connie U. Smith, build around modeling software performance on early stages of development,
Performance Engineering for Systems in the Early Stages
by Sudha Paidipati) and “classic” system performance/load testing.

As for the second point, I believe that performance testing is somewhat agile by definition, more like scientific research. I guess the only case when you really can do it in a formal way is when you test a well-known system (some kind of performance regression testing). You really can’t plan much – you never know at what load level you face problems and what you would be able to do with them. So it becomes an iterative process involving tuning and close work with development. Moreover, I am not sure that the term performance/load testing is good for that whole process at all: really it is a mix of testing, tuning, and diagnostics.

You can check my paper "Effective Load Testing" (see p.33-39) – I discussed these points there.

What Scott Barber’s idea do you refer to? UCML? I am a little confused here, perhaps I missing your point. I’d say that documenting load is useful for any kind of performance/load testing, formal or agile, and don’t see as it add more agile-ness.

Comments

Alex,
I like the way you have taken the ideas and re-factored/ reframed them to your context to assist with understanding, something I do myself often to assist in my learning.
Yes the first is indeed is performance test early, this has also been blogged about and raised as part of a presentation where it was touched on by Stevan Zivanovic [http://www.testingreflections.com/trackback/1989?PHPSESSID=334da8f1145663b4cb266c66b9e3a9e0] in one of his blogs here. What I do is similar to what you suggest, but it is to user stories and later the "linked" stories that become use cases in Agile-Rup projects, in my context. We also look to exercise some early volumes and multiple connections onto early artefacts to the technical stories or constraint stories on the system and repeat these as we iterate through builds, whilst running the monitors. I use the term Toolsmith for this role as have James Bach and Danny Faught in one of their Stickyminds articles: Agile Automation - not your fathers automation, it works for me but any term will do.
It is interesting you find performance testing slightly agile anyway, I always thought of it as tending towards it, however I have mainly seen it in action as big design upfront even if it does not actually run till the end of the project.
The "agile" ness I gain in Scott's approaches, which was the process flow he uses : was the freeing up of the exploratory element and how these are used to generate new test areas for performance testing in complex systems. The UCML I use in conjunction with the use-cases to see what "stories" combine to help get a realistic model of usage quickly in workshops. I agree it is not an XP agile trait, as in Test Driven Design but it does fit with Scott Amblers "agile modelling" view and practices.
You have also given me three excellent new references I need to go and read up on now as well, thanks for the comments and thoughts. I will post more as it develops.
Neill McCarthy
"Agile Testers of the World UNIT !"

Neill,

I'd say that it is agile more than slightly for new systems (in common sense meaning of agile, I haven't seen the exact definition when it is used with words programming and testing). You start from collecting use-cases and scenarios ("stories"), figure out a way to create load, and... you never know what. Usually after long experiments to tune the system and trace down the problems (quite often you don't know your next step until you finish previous one) and negotiations with "customer" you end up with performance "solution
": a combination of hardware/software/settings/process that will satisfy at least minimal requirements (subject of long negotiations itself). Although probably a big part of collaboration with development is because all components (units) weren't properly tested for performance.

Yeah, there are many things that should be ready to make meaningful performance testing. You need reasonably good data, functionality included in the use cases working, proper hardware and infrastracture, a load testing tool, etc. And yes, people usually start to think about it just before the end of the project.

But if you are going to do unit performance testing, you need to think about that well before. If you are going to test a database design with two records in the database (the way it is usually done)it would be meaningless - have nothing common with the time when you will have many GBs there. If you are going to monitor unit performance tests on a developer machine or a shared server that everybody use it can be misleading too - who knows what is going on in parallel with your tests. If you are going to do unit performance testing you need to figure out use cases ("stories") you are going to use (and how they would impact that component). So if you get all this pieces in place (so probably should be for the test-driven development approach at least in the theory), you probably can start "system" (traditional) performance testing early enough with some subset of functionality working.

By the way, have you seen any reference to performance/load testing in agile/XP/etc books/papers?

References so far on this seem to be few and far between, I found this interesting blog by Grig Gheorghiu [http://agiletesting.blogspot.com/] which contains some specific ideas and examples. I usually enjoy Grig’s comments on the agile-testing yahoo group.
I did notice that the points Michael Kelly made in the May/June 2005 Better Software (which I read last night on the train home) in the article “How to win friends and automate testing”, though not aimed at performance testing per se are applicable to it and this iteration build of the tests from Developer code and where possible extending and enriching via re-factoring into other tests is part of the exercise I have been using in the “agile” performance testing I do. The sections on the Perform Runtime Analysis together and the Use log files were particularly pertinent.
In the same Better Software, this coming only a few weeks after I said I was less impressed in its new generic direction and I read two of the most pertinent articles to me for a couple of years, Scott Barber’s High-Performance Testing also raised some of the points about being able to test “performance” early
I have also come across a couple of other comments and items on it in various “google” sessions when I am looking for inspiration but nothing else has really struck me yet.
With regards to the “risks” of this early testing, I agree as you point out and Scott did in his article it is not possible to always, or often, to project the future effect from the early behaviors. However we can spot emergent patterns in the micro-behaviors of stories and classes/methods through these early iterations and see the effect of the increases in complexity and dependencies between artifacts. By charting these impacts we can make assessments and create patterns that we can use to look for issues or to optimize areas of coding. These micro-patterns also become useful heuristics for the areas we are looking to test more fully as the software grows and reaches points of stability before we start a new spike. I am also still working with the impact and effects of emergent architectures in this respect; I will blog separately on these later.
As for the issues of “other peoples actions”, this can be solved with environmental controls and monitoring to see the effects, even this “some one was doing something unexpected” activity whilst x was performance testing can still lead to interesting questions that lead to more investigative testing and exploratory sessions.I will blog on the ideas and benefits of Exploratory Testing: performance charters later and trackback.

Neill McCarthy
"Agile Testers of the World UNIT !"

Alex, I need to weigh in strongly opposing the title of Performance Engineer for several reasons.

1) We don't engineer performance - at best, we engineer performance tests.

2) Titles don't change perceptions, people do.

3) Engineers are licenced to practice their particular field of engineering. With that licence comes professional liability (and thus a TON of liability insurance). In the state of Texas, no one who is not so licenced can legally use the word Engineer as part of their official professtional title. Doing so is a prosecutable offense.

Personally, I've taken to refering to myself as a Performance Tester/Analyst. That is what I do. I test performance and I analyze performance. I may or may not diagnose symptoms of poor performance, but I'm unaware of an acceptable term for "one who diagnoses" and another slash and fancy word just feels gaudy.

Bottom line, be careful about adding Engineer to a title. Whatever it may imply to you, it may not imply the same thing to the rest of the world.

--
Scott Barber
Chief Technology Officer
PerfTestPlus, Inc.
sbarber@perftestplus.com
www.perftestplus.com

IBM-Rational just released their new performance testing tool that is fully integrated into eclipse. Microsoft will be releasing a new performance testing tool that is fully integrated into Visual Studio in September. Both deliver with frameworks for performance unit tests. This is no longer an obscure, taboo concept. Within a year it will be the norm (at least for those devloping in Eclipse and Visual Studio) rather than the exception.

--
Scott Barber
Chief Technology Officer
PerfTestPlus, Inc.
Software Performance Specialist,
Consultant, Author, Educator
sbarber@perftestplus.com
www.perftestplus.com

This is EXACTLY the way I had hoped UCML would be used. It's meant to be quick, intuitive and descriptive. One can then add depth, documentation, data, etc. when the additional documentation adds value. If you have an example of this being successful applied in an XP/Agile project that uses stories, I'd love to hear about it!

--
Scott Barber
Chief Technology Officer
PerfTestPlus, Inc.
Software Performance Specialist,
Consultant, Author, Educator
sbarber@perftestplus.com
www.perftestplus.com

Yes, I saw a couple of discussions about that. Still programmers and tester in the company I work for have titles "software engineer" and "QE engineer" and I am not aware that those who work in Texas have other titles. Looks like "software engineer" is still a mainstream title and I haven't heard that somebody was sued for that.
For me, looks like these discussions are rather academical and rhetorical (at least so far).

If you add Software Performance Engineering (SPE) approach by Dr. Connie U. Smith to the picture, the idea to "engineer performance" doesn't sound too absurd.

Agree that "titles don't change perceptions, people do", so probably not much sense to spent too much efforts arguing what title is better.

Very interesting. Is there any details about "frameworks for performance unit tests"?

By the way, I heard that the number of supported protocols in the new Rational tool was decreased significanly. What is now supported except HTTP?

Thanks!

Alex

http://www.clarkware.com/software/JUnitPerf.html

An example of a performance unit test.

-Mike

Scott,
I will blog on this later in the month and how I have used it. I have also been experimenting with the linking of the UCML through the systems architecture and linking it with the stories and the UML models.
As these are enriched then I try to run the models for the performance tests from these "stories" and happy paths by putting some simple expected % usages to them and then refine via execution. At the moment it is still an imperfect art, for me, and I am refining the craft across projects and looking to quantify the benefits of the approach.
The visual modeling approach has worked fairly well so far in my limited experience and I have also tried to supplement this further with “curves” of day usage patterns to assist with understanding and to get rapid acceptance of the concepts and remit of testing early. At the moment I have a good feeling on its usage as a tool to aid understanding and for demystifying the “black art” of performance testing.
Neill McCarthy
"Agile Testers of the World UNIT !"

Scott,
I am not so optimistic that this will become the norm, at least not in the next 12 months. Though I am of the view it should be and that the “there is not the tooling” argument is weakening by the month.
There are still few practitioners in this area or if there are there is little written or discussed on it at this stage. May be it is one of those things that is done more then discussed, however if that is the case then “performance impaired” systems would not be seen so often.
Tooling is improving and I have been following the performance tools in eclipse for a while since it was the Hyades project, I am also aware of the Microsoft tooling but have yet to get to use this myself.
As pointed out by Mike Kelly there is also a performance plug in for Junit and I have seen similar ideas being discussed as plug ins around other frameworks. However there is still the need to have these tools and ideas adopted and acted on as practice, without these being discussed and championed I do not feel it will be as pervasive as you perceive, at least not rapidly as you would suggest. I will be happy to be proved wrong though.
Neill McCarthy
"Agile Testers of the World UNIT !"

Scott,

Looking through you papers and presentations, I noticed that you use "Performance Engineering" quite often. For example, in A Performance Engineering Strategy (although I understand "Performance Engineering" wider than described in this document). Have you re-evaluated using of this term or your concerns are only about the "performance engineer" title?

Alex

I looked through Continuous Performance Testing With JUnitPerf. While probably it is a big step forward, it looks like the only output is timestamps for each particular TimedTest. Good for 5 users, but rather is a problem for real load - say, dozens of "TimedTest"s and hundreds of users. The ability to aggregate results in a meaningful way looks extremely important for me for any kind of load/performance testing (and on unit level too). Unfirtunately I wasn't able to play with the product so quite possible that I missed something.