Visualizing Quality – keynote by Gojko Adzic at BDD Exchange 2011

How do you measure quality?  Number of defects? Customer happiness? Money earned? Developer smiles? That’s the question raised by @gojkoadzic in his keynote at the recent BDD and Agile Testing Exchange in London, to make us think and propose some elements of response.

We tend to ignore information

We are used to ignore automatic alarms, even fire alarms:  we just don’t care, but we care when a person tells us about an issue or is shouting “fire in the building!”.

Security Poster in a mine

As in the story of the Gloves on the Boardroom Table,  making the problem tangible and visible help trigger reactions. The book Switch gives examples of such surprising techniques. Sharing information is not enough, a company is like an elephant, you don’t really drive it, you motivate it to go where you’d like to go. So visualization can help there.

Best practices change over time, as the techniques in athleticism replace each other. For example at one company they disable most of their functional tests after a feature is developed, because they impede the frequency of their release cycle; on the other hand, they closely monitor the conversion rate of the website, which is for them the best metrics since it encompasses both the technical bugs and the business bugs.

This does not mean you should disable your functional tests, but it illustrate that what may seem nonsense today may be the next best practice tomorrow.

Absence of bugs ? Presence of quality

The absence of bugs is completely different than the presence of quality. Twitter has lots of bugs but most people are happy with it so it has quality; on the other hand, Nokia phones have no bug, yet if nobody likes them they have no quality either.

Onboard screens in a plane all showing a computer crash
Do we have quality here?

So how to measure quality? If you put 20 people in a room with 3 post-it’s each and ask them to write what attributes of the product they think are the most important, and if you get a small enough group of common answers then they are candidates of things you need to measure.

Usually quality c an be measured at 3 levels (the 3 P’s):
  1. Process effectiveness
  2. Product status
  3. Production performance

Here I have the feeling that each of the first two is just a proxy for the next, and only the last is the real one, though I’m not sure.

Test coverage is crap

Taking an example of a UI, a window with 5 buttons on it it will likely requires 5 time more testing than one with only one button, even if the later brings a lot more value, which suggests that test coverage is crap.

If test coverage itself is crap, you may however monitor your risk coverage, that is the amount of testing compared to the associated risk. This metrics helps identify where you need to test more.

Visualize to make information actionable

Once you know what to measure, you want to visualize it. James Bach proposes his technique of low-tech testing dashboard. You divide your system into high-level subsystems, and for each you monitor the “progress of testing”: not yet started, half done or done, and the “level of confidence so far” with 3 smileys from happy to anxious. That’s better than a bug-tracking tool with 500 tickets mixed altogether.

Another visualization mode is the use of effects maps, or ACC matrix. You may also monitor in realtime the feed of sentiment of every Twitter message about your company or product as they do at FINN Technology in Oslo.

As a conclusion, Gojko introduces the brand new visualisingquality.org website where you can find many ways to visualize your measures of quality and where you can contribute additional ones. Since making things visible is so essential for action, let’s try ideas and share them through this website!

References:

The podcast of the full talk is available here: http://skillsmatter.com/event/agile-testing/agile-testing-bdd-exchange-2011/js-2990

Read More

The Arolla “ancient world map” of software development

Software development technologies and trends are not particularly tangible things, yet we need to reason on them. At Arolla, the company I’m part of, we’ve designed an “ancient world map” of software development, as a cartography of the universe of software development we live in. Built for our own purpose, we also share it so you can benefit from it.

 The Arolla "ancient world map" of our software development universe
The Arolla "ancient world map" of our software development universe

If you want to use the map with your own teams, please do so (it’s licensed CC-NC-ND). If you need a high-resolution file for print, just ask (the file is quite big). We’d love to get your feedback!

The metaphor of an ancient world map

Agile and in XP suggest using metaphors to help materialize abstract stuff, and make it easier to grasp. You’ve probably seen Eric Evans (Domain-Driven Design) showing a picture of an ancient world map when presenting his concept of “a model built for a purpose”. We wanted to materialize software development technologies and trends, for which we have no clear and accurate visualisation yet, just like explorers in the middle age had no accurate knowledge of the world, lands and oceans. Ancient world maps had to represent that part of ignorance, with dragons and strange creatures on the less-known areas.

So we chose this metaphor to represent our universe. And ancient world maps are beautiful too!

A map conveys meaning

In cartography, the role of map design is to:

Orchestrate the elements of the map to best convey its message to its audience.

In our map, each continent represents a particular chapter of related technical stuff , and oceans in between represents “soft” techniques that complement them best. Of course the universe of software development has many more dimensions than the two dimensions available on a map. This means that such map is quite subjective, it depends on our own mental model. On the other hand, this is also true for any map of the physical world, that is also supposed to represents a 3D planet onto a plane, with some deep decision on whether to preserve angles or distances.

We tried to put most related technologies as close as possible to each other. Regions in the middle of the map represent the core of a developer daily work, in contrast with the regions closer to the poles which are more specialized.

Of course not every existing technology and trend was included on the map, especially the ones that we do not want to offer to our clients or that are of less importance. As a fan of DDD I regret we could not include our clients domains (finance, e-commerce, media, e-advertising, online games…) on the map, but a map is for a purpose, just like a model. A map of everything would just be useless.

Drawing the map proved a lot more work than expected, with hundreds of layers and lots of little adjustments everywhere, resulting into a huge file for printing.

To discuss skills and areas of expertise

Why this map int the first place? At Arolla we wanted to organise a session for all our consultants to discover more about each other from a skills point of view. We also wanted more senior consultants to volunteer and take ownership on some areas of their technical expertise.

We didn’t want another boring evening with slides shows in front of a passive audience. We wanted a more concrete, more fun and more engaging way to talk about technical skills and areas of knowledge.

So we printed the map on a big poster laid on a table, like in the captain room in an old vessel, and made it into a game.

Map + Lego = game

During last June DDD Immersion course at skillmaster, Alberto told me about Lego Serious Play. This game helps people express their ideas and be creative using Lego bricks as a medium, to help say crazy things in front of other people. It also makes any discussion a lot more fun! So we bought a set of Lego mini-figures, including some crazy figures, for our consultants to play with on the map (we did not follow the actual Lego Serious Play game at all).

The first part of the game was to get familiar with the map. I was shouting the name of various technologies at random, and the first to find out where it is on the map would take a marshmallow and place it as a flag (we had lots of sweeties everywhere too). This part was fast-paced and didn’t take long as it was quickly obvious that everyone had understood and indexed the map in their mind already.

Putting marshmallow on the map

In the next part, we had to create our own Lego persona, using the Lego building blocks available. It was really fun, and everyone was really happy to be given the opportunity to play with Lego again. It was nice to see that nobody built a mini-figure as planned on the box, they were all very personal, ironical or really weird.

Each consultant then told his/her career by moving his/her Lego custom figure over the big map, starting from some technology (e.g. C++) “in this continent” before “moving to another territory” (e.g. Java or .Net), then “crossing oceans” to gain additional skills in software factory and agile.

I particularly enjoyed the relaxed atmosphere, where anyone would interrupt to ask questions or comment, or throw a joke out loud. Perhaps the map, the Lego and the marshmallow made it clear that it was not too serious an exercise. The map and the Lego figures were able to make abstract skills more tangible and more fun for the time of the game.

We could have done the same exercise without all that, but the feedback we had from our consultants was very positive, they really enjoyed the game.

What’s next?

We’re going to put the big poster of the map on the wall, as a decoration. Smaller printed maps will help scope some discussions, perhaps for interviews, or when discussing with less technology-involved people. It could also be useful for developers to self-assert how much they know about the current state of the art in our craft.

Read More

Documenting the design of software using patterns?

I have experimented an approach that considers every design pattern as the recursive composition of smaller patterns. This led to a prototype tool to illustrate its benefits by generating design-level documentation of annotated source code.

Eat your own dog food

The source code of this tool itself was used as the code base to apply the tool on, in order to generate its own design documentation. The Java Doclet API was used to retrieve the Javadoc tags in the class Javadoc comments, e.g. :

@pattern KnowledgeLevel OperationLevel=MyOtherClass some free text comment…

The pattern catalogue

A subset of design patterns and some other patterns such as the Knowledge Level (Fowler) has been defined in small definitions files then loaded at tool startup. We call this set of patterns definitions the pattern catalogue.

Here is an example of the definition file for the Builder pattern, notice the declaration of the member roles at the bottom, that also declares the expected pattern kind:

Encapsulates the creation of a complex object from a source object.
@extends DesignPattern
@category DesignPattern
@category Creational
@author GoF
@book [GoF 95]  E. Gamma, R. Helm. R. Johnson, and J. Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley 1995.
@style ObjectOrientedProgramming
@reference [http://en.wikipedia.org/wiki/Builder_pattern ]
@defaultrole Builder
@role Director=TypeHierarchy
@role Builder=TypeHierarchy
@role Product=TypeSet
@role Factory=AbstractFactory

Parsing the annotations to build the pattern occurrences graph

Every time a Javadoc tag “@pattern Pattern” is met in the Javadoc part of a class, an occurrence of the corresponding pattern is created in memory, linking to the corresponding class in the Java programming language metamodel.

Whenever such pattern uses a Type Hierarchy (this is declared in the definition of the pattern), then the tool will look for every subtype of the declared (super) type in order to complete the pattern occurrence as well as possible.

Even patterns that are usually considered as atomic patterns can be explained as the composition of smaller “elemental” patterns. For instance, A Builder [GoF] usually defines an AbstractBuilder role and a ConcreteBuilder role that implements it. We can express this part of the Builder pattern as a TypeHierarchy elemental pattern in the role Builder; the supertype of the TypeHierarchy therefore represents the AbstractBuilder role, while every other concrete type of this hierarchy represents ConcreteBuilder roles.

By analogy with the members of a class, we call members patterns and member occurrences the patterns and occurrences (respectively) a pattern or occurrence is composed of. The tool totally ignore fields and methods, it only considers types (interfaces and classes).

The benefits of this approach are to reuse a bigger part of the pattern definitions through composition and to help address the variant problem.

Once the pattern occurrences graph is completely built, it is rendered into an UML class diagram using Graphviz dot. Here is an example of overview for the full project:

coremodulediagram_final
Example of overview diagram for the Core module of the prototype source code

In the above diagram we can see how the patterns naturally sit on top of the several type hierarchies while connecting them together. Also the relationships between patterns become very obvious, such as the Interpreter that references the Builder and the Composite sub-patterns. This suggests that such diagrams are very valuable to document the design of a code base.

Topological sort

The approach of representing every pattern as the composition of other patterns yields to directed graphes, at the pattern level (the “meta” level) and at the pattern occurrences level. One consequence is that we can apply a topological ordering on such graphs. In the current prototype we need to do a topological sort on the patterns definitions so that we can know in advance in what order to process pattern occurrences declarations: the patterns used must be built before the patterns using them.

The pattern occurrences graph can also be navigated using the usual Depth-First search or Breadth-First search. DFS is convenient to generate the Overview diagrams such as the diagram shown above, whereas BFS is convenient to drill down the design from the top to the bottom, one level at a time. This can therefore generate the following table of contents:

Patterns graph navigation index
Patterns graph navigation index

Natural language generation alternative

As an alternative to visual diagrams, a tool can also generate natural language text to communicate the exact same information. I have tried this approach using simple text templates in each pattern definitions. Each pattern therefore defines its own design description, but using variable names to be filled later with the data from the pattern occurrences graph.

The biggest work in this approach is to deal with the plural, the enumeration of collections (comma between each item but the last etc.), truncating collections that are too long, and the complete omission of a section if a pattern member is totally missing.

Here is a sample of text generation for the prototype tool itself. Notice how the structure is rigid for each pattern occurrence: Pattern name, short description, then template text evaluated with the actual types names, followed by the comment that was put after the @pattern declaration:

Design of the module Core
This module is essentially made of 9 pattern occurences: Strategy, Visitor, KnowledgeLevel, Interpreter, Builder and AbstractFactory.

Visitor
The visitor design pattern is a way of separating an algorithm from an object structure. A practical result of this separation is the ability to add new operations to existing object structures without modifying those structures
This module defines a Visitor OccurrenceVisitor. comment

Strategy
A Strategy object encapsulates an algorithms that can be selected on-the-fly at runtime.
This module defines the interface of a Strategy: Plugin. It enables plugins to manipulate the Patternity metamodel.

AbstractFactory
Provides one or more method(s) that encapsulate(s) the creation of object.
The type OccurrenceFactory defines the interface of an AbstractFactory that creates instances of Occurrence and TypeOccurrence.

Interpreter
Define a representation for a language along with an interpreter that uses the representation to interpret sentences in the language.
The type(s) Occurrence and TypeOccurrence define(s) an object representation of the considered language. The considered language describes a graph of patterns occurrences. It uses the Visitor Visitor to add new operations to the object representation without modifying this structure.

Strategy
A Strategy object encapsulates an algorithms that can be selected on-the-fly at runtime.
This module defines the interface of a Strategy: Loader. Encapsulates how to load every pattern definition into the pattern.repository.

[...]

KnowledgeLevel
A Knowledge Level is a group of objects that describes how another group of objects should behave.
This module defines a KnowledgeLevel (also known as metamodel). It is made of two levels of objects: objects in the knowledge level, i.e. the instances of Pattern define how objects in the operational level can behave, i.e. instances of Occurrence. Typically each type in the operation level maintains a reference to its corresponding type in the knowledge level in order to know how to behave.

Conclusion

Later, at work, when asked to produce a design documentation targeted at other developers, I have experimented doing a similar effort manually, using nothing but patterns. I am still impatient to receive some feedback from the people that will read the resulting documentation in order to validate or infirm this approach.

Read More