The Arolla “ancient world map” of software development

Software development technologies and trends are not particularly tangible things, yet we need to reason on them. At Arolla, the company I’m part of, we’ve designed an “ancient world map” of software development, as a cartography of the universe of software development we live in. Built for our own purpose, we also share it so you can benefit from it.

 The Arolla "ancient world map" of our software development universe
The Arolla "ancient world map" of our software development universe

If you want to use the map with your own teams, please do so (it’s licensed CC-NC-ND). If you need a high-resolution file for print, just ask (the file is quite big). We’d love to get your feedback!

The metaphor of an ancient world map

Agile and in XP suggest using metaphors to help materialize abstract stuff, and make it easier to grasp. You’ve probably seen Eric Evans (Domain-Driven Design) showing a picture of an ancient world map when presenting his concept of “a model built for a purpose”. We wanted to materialize software development technologies and trends, for which we have no clear and accurate visualisation yet, just like explorers in the middle age had no accurate knowledge of the world, lands and oceans. Ancient world maps had to represent that part of ignorance, with dragons and strange creatures on the less-known areas.

So we chose this metaphor to represent our universe. And ancient world maps are beautiful too!

A map conveys meaning

In cartography, the role of map design is to:

Orchestrate the elements of the map to best convey its message to its audience.

In our map, each continent represents a particular chapter of related technical stuff , and oceans in between represents “soft” techniques that complement them best. Of course the universe of software development has many more dimensions than the two dimensions available on a map. This means that such map is quite subjective, it depends on our own mental model. On the other hand, this is also true for any map of the physical world, that is also supposed to represents a 3D planet onto a plane, with some deep decision on whether to preserve angles or distances.

We tried to put most related technologies as close as possible to each other. Regions in the middle of the map represent the core of a developer daily work, in contrast with the regions closer to the poles which are more specialized.

Of course not every existing technology and trend was included on the map, especially the ones that we do not want to offer to our clients or that are of less importance. As a fan of DDD I regret we could not include our clients domains (finance, e-commerce, media, e-advertising, online games…) on the map, but a map is for a purpose, just like a model. A map of everything would just be useless.

Drawing the map proved a lot more work than expected, with hundreds of layers and lots of little adjustments everywhere, resulting into a huge file for printing.

To discuss skills and areas of expertise

Why this map int the first place? At Arolla we wanted to organise a session for all our consultants to discover more about each other from a skills point of view. We also wanted more senior consultants to volunteer and take ownership on some areas of their technical expertise.

We didn’t want another boring evening with slides shows in front of a passive audience. We wanted a more concrete, more fun and more engaging way to talk about technical skills and areas of knowledge.

So we printed the map on a big poster laid on a table, like in the captain room in an old vessel, and made it into a game.

Map + Lego = game

During last June DDD Immersion course at skillmaster, Alberto told me about Lego Serious Play. This game helps people express their ideas and be creative using Lego bricks as a medium, to help say crazy things in front of other people. It also makes any discussion a lot more fun! So we bought a set of Lego mini-figures, including some crazy figures, for our consultants to play with on the map (we did not follow the actual Lego Serious Play game at all).

The first part of the game was to get familiar with the map. I was shouting the name of various technologies at random, and the first to find out where it is on the map would take a marshmallow and place it as a flag (we had lots of sweeties everywhere too). This part was fast-paced and didn’t take long as it was quickly obvious that everyone had understood and indexed the map in their mind already.

Putting marshmallow on the map

In the next part, we had to create our own Lego persona, using the Lego building blocks available. It was really fun, and everyone was really happy to be given the opportunity to play with Lego again. It was nice to see that nobody built a mini-figure as planned on the box, they were all very personal, ironical or really weird.

Each consultant then told his/her career by moving his/her Lego custom figure over the big map, starting from some technology (e.g. C++) “in this continent” before “moving to another territory” (e.g. Java or .Net), then “crossing oceans” to gain additional skills in software factory and agile.

I particularly enjoyed the relaxed atmosphere, where anyone would interrupt to ask questions or comment, or throw a joke out loud. Perhaps the map, the Lego and the marshmallow made it clear that it was not too serious an exercise. The map and the Lego figures were able to make abstract skills more tangible and more fun for the time of the game.

We could have done the same exercise without all that, but the feedback we had from our consultants was very positive, they really enjoyed the game.

What’s next?

We’re going to put the big poster of the map on the wall, as a decoration. Smaller printed maps will help scope some discussions, perhaps for interviews, or when discussing with less technology-involved people. It could also be useful for developers to self-assert how much they know about the current state of the art in our craft.

Read More

Patterns for using custom annotations

If you happen to create your own annotations, for instance to use with Java 6 Pluggable Annotation Processors, here are some patterns that I collected over time. Nothing new, nothing fancy, just putting everything into one place, with some proposed names.

annotation

Local-name annotation

Have your tools accept any annotation as long as its single name (without the fully-qualified prefix) is the expected one. For example com.acme.NotNull and net.companyname.NotNull would be considered the same. This enables to use your own annotations rather than the one packaged with the tools, in order not to depend on them.

Example in the Guice documentation:

Guice recognizes any @Nullable annotation, like edu.umd.cs.findbugs.annotations.Nullable or javax.annotation.Nullable.

Composed annotations

Annotations can have annotations as values. This allows for some complex and tree-like configurations, such as mappings from one format to another (from/to XML, JSon, RDBM).

Here is a rather simple example from the Hibernate annotations documentation:

@AssociationOverride( 
   name="propulsion", 
   joinColumns = @JoinColumn(name="fld_propulsion_fk") 
)

Multiplicity Wrapper

Java does not allow to use several times the same annotation on a given target.

To workaround that limitation, you can create a special annotation that expects a collection of values of the desired annotation type. For example, you’d like to apply several times the annotation @Advantage, so you create the Multiplicity Wrapper annotation: @Advantages (advantages = {@Advantage}).

Typically the multiplicity wrapper is named after the plural form of its enclosed elements.

Example in Hibernate annotations documentation:

@AttributeOverrides( {
   @AttributeOverride(name="iso2", column = @Column(name="bornIso2") ),
   @AttributeOverride(name="name", column = @Column(name="bornCountryName") )
} )

annotationbis

Meta-inheritance

It is not possible in Java for annotations to derive from each other. To workaround that, the idea is simply to annotate your new annotation with the “super” annotation, which becomes a meta annotation.

Whenever you use your own annotation with a meta-annotation, the tools will actually consider it as if it was the meta-annotation.

This kind of meta-inheritance helps centralize the coupling to the external annotation in one place, while making the semantics of your own annotation more precise and meaningful.

Example in Spring annotations, with the annotation @Component, but also works with annotation @Qualifier:

Create your own custom stereotype annotation that is itself annotated with @Component:

@Component
public @interface MyComponent {
String value() default "";
}
@MyComponent
public class MyClass...

Another example in Guice, with the Binding Annotation:

@BindingAnnotation
@Target({ FIELD, PARAMETER, METHOD })
@Retention(RUNTIME)
public @interface PayPal {}

// Then use it
public class RealBillingService implements BillingService {
  @Inject
  public RealBillingService(@PayPal CreditCardProcessor processor,
      TransactionLog transactionLog) {
    ...
  }

Refactoring-proof values

Prefer values that are robust to refactorings rather than String litterals. MyClass.class is better than “com.acme.MyClass”, and enums are also encouraged.

Example in Hibernate annotations documentation:

@ManyToOne( cascade = {CascadeType.PERSIST, CascadeType.MERGE}, targetEntity=CompanyImpl.class )

And another example in the Guice documentation:

@ImplementedBy(PayPalCreditCardProcessor.class)

Configuration Precedence rule

Convention over Configuration and Sensible Defaults are two existing patterns that make a lot of sense with respect to using annotations as part of a configuration strategy. Having no need to annotate is way better than having to annotate for little value.

Annotations are by nature embedded in the code, hence they are not well-suited for every case of configuration, in particular when it comes to deployment-specific configuration. The solution is of course to mix annotations with other mechanisms and use each of them where they are more appropriate.

The following approach, based on precedence rule, and where each mechanism overrides the previous one, appears to work well:

Default value < Annotation < XML < programmatic configuration

For example, the default values could be suited for unit testing, while the annotation define all the stable configuration, leaving the other options to  configure for deployments at the various stages, like production or QA environments.

This principle is common (Spring, Java 6 EE among others), for example in JPA:

The concept of configuration by exception is central to the JPA specification.

Conclusion

This post is mostly a notepad of various patterns on how to use annotations, for instance when creating tools that process annotations, such as the Annotation Processing Tools in Java 5 and the Pluggable Annotations Processors in Java 6.

Don’t hesitate to contribute better patterns names, additional patterns and other examples of use.

EDIT: A related previous post, with a focus on how annotations can lead to coupling hence dependencies.

Pictures Creative Commons from Flicker, by ninaksimon and Iwan Gabovitch.

Read More

Patterns as stored arrangements: toward more efficient tools

In nature, out of every possible arrangement of several elements, only a few arrangements are stable. This is illustrated with atoms combined together, or smaller particles arranged together into atoms, where not every combination is sustainable.

Unstable arrangements tend to move toward stable ones over time. Whenever you observe the elements, you mostly see stable arrangements of them. Because there is only a relatively small number of stable arrangements, a brain can be trained to recognize them, and they can even be named and incorporated into the language.

Better with a brain

The capability to recognize common arrangements of elements is beneficial because it saves a lot of time and thinking. Rather than describing in the details each arrangement each time, it is therefore very economical -cheaper- to describe each stable arrangement once, and then to declare that such arrangement happens in this or that case. Saying: “this is an equilateral triangle” is times more efficient than explaining what it is: “there are 3 lines of the same length, each connected to the two others such as they form a closed path, etc.” It also enables thinking at a higher level.

In software development

Small parts arranged together into bigger (and higher-level functionality) parts
Small parts arranged together into bigger (and higher-level functionality) parts

In software development, the usual elements are classes, interfaces, methods, fields, associations (implementation, delegation, instantiation) and various constraints between them. Given a few of these elements we can form many possible arrangements, however only a relatively small number of them is useful and of interest. This happens because the useless arrangements tend to be quickly replaced by the skill-full developer into other that are more useful. For example, any arrangement of two distinct classes that depend to each other, forming a cycle, is usually not desirable. Patterns authors have been working for almost two decades to inventory as many useful arrangements as they could find, resulting into many books and publications in every domain.

Common and stable arrangements of two to three classes together form the basis for design patterns as in the GoF book, an idea I have experimented in a previous post: systematic combination of subpatterns generates design patterns.

Common stable arrangements of methods, fields and how they relate with each other within one class are simply stereotypes of classes, which we tend to call patterns anyway like the ValueObject pattern in the Domain Driven Design book.

Note that in this discussion we are focusing on arrangements of programming elements in the solution space, not in the problem space, but pattern express intents too.

Harnessing stable arrangements of things: toward more efficient tools

I believe that making explicit the use of predefined common stable arrangements of programming elements, in the coding process, can boost the efficiency of many tools. I also believe that such common and stable arrangements of programming elements have already been identified and are already well-documented in the existing pattern literature.

Rather than configuring tools at the programming element level (class, field, method etc.), if the code is augmented with explicit declarations of the patterns used, the tools can then be configured at the pattern level. For each tool, the idea is to prepare in advance how to configure it for each supported pattern. This preparation must be automated, so that given an occurrence of a known pattern in the code base, the configuration of the tool can be automatically derived from the particular details of the pattern occurrence.

In other words:

tool configuration=
  auto-configuration(pattern, tool) + pattern occurrence

A simple case study

To start with an example, let us consider the pattern Abstract Factory, that defines an Abstract Factory interface and one or more Product interface(s). Then assume that in our code base we have an occurrence of this pattern, where the interface WidgetFactory participates as the Abstract Factory, and the interfaces Window and Button participate as Product. Concrete classes form two families, one Linux family (LinuxWidgetFactory, LinuxWindow, LinuxButton) and one Mac family (MacWidgetFactory, MacWindow, MacButton), where each concrete class participates as ConcreteFactory, ConcreteProduct and ConcreteProduct respectively.

Dependencies restrictions (à la Jdepend + Junit)

The auto-configuration(AbstractFactory pattern, dependency checker tool) could be programmed like the following:

//Given a pattern occurrence from the actual base: occ

//Factory interface knows about the Product interface, not the other way round
For each Product in occ, add constraint (Product, Must not depend upon, AbstractFactory)

//ConcreteProduct must not know about the AbstractFactory
For each ConcreteProduct participant in occ, add constraint (ConcreteProduct, Must not depend upon, AbstractFactory)

//ConcreteProduct must not know about the ConcreteFactory
For each ConcreteProduct participant in occ, add constraint (ConcreteProduct, Must not depend upon, ConcreteFactory)

//Interfaces must not depend upon their implementor
For each abstract participant in occ, add constraint (participant, Must not depend upon, implementor of participant)

I have expanded the auto-configuration script to highlight how we can do more sophisticated configuration as soon as it is supposed to be reused many times, something we would never afford for one-shot configuration. Also in the above script, it is quite obvious that we can extract more powerful primitives to simplify the declaration of the script itself.

I have already presented this idea: toward smarter dependency constraints (patterns to the rescue).

Dependency injection (à la Spring)

The auto-configuration(AbstractFactory pattern, IoC container) could be programmed like the following:

//Given a pattern occurrence from the actual base: occ
//Given the selected family (Mac or Linux) we want to inject: F

//Bind the ConcreteFactory from F to the AbstractFactory interface
For the AbstractFactory participant in occ, and for the ConcreteFactory in occ that belongs to F, bind (ConcreteFactory, AbstractFactory)

//Bind each ConcreteProduct from F to the Product interface
For each ConcreteProduct in occ that belongs to F, bind (ConcreteProduct, corresponding Product)

Again we can see that we can automate the binding of each implementation to its corresponding interface from the single explicit selection of the family F we want to inject. This is a great way to factor out dependency injection to make it more manageable. In particular, the configuration is closer to the way we actually think.

Other tools

The same idea can be applied for many other tools, in particular:

Conclusion

In this post I described how it makes sense to consider overall arrangements of programming elements as higher-level constructs, which I identify with patterns (the solution part of patterns in fact). I emphasize the fact that the useful arrangements are not that numerous, and that many of them are already documented in the pattern literature. Finally I present how such arrangements or patterns, if declared explicitly in the code, can be leveraged to automate tools configuration in a powerful way.

As usual, any feedback is highly welcome.

Read More

Toward smarter dependency constraints (patterns to the rescue)

Low coupling between objects is a key principle to help you win the battle against software entropy. Making sure your dependencies are under control matters. Several tools can enforce dependencies restrictions, such as JDepend. However in a real project with many classes, packages and modules, the real issue is how to decide and configure the allowed and forbidden dependencies. Per class? Per package? Per Module? Based on gut feeling? Is there a theory for that?

Of course, in a layered architecture, the layers specify the dependencies. This is not bad, but I am sure we can do better.

Smarter dependencies

To go further, I suggest expanding our vocabulary of concepts. In OO languages such as Java, everything is a class (or an interface), grouped into packages. Such classification is not really helpful. Fortunately, several books provide ready to use vocabularies in the form of patterns languages (not only design patterns, but patterns in general). Some of these patterns are foundations on which rules to manage dependencies can be proposed.

Disclaimer: the dependencies rules suggested below are hypothesises to be debated and verified against a corpus of actual projects, I would be happy to be given counter-examples and counter arguments.

The child really depend upon the mother
The child really depend upon the mother

Domain Driven Design

The book Domain Driven Design by Eric Evans defines a rich vocabulary of concepts used in every application, and we can leverage that vocabulary to propose some dependencies principles between them:

  • ValueObject never depends upon Entity nor Services
  • Entities should not depend upon Services (maybe not a hard rule)
  • Generic SubDomain should not depend upon Core Domain
  • Core Domain should not depend upon Cohesive Mechanism (the “What” should not depend upon the “How”)
  • Domain Layer should not depend on any infrastructure code
  • Abstract Core module never depends on its specialized or implementation modules

Analysis Patterns

The book Analysis Patterns by Martin Fowler also provides patterns as a richer vocabulary, from which we could propose:

  • Elements from a Knowledge Level should not depend upon elements from the corresponding Operation Level

I did not find that rule written in the book but every example appears to support it. Considering that classes and subclasses in usual OOP are a special case of Knowledge Level built-into the language, this would lead to:

  • Abstraction never depends upon their Implementations

which is similar to the second part of the Dependency inversion principle by Robert C. Martin:

Abstractions should not depend upon details. Details should depend upon abstractions.

Since many analysis patterns in the Analysis Patterns book involve the Knowledge Level pattern, this single dependency rule already applies to many analysis patterns: Party Type Generalizations, Measurement, Observation, Protocol, Associated Observation, Measurement Protocol etc. The pattern Quantity can be seen as a specialized ValueObject (see Domain Driven Design above) hence should also not depend on any Entity nor Service.

Design Patterns

The book Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma et al. presents the classic design patterns. These patterns define participants which are named.  In the pattern participant ignorance principle I discussed the concepts of ignorant Vs. dedicated participants within a pattern, and their consequences for dependencies:

  • Ignorant pattern participants should never depend on dedicated participants
  • Pattern participant never depend on the “Client” participant
  • For each ConcreteX participant, the corresponding abstract X never depends on it (Abstraction never depends upon their Implementations)

In practice, this means:

  • In the Adapter pattern, the Adaptee should not depend upon the Adapter, and the Target should depend upon nothing
  • In the Facade pattern, the sub systems should not depend upon the Facade
  • In the Iterator pattern, the Aggregate should not depend upon the Iterator; however every Java collection is a counter example as they contain their own ConcreteIterator.
  • In creational patterns (Abstract Factory, Prototype, Builder etc.), the Product and ConcreteProduct should not depend on the dedicated participant that does the allocation (the Factory etc.)
  • And so on for other patterns, some of which being already discussed in the pattern participant ignorance principle.

In short, if we look at the design patterns as a set of types with inheritance/implementation, invocation/delegation and creation relationships between them, the dependencies should not flow in the reverse direction of the relationships; in other words, using UML arrows, the dependencies should only be allowed in the direction of the arrows.

Addiction to sugar is a kind of dependency
Addiction to sugar is a kind of dependency

Patterns of Enterprise Architecture

In the book Patterns of Enterprise Application Architecture by Martin Fowler, the Separated Interface Pattern proposes a way to manage dependencies by defining the interface between packages in a separate package from its implementation. This is similar to the Dependency inversion principle, as discussed here, which states:

A. High-level modules should not depend upon low-level modules. Both should depend upon abstractions.

By the way this is also very similar to the recommendation in Domain Driven Design:

Abstract Core module never depends on its specialized or implementation modules.

Finally, in the spirit of UML stereotypes that we sometimes put on packages to express their intent:

  • Utils never depends on anything but other Utils

What for?

If we manage to make every use of the above pattern explicit in the source code, for instance using Java annotations or simply Javadoc tags, then it becomes possible for a tool to deduce dependencies constraints and automatically enforce them.

Imagine, just add @pattern ValueObject in your Javadoc comment, and voila! A tool is now able to deduce that if you happen to import anything but basic java.* you must be warned.

Of course the fine tuning of the default behavior can take some time (do we accept that ValueObjects may depend upon low level utils like StringUtils? Probably yes), but the result will at least be stable regardless of the refactorings.

Given the existing variety of patterns over there, I am confident that just any class or interface within a project can be declared as being a participant in at least one pattern, and have therefore its dependency constraints deduced at the same time.

Read More

Systematic combination of subpatterns generates design patterns

Composite patterns, such a the Bureaucracy pattern, are patterns built by the composition of other “smaller” patterns. However even usual design patterns can be considered composite patterns made of smaller subpatterns.

The goal is therefore to find out which are the main subpatterns that enable to reconstruct as many design patterns as possible.

The subpatterns

From an early analysis I believe that four subpatterns form the foundation of many GoF design patterns:

  1. Type Hierarchy and its variants (interface, abstract class, non final class…)
  2. Type Set, a set of types with no particular relationship between them
  3. Delegation and its variants (one-to-one, one-to-many)
  4. Allocation (creation of instances) and its variants

The experiment

To validate this proposal, I have coded a small experiment to generate every combination of two subpatterns out of (1) and (2) plus one relationship between them out of (3) and (4), or every combination of one subpattern out of (1) and (2) plus a relationship of (3) and (4) to either itself, or in the case of (1), between a super type and a subtype or the other way round.

For each generated combination I have automatically generated its corresponding class diagram, and grouped them together into one SVG document, converted into PDF using Inkscape. The resulting picture is available here, and you can see a partial preview below:

combinations_thumb

Results

The interesting finding is that this simple combinatorial method re-discovers 11 well-known design patterns (in the reading order of the picture):

  • Template Method (delegation to self, expecting a subtype to provide the service)
  • Composite (Considering a one-to-many delegation) or Decorator / Proxy (one-to-one delegation)
  • Prototype (creates instance of this type)
  • Bridge (one Abstraction hierarchy delegates to another Implementor hierarchy)
  • Abstract Factory (one Factory hierarchy creates instances of another (or several others) Products hierarchy)
  • … skip 4 next …
  • Strategy or State or Builder (some client delegates to some hierarchy)

Note that there are several cases where we could see the Factory Method, Adapter or Facade patterns. This would lead to some 13 design patterns out of the 23 in the GoF book.

Discussion

It is obvious anyway that the four subpatterns used here are not enough to rebuild the full set of the 23 design patterns. The Command pattern is unique in its use of a separate invoker, and the Chain of Responsibility is clearly about how handlers self-assign a given task without any centralized management. In these two examples however, the unique characteristics is not in their structure but rather in their dynamics. Sequence diagrams would be much more relevant to convey that kind of information.

The full test runs in one go in a fraction of second. The source code is available here: patternitygraphic_src, the code of the experiment is in CombinationTest.

EDIT: the plain SVG file that was generated is also available here for download:

Read More

New: Java API for UML diagrams

As part of the Patternity effort, I spent some time creating a simple Java API to generate UML diagrams programmatically from Java, in SVG format.

This small API called for now Patternity Graphic is working and available here: patternitygraphic_src as a source Zip (alpha release of course).

It can render small class diagrams with hierarchic, flow and radial layouts, and arbitrary sequence diagrams with unlimited nesting of method calls and embedded comments. My focus was to support the subset of diagram elements and capabilities required to display patterns occurrences.

Here is a sample sequence diagram with a nice call stack:

sequence1

And here is a class diagram for a simple dummy hierarchy:

hierarchy

Boxes and links have various styles, defined in a template.svg template file, here is a random display of the boxes styles:

boxstyles

Apart from unit tests there is no documentation. If you are interested to reuse that please contact me for help. The Zip was exported in Eclipse with the project Export… function, and the project requires Commons Collections. Svg diagrams can be converted into images using Batik.

Other projects you might consider to generate UML diagrams: UMLGraph, modsl, umlspeed, jsigner, MetaUML, and of course GraphViz.

Any feedback appreciated!

Read More

Patternity is back!

After my initial attempt at doing something useful and automated around design patterns, I started working again on a brand new version of a pattern-aware tool: Patternity.

No more code generation, the focus is now on the generation of documentation artifacts (UML class and sequence diagrams, reports, enhanced Javadoc…) from the source code annotated with explicit pattern occurrences declarations. More like a research project, the goal is to investigate how a tool can be valuable for developers by being aware of the pattern and their occurrences in the source code.

Roadmap

Challenges

Unlike the initial tool I did in 2002 that was really basic, there are many challenges and concerns to be addressed for a tool to deal with pattern in a clever way:

  1. Patterns variants: a pattern can be realized in the source code in many different ways, while still being the same pattern. For instance many patterns rely on polymorphism,which in practice can be achieved with an interface, an abstract class or just a non-final class. Many patterns also use some form of delegation, typically through an association (member field), but the pattern still holds if the delegation occurs through a lookup mechanism.
  2. Representation of patterns: creating a patterns database of a priori knowledge about patterns, including their static and dynamic aspects in a formal way that a tool can interpret – for instance to generate diagrams. Such representation must also be small enough to be feasible manually (of course only once).
  3. How to declare patterns occurrences within the code without interfering with the developer flow of work: easy and logical annotation syntax, good defaults values, code completion in IDE.
  4. How to generate useful artifacts: documentation class and sequence diagrams, reports…

Pattern metamodel

An analysis of a pattern metamodel (points 1 and 2 above) reveals other concerns that add up to the task:

  • Occurrences: What is a pattern occurrence in the code? How many occurrences of the same pattern can be considered the same occurrence? It sounds obvious that a Strategy class and all its subclasses are indeed one same pattern occurrence, but this is a priori not true for the subclasses of an Immutable class (I take a very wide definition of patterns that includes many idioms and other UML-like stereotypes).
  • Classification: Patterns usually couple a Problem (or Intent) and a Solution for it within a Context; how to create the most useful classification of patterns? (I tend to favour a classification by Problem, but this means a SecurityProxy will probably not be related with a VirtualProxy, which is not very intuitive)
  • Identification: How to identify a pattern occurrence? This is useful so that a declaration of a pattern occurrence can reference another pattern occurrence as a member part of it, just like the Bureaucracy pattern is composed of a Composite, a Mediator, a Chain of Responsibility and an Observer.

Pattern declaration

Declaration of pattern occurrences in the source code (point 3 above) also has its issues:

  • Syntax: Definition of a good syntax, easy to grasp with as little explanation as possible is hard
  • Consistency: many patterns introduce additional elements (we can call them active participants) around existing elements (passive elements) that must not know about the pattern. For instance in the Decorator pattern, the Decorator participant (role) participates actively in the pattern, whereas the decorated element should NOT be touched at all, which means that the source file for the decorated class should not be touched in the version control when adding the pattern.
  • Where to declare: in the source code, using Javadoc tags for classes or methods, or in an external file when this is more convenient (e-g to annotate third-party libraries).
Example of a declaration of a Builder pattern occurrence
Declaration of a Builder pattern occurrence

Generation of documentation artifacts

Generation of artifacts (point 4 above) from both the pattern occurrence declaration and the a priori knowledge about the pattern in the pattern database also yields its own challenges:

  • Reuse, adapt or create a Java API to generate diagrams
  • Find out a logic to interpret pattern data into diagrams or other reports
  • How to deal with huge code bases: use ellipsis, continuation marks and move details to addendum to reduce the size of a diagram

Discussion

Why use patterns for documentation?

Patterns can store pre-digested documentation as “templates”, either textual or visual (diagrams); then for each actual pattern occurrence these templates can be “evaluated” to generate real and complete documentation artifacts. When we want to create useful documentation, the need for developers to inject design information in addition to the information that is already present in the code base can therefore be reduced, since a significant part of these additional information can be factored out and stored once into the patterns metamodel. I expect that patterns are a convenient concept to encapsulate reusable design information and how it can be used, however this has still to be demonstrated in practical use.

I expect that patterns can help create better diagrams than usual case tools, just because patterns know more about the design: knowing that the Bridge pattern is made of type hierarchies (to render vertically) and a delegation betweem them (to render horizontally) makes it easy to draw its class diagram this way:

Generated class diagram for the Bridge pattern
Generated class diagram for the Bridge pattern

Experimental validation

Possible validation of the utility of the tool can be:

  • Design documentation reconstitution: on a well-known and well-documented Open-Source project in Java, annotate the source code with pattern occurrences declarations, then generate a documentation. By comparing this generated documentation with the official documentation of the project one can manually assert the relevance of our approach
  • Maintenance experiments: similar to this study.
  • From the sole pattern database, generate default documentation for each “default” (generated) pattern occurrence, and compare with the patterns books, the more similar the better.

Future work

We could imagine several other potential benefits from using a pattern-aware tool:

  • The requirement to explicitly declare each pattern occurrence while coding also leads to a more motivated design, where each design decision has to be deliberate rather than more or less random or automatic. For beginners, this also represents an incitive to open the books more often to lookup “what is the pattern I am trying to do right now?” because of the need to document it.
  • Provided that the tool can link a pattern with the Problem it is supposed to address, the pattern occurrences can then trace back to their Problem, and from there, link to other alternative patterns that could have been used as well. This could be valuable to do design-level refactoring.
  • The explicit knowledge of the pattern occurrences in the code base can also give insight to configure other tools: for instance, since the patterns know what participant is supposed to know or not the other participants, this knowledge could be used to configure Jdepend automatically.

Please let me know your ideas, remarks or scepticism on this topic!

The official Patternity website still presents the old version; I expect to update it when there is good progress on the new version, however the new source code will be very soon under the same Sourceforge CVS.

Read More

Sharing a common architecture or vision within a team

What do you think about when you hear the word “architecture” about software? Fowler defines it: whoNeedsArchitect “In most successful software projects, the expert developers working on that project have a shared understanding of the design system design. This shared understanding is called ‘architecture.'”.

However for most people the word “architecture” comes full of middleware connotations, hence I consider this word now inappropriate. Let us consider here the concept of a shared “vision” of the design.

First it has to be a shared vision of the domain, for instance the interest rate swaps trading business. Words must have only one meaning, known by everyone, from IT, marketing or support. A shared glossary is key for that.

Then the IT team must share a consistent set of practices: consistent between developers, and consistent between practices themselves. This also means that the set of practices de-facto excludes other practice that conflict with it, and this has to be clearly acknowledged.

Even though any set of practices can be considered arbitrary, everybody in a team must adhere to it, preferably wholeheartedly. Of course the more accepted and documented the practices, the easier it is for everybody to adhere to them.

For example, a team can have a set of practices that includes Agile programming (be prepared for change, no big design upfront), automated testing (unit tests, stress tests), domain-driven analysis as devised by Eric Evans, the use of proven solutions when they exist (hence design and analysis patterns from GoF, POSA, Fowler, Doug Lea…), a bit of functional-programming style (favour immutability and side-effect-free functions, pipe and filters…), and Eclipse default coding conventions.

Some practices that also belong to the shared set of practices might not be directly documented, though they derived from articles, papers, blogs and some personal opinions. Examples of such custom practices that I like a lot include “Align the code with the business” and “think options not solutions”.

“Align the code with the business”, focuses at the smallest level, i.e. at the class and method level. Just every bit of the application must make sense with the domain considered. A class must reflect some relevant concept, concrete or more abstract, of the domain, not of the requirement. The rationale is that while requirements change all the time, the domain concepts are very stable; they are not true or false, they just exist!

“Think options not solutions” refers to the very basics of Agile and XP: “be prepared for change”. It also acknowledges the fact that most of the code in a well done software project provides standalone, encapsulated and reusable building blocks that are then assembled together using a smaller amount of glue code, that can even be reduced by using a dependency-injection framework such as Spring. Typically, a concept from the domain, such as an interpolation algorithm in the finance domain, translates into an Interpolation interface along with several alternate implementations. Most of the time, only one implementation out of the ten possible is really used at a time. However the option to use the other nine alternatives is there. This also means that we can assemble a release of the software using some subset of its building blocks, and assemble another release using another subset, at the same time. One codebase, several possible applications: this is the goal.

In addition to every explicit practices, the existing code base implicitly shows practices that can either be reused just for the sake of consistency, or replaced as widely as possible when they conflict. Prior art, when of good-enough quality, can form a skeleton for a shared vision of how to create software. Again this is arbitrary, but consistent, and consistent design is valuable in itself, and often more than the design decisions themselves.

Of course there is a point where more shared practices become counter-productive, for instance when there are too many of them (more efforts to learn by newcomers, feeling of loosing freedom, lack of flexibility). However when they are standard enough, or becoming standard enough, developer should consider that acquiring these practices a long term, beneficial, investment.

Initially published on Jroller on Monday January 21, 2008

Read More

Documenting the design of software using patterns?

I have experimented an approach that considers every design pattern as the recursive composition of smaller patterns. This led to a prototype tool to illustrate its benefits by generating design-level documentation of annotated source code.

Eat your own dog food

The source code of this tool itself was used as the code base to apply the tool on, in order to generate its own design documentation. The Java Doclet API was used to retrieve the Javadoc tags in the class Javadoc comments, e.g. :

@pattern KnowledgeLevel OperationLevel=MyOtherClass some free text comment…

The pattern catalogue

A subset of design patterns and some other patterns such as the Knowledge Level (Fowler) has been defined in small definitions files then loaded at tool startup. We call this set of patterns definitions the pattern catalogue.

Here is an example of the definition file for the Builder pattern, notice the declaration of the member roles at the bottom, that also declares the expected pattern kind:

Encapsulates the creation of a complex object from a source object.
@extends DesignPattern
@category DesignPattern
@category Creational
@author GoF
@book [GoF 95]  E. Gamma, R. Helm. R. Johnson, and J. Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley 1995.
@style ObjectOrientedProgramming
@reference [http://en.wikipedia.org/wiki/Builder_pattern ]
@defaultrole Builder
@role Director=TypeHierarchy
@role Builder=TypeHierarchy
@role Product=TypeSet
@role Factory=AbstractFactory

Parsing the annotations to build the pattern occurrences graph

Every time a Javadoc tag “@pattern Pattern” is met in the Javadoc part of a class, an occurrence of the corresponding pattern is created in memory, linking to the corresponding class in the Java programming language metamodel.

Whenever such pattern uses a Type Hierarchy (this is declared in the definition of the pattern), then the tool will look for every subtype of the declared (super) type in order to complete the pattern occurrence as well as possible.

Even patterns that are usually considered as atomic patterns can be explained as the composition of smaller “elemental” patterns. For instance, A Builder [GoF] usually defines an AbstractBuilder role and a ConcreteBuilder role that implements it. We can express this part of the Builder pattern as a TypeHierarchy elemental pattern in the role Builder; the supertype of the TypeHierarchy therefore represents the AbstractBuilder role, while every other concrete type of this hierarchy represents ConcreteBuilder roles.

By analogy with the members of a class, we call members patterns and member occurrences the patterns and occurrences (respectively) a pattern or occurrence is composed of. The tool totally ignore fields and methods, it only considers types (interfaces and classes).

The benefits of this approach are to reuse a bigger part of the pattern definitions through composition and to help address the variant problem.

Once the pattern occurrences graph is completely built, it is rendered into an UML class diagram using Graphviz dot. Here is an example of overview for the full project:

coremodulediagram_final
Example of overview diagram for the Core module of the prototype source code

In the above diagram we can see how the patterns naturally sit on top of the several type hierarchies while connecting them together. Also the relationships between patterns become very obvious, such as the Interpreter that references the Builder and the Composite sub-patterns. This suggests that such diagrams are very valuable to document the design of a code base.

Topological sort

The approach of representing every pattern as the composition of other patterns yields to directed graphes, at the pattern level (the “meta” level) and at the pattern occurrences level. One consequence is that we can apply a topological ordering on such graphs. In the current prototype we need to do a topological sort on the patterns definitions so that we can know in advance in what order to process pattern occurrences declarations: the patterns used must be built before the patterns using them.

The pattern occurrences graph can also be navigated using the usual Depth-First search or Breadth-First search. DFS is convenient to generate the Overview diagrams such as the diagram shown above, whereas BFS is convenient to drill down the design from the top to the bottom, one level at a time. This can therefore generate the following table of contents:

Patterns graph navigation index
Patterns graph navigation index

Natural language generation alternative

As an alternative to visual diagrams, a tool can also generate natural language text to communicate the exact same information. I have tried this approach using simple text templates in each pattern definitions. Each pattern therefore defines its own design description, but using variable names to be filled later with the data from the pattern occurrences graph.

The biggest work in this approach is to deal with the plural, the enumeration of collections (comma between each item but the last etc.), truncating collections that are too long, and the complete omission of a section if a pattern member is totally missing.

Here is a sample of text generation for the prototype tool itself. Notice how the structure is rigid for each pattern occurrence: Pattern name, short description, then template text evaluated with the actual types names, followed by the comment that was put after the @pattern declaration:

Design of the module Core
This module is essentially made of 9 pattern occurences: Strategy, Visitor, KnowledgeLevel, Interpreter, Builder and AbstractFactory.

Visitor
The visitor design pattern is a way of separating an algorithm from an object structure. A practical result of this separation is the ability to add new operations to existing object structures without modifying those structures
This module defines a Visitor OccurrenceVisitor. comment

Strategy
A Strategy object encapsulates an algorithms that can be selected on-the-fly at runtime.
This module defines the interface of a Strategy: Plugin. It enables plugins to manipulate the Patternity metamodel.

AbstractFactory
Provides one or more method(s) that encapsulate(s) the creation of object.
The type OccurrenceFactory defines the interface of an AbstractFactory that creates instances of Occurrence and TypeOccurrence.

Interpreter
Define a representation for a language along with an interpreter that uses the representation to interpret sentences in the language.
The type(s) Occurrence and TypeOccurrence define(s) an object representation of the considered language. The considered language describes a graph of patterns occurrences. It uses the Visitor Visitor to add new operations to the object representation without modifying this structure.

Strategy
A Strategy object encapsulates an algorithms that can be selected on-the-fly at runtime.
This module defines the interface of a Strategy: Loader. Encapsulates how to load every pattern definition into the pattern.repository.

[...]

KnowledgeLevel
A Knowledge Level is a group of objects that describes how another group of objects should behave.
This module defines a KnowledgeLevel (also known as metamodel). It is made of two levels of objects: objects in the knowledge level, i.e. the instances of Pattern define how objects in the operational level can behave, i.e. instances of Occurrence. Typically each type in the operation level maintains a reference to its corresponding type in the knowledge level in order to know how to behave.

Conclusion

Later, at work, when asked to produce a design documentation targeted at other developers, I have experimented doing a similar effort manually, using nothing but patterns. I am still impatient to receive some feedback from the people that will read the resulting documentation in order to validate or infirm this approach.

Read More