The Arolla “ancient world map” of software development

Software development technologies and trends are not particularly tangible things, yet we need to reason on them. At Arolla, the company I’m part of, we’ve designed an “ancient world map” of software development, as a cartography of the universe of software development we live in. Built for our own purpose, we also share it so you can benefit from it.

 The Arolla "ancient world map" of our software development universe
The Arolla "ancient world map" of our software development universe

If you want to use the map with your own teams, please do so (it’s licensed CC-NC-ND). If you need a high-resolution file for print, just ask (the file is quite big). We’d love to get your feedback!

The metaphor of an ancient world map

Agile and in XP suggest using metaphors to help materialize abstract stuff, and make it easier to grasp. You’ve probably seen Eric Evans (Domain-Driven Design) showing a picture of an ancient world map when presenting his concept of “a model built for a purpose”. We wanted to materialize software development technologies and trends, for which we have no clear and accurate visualisation yet, just like explorers in the middle age had no accurate knowledge of the world, lands and oceans. Ancient world maps had to represent that part of ignorance, with dragons and strange creatures on the less-known areas.

So we chose this metaphor to represent our universe. And ancient world maps are beautiful too!

A map conveys meaning

In cartography, the role of map design is to:

Orchestrate the elements of the map to best convey its message to its audience.

In our map, each continent represents a particular chapter of related technical stuff , and oceans in between represents “soft” techniques that complement them best. Of course the universe of software development has many more dimensions than the two dimensions available on a map. This means that such map is quite subjective, it depends on our own mental model. On the other hand, this is also true for any map of the physical world, that is also supposed to represents a 3D planet onto a plane, with some deep decision on whether to preserve angles or distances.

We tried to put most related technologies as close as possible to each other. Regions in the middle of the map represent the core of a developer daily work, in contrast with the regions closer to the poles which are more specialized.

Of course not every existing technology and trend was included on the map, especially the ones that we do not want to offer to our clients or that are of less importance. As a fan of DDD I regret we could not include our clients domains (finance, e-commerce, media, e-advertising, online games…) on the map, but a map is for a purpose, just like a model. A map of everything would just be useless.

Drawing the map proved a lot more work than expected, with hundreds of layers and lots of little adjustments everywhere, resulting into a huge file for printing.

To discuss skills and areas of expertise

Why this map int the first place? At Arolla we wanted to organise a session for all our consultants to discover more about each other from a skills point of view. We also wanted more senior consultants to volunteer and take ownership on some areas of their technical expertise.

We didn’t want another boring evening with slides shows in front of a passive audience. We wanted a more concrete, more fun and more engaging way to talk about technical skills and areas of knowledge.

So we printed the map on a big poster laid on a table, like in the captain room in an old vessel, and made it into a game.

Map + Lego = game

During last June DDD Immersion course at skillmaster, Alberto told me about Lego Serious Play. This game helps people express their ideas and be creative using Lego bricks as a medium, to help say crazy things in front of other people. It also makes any discussion a lot more fun! So we bought a set of Lego mini-figures, including some crazy figures, for our consultants to play with on the map (we did not follow the actual Lego Serious Play game at all).

The first part of the game was to get familiar with the map. I was shouting the name of various technologies at random, and the first to find out where it is on the map would take a marshmallow and place it as a flag (we had lots of sweeties everywhere too). This part was fast-paced and didn’t take long as it was quickly obvious that everyone had understood and indexed the map in their mind already.

Putting marshmallow on the map

In the next part, we had to create our own Lego persona, using the Lego building blocks available. It was really fun, and everyone was really happy to be given the opportunity to play with Lego again. It was nice to see that nobody built a mini-figure as planned on the box, they were all very personal, ironical or really weird.

Each consultant then told his/her career by moving his/her Lego custom figure over the big map, starting from some technology (e.g. C++) “in this continent” before “moving to another territory” (e.g. Java or .Net), then “crossing oceans” to gain additional skills in software factory and agile.

I particularly enjoyed the relaxed atmosphere, where anyone would interrupt to ask questions or comment, or throw a joke out loud. Perhaps the map, the Lego and the marshmallow made it clear that it was not too serious an exercise. The map and the Lego figures were able to make abstract skills more tangible and more fun for the time of the game.

We could have done the same exercise without all that, but the feedback we had from our consultants was very positive, they really enjoyed the game.

What’s next?

We’re going to put the big poster of the map on the wall, as a decoration. Smaller printed maps will help scope some discussions, perhaps for interviews, or when discussing with less technology-involved people. It could also be useful for developers to self-assert how much they know about the current state of the art in our craft.

Read More

Thinking functional programming with Map and Fold in your everyday Java

In functional programming, Map and Fold are two extremely useful operators, and they belong to every functional language. If the Map and Fold operators are so powerful and essential, how do you explain that we can do our job using Java even though the Java programming language is lacking these two operators? The truth is that you already do Map and Fold when you code in Java, except that you do them by hand each time, using loops.

Disclaimer: I’m not a reference in functional programming and this article is nothing but a gentle introduction; FP aficionados may not appreciate it much.

You’re already familiar with it

Imagine a List<Double> of VAT-excluded amounts. We want to convert this list into another corresponding list of VAT-included amounts. First we define a method to add the VAT to one single amount:

public double addVAT(double amount, double rate) {return amount * (1 + rate);}

Now let’s apply this method to each amount in the list:

public List<Double> addVAT(List<Double> amounts, double rate){
  final List<Double> amountsWithVAT = new ArrayList<Double>();
  for(double amount : amounts){
     amountsWithVAT.add(addVAT(amount, rate));
  }
  return amountsWithVAT;
}

Here we create another output list, and for each element of the input list, we apply the method addVAT() to it and then store the result into the output list, which has the exact same size. Congratulations, as we have just done, by hand, a Map on the input list of the method addVAT(). Let’s do it a second time.

Now we want to convert each amount into another currency using the currency rate, so we need a new method for that:

public double convertCurrency(double amount, double currencyRate){return amount / currencyRate;}

Now we can apply this method to each element in the list:

public List<Double> convertCurrency(List<Double> amounts, double currencyRate){
   final List<Double> amountsInCurrency = new ArrayList<Double>();
   for(double amount : amounts){
      amountsInCurrency.add(convertCurrency(amount, currencyRate));
   }
   return amountsInCurrency;
}

Notice how the two methods that accept a list are similar, except the method being called at step 2:

  1. create an output list,
  2. call the given method for each element from the input list and store the result into the output list
  3. return the output list.

You do that often in Java, and that’s exactly what the Map operator is: apply a given method someMethod(T):T to each element of a list<T>, which gives you another list<T> of the same size.

Functional languages recognize that this particular need (apply a method on each element of a collection) is very common so they encapsulate it directly into the built-in Map operator. This way, given the addVAT(double, double) method, we could directly write something like this using the Map operator:

List amountsWithVAT = map (addVAT, amounts, rate)

Yes the first parameter is a function, as functions are first-class citizens in functional languages so they can be passed as parameter. Using the Map operator is more concise and less error-prone than the for-loop, and the intent is also much more explicit, but we don’t have it in Java…

So the point of these examples is that you are already familiar, without even knowing, with a key concept of functional programming: the Map operator.

And now for the Fold operator

Coming back to the list of amounts, now we need to compute the total amount as the sum of each amount. Super-easy, let’s do that with a loop:

public double totalAmount(List<Double> amounts){
   double sum = 0;
   for(double amount : amounts){
      sum += amount;
   }
   return sum;
}

Basically we’ve just done a Fold over the list, using the function ‘+=’ to fold each element into one element, here a number, incrementally, one at a time. This is similar to the Map operator, except that the result is not a list but a single element, a scalar.

This is again the kind of code you commonly write in Java, and now you have a name for it in functional languages: “Fold” or “Reduce”. The Fold operator is usually recursive in functional languages, and we won’t describe it here. However we can achieve the same intent in an iterative form, using some mutable state to accumulate the result between iterations. In this approach, the Fold takes a method with internal mutable state that expects one element, e.g. someMethod(T), and applies it repeatedly to each element from the input list<T>, until we end up with one single element T, the result of the fold operation.

Typical functions used with Fold are summation, logical AND and OR, List.add() or List.addAll(), StringBuilder.append(), max or min etc.. The mindset with Fold is similar to aggregate functions in SQL.

Thinking in shapes

Thinking visually (with sloppy pictures), Map takes a list of size n and returns another list of the same size:

On the other hand, Fold takes a list of size n and returns a single element (scalar):

You may remember my previous articles on predicates, which are often used to filter collections into collections with less elements. In fact this filter operator is the third standard operator that complements Map and Fold in most functional languages.

Eclipse template

Since Map and Fold are quite common it makes sense to create Eclipse templates for them, e.g. for Map:

Getting closer to map and fold in Java

Map and Fold are constructs that expect a function as a parameter, and in Java the only way to pass a method is to wrap it into a interface.

In Apache Commons Collections, two interfaces are particularly interesting for our needs: Transformer, with one method transform(T):T, and Closure, with one single method execute(T):void. The class CollectionUtils offers the method collect(Iterator, Transformer) which is basically a poor-man Map operator for Java collections, and the method forAllDo() that can emulate the Fold operator using closures.

With Google Guava the class Iterables offers the static method transform(Iterable, Function) which is basically the Map operator.

List<Double> exVat = Arrays.asList(new Double[] { 99., 127., 35. });
 Iterable<Double> incVat = Iterables.transform(exVat, new Function<Double, Double>() {
   public Double apply(Double exVat) {
     return exVat * (1.196);
   }
 });
 System.out.println(incVat); //print [118.404, 151.892, 41.86]

A similar transform() method is also available on the classes Lists for Lists and Maps for Maps.

To emulate the Fold operator in Java, you can use a Closure interface, e.g. the Closure interface in Apache Commons Collection, with one single method with only one parameter, so you must keep the current -mutable- state internally, just like ‘+=’ does.

Unfortunately there is no Fold in Guava, though it is regularly asked for, and there even no closure-like function, but it is not hard to create your own,  for example, you can implement the grand total above with something like this:

// the closure interface with same input/output type
public interface Closure<T> {
 T execute(T value);
}

// an example of a concrete closure
public class SummingClosure implements Closure<Double> {
 private double sum = 0;

 public Double execute(Double amount) {
   sum += amount; // apply '+=' operator
   return sum; // return current accumulated value
 }
}

// the poor man Fold operator
public final static <T> T foreach(Iterable<T> list, Closure<T> closure) {
 T result = null;
 for (T t : list) {
   result = closure.execute(t);
 }
 return result;
}

@Test // example of use
public void testFold() throws Exception {
 SummingClosure closure = new SummingClosure();

 List<Double> exVat = Arrays.asList(new Double[] { 99., 127., 35. });
 Double result = foreach(exVat, closure);
 System.out.println(result); // print 261.0
}

Not only for collections: Fold over trees and other structures

The power of Map and Fold is not limited to simple collections, but can scale to any navigable structure, in particular trees and graphs.

Imagine a tree using a class Node with its children. It may be a good idea to code once the Depth-First and the Breadth-First searches (DFS & BFS) into two generic methods that accept a Closure as single parameter:

public class Node ...{
   ...
   public void dfs(Closure closure){...}
   public void bfs(Closure closure){...}
}

I have regularly used this technique in the past, and I can tell it can cut the size of your classes big time, with only one generic method instead of many similar-looking methods that would each redo their own tree traversal. More importantly, the traversal can be unit-tested on its own using a mock closure. Each closure can also be unit-tested independently, and all that just makes your life so much simpler.

A very similar idea can be realized with the Visitor pattern, and you are probably already familiar with it. I have seen many times in my code and in the code of several other teams that Visitors are well-suited to accumulate state during the traversal of the data structure. In this case the Visitor is just a special case of closure to be passed for use in the folding.

One word on Map-Reduce

You probably heard of the pattern Map-Reduce, and yes the words “Map” and “Reduce” in it refer to the same functional operators Map and Fold (aka Reduce) we’ve just seen. Even though the practical application is more sophisticated, it is easy to notice that Map is embarrassingly parallel, which helps a lot for parallel computations.

Read More