The Self-descriptiveness pattern

The Self-Descriptiveness pattern can save your life many times in the course of any software project. The power behind it is to harness the computer, rather than yourself.

What is Self-Descriptiveness?

Self-Descriptiveness is a property of any system that is able to describe itself with no external help. Just ask it “describe yourself” and it will. In other words, it is a system where its documentation is embedded into itself.

This pattern is so essential, however it is not expressed often. I imagine it is too obvious for those that appreciate it, whereas other do not imagine its benefits and focus on how memory or bandwidth they save instead.

You can find this pattern in many areas:

Database

Database tables in many databases are self-descriptive, thanks to definitions or system tables. This enables to query the database for its own schema, without any other documentation or explanation. This also enables tools to browse a database just by connecting to it. Databases try to be user-friendly (SQL), so here Self-Descriptiveness also means affordability for users.

XML

XML is one of the most famous example of self-descriptive format. The use of named tags around the values to describe them is supposed to enable humans or even tools to read and extract their meaning even with no prior knowledge of the exact format. The ordering could vary, elements may be missing, it would remain readable. In this case Self-Descriptiveness means robustness against variations, perhaps to be more future-proof.

This key holder is self-descriptive
This key holder is self-descriptive

Spreadsheets, associative arrays

Spreadsheets with header columns, Map and associative arrays in general are other low-tech forms of self-descriptiveness we use all the time. You don’t have to remember what each columns represents (or each slot in a Map), the headers (or keys) tell you that directly. Here Self-Descriptiveness means convenience for the developers.

Reflection

Reflection built into languages such as Java enables the software to query its own structure. It becomes then possible to introspect each class to find out which fields and methods it has, and what class or interface its extends and implements. The metamodel of Java makes class definitions self-descriptive. This feature is essential to load at runtime code that was not known at compile time. Here Self-Descriptiveness means extensibility.

Analysis patterns

This leads to the Knowledge Level pattern (also known as Metamodel), which uses objects to describe the behavior of other objects. The Operation Level becomes self-descriptive thanks to its Knowledge Level. This is typically how databases and virtual machines implement their Self-Descriptiveness. The Knowledge Level pattern is key to achieve Self-Descriptiveness for systems.

The Quantity patterns is a simple yet very powerful form of Self-Descriptiveness for your ordinary project, especially in the Domain Layer. The idea is to keep the unit attached with the value. In the case of the Money pattern, the unit is the currency. If your application deals with percents, by all means introduce from the beginning a class Percentage that follows the Quantity pattern. This simple class has the potential to completely eliminate every percent-related bug! In the finance domain, interest rates, usually expressed in percents but that are actually in percent per year, deserve their own Quantity class as well. This class should be distinct from the usual percentage. Again this simple decision wil ensure that no one will ever confuse one for another. You could even go further and type each kind of interest rate by its destination, either a coupon rate, or a yield, and therefore have the computer verify they must not be confused.

Be explicit!

Self-Descriptiveness could be generalized into one principle: “Be explicit!”. This means that everything implicit must be identified and made explicit. Your application only deals with one currency? Make the currency explicit anyway. Your time periods are all an integer number of years? Make the time unit (years) explicit anyway. All your data come from Reuters? Add a field “Source” to track the source of the data anyway. Even if YAGNI, the consistency of your domain matters. And perhaps you will gonna need it!

This chair is self-descriptive, and explicit!
This chair is self-descriptive, and explicit!

Self-Descriptiveness is obviously very precious for debugging and maintenance. First you reduce the possibility of bugs: you can try to sum 0.05 +2% + 35bp (basis points are one-hundreds of percent) and still have a valid result (7.35%) if you use Quantity objects for them. Then debugging becomes a breeze, everything tells its story, especially if you have written judicious toString() methods.

Optimization on the other hand appears to dislike Self-Descriptiveness. XML is very verbose, a Quantity object consumes more memory than just the primitive value inside, etc. In most cases, it is not worth compromising Self-Descriptiveness for the sake of optimization. And even when you need real performance, there are solutions to retain the benefits of Self-Descriptiveness without its cost. XML compression works very well, associative maps can sometimes be implemented as random access in an array, etc. And extreme needs can be satisfied with extremely sophisticated solutions*.

To be fully honest, Self Descriptiveness can hardly be achieved in practice as explained by Mark Baker in his blog, but your project deserves as much Self Descriptiveness as you can possibly afford!

*For instance the FIX protocol, used for financial data feeds, is a very old key-value format. It is not really self-descriptive as the keys are numbers, not labels. However just like self-descriptive formats, the keys impede efficiency: they consume bandwidth for almost nothing. So instead of changing the format in favour of efficiency, the protocol FAST has been invented. This optimization done by the computer, not by developers, extracts in real time the values using a message template, compress them and send them in fixed order, in other words in a fully implicit way. On reception the message is automatically rebuilt against the template. More details about FIX and FAST can be found here and here. This is a fairly sophisticated solution.

Pictures taken at the Milano Furniture Fair 2009.

Read More

Patterns as another kind of types

Patterns represent a couple (intent, solution), where the intent matters most. Based on that intents, that can be generic or specialized, I propose to consider patterns like types in languages with strong typing, for the compiler to enforce their constraints.

Declaring patterns: what for?

Consider the very simple Quantity pattern from Analysis Patterns (Fowler):

Represent dimensioned values with both their amount and their unit

By simply declaring this pattern, we immediately express many things:

  1. We wish to represent a quantity with its unit
  2. Quantity is a special kind of ValueObject (as in Fowler or DDD), hence:
    1. It is Immutable, therefore it has no setter, only a valued constructor
    2. Identity is based solely on its values, two value objects are equal if they share the same values
  3. In Java, the hashcode() and equals() methods must be implemented solely on the values of the immutable fields, and the hashcode can be cached internally; the toString() result can be cached as well
  4. In the spirit of Domain Driven Design, it also probably follows Closure of Operation and Side -Effect-Free Functions
  5. It must not depend on any heavyweight dependencies such as middleware, database or gui classes
  6. It must not depend on Entities and Services
Strangely typed furniture
Strangely typed furniture

If we had an explicit way to mark the use of the pattern in the code, and if the compiler was able to know about this pattern, then it could enforce all the above. Moreover, it could also generate corresponding unit tests to assert the dynamic and runtime aspects constrained by the pattern.

This approach can be considered similar to the typing used in programming languages, where the compiler can enforce many constraints according to the type declarations given by the developer.

In practice

In current languages there is no explicit way to declare the use of patterns in the source code. The reader of the source must pay attention to various hints and compare to his/her prior knowledge to decide whether or not a pattern is used. Typical hints include naming conventions, where the class or interface names ends with the name of the pattern participant: an interface TradeFactory is probably a Factory that creates Trade instances.

Since many years I have been using a custom Javadoc tag @pattern to explicitly mark the patterns I use in my source code, hoping that a tool could use it in the future. Over time I have noticed several benefits from doing that, the biggest one being that I am forced to be 100% clear about what I want. Many times I noticed that asking myself to explicitly declare in patterns what I was doing required me to think deeper. Though I know what I am doing, when trying to express that intent using patterns from the books I am forced to clarify my intent to decide which exact pattern I am applying.  This is fully in the spirit of the Pragmatic Programmer‘s “Programming by Coincidence“, and also in the spirit of Test Driven Development where the unit test is forcing you to clarify what you really want.

In other words, we could call that Pattern Driven Development. PDD, yet another acronym!

Yet another strangely typed object (the disco vaccum cleaner)
Yet another strangely typed object (the disco vaccum cleaner)

Discussion

It may not sound intuitive that there exists a documented pattern for each and every possible design decision, but I can confirm that the panel of existing patterns is very wide and already covers a lot of common design decisions. Such patterns are far from sophisticated solutions or tricks, they usually just name intents and known practices. But this is exactly what we need. Dirk Riehle, in a recent paper entitled Design Pattern Density Defined, found that in the case of open-source frameworks, and considering only classic design patterns, the density of such patterns was between 40 and 70%, where density is defined by “The design pattern density of an object-oriented framework is the percentage of its collaborations that are design pattern instances.

In addition, it becomes increasingly common for software teams to use custom patterns as a convenient way of documenting their common design practices. Actually this is how the pattern story began when Kent Beck and Ralph Johnson decided to document the design of HotDraw in the now classic paper: Patterns generate architectures. To illustrate that, the Drupal team is using custom patterns in addition to standard patterns to document their design in an efficient way, as described in the program of the DrupalCon Paris 2009 conference:

This session will cover the how and why of design patterns, and review the most common patterns used in Drupal as well as a number of common patterns we could be using. Some are Drupal-specific patterns and some more more general software design patterns.

Another major example of patterns dedicated for a particular project is the the Eclipse IDE. Erich Gamma, the main Eclipse designer -one of the Gang of Four- wrote the book Contributing to Eclipse: principles, patterns, and plug-ins (together with Kent Beck) where Eclipse-specific and standard patterns are used together as the primary mean of documenting the design.

Conclusion

Patterns are signs to denote intents, and I propose to promote the use of patterns (patterns already existing or to be defined in the context of a particular project) to trace these intents explicitly, and to enable tools to enforce pattern-specific constraints.

Pictures taken at la Biennale Internationale Design, Saint Etienne 2008

Read More

Toward smarter dependency constraints (patterns to the rescue)

Low coupling between objects is a key principle to help you win the battle against software entropy. Making sure your dependencies are under control matters. Several tools can enforce dependencies restrictions, such as JDepend. However in a real project with many classes, packages and modules, the real issue is how to decide and configure the allowed and forbidden dependencies. Per class? Per package? Per Module? Based on gut feeling? Is there a theory for that?

Of course, in a layered architecture, the layers specify the dependencies. This is not bad, but I am sure we can do better.

Smarter dependencies

To go further, I suggest expanding our vocabulary of concepts. In OO languages such as Java, everything is a class (or an interface), grouped into packages. Such classification is not really helpful. Fortunately, several books provide ready to use vocabularies in the form of patterns languages (not only design patterns, but patterns in general). Some of these patterns are foundations on which rules to manage dependencies can be proposed.

Disclaimer: the dependencies rules suggested below are hypothesises to be debated and verified against a corpus of actual projects, I would be happy to be given counter-examples and counter arguments.

The child really depend upon the mother
The child really depend upon the mother

Domain Driven Design

The book Domain Driven Design by Eric Evans defines a rich vocabulary of concepts used in every application, and we can leverage that vocabulary to propose some dependencies principles between them:

  • ValueObject never depends upon Entity nor Services
  • Entities should not depend upon Services (maybe not a hard rule)
  • Generic SubDomain should not depend upon Core Domain
  • Core Domain should not depend upon Cohesive Mechanism (the “What” should not depend upon the “How”)
  • Domain Layer should not depend on any infrastructure code
  • Abstract Core module never depends on its specialized or implementation modules

Analysis Patterns

The book Analysis Patterns by Martin Fowler also provides patterns as a richer vocabulary, from which we could propose:

  • Elements from a Knowledge Level should not depend upon elements from the corresponding Operation Level

I did not find that rule written in the book but every example appears to support it. Considering that classes and subclasses in usual OOP are a special case of Knowledge Level built-into the language, this would lead to:

  • Abstraction never depends upon their Implementations

which is similar to the second part of the Dependency inversion principle by Robert C. Martin:

Abstractions should not depend upon details. Details should depend upon abstractions.

Since many analysis patterns in the Analysis Patterns book involve the Knowledge Level pattern, this single dependency rule already applies to many analysis patterns: Party Type Generalizations, Measurement, Observation, Protocol, Associated Observation, Measurement Protocol etc. The pattern Quantity can be seen as a specialized ValueObject (see Domain Driven Design above) hence should also not depend on any Entity nor Service.

Design Patterns

The book Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma et al. presents the classic design patterns. These patterns define participants which are named.  In the pattern participant ignorance principle I discussed the concepts of ignorant Vs. dedicated participants within a pattern, and their consequences for dependencies:

  • Ignorant pattern participants should never depend on dedicated participants
  • Pattern participant never depend on the “Client” participant
  • For each ConcreteX participant, the corresponding abstract X never depends on it (Abstraction never depends upon their Implementations)

In practice, this means:

  • In the Adapter pattern, the Adaptee should not depend upon the Adapter, and the Target should depend upon nothing
  • In the Facade pattern, the sub systems should not depend upon the Facade
  • In the Iterator pattern, the Aggregate should not depend upon the Iterator; however every Java collection is a counter example as they contain their own ConcreteIterator.
  • In creational patterns (Abstract Factory, Prototype, Builder etc.), the Product and ConcreteProduct should not depend on the dedicated participant that does the allocation (the Factory etc.)
  • And so on for other patterns, some of which being already discussed in the pattern participant ignorance principle.

In short, if we look at the design patterns as a set of types with inheritance/implementation, invocation/delegation and creation relationships between them, the dependencies should not flow in the reverse direction of the relationships; in other words, using UML arrows, the dependencies should only be allowed in the direction of the arrows.

Addiction to sugar is a kind of dependency
Addiction to sugar is a kind of dependency

Patterns of Enterprise Architecture

In the book Patterns of Enterprise Application Architecture by Martin Fowler, the Separated Interface Pattern proposes a way to manage dependencies by defining the interface between packages in a separate package from its implementation. This is similar to the Dependency inversion principle, as discussed here, which states:

A. High-level modules should not depend upon low-level modules. Both should depend upon abstractions.

By the way this is also very similar to the recommendation in Domain Driven Design:

Abstract Core module never depends on its specialized or implementation modules.

Finally, in the spirit of UML stereotypes that we sometimes put on packages to express their intent:

  • Utils never depends on anything but other Utils

What for?

If we manage to make every use of the above pattern explicit in the source code, for instance using Java annotations or simply Javadoc tags, then it becomes possible for a tool to deduce dependencies constraints and automatically enforce them.

Imagine, just add @pattern ValueObject in your Javadoc comment, and voila! A tool is now able to deduce that if you happen to import anything but basic java.* you must be warned.

Of course the fine tuning of the default behavior can take some time (do we accept that ValueObjects may depend upon low level utils like StringUtils? Probably yes), but the result will at least be stable regardless of the refactorings.

Given the existing variety of patterns over there, I am confident that just any class or interface within a project can be declared as being a participant in at least one pattern, and have therefore its dependency constraints deduced at the same time.

Read More

After Technical Debt, technical Options

In finance, options are powerful tools for traders, and many design practices including design patterns can be seen as options. Options can -perhaps- yield a great benefit for a certain and immediate cost. If this cost is cheap enough it can be quite attractive.

An option to buy a stock is a right to buy the stock at a predefined price (let’s say EUR25) before a given date in the future. Until this date, you can freely exercise the option (i.e. buy the stock at EUR25), or not (do nothing). Of course if it happens that the stock price is greater than EUR25 (let’s say the stock is now EUR35), if you exercise the option you earn money (EUR10), and for that reason the option is worth something (EUR10 here, or even more if we expect the price to go even higher in the future).

In finance, the price of the option is directly linked to the expected benefit it can bring, even though this benefit is not certain. To be more precise, the more the stock price moves up and down, the bigger the option value: this instability of the price is called volatility, and you can think of it like a standard deviation in statistics. Options traders actually trade volatility: the higher the volatility (i.e. the less stable the prices), the more valuable the option.

Many design practices can be seen as options, and some are indeed cheap, almost free options. A free option is something that a reasonable investor cannot refuse to get, since it costs nothing while it can perhaps yields a benefit: this is actually an uncertain version of a “free lunch”. A design practice that would cost nothing is hard to resist, since it yields potential return in exchange for nothing.

In the spirit of the Technical Debt, we will focus on the cost of maintaining the code over time. Any change to the code must be weighted in accordance to the burden it will bring compared to its benefit.

Plenty of options here
Plenty of options here

Technical options in the code

In practice, everything that improves the flexibility of the software is de facto like an option: it creates additional opportunities that we can use or not in the future. Typically, it can enable the designer to change his mind at a later stage at no cost, or to defer a design decision.

The design principle to code against abstractions rather than implementation comes immediately to mind: coding against interfaces is probably the number one technical option. Therefore the question is: given this piece of concrete code, should I introduce an interface and reference the code through the interface, or should I leave it like that? Just like in finance, the answer is: it depends on the upfront cost of the option (adding the extra interface) compared to the anticipated stability of the feature the code is doing (is it likely to change in the future?).

In this case, the upfront cost depends on the tools (with modern IDEs with wizards and refactoring capabilities it is easier to add or even extract an interface automatically), and perhaps on what you like to do or not (if you hate having too many classes then adding an interface will cost you more, subjectively). Some commercial source control systems that are not convenient with widespread refactoring also make the cost of future changes much higher, and as such they encourage building flexibility upfront.

Going further, many design patterns can be considered as options to improve flexibility for a minimal cost. Out of the GoF patterns, the patterns Abstract Factory, Strategy, State, Visitor, Iterator, Bridge and Builder directly plan for easy change of behaviour or extension in the future (typically through additional interfaces as well), and thus are simple yet efficient ways to buy technical options.

Various elements combine together for bigger value added
Various elements combine together for bigger value added

Technical options Vs. YAGNI

Knowing the future is hard, and YAGNI is there to remind us that. If there are practices that cost a little to introduce and that can bring benefits later, and if we are so poor at anticipating change, why not just introduce them later? This is right, in most cases YAGNI applies, unless we can anticipate a need. The funny thing is that while the introduction of interfaces (directly or through the patterns mentioned before) creates opportunities to vary the implementations, it also creates opportunities for other opportunities. The new interface enables the addition of non-intrusive structural patterns that revolve around an existing interface, such as Decorator, Composite, Proxy, Adapter, Null Object, and every combination of them as suggested by the Interpreter pattern. (This introduction of such patterns, by the way, is precisely what most IoC containers, application servers and AOP frameworks do to add functionalities). How these patterns combine together under the common “umbrella” interface is kind of fascinating to me, and it always reminds me about the Network Effect: the more we use an interface, with implementations or opportunistic patterns, the more useful it becomes. I believe that a net of patterns like that can become so useful that it becomes irresistible. You will gonna need it.

Of course there is nothing really new in all that, nothing more than a new metaphor to help explain to managers. Even if they do not know what an option is, I am sure they can understand the concept very quickly.

I have been pushing this approach that I called “Think options, not solutions” in my former team to emphasize that we can build the system even if some of its business behaviours are not completely known yet, as long as we can define their contours and start with an arbitrary solution. The goal is that we are confident we can change it later with no pain. This was justified from a business point of view because we did not know what the market was expecting, for the product (an interest rate swaps multilateral trading facility) was too novel. In that environment, change was definitely not something we feared, rather the opposite: it was stimulating.

Just like traders speculate on volatility using options, developers can speculate on the stability of the business features using (or not) design practices that increase flexibility. The little extra effort to introduce these practices must be weighted by the expectation that it will be useful. And just for the fun, anyone interested in computing the value of a technical option using the Black-Scholes pricing model?

Read More

Akai MPC program editor in Java

I juste wanted to automate the creation of programs for my Akai MPC500 sampler/groove machine, for my personal needs, and I ended up releasing a piece of software to sourceforge. It is called MPC Maid, read “assistant of the MPC”, here is a screenshot of the program editor:

MPC Maid program editor view
MPC Maid program editor view

The MPC file format was freely available on the website of Stephen Norum, very well documented. However my day job is to design and code big financial server-side systems, hence coding at the byte-level was refreshing…

Chop loop into slices

I had been playing around with audio/Music Information Retrieval (MIR) recently for fun, so I took the opportunity to integrate some of that, in the form of a really simple loop slicer.

Audio Slice Editor view
Audio Slice Editor view

You just drag and drop a WAV file from the file explorer into its waveform editor, then it analyzes the audio samples to detects the beats. There is a slider to adjust the sensitivity of the detection, which means this is not a totally automatic algorithm.

The idea for the detection comes from the article “Beat Detection Algorithms” from Frédéric Patin, with some personal modifications. In particular I have added a toggle to prevent a detection immediately after another, and a zero-cross detection to cut the slices on a zero-crossing, in order to reduce the glitches (but of course this is not enough to be perfect, fortunately the MPC has a convenient small decay at the very end of every sample, which solves the problem).

The idea is to compute the energy in sliding time-domain windows,then compare the energy of each window against the average in the surrounding; when it exceeds it enough, there is a beat.

Since many loops available are commercial quality, their length gives the tempo with an extreme accuracy. I used that to generate a MIDI file of the groove of the loop. Coupled with the program edition this allows for exporting the slices, the MPC program that uses them, and the MIDI sequence of their placements, in other words this is roughly equivalent to the REX file idea.

Native-looking but fully portable

Making sure that the software is fully portable and looks native in Mac OS X and in Windows requires careful attention, especially on the Mac side, where the menu is on top of the screen, not the window, the shortcuts are different, and even worst, the menu do not follow the same conventions! For instance the “About” menu is in the Application special menu, not in the Help menu as on Windows. To achieve that in Java requires the Apple OSXAdapter to call by reflection an API that will not be available on non Mac platforms!

Again I hardly do any user interface usually at work, so this little project was overal a good experience to me, and now the users have begun to play with it it is soon to become even more interesting!

Update: here is a video of how to chop a loop almost automatically into slices, then how to play it back at a different tempo on the MPC:

Read More

Essential patterns books for today object-oriented developers

The more patterns developers know, the most efficient they become within a team: it only takes one or two words (the pattern name) to communicate a design decision or proposal, instead of 10 mn of explanations. Communication also gets much more accurate and to-the-point (or less fuzzy).

Because patterns often form a pattern language, not only they offer a standard vocabulary but they also help structuring the mind by their relationships: patterns can be related as exclusive alternatives, or rather often-go-together, which is useful.

Patterns are not always supposed to be “tricks” to learn, or extra complication to introduce to a design; indeed even obvious and uninteresting pattern are worth reading in books, just for the social advantage of being able to reference them. Going further, sometime you can just go to the book to find out which is the pattern that you just did without knowing its name…

Design Patterns: Elements of Reusable Object-Oriented Software (Addison-Wesley Professional Computing Series)
1.  Design Patterns: Elements of Reusable Object-Oriented Software (Addison-Wesley Professional Computing Series) by Erich Gamma et al.
The most essential book, a definitive must-have. Every experienced software developer must know these 23 design patterns.

Analysis Patterns: Reusable Object Models (Addison-Wesley Object Technology Series)
2.  Analysis Patterns: Reusable Object Models (Addison-Wesley Object Technology Series) by Martin Fowler
The essential complement to the GoF, to extend the benefits of patterns from design to analysis (closer to the problem to solve in the domain). Each pattern will not necessarily teach you a new solution, but will always give you have a standard name for each solution, which is already worth the book.

Domain-Driven Design: Tackling Complexity in the Heart of Software
3.  Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans
A book that can change the way you deal with software. Considering the domain as the main driver of the design is so powerful an approach! Works best together with the analysis patterns.

Patterns of Enterprise Application Architecture (Addison-Wesley Signature Series)
4.  Patterns of Enterprise Application Architecture (Addison-Wesley Signature Series) by Martin Fowler
Other analysis patterns to solve many technical problems, useful complement to Analysis Patterns.

Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series)
5.  Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley Signature Series) by Gregor Hohpe
Very complete pattern language (both test and visual vocabulary) about messaging, will improve the communication within a team a lot, both in efficiency and in accuracy.

Pattern-Oriented Software Architecture Volume 2: Patterns for Concurrent and Networked Objects
6.  Pattern-Oriented Software Architecture Volume 2: Patterns for Concurrent and Networked Objects by Douglas Schmidt
Important book, more specialized toward optimized low-level problems (web servers…). often quoted. The other volumes are less relevant for today software development on modern languages or virtual machines.

Patterns in Java: A Catalog of Reusable Design Patterns Illustrated with UML, 2nd Edition, Volume 1
7.  Patterns in Java: A Catalog of Reusable Design Patterns Illustrated with UML, 2nd Edition, Volume 1 by Mark Grand
The book I discovered design patterns in, quite affordable, but some colleagues do not like it that much.

Read More

Software development is more than mastering syntax

When a junior developer joins our team, it is interesting to realize how mastering language syntax and API is just a small part of the skills that matter.

Just after the syntax and API knowledge (actually knowing where to find what you want in the API is enough), there are a few other skills you definitely must know.

First is unit test. It is not difficult to begin with that, it is not that easy to tune tests so that they dont depend on too many things, to save time when refactoring. Also mocking objects is the next step.

Mastering dependencies is another key to successful software development. Dependency Injection is a buzzword for a very simple idea, that everyone can apply only by being aware to do the ”new” where is it most convenient to change, using valued constructors or setters.

Design patterns are of course an essential tool to build flexible software. Everytime there is a design problem, design and analysis patterns often provide a good solution. They also guide you to good object-oriented thinking. However, when there is no design problem, there is no need for patterns.

The last very important thing to understand in software development is to look for simplicity, always. It is easy to build a complicated solution, it is difficult to build a simple one. But a simple solution has so many advantages… To simplify a design, domain analysis is a great tool; the more you understand the domain, the best you can align your design to it, and the simplest it will be. By separating orthogonal things or by unifying similar things you can also simplify.

There are also things that we must unlearn, like doing design first and testing at the very end of a project; it is most of the time also useless to create extensive UML diagrams, good design can hardly come from diagrams, if you are not close enough to the code you may miss the point and have a flawed design, even though it looks great as a diagram.

It is counter-intuitive to beginners to focus on anything but syntax. Therefore, unless you have a colleague to introduce you to other concerns, or unless you are really motivated to look for advises on the Internet, you can live a long time without even knowing there is more than that…

Initially published on Jroller on Tuesday June 28, 2005

Read More

Software considered biological

As software become more complex it increasingly gains biological features, and we increasingly need to rethink the way we consider it.

Complexity is the logical consequence of our wish to always delegate more work to computers. Historically computers did payroll, simply computing the amount of money to give to each employee given its rate and working time. It was easy to tell the machine to get each value, then multiply and print the result.

Now we have networks of software agents that interact in ways we can hardly represent, even inside a piece of software we have mecanisms like auto retry on failure that ultimately run their own life. In financial pricing systems the computations may involve so many calculation conventions mixed together (day counts, interest rate fixing, calendar folk…) that we can hardly tell what particular formula is used for a given instrument, since the overal formula is highly flexible. Sure each simple part was told step by step what to do, but as we consider the entire system, it nearly becomes a new kind of living creature !

If you just put many pieces that work together in the same box, then you get complexity since our limited brain cannot see, nor think, about each piece at the same time. So we say the system is complex, which also means: “I cannot understand it completely at a time”.

Like biological entities, softwares are increasingly made of many simple parts, similar to cells, that together create a useful and complex system with a purpose. Moreover the overal system can adapt to its changing environment (I’m not talking about mutation, but just about the way insects can cope with many various situations, that is similar to how modern software can cope with a wide variety of data, data patterns, incidents…).

Another essential quality that software increasingly share with biological creatures, e-g ants or bees, is how an overal ability, or intelligence, is emerging from a group of simple, dumb parts interacting together. Simple cells together become an insect that can move, eat, follow its mates. Simple insects together become a collective creature that can hunt bigger animals, survive floods or other insects attacks. Similarly, simple objects group together to compose a complex application.

In complex animals like humans, in our brains simple neurons together become brain functions that are capable of recognising faces, symbols, sounds. Simple brain functions together become our mind that is capable of reasoning, communicate with symbols… and have emotions ! Such comparision is not incidental, searches have proven that human mind is not One mind, but actually a group of simpler mind parts working together in sync, this explains why we may change our mind depending on which part is talking louder than the others, by the way !

Comparing all that to most complex software does make sense. After all, building complex software is indeed an attempt to re-create artifical brains to work on our behalf. And to build software we put together simple parts into modules that we put together into more complex modules, and so forth until it can solve the business problem it is expected to solve. Will this help me in my everyday work ? Sure not yet.

Initially published on Jroller on Tuesday May 17, 2005

Read More

Design Patterns cannot be reused through code reuse

It’s a complete mistake to think design patterns could be written in a library for later reuse, it just does not work !

In theory, first, design patterns are expected to be tailored to each particular situation. What you reuse is the abstract solution, not the code itself.

And it is true, whatever the design pattern you may consider, it is just not possible to implement it once and then reuse the code.

When I first thought about this topic I remember I immediatly thought of Observer and Iterator patterns as possible objections since they are both defined as interfaces and classes in the java API itself. Are they counter-examples of what is stated before ?

I don’t think so. I tried to use the Observer/Observables classes built-in in the Java API several times, but every time they were not suited for my needs so I had to define my very own for the needs of my application. This is a good example of how a pattern can be expressed in code, for one particular case, but cannot be really reused accros other cases. The Iterator pattern proves that again, as there is not one but already two interfaces in the Java API for this simple thing. And if you take a look at the commons collections you will find many other variations on this pattern, such as OrderedIterator and ResettableIterator.

So everytime I see a framework trying to freeze patterns in code, which is what a framework does in essence, it becomes obvious that this reduces reusability a lot (to be honest I am now rather defiant about frameworks).

Initially published on Jroller on Thursday May 12, 2005

Read More

My definition of abstraction

One thing human are particularly pride of is our ability to use abstraction to solve problems. We in software development all know that “another level of abstraction can solve my problem”. But what do you call abstraction ?

Abstract is the opposite of concrete. Concrete things are simply things, the kind of things a dog can see, bite and eat: me, the bone on the floor, the bee it’s trying to catch on the fly… In my job, concrete things are the formula to compute the price of a fixed income bond, the interest rate risk in a portfolio or the report the manager wants the tool to generate.

When I first work on a thing, this thing is concrete to me. The first bond I am computing the price for is just a simple concrete bond. But the second, different bond is a revelation: while it is different to the first bond, it also has lots in common ! wow ! I can notice it, and especially I’d like to save work from this, as I am a lazy developer. So I gonna use abstraction…

Concretely, I would define abstraction as the mental action of finding the commonalities and the differences between two or more things. The common part of every thing is what I will call the abstract thing, and I will then call each thing with its differences a concrete thing.

So my (very practical) definition of abstraction has to be explicit to be of value to me, so that I can reuse the common part of things. The most obvious example of abstraction is the very notion of class against instances of a class, aka objects: the class represents the common part (the structure) of a group of objects, while each object has its own values that make it different to its mates. What is common between every bond I will be given is that they all share a group of parameters and calculation conventions, thus I will create my pricing software for this abstract bond in order to be able to price every bond by just describing its differences against the common bond.

Young children early develop their abstract skills by playing with things; They soon see the difference between one thing and two things as the notion of numbers, the difference between a red shape and a green same shape as the notion of colour.

As said before, abstraction is very much about classification: given the red square and the green square, we can see they both have a square shape, so we can classify them in the shape = square category. Given another green circle, we can also classify the green square and the green circle in the colour = green category. This way, we have identified two abstract notions, shape and colour, from very concrete things.

As I am -more and more- a lazy developer, my next question is: can my computer abstract things for me ? After all abstraction helps in handling what’s common in things in order to handle this common part only once, and I have identified the very notion of abstraction itself as an abstract notion, maybe in order to reuse it ? (by the way what I am doing at this point in this writing also follows the abstract notion of recursion, talk about abstraction, then talk about the abstraction of abstraction). Some clues: in common spreadsheets, when we enter a number sequence, say 1, 2, 3, the spreadsheet recognizes the arithmetic sequence so that it can guess its next values. I’m tempted to claim it is an abstraction ability already… Not sure in fact. Obviously, it is in AI field (artificial intelligence) that neuronal networks have natural classification abilities, so the answer is yes, software can abstract concepts.

Initially published on Jroller on Monday April 11, 2005

Read More