Tuesday, August 25, 2009

Delivering Business Value Through Polyglot Systems (part 4 / conclusion)

In my previous three posts, I described my experience with maintenance, refactoring, and installation on a large project that used both Java and Groovy. In this post, I will discuss the "support nightmare" raised by Bill Burke in his blog post "Polyglot programming is the worst idea I ever heard". As I illustrate below, adding moving parts of any kind can make support difficult. But this shouldn't scare us off from adding another language - most of us work with multiple languages like Javascript, CSS, SQL and XML already.

Mr. Burke describes the "support nightmare" as follows:
A support call comes in for your product or application. It's obviously a bug, but where is the problem? Is it your application code? Your JVM? Your Ruby VM? Your Java library? Your Ruby gem?
To paraphrase, adding a language will make it more difficult to track down bugs when support calls are made. Of course, with the decision to add any framework, runtime or library, we add complexity to our applications. In my experience, I think adding another language is no different. Even if I have an all Java stack, that doesn't make it right to add any Java library that I want to the application environment. Too many times, I have seen frameworks injected into applications to solve problems that could have been coded by hand in less than fifty lines of Java. In situations like this, the conceptual complexity of learning a new framework doesn't make sense. The critical aspect about the decision to add any toolset to an application is that it must be evaluated relative to its benefit.

In reality, we already deal with multiple runtimes when we resolve support calls. If a value doesn't display correctly in a typical Java web application, the problem could be in any number of places. It might be a problem with HTML, a JSP tag library, a JavaScript function, a JavaScript library, server side controllers, business logic embedded in a service, a library utilized by these services, messaging to other systems, the ORM library or the database itself. This is why feel strongly that the decision to add anything to a system must be evaluated relative to its benefit. The expertise we utilize when evaluating a library can and should be applied when adding a language as well.

This final point about the "support nightmare" provides a nice segue to some concluding thoughts about building polyglot systems. The fact is we already use multiple runtimes and multiple languages. Even simple two-tier desktop systems will typically use SQL (the language), a SQL database runtime and whatever language the application is being built in. The reasons we already do it are so obvious, we rarely debate them. I remember the excitement I had when I was able to embed SQL in my C code instead of using Btrieve, an indexed record manager. Sure, the SQL based RDMS was more complex to install, hid more details under the covers, required a preprocessor for the C code and involved effort to setup and maintain. But the benefits easily out-weighed the effort for the applications I was building. In the same way, adding another language like Groovy or Python may require some effort, but the benefits in terms of a smaller, more expressible code base that directly reflects the problem domain can easily be worth it for the long run. Ultimately, we deliver greater business value to our customers with systems that are easier to comprehend, support, enhance and maintain. I've found that using the right language for the right job can help deliver such systems.

Monday, August 17, 2009

Delivering Business Value Through Polyglot Systems (part 3)

In my previous two posts, I described my experience with maintenance and refactoring on a large project that used both Java and Groovy. In this post, I will discuss the installation concerns raised by Bill Burke in his blog post "Polyglot programming is the worst idea I ever heard".

In describing the "installation nightmare", Burke states:
But you still need to package and install all the appropriate Java libraries, Ruby gems, and Python libraries on the appropriate places of your user’s machine. You also need to hope that all the desparate environments don’t conflict with anything the user has already installed on his machine. And that’s just with languages running in the JVM.
These are valid concerns, but I don't think they are exclusive to polyglot programming.

The problems of developing code that "works on my machine" but doesn't work anywhere else are well documented. In my opinion, the first place to start with any project is a one-step build that provides for automated testing plus some form of continuous integration. This isn't a particularly original idea these days. In our project, we used Ant to implement a one-step build using a clean checkout from version control. It took effort to get this working, but the effort involved would have been required whether Groovy was involved or not. In my experience, migrating from desktop Windows to Linux and then to AIX became a non-event once I had a one-step build in place.

In response to my previous blog posts on this topic, it has been pointed out that Groovy might be a special case. This is true to some degree in that Groovy was designed from the beginning for easy integration with Java. However, I think the concern about conflicts among libraries on different platforms is valid with or without Groovy. I've experienced first hand the unexpected differences in string truncation among IBM's DB2 drivers running on different platforms. On Java projects, there is always a concern about a conflict with Antlr or ASM when using Hibernate and some other library that uses byte code generation. Adding Groovy is no different than adding any other library or framework to a project. There are always risks. Fortunately, one step builds and continuously integrating on all of your project platforms are tried and true strategies for mitigating this risk.

Finally, my thoughts with regards to non-JVM languages are different than those expressed by Mr. Burke:
Imagine if you used vanilla Ruby, Python and Java all as separate silos!

I would encourage you to take your imagination out of the decision making process and base it instead on facts by testing things for yourself. For example, with Python, you might find that not only is it already installed on your test and production platforms, but your required libraries might be there as well. It also might be that installing the required libraries is trivial. Of course, it is possible that it is not trivial. The important point is to try it out and then evaluate the cost/benefit to your particular project. If the benefit is replacing 10,000 lines of the most complicated part of your project with 2,000 lines of code that is more readable and easier to "reason about", then even the extra effort of engaging your infrastructure team to install a required Python library might be worth. Remember, the costs don't exist in isolation but in relation to the benefit of adding another of language. The importance of this cannot be overstated.

At the end of the day, the business doesn't exist for IT's benefit and comfort. It is our job as developers to deliver systems that represent a good value to those making the investment. Where appropriate, we shouldn't be afraid to utilize more than one language if it adds value to the project and take advantage of techniques like one-step builds and continuous integration to mitigate the risks.

In my next post, I'll address the "support nightmare".

Monday, August 10, 2009

Delivering Business Value Through Polyglot Systems (part 2)

In part 1, I described how my experiences using both Groovy and Java delivered business value to the customer by letting the project team work with code that closely reflected the domain. We found the new system didn't result in a "maintenance nightmare", but rather made maintenance much easier than if only Java had been used.

In this post, I'll address the "refactoring nightmare" as described by Bill Burke in "Polyglotism is the worst idea I ever heard". In that post, he states:
So now your developers can use any one of these language to build out extensions to your Java app. But what happens when you want to refactor one of your re-used Java libraries? OOPS!!!

I've seen these type of statements in blogs dating back to 2004. Let's take the hypothetical one step further: What happens when I use Java to build out an extension to my Java app and the library doesn't live in the same project or project workspace as my extension? OOPS!!! There are all kinds of hypothetical scenarios that we can imagine to scare ourselves into not using this or that technology. It is more insightful to see how actual project experience can substantiate or remove the fear, uncertainty and doubt.

I do think it is reasonable to expect refactoring could be different with a dynamic language in play. The lack of tooling was one of the major concerns introducing Groovy into our project. Personally, I was used to coding in Eclipse and was very comfortable with the tools it provides. I couldn't imagine not working in Eclipse so I expected the worst. However, reality was different from expectations in ways I never anticpated.

When the project started out, I made sure all of my Groovy code always explicitly declared types. I also created interfaces following Java best practices. As a programmer who had been using Java since 1998, I was willing to accept Groovy but I was going to make sure our team followed "good" programming practices. I didn't want our team to get too carried away with this dynamic stuff. I was quite pleased as the project proceeded quite well for almost nine months without any major refactoring required. Then, the inevitable unforeseen change in requirements appeared that was going to rip its way through most of our existing code. At the time, I only had JEdit, Ant and JUnit at my disposal. This change resulted in my spending a long, exhausting weekend refactoring the code. Luckily, our team had almost 100% code coverage between our unit tests and integration tests. Without those tests, I'm sure it would have taken weeks to make sure the new changes were correct.

The insight gained from this long refactoring weekend was completely unexpected. If I had not coded to interfaces and had not included types on all the method signatures, the coding changes would have been minimal. Our team did a really good job of writing lots of small methods that operated on a few properties at a time. It became obvious that the successful execution of these methods didn't depend on the objects containing the properties.

Consider the following example: A method might look up a rate adjustment in an insurance rate table based on the gender and age of the insured. The only things that mattered were the the gender, the age and the lookup table. It didn't really matter if age and gender were part of a Person object, an InsuredPerson object or a hash map. With Groovy and duck-typing, as long the object passed into the method had the properties "age" and "gender", the method could do its job.

I kept my eye on this going forward and realized that as objects with properties flowed through the system, it didn't really matter what the objects were. As long as the objects passed into the business calculations had the expected properties, we got the behavior we expected from the system. Thus, my experience for our system over the next two years was that we had an easy time of refactoring when we avoided specifying types in our system. Never again did I waste a whole weekend trying to refactor our system to accommodate a change in our object model.

I have given presentations about this project many times during the last two and half years and this concept is always the hardest thing for the audience to comprehend. I don't think its something that a person can craft an argument for and expect to have everyone nod in an agreement. There are some things that just have be experienced to be understood. I am certainly willing to grant that I may be the lucky recipient of the perfect problem domain, programming languages, and programming team for this to happen. However, I have had similar experiences doing other, smaller projects in Groovy and Java.

In summary, on a large polyglot project using Groovy and Java, I didn't experience a "refactoring nightmare". I found that within a given coarse-grained component, if I (a) removed all types, (b) only used interfaces sparingly, and (c) had a dependable test suite, the time spent refactoring became a non-issue for my project. How does this relate to delivering business value? Time spent refactoring is time spent focusing on technical issues, not business issues. In our regular meetings with our business experts and project manager, the team stayed focused on working through the business logic and not getting sidetracked on technical issues that don't deliver business value.

In the next segment, I will focus on my experiences as they relate to the "installation nightmare".

Wednesday, August 5, 2009

Delivering Business Value Through Polyglot Systems (part 1)

My answer to Peter Ledbrooks's blog post "Are polyglot systems a good idea?" is "Yes!".

He wrote his post after reading "Polyglotism is the worst idea I ever heard" by Bill Burke. I don't know Bill Burke personally but the post has a troll-like feel. However, beyond the rhetoric there are concerns that I see raised over and over again. These concerns are nothing new. Almost three years ago, they were the inspiration for developing my presentation "Real World Groovy". In that presentation, I raise concerns about introducing Groovy into a Java project and I address each concern by relating my own experience having worked with Java and Groovy on paid projects.

Burke describes four "nightmares" in his post. These nightmares are based on the assumption that introducing another programming language into a project
...is just a big excuse so the developer can learn and play with a new language, or for a language zealot/missionary to figure out a way to weasel in his pet language into a company. Plus, you’d probably end up being average or good at many languages but a master of none.
Personally, I never had trouble sleeping when using Groovy and Java on the same project. In this series of blog posts, I'll examine these nightmares one at time.

The "Maintenance Nightmare" concerns staffing and training. He describes the hypothetical situation that after the polyglot programmer has left the team,
...your group is a bunch of Java developers.For any bug that needs to be fixed, these developers need to be retrained in Ruby, a new Ruby developer needs to be hired, and/or a Ruby consultant/contractor needs to be brought it. Multiply this by each language you’ve introduced to your project.
Hypothetically, this group of Java developers won't be able to read and update the code written in another language without training.

My experience
is the exact opposite. The code written in Groovy was readable by the other developers on my team. No formal training was provided. The developers on my team were good, but they certainly weren't language zealots. One had about 8 years of DB2 & Cobol with less than one year of Java experience. Another developer had 10+ years in VB and various SQL databases, as well as some Java. The subject matter experts were insurance actuaries with programming experience in PL-SQL and Foxpro. On our project, all of these people were able to write and maintain the Groovy code without formal training.

How is this possible? Groovy wasn't added to the project because of some technical itch that needed scratching. The process of choosing an additional language involved taking a non-trivial piece of functionality and coding it in several languages, including Java. Each version was reviewed by those who would be developing and maintaining the new system. At that point in time, no one on the project had any experience with Groovy. If anything, there was prejudice against Groovy because of the uncertainty. Groovy was selected because it allowed us to easily express the algorithms and business logic in a way that reflected the business domain. The Groovy code was both concise and readable, with a very high signal-to-noise ratio.

Circling back to the maintenance concern, my experience has been that Groovy hasn't made maintenance harder. In fact, it has made maintenance easier. Even with significant time passing between authoring and maintenance, the Groovy code has been easy to comprehend because it reflects the business domain. I try to write my Java code this way as well. However, Groovy doesn't have a lot of ceremony around it, so it is accessible for developers with experience in other languages. My team did not struggle to decipher what was going on in a given piece of code because it was written in Groovy.

It is my experience that bugs can be resolved in minutes or hours that would have otherwise taken hours and days. The problems were solved quickly because those with domain expertise - my client's developers and subject matter experts - could see at a glance what was wrong in the code. Note that none of this required them to be masters of Groovy. I have heard similar stories from others who had seen performance improvements rewriting C modules in Lisp or Smalltalk. The performance gains came because it was easier to implement logical efficiencies that just weren't practical in C.

Based on what I've seen over the last 20+ years, technology issues are not what doom projects. I can't tell you how many times I've heard developers say, "If the client could just decide what they want, I could build it. I just need to know what to build." The hard part is getting systems to do what our client's intend them to do. In order to bring a system into production, the team must be able to reason easily and quickly through the code to fix business logic errors.

In summary, the readability and expressiveness of languages like Groovy make maintenance of polyglot systems easier. There is less code to write and maintain and it is more readable. The benefit to less code isn't in less effort typing at the keyboard. Rather, it is the ability to get one's head around the various parts of the system. In addition, this enables a broader group of people with business expertise to be able to understand what the program is doing. All of this helps avoid creating a big ball of mud. Keeping the code easy to read and understand is the key to avoiding the "maintenance nightmare".

In the next post, I'll address the "Refactoring Nightmare".