.::output >> /dev/null::.

About 9 months ago, I said that I will be looking into collecting different patterns for screencasting. So finally....

I just finished writing a pattern language for screencasting based on my experiences producing and observing various screencasts on the internet. Through my experiences I have noticed various patterns that myself and other screencasters have used. Applying those patterns would certainly help the presentation of your screencast but, more importantly, I believe that the patterns enhance the teaching-learning experience of screencasts. And ultimately that is what matters most.

Interested readers may view the patterns here. The pictures that I have included are based on software for the Mac since that is what I use for my screencasting.

At the moment, it's a rather complete version of what I had originally envisioned for such a document. It can definitely be improved but I would like to get a version out on the web for people to read in case anyone is interested in providing feedback or suggestions.

I do maintain copyright on the document in case I ever decide to publish it at some pattern conference.

Refactoring Dynamic Code:

"These tools, along with the relatively small amount of Ruby code required to implement the desired features, made life easy during the first few iterations. With each new feature, however, the amount of refactoring required to keep the codebase clean increased. Although Rails' adherence to the DRY principle helped reduce duplication, each refactoring had to change numerous code artifacts. Some of the changes, admittedly, were needed to keep the code in line with Rails' convention-over-configuration philosophy.
Renaming an entity class, for instance, requires renaming the corresponding database tables, including other columns and indexes referencing the model's table, renaming the model class, changing the corresponding controller, helper classes, and every reference to the entity class in the views, as well as renaming the directory containing the view classes and templates. And, of course, you have to similarly rename the associated tests and all references to the model, controller, and views in the tests and fixtures."

One of the advantages of being under Prof. Ralph Johnson is that we are indoctrinated to think of programming as a series of program transformation. From the humble HelloWorld program (just think of when you are reediting the program to add some comments or change what gets printed out to the console) to more complex programs out there, almost everything can be viewed as a series of transformations.

Refactoring is as a form of program transformation. Refactoring is a process where you restructure your code without affecting its observable behavior -- where behavior is defined pretty loosely. Why would anyone want to do something like this? If you have never done refactoring before this might seem like a worthless task. After all, if you aren't modifying the behavior of the program, you aren't adding new features to it. And features are what the customer cares about, right?¹

Well, turns out that you refactor to make your code easier to understand. Code that is easier to understand is also easier to modify. So essentially, refactoring helps prepare your program for the next feature, and the next feature, and the next feature... that you want to add; making those features easier to add.

Certain refactorings can now be automated through an intelligent IDE. And IDEs have generally done a very good job of it -- until you being to cross the traditional language barrier. For instance, look at Rails. Rails is a framework that relies on Ruby, erb, Javascript, SQL, etc. When you perform a refactoring on your Ruby code does that refactoring percolated correctly through the entire program? If you rename a method in your Ruby code, will it change the name of the method to the new one everywhere that it was called before? Would it also be able to change the method call inside the erb file?

Furthermore, because Rails relies on the convention-over-configuration idea, it is desirable that your refactoring preserves those conventions as well. So now, your refactoring tool not only needs to know Ruby but it must be configured to understand the Rails framework itself! Renaming your Rails model should also make the appropriate changes to the controller and your erb file just because the framework expects them to be related by name in a predetermined manner. Failing to do so would actually cause your program to fail.

If another framework comes along, your tool has to be taught to function for that framework too. And a programmer might choose to use numerous frameworks together. How would you ensure that the refactoring tool would work for any combination of frameworks?

The point of this post is to illustrate the need for multi-language refactoring support. We can think of complex frameworks as basically an extension of the language itself. Frameworks have their own set of conventions to be followed. Following those conventions is essential for a framework such as Rails. For other frameworks, following those conventions would be desirable to make the code easier to reason about.

So the main challenge is how to capture those rules that a framework specifies so that our refactoring tool can understand them and make the appropriate changes toward the entire code base.

MF Bliki: OneLanguage:

"For many developers, the one-language notion is a sign of lack of professionalism. This is best exemplified by the Pragmatic Programmers' advice to learn a new language every year. The point here is that programming languages do affect the way you think about programming, and learning new languages can do a lot to help you think about solving problems in different ways. (It's important to learn languages that are quite different in order to get the benefit of this. Java and C# are too similar to count.)"

How To Become A Hacker:

"LISP is worth learning for a different reason - the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot. (You can get some beginning experience with LISP fairly easily by writing and modifying editing modes for the Emacs text editor, or Script-Fu plugins for the GIMP.)
It's best, actually, to learn all five of Python, C/C++, Java, Perl, and LISP. Besides being the most important hacking languages, they represent very different approaches to programming, and each will educate you in valuable ways."

For simplicity, let's assume that there are two types of programmers out there: 1)those who are only extremely proficient in one language (most likely C++, Java or C#) and know everything about the language including the obscure corner cases of its implementation 2) those who know multiple programming languages and enough of each to be able to write applications with it with some form of reference by their side. Now, these are two extreme ends of the gamut and any programmer worth his or her pay will definitely be able to use multiple languages.

Imagine that you are trying to solve a programming problem. You go ask a developer from category 1). He replies that you should try to use language feature #x together with library #z and that you should be careful of some of the dependencies - make sure that you use version 2.1 even if it is in the unstable branch since previous versions cannot work with library #z. Now, you go ask a developer from category 2). He replies that he has seen this problem in one of the languages that he knows and that it is called the "some person's name problem". He explains what the gist of the problem is, some solutions that people have proposed and what he thinks is the best approach to the problem without suggesting a specific language or library (he gives a few that you could try but never ever just one).

Which developer would you prefer? If you go with developer 1), you get the exact language feature and library to use to solve your problem. You might of course run into problems later but for the most part you can get started immediately. If you go with developer 2), you are now able to understand the problem better, possibly google the "some person's name problem" and be able to form you own conclusions on the matter. With 1) you get an immediate solution but with 2) you get a solution that you can potentially apply again and again in different environments.

Of course, the choice of which developer to go with depends on your current situation. The gist of the example above, is that by knowing a few more languages, you are able to distill the actual problem (the "some person's name problem"). From my experience this comes from the fact that most languages are able to solve certain problems better. Furthermore, if you ever read the literature, you will discover that programming languages were invented to address certain inadequacies in other programming languages. Programming languages are turing-complete so you can solve all the solvable problems (problems that do not reduce to the halting problem) with them.

Consider the simple Quicksort algorithm. I remember learning about this in my introductory CS class. The explanation was easy to understand but when we had to actually implement it in Java, the beauty of the algorithm was lost amidst all the noise that the programming language had. Contrast this to the naive - it's naive because it uses list concatenation which is slow - Haskell implementation below:

Introduction - HaskellWiki:

qsort [] = []
qsort (x:xs) = qsort (filter (< x) xs) ++ [x] ++ qsort (filter (>= x) xs)"

While you do need to know a bit of Haskell syntax to comprehend the snippet above, once you do, it becomes almost ethereally clear and concise. The Haskell snippet expresses not only the implementation but also the algorithm itself clearly. The time that you invested in learning a different language like Haskell pays off because it allows you to think about the problem and how to solve it. In the Haskell implementation you solve it using list comprehension whereas in the Java/C version you manipulate data structures with indices or pointer arithmetic. Don't worry too much about the syntax since it is just a way to express the underlying idea.

I claim that the ability to think about the problem itself is an important skill. In the simplest case, it allows you to formulate a query to feed into google. You will be amazed at how many people do not know how to formulate a good query to feed into google.

But isn't it a waste to know so many languages but not be able to use them together? Sure, most languages are able to interface with C because C is still the language to use when you need performance (Fortran programmers will probably disagree since they live in their own world of super high performance computing where the laws of normal CS no longer apply to them). So while the bridge from C to "your favorite language" is pretty good, the bridge from "your favorite language" to "another language that you need" is not that good.

Things are beginning to change of course. I think that the advent of projects like the CLR and JLR will be able to make multilingual programming possible. There might be other implementations out there but admit it, the Java Virtual Machine and the .NET virtual machine are the two most popular ones right now.

As far as I know, right now the way to use different languages would be to create libraries in one language and then export it. You can then use that library in either Java or one of the .NET core languages. Even the JVM supports Javascript and Ruby, I don't think that you can use them together yet.

MF Bliki: OneLanguage:

"For many developers, the one-language notion is a sign of lack of professionalism. This is best exemplified by the Pragmatic Programmers' advice to learn a new language every year. The point here is that programming languages do affect the way you think about programming, and learning new languages can do a lot to help you think about solving problems in different ways. (It's important to learn languages that are quite different in order to get the benefit of this. Java and C# are too similar to count.)"

How To Become A Hacker:

"LISP is worth learning for a different reason - the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot. (You can get some beginning experience with LISP fairly easily by writing and modifying editing modes for the Emacs text editor, or Script-Fu plugins for the GIMP.)
It's best, actually, to learn all five of Python, C/C++, Java, Perl, and LISP. Besides being the most important hacking languages, they represent very different approaches to programming, and each will educate you in valuable ways."

For simplicity, let's assume that there are two types of programmers out there: 1)those who are only extremely proficient in one language (most likely C++, Java or C#) and know everything about the language including the obscure corner cases of its implementation 2) those who know multiple programming languages and enough of each to be able to write applications with it with some form of reference by their side. Now, these are two extreme ends of the gamut and any programmer worth his or her pay will definitely be able to use multiple languages.

Imagine that you are trying to solve a programming problem. You go ask a developer from category 1). He replies that you should try to use language feature #x together with library #z and that you should be careful of some of the dependencies - make sure that you use version 2.1 even if it is in the unstable branch since previous versions cannot work with library #z. Now, you go ask a developer from category 2). He replies that he has seen this problem in one of the languages that he knows and that it is called the "some person's name problem". He explains what the gist of the problem is, some solutions that people have proposed and what he thinks is the best approach to the problem without suggesting a specific language or library (he gives a few that you could try but never ever just one).

Which developer would you prefer? If you go with developer 1), you get the exact language feature and library to use to solve your problem. You might of course run into problems later but for the most part you can get started immediately. If you go with developer 2), you are now able to understand the problem better, possibly google the "some person's name problem" and be able to form you own conclusions on the matter. With 1) you get an immediate solution but with 2) you get a solution that you can potentially apply again and again in different environments.

Of course, the choice of which developer to go with depends on your current situation. The gist of the example above, is that by knowing a few more languages, you are able to distill the actual problem (the "some person's name problem"). From my experience this comes from the fact that most languages are able to solve certain problems better. Furthermore, if you ever read the literature, you will discover that programming languages were invented to address certain inadequacies in other programming languages. Programming languages are turing-complete so you can solve all the solvable problems (problems that do not reduce to the halting problem) with them.

Consider the simple Quicksort algorithm. I remember learning about this in my introductory CS class. The explanation was easy to understand but when we had to actually implement it in Java, the beauty of the algorithm was lost amidst all the noise that the programming language had. Contrast this to the naive - it's naive because it uses list concatenation which is slow - Haskell implementation below:

Introduction - HaskellWiki:

qsort [] = []
qsort (x:xs) = qsort (filter (< x) xs) ++ [x] ++ qsort (filter (>= x) xs)"

While you do need to know a bit of Haskell syntax to comprehend the snippet above, once you do, it becomes almost ethereally clear and concise. The Haskell snippet expresses not only the implementation but also the algorithm itself clearly. The time that you invested in learning a different language like Haskell pays off because it allows you to think about the problem and how to solve it. In the Haskell implementation you solve it using list comprehension whereas in the Java/C version you manipulate data structures with indices or pointer arithmetic. Don't worry too much about the syntax since it is just a way to express the underlying idea.

I claim that the ability to think about the problem itself is an important skill. In the simplest case, it allows you to formulate a query to feed into google. You will be amazed at how many people do not know how to formulate a good query to feed into google.

But isn't it a waste to know so many languages but not be able to use them together? Sure, most languages are able to interface with C because C is still the language to use when you need performance (Fortran programmers will probably disagree since they live in their own world of super high performance computing where the laws of normal CS no longer apply to them). So while the bridge from C to "your favorite language" is pretty good, the bridge from "your favorite language" to "another language that you need" is not that good.

Things are beginning to change of course. I think that the advent of projects like the CLR and JLR will be able to make multilingual programming possible. There might be other implementations out there but admit it, the Java Virtual Machine and the .NET virtual machine are the two most popular ones right now.

As far as I know, right now the way to use different languages would be to create libraries in one language and then export it. You can then use that library in either Java or one of the .NET core languages. Even the JVM supports Javascript and Ruby, I don't think that you can use them together yet.

The past week I had the opportunity to play around with Groovy. Groovy is an "agile dynamic language for the Java Platform". In short, you can think of Groovy as adding dynamic features to the Java language with some syntactic sugar to the Java syntax. Like JRuby and Jython, Groovy runs on the JVM and plays well with Java code.

Groovy offers functionality similar to that of Ruby, Python and other dynamic languages. Of course, Groovy has its own way of doing things but some of the most interesting things that I have seen are:

• Integrated support for XML processing directly via GPath (akin to XPath)
• Support for multimethods. The type of an object is determined at run-time. This leads to the interesting behavior that even if you write Object foo = "Hello", foo actually has the type String instead of Object.
• Option to declare the type for variables. Though not required, you have the option of declaring the static type of an object. The only other system that I have seen that offers this option is Strongtalk.

While Groovy is certainly an interesting and useful language (personally, any language that reduces the verbosity of Java is useful), what is more interesting is the way in which the dynamic features are done. Groovy relies on an internal implementation of the Metaobject Protocol. As far as I know, the term Metaobject Protocol was first made popular in the book The Art of the Metaobject Protocol. You can read more about this concept from any of the previous links but in a nutshell, designing a programming language so that it conforms to some Metaobject Protocol is a good thing. It allows the programmer (and not only the designer) to change the semantics of the language so that it can not only modify the current features of the language but also incorporate new features easily.

This might seem like an absurd thing to do if you are used to programming in more traditional languages like C which does not even offer the ability for reflection. However, the ability for actually modifying the programming language itself is becoming more useful. Modifying does not only mean adding some form of syntactic sugar on top of the existing language but also the ability to add new features to the programming language itself. For instance, the foreach control structure could be viewed as some form of syntactic sugar. foreach could be easily implemented as a macro that is expanded before execution into the normal for loop or while loop. On the other hand, adding object-oriented features on top of a "non-object-oriented" language would be viewed as adding a new feature (and not just syntactic sugar) to the language itself. This was what was done with CLOS in Lisp.

One could always argue that there is no need for such a feature to be an inherent part of the programming language. A valid argument against implementing something like the Metaobject Protocol is the issue of speed and performance. Having the infrastructure to modify the behavior of the programming language is going to have some form of overhead on the execution of the program. Furthermore, the ability to change the programming language would lead to problems with a proliferation of different versions of the same programming language with subtle differences. Moreover, one might also argue that if a feature is that important, why not just add it in the next version of the programming language. That is exactly the case with Java and C#; newer editions keep adding features to the language.

While those arguments are certainly valid, the ability to be able to modify the semantics of the programming language should not be discounted. As computers get faster, the speed overhead from having the Metaobject Protocol built-in to the language is negligible save for the most performance intensive machines. And how many people actually want to wait for the designers of the programming language to add some new feature? The desired feature might not be important for everyone and might not be added to the core of the language. Should that be a reason why it is not there if your application actually requires it? For instance, as the ability to express domain knowledge clearly in programming languages become more important, more and more forms of domain-specific languages are surfacing. These domain-specific languages provide a concise way to express essential concepts of the domain which might otherwise be hidden through the syntax and semantics of a traditional programming language. No sane programming language designer is going to design multiple domain languages and give them multiple names such as C-Telecommmunication, C-Accounting, C-Transactions¹.

While the idea of having a full-fledged Metaobject Protocol might be too idealistic for some, some of the newer ideas in the software engineering world are taking advantage of a more moderate form of it. The designers of programming languages can certainly vary the degree of dynamic behavior. For instance, Ruby does not have a fully modifiable underlying semantics but it does provide enough for programmers to accomplish most of what needs to be done.

In conclusion, I feel that more and more languages would incorporate some form of the Metaobject protocol. Dynamic languages are becoming increasingly prominent in the software engineering world today and the logical next step would be to expand on their dynamic behavior to improve on the extensibility². It should be possible for both programming language designers and programmers to modify and/or features to the programming language in a hygienic way without resorting to ugly hacks.

¹ An alternative would be to design your own language from scratch if it is simple enough. Tools like Antlr and even Maude have made this process less painful. Even then, designing a language from scratch -- albeit being fun -- requires a lot more devotion than modifying an existing language.

² Some features might also be extremely hard if not impossible to add to an exiting system whether or not it has an Metaobject Protocol. For instance, (I think) having the ability to do multi-stage programming might require extensive changes to the language itself and is not accomplishable with just some hacking on the language itself.

The past week I had the opportunity to play around with Groovy. Groovy is an "agile dynamic language for the Java Platform". In short, you can think of Groovy as adding dynamic features to the Java language with some syntactic sugar to the Java syntax. Like JRuby and Jython, Groovy runs on the JVM and plays well with Java code.

Groovy offers functionality similar to that of Ruby, Python and other dynamic languages. Of course, Groovy has its own way of doing things but some of the most interesting things that I have seen are:

• Integrated support for XML processing directly via GPath (akin to XPath)
• Support for multimethods. The type of an object is determined at run-time. This leads to the interesting behavior that even if you write Object foo = "Hello", foo actually has the type String instead of Object.
• Option to declare the type for variables. Though not required, you have the option of declaring the static type of an object. The only other system that I have seen that offers this option is Strongtalk.

While Groovy is certainly an interesting and useful language (personally, any language that reduces the verbosity of Java is useful), what is more interesting is the way in which the dynamic features are done. Groovy relies on an internal implementation of the Metaobject Protocol. As far as I know, the term Metaobject Protocol was first made popular in the book The Art of the Metaobject Protocol. You can read more about this concept from any of the previous links but in a nutshell, designing a programming language so that it conforms to some Metaobject Protocol is a good thing. It allows the programmer (and not only the designer) to change the semantics of the language so that it can not only modify the current features of the language but also incorporate new features easily.

This might seem like an absurd thing to do if you are used to programming in more traditional languages like C which does not even offer the ability for reflection. However, the ability for actually modifying the programming language itself is becoming more useful. Modifying does not only mean adding some form of syntactic sugar on top of the existing language but also the ability to add new features to the programming language itself. For instance, the foreach control structure could be viewed as some form of syntactic sugar. foreach could be easily implemented as a macro that is expanded before execution into the normal for loop or while loop. On the other hand, adding object-oriented features on top of a "non-object-oriented" language would be viewed as adding a new feature (and not just syntactic sugar) to the language itself. This was what was done with CLOS in Lisp.

One could always argue that there is no need for such a feature to be an inherent part of the programming language. A valid argument against implementing something like the Metaobject Protocol is the issue of speed and performance. Having the infrastructure to modify the behavior of the programming language is going to have some form of overhead on the execution of the program. Furthermore, the ability to change the programming language would lead to problems with a proliferation of different versions of the same programming language with subtle differences. Moreover, one might also argue that if a feature is that important, why not just add it in the next version of the programming language. That is exactly the case with Java and C#; newer editions keep adding features to the language.

While those arguments are certainly valid, the ability to be able to modify the semantics of the programming language should not be discounted. As computers get faster, the speed overhead from having the Metaobject Protocol built-in to the language is negligible save for the most performance intensive machines. And how many people actually want to wait for the designers of the programming language to add some new feature? The desired feature might not be important for everyone and might not be added to the core of the language. Should that be a reason why it is not there if your application actually requires it? For instance, as the ability to express domain knowledge clearly in programming languages become more important, more and more forms of domain-specific languages are surfacing. These domain-specific languages provide a concise way to express essential concepts of the domain which might otherwise be hidden through the syntax and semantics of a traditional programming language. No sane programming language designer is going to design multiple domain languages and give them multiple names such as C-Telecommmunication, C-Accounting, C-Transactions¹.

While the idea of having a full-fledged Metaobject Protocol might be too idealistic for some, some of the newer ideas in the software engineering world are taking advantage of a more moderate form of it. The designers of programming languages can certainly vary the degree of dynamic behavior. For instance, Ruby does not have a fully modifiable underlying semantics but it does provide enough for programmers to accomplish most of what needs to be done.

In conclusion, I feel that more and more languages would incorporate some form of the Metaobject protocol. Dynamic languages are becoming increasingly prominent in the software engineering world today and the logical next step would be to expand on their dynamic behavior to improve on the extensibility². It should be possible for both programming language designers and programmers to modify and/or features to the programming language in a hygienic way without resorting to ugly hacks.

¹ An alternative would be to design your own language from scratch if it is simple enough. Tools like Antlr and even Maude have made this process less painful. Even then, designing a language from scratch -- albeit being fun -- requires a lot more devotion than modifying an existing language.

² Some features might also be extremely hard if not impossible to add to an exiting system whether or not it has an Metaobject Protocol. For instance, (I think) having the ability to do multi-stage programming might require extensive changes to the language itself and is not accomplishable with just some hacking on the language itself.

defmacro - Why Exotic Languages Are Not Mainstream:

"Haskell is an excellent programming language. It has features that allow for a tremendous productivity boost. Type inference figures out and lets you examine types of every piece of code that you write so you don't have to baby sit the compiler, yet you get the benefits of static typing. Functions are first class objects so you can build abstractions without writing useless glue code. The language is elegant and has plenty of syntactic sugar that makes it a pleasure to work with. It even has libraries for things you may want to do in a real world project. So why wouldn't you want to pick Haskell for your projects?"

There are a lot of great programming languages out there. But having a great language alone will not be able to get people to program in it. There must be the tools to help you be productive in it. No matter how many lines of code savings you could potentially shave from a more expressive language, the returns are not going to be all that great if you have to invest the time to write your own libraries for common tasks. It's actually kind of like a catch-22: people cannot really start using the language unless there are good libraries for it; but libraries will not be written if they aren't enough people using it.

One of the reasons for Java's success was backing from Sun Microsystems. They had this incipient language out in 1995 when C++ was still reigning champ of programming languages. However, Java was quickly bundled with lots of libraries that worked together nicely. There was the GUI layer, the enterprise architecture, the sound, etc. At that time, none of the libraries was particularly mature yet but Java was able to entice people who were already were tired of trying to find out how to get different libraries to work together.

In the article by defmacro, he also lists another tool that is missing for esoteric languages: a good IDE. A normal text editor - as Vim and Emacs users will tell you - is sufficient for most tasks. But what happens when you need to write lots of code in different packages? Actually, Vim and Emacs are great because of the underlying unix tool support that is built-in. For instance, ctags and cscope really help make a project easier to navigate and Vim and Emacs have support for them. However, ctags and cscope do not support a lot of languages. For people who are spoilt with all the features from an IDE, using a normal text editor that just offers syntax highlighting is barely going to be fun.

It's hard to get people to abandon their mindset of what a development environment should be. Most people are already used to the idea of having a debugger, IDE, good reference and libraries at their fingertips for serious productions work. So, if esoteric languages want to really take off some of these things that people take for granted have to be offered first.

But then again, esoteric languages might not really want to take off. True, you can use Erlang and Haskell and Forth and ... in everyday situations but that might not be the original intentions of those languages. All I am saying is that certain languages were not created to be mainstream languages. They function well in their own little niche and the few people who use it are already happy with the tools that are provided.