Level: Introductory Jonathan Bartlett (johnnyb@eskimo.com), Director of Technology, New Medio
28 Feb 2006 Enterprise metaprogramming is becoming more common all the time as graphical and textual utilities make programming tasks easier and more descriptive, all because of the continuing formalization process occurring under the Object Management Group's Model Driven Architecture (MDA). This article, the third in a three-part series, explores the limits of metaprogramming, describes MDA and the problems it can solve, and presents a short example of a textual system that uses MDA.
Oh, no! Not another "theoretical" discussion! Keep reading, though -- this discussion of metaprogramming theory can help you separate fact from fiction in the sometimes wild marketing claims of various metaprogramming tools.
This article explores the theoretical limits of metaprogramming and the Object Management Group's (OMG) Model Driven Architecture (MDA). This architecture separates business and application logic from the underlying technology platform. Learn about the problems that the MDA can solve and see an example of a textual system that uses MDA.
The theoretical limits of metaprogramming
Knowing the possibilities for the future of metaprogramming and the inherent limits of metaprogramming can help you choose the right tools and ask vendors the right questions.
Graphical and textual metaprogramming
Parts 1 and 2 in this series focused on textual
metaprogramming systems; however, many metaprogramming
systems are graphical in nature. Let's begin by
examining a program represented both textually and graphically and see
what the similarities and differences are. This example uses regular expressions.
Regular expressions are basically a small sub-language built for text
processing, used to match and replace strings of characters. Learn more about regular expressions in developerWorks tutorials (see Resources).
Listing 1 a small regular expression in Perl that substitutes every occurrence of <h1> with
<h2> in an HTML string:
Listing 1. Perl regular expression for substituting HTML tags
$html_str =~ s!<(/?)h1>!<${1}h2>!g;
|
The language for doing a regular expression is not Perl; it is a
domain-specific language that Perl happens to have a hook into.
However, the exact syntax of the regular expression is not that
important; regular expressions can be written in many ways.
Basically,
whether you are communicating to the computer in a full-featured
programming language or a small domain-specific language, you need a way to communicate to the compiler or interpreter the
symbols of that language. You can do by typing code, but you could also express the above regular expression like this:
Figure 1. A Perl regular expression as a diagram
This works not only for domain-specific languages, but also for
general programming languages. Because compilers operate on symbols,
you can express symbols textually or graphically. Reasons for text expressions include:
- Typing is often faster than having to position graphs.
- More training and tools are geared toward text-based programming systems.
- Text forms are often easier to think about and manipulate programmatically.
In fact, for regular expressions, entering the code graphically is indeed more difficult. In the two representations of the regular
expression, each graph has the same amount of programming information. Thus, graphical representations of programming concepts do not reduce the amount of programming involved.
If you have used visual tools for programming, your experience may differ, due to the benefits of domain-specific languages, not specifically the graphical approach. Let's look at an example from Windows® programming, a Win32 resource script for building dialog boxes:
Listing 2. Example Win32 resource script
IDD_TEST DIALOG DISCARDABLE 0, 0, 200, 200
STYLE DS_MODALFRAME | WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU
CAPTION "Hello There!"
FONT 8, "MS Sans Serif"
BEGIN
DEFPUSHBUTTON "OK",IDOK,108,8,50,14
END
|
The script in Listing 2 requires you to "put in" the same amount of
information as if you had used a graphical tool to build the dialog box
and position it. The difference is that it is much easier to select
widgets and position them with a mouse than it is to type out their
names and then type in their coordinates. The amount of
information you convey to the computer is the same in
both cases.
So why does the operation require less programming than coding it by hand in C? The restricted, domain-specific sub-language requires less programming, not the visual or textual nature of the tool. This suggests the following conclusions:
- Domain-specific languages may reduce the amount of programming involved.
- Some domain-specific languages are easier to enter textually, while others are easier to enter graphically.
- Whether a program in a domain-specific language is entered textually or graphically, the same amount of information flows from the programmer to the computer.
- Because graphical languages still must provide symbols to the computer for compilation or interpretation, graphical specifications are still be required to follow formal language parameters. Such languages cannot be ad-hoc any more than computer languages can be. Graphical representations have syntax and semantics that are just as formalized as those of textual languages.
However, you can add more non-programmatic information to graphical representations. For example, you can:
- Spatially arrange different components to make
more sense to other people in a graphical environment.
- Add graphs, drawings, and other graphical explanatory
elements and visual cues directly to the source code to help explain a process.
- Assign code to regions that are hidden or
exposed, depending on the detail of the view (for example, you can hide
error-checking code when you want to analyze just the algorithm).
Graphical models take programming to a new
level, not because less information is passed from the
programmer to the computer, but because more information can
be passed from programmer to programmer in a more natural way.
Metaprogramming, programmers, and domain experts
Now that we've established that it is the domain-specific sub-language
that reduced the amount of programming involved, not necessarily the
graphical nature, can we ever reach a point
where programming is done completely by domain experts? In other words, can we skip the
actual programmers to do the work, and let the domain experts
simply use metaprogramming tools to design and build their entire
applications?
To answer this question, think about what programmers do for a programming project other than write code. While the domain expert may try to specify all possible scenarios for a particular transaction, some may be missed. For example, let's say you created an application that automatically filled in the city for a given ZIP code. The person who requested the feature may not be aware that some ZIP codes lie in multiple cities. What happens then? Is the user prompted? Is one automatically chosen? These underspecified corners of programming are called edge cases. In my own experience, the most important role of the programmer/analyst is to identify all of the edge cases that the domain expert has not thought of. Programmers tend to think about issues of process more methodically and therefore are often in the best position to help the domain expert identify and expound upon all of the edge cases.
Domain experts, left to their own devices, will often leave large areas of the system underspecified no matter how hard they try not to. Performing ad-hoc solutions is usually
just a part of doing business, and the need to establish absolute specifications just isn't in their realm of everyday thinking.
The second thing that programmers do is understand fully the
syntax, semantics, abilities, and limitations of the languages that
they program with. As mentioned above, graphical
languages are no less subject to issues of exact semantics than
textual languages are since they are both ultimately reduced to
symbols. Therefore, even in the best, most simplified graphical
environments, it is still utterly necessary to have a full
understanding of the tool being used and its semantics.
Based on these two observations, I am confident that there will always
be a need for programmers as long as there are programs to write.
Eventually programming may be embedded well enough into the
schools and curriculums that there won't be a need for programming as
a separate discipline. Instead of programming being simplified
to a point where programmers are not needed, public
education would be expanded to a point where such simplifications would not be necessary!
The important point in all of this discussion is that the
role of being a domain expert is and will continue to be separate from
the role of being a programmer, though those roles are sometimes
played by the same people.
Ultimately, graphical metaprograms are still
programs, thus they require a programmer to implement them
correctly.
Working at multiple levels of abstraction
One of the biggest problems in metaprogramming is the fact that the
programmer almost always has to work with multiple levels of
abstraction. The programmer not only needs to know the details of
how to program the
domain-specific languages and the general-purpose languages, but also:
- The details of how they are each implemented
- How to communicate between them
- What sort of impedance mismatches are between them.
Often, most of the code can use the metaprogramming
tools available to accomplish the task. But the hardest part comes
when they have to be mixed. For example, if the metaprogramming
system auto-generates a method within a class, but later that method
must be tweaked, how do you implement something like that?
In some instances and with some tools, modifying the generated
code immediately removes the program from the purview of the
metaprogramming system. The metaprogram is no longer of any use
because the change must be made at a lower level. There are some
solutions to this problem, but ultimately it is very difficult if not
impossible to solve completely.
The best systems are those that allow you to work at multiple levels
of abstraction within the metaprogramming system itself. For example,
Part 2 of this series discussed how the Scheme macro system
allows an easy mixing of macro and non-macro code. Ideally you need to work at both a high and a low level
from within the metaprogramming tool itself and simply to mark for the
tool where you need to transition to a lower level.
Now let's look at the OMG's Model Driven Architecture and the problems it can help you solve.
A high-level view of the Model Driven Architecture
Now that you have a better understanding of what is and isn't
possible with metaprogramming tools, let's turn to
the concept of enterprise metaprogramming and see how it is both similar to
and different from other metaprogramming practices.
Enterprise metaprogramming ultimately is just
metaprogramming. What differentiates it are these two defining qualities:
- The nature of the problems they try to solve
- The cardinality of the relationship between source and destination languages
Problems solved by the MDA
The first difference between traditional metaprogramming and
enterprise metaprogramming are the problems each tries to solve. Traditionally, metaprogramming has been mostly about these
things:
- Reducing the software-development release cycle
length
- Forcing programs to follow system
constraints
- Making the programming language follow the domain model
more closely
Enterprise metaprogramming involves these as well, but also adds other
considerations. First of all, one problem has always been that models are often
developed at the beginning of a project and then become irrelevant as the project progresses. The realities of coding
mean that the model is not very accurate, and the act of changing the model
continually only delays the project.
It seems almost wasteful
to design a model at the beginning and then not use it. It also seems
wasteful to develop a model once in a charting program and then
redevelop it later as code. Therefore, the Model Driven
Architecture aims to make the models developed at the beginning of the
project part of the code itself, so that it stays current and is
not redundant to the actually coding process.
Another problem is a related one, that of communication.
Code is very difficult for a non-programmer to read. However, if code
were written in a graphical manner, then it would be much easier to
discuss coding matters with non-coders. This would actually
make the code more difficult to develop, though, since aesthetics would have
to be included in the coding, but ultimately it would make the
communication between the programmer and the domain experts much more
efficient since the domain expert could comment on the code itself, represented in a graphical, domain-friendly nature.
Another problem is portability. Businesses often
use very different tools. This is not necessarily because of a lack
of technological forethought, but is simply a reality of business since:
- Many businesses purchase other businesses, and therefore their ITs must be combined.
- Businesses have so many small needed that it is impossible for an IT department to come up with a cohesive solution to all needs.
Therefore, it should simply be assumed that in a large organization there will be a diverse set of technologies that must be brought together. Because of this, the metaprogramming system should be customizable to produce output for a wide variety of systems.
The final problem is maintenance. Ultimately, systems are not just built; they must be maintained. Because the model is tied directly
into the code itself, maintenance becomes easier. Programmers can
communicate to future programmers graphically and use not only text,
but also graphics and charts to explain their programs. In fact, in
many cases the graphs and charts will be the program!
Platform-specific and -independent models
The core concept of the Model Driven Architecture -- which can solve or alleviate the problems mentioned above -- is the platform-independent model. Essentially, the MDA has three tiers:
- The code. Ultimately, all of our models will become code. This is the bottom-most tier of the MDA.
- The platform-specific model, or PSM. The metaprogramming systems described in the previous articles in this series are essentially platform-specific models. Although the models exist at a higher level than the programming languages themselves, they are ultimately directly tied to them. The translation and relationship between the platform-specific model and the code are usually fairly obvious. This model is often abbreviated as PSM.
- The platform-independent model, or PIM. This makes enterprise metaprogramming unique. PIMs, as you may have
guessed, get translated into PSMs. What makes them different is that the PIM is usually translated into multiple PSMs and is usually not tightly tied to any destination PSM or language.
Having a PIM that can be translated into multiple PSMs is useful for two reasons:
- Using the same program for two different platforms -- because of heterogeneous systems or the need to transition from a legacy system to a newer system
- Having a single descriptive model translate into multiple aspects of a system
The second reason is by far the more important of the two. For example, let's say that you have a data model that you want to use in
a C++ database program. With a platform-independent model, you can
write a single source file and have it translate into multiple aspects
of the final system. For example, the data model could translate
into:
- A skeleton C++ class, including operations for
serializing the class to the database
- An SQL script for creating the database on which the
data model is based
- A simple user interface for data
loading
So, from a single platform-independent model, you would get three
separate platform-specific models. Note that the
difference between a platform-specific model and code is merely
a difference of degree (think of a programming language as a platform-specific model for its object code). The important idea is that of translating a single
source file into multiple platform-specific ones, each covering a
different aspect of the system.
Coding in the Model Driven Architecture requires an ability to program on all three levels based on the task at hand, and perhaps even an ability to write translators between source models and
destination models:
- Some parts of an application can be written as a PIM in which most of the legwork is done by the metaprogramming system.
- Other parts must be done as a PSM in which the features of the model are heavily tied to the purpose and abilities of the language.
- And some things can be written in code, as usual.
The benefits to the Model Driven Architecture are that:
- The models for the system have continued
usefulness for the life of the software.
- Much of the redundancy that is common in
applications with multiple aspects is eliminated.
- Communication between the programmer, other
programmers, and domain experts is enhanced.
Example: Building code from Dia diagrams
One of the most interesting things about the MDA is that while there are specifications involved, it is at its core
mostly an idea, a vision, and a process. A tool is an MDA tool if it
implements some or all of the vision of the MDA. Of course, being an
OMG standard, UML (the Unified Modeling Language) is the preferred modeling language, but the vision
of MDA greatly supersedes the technologies that it uses.
 |
Dia
Dia is a gtk+-based diagram-creation program released under GPL. It is designed to work like Visio and can draw many types of diagrams, including entity relationship diagrams, UML diagrams, flowcharts, network diagrams, and simple circuits. It can load and save diagrams to a custom XML format and can export diagrams to EPS and SVG formats. It contains printer support, including the ability to span multiple printed pages.
|
|
For our example, we are going to create an MDA tool that takes
class diagrams from the graphing program Dia and produces both SQL
code for database tables and C++ code for simple C++ classes. This
doesn't live up to the full expectation of MDA tools, but it gives a
good example of generating multiple platform-specific models (well, we're
generating actual code in this case) from a single
platform-independent model.
Creating the diagram
To create the diagram, fire up Dia and use its UML
graphing capabilities. When you start Dia, it looks like this:
Figure 2. The Dia program
There is a menu of object types on the main toolbox. Choose the UML object set. The first object is a UML class. Select it and then
add it to your diagram. Your diagram will look something like this:
Figure 3. A UML class in a new drawing
If you double-click the new class glyph, you see a screen
like this:
Figure 4. Basic class information
This dialog box lets you set up the class information. Since this
example is limited, the only field on this tab that is of
use is the Name field. For our example, we're calling this the
Person class. Next, add attributes, so go
to the Attributes tab. Here we will add three fields: id, name, and address of types int, text, and text, respectively. Your
screen should look like this:
Figure 5. Editing class attributes
Click OK on the dialog box to finish. You can add additional
classes if you want. Once you are done, save the
diagram using this dialog box:
Figure 6. Dia's save dialog box
Be sure when saving that the "compress diagram files" box is
not checked.
A quick overview of the Dia file format
A Dia file is simply an XML file using Dia's own schema and
namespace, either compressed or uncompressed (our program only
deals with uncompressed files). The layout is simple. Here are
some portions of the file we just made:
Listing 3. Example Dia file
<?xml version="1.0" encoding="UTF-8"?>
<dia:diagram xmlns:dia="http://www.lysator.liu.se/~alla/dia/">
<dia:diagramdata>
<dia:attribute name="background">
<dia:color val="#ffffff"/>
</dia:attribute>
...
...
</dia:diagramdata>
<dia:layer name="Background" visible="true">
<dia:object type="UML - Class" version="0" id="O0">
<dia:attribute name="obj_pos">
<dia:point val="11.6,6.35"/>
</dia:attribute>
...
...
<dia:attribute name="name">
<dia:string>#Person#</dia:string>
</dia:attribute>
...
<dia:attribute name="attributes">
<dia:composite type="umlattribute">
<dia:attribute name="name">
<dia:string>#id#</dia:string>
</dia:attribute>
<dia:attribute name="type">
<dia:string>#int#</dia:string>
</dia:attribute>
<dia:attribute name="value">
<dia:string>##</dia:string>
</dia:attribute>
...
...
</dia:composite>
</dia:attribute>
<dia:attribute name="operations"/>
<dia:attribute name="template">
<dia:boolean val="false"/>
</dia:attribute>
<dia:attribute name="templates"/>
</dia:object>
</dia:layer>
</dia:diagram>
|
This file has a lot of information in it, but only a small part of it
is useful to our purposes. Notice the
XML trees that begin with something like <dia:object type="UML - Class" version="0" id="O0">.
And from these, notice the name of the
class and the name and type of each attribute.
The name of the class
is in an attribute named name. The attributes are all in a subtree
called attributes with each attribute being a umlatrribute with
corresponding name and type fields. Therefore, within the sea of
XML tags and data, sort through them and grab the portions of interest.
An MDA implementation
The MDA implementation is a simple SAX parser written in
Scheme using the SSAX module (for more information about Scheme, SAX,
and the SSAX module, see Resources). The parser reads from one .dia file and writes to an SQL file and a
C++ source file. For each class it creates a C++ class and a
database table.
To make the example as simple as possible,
only attributes are supported, not operations, and no database
loading/saving functions are implemented. However, adding those
features are only incremental steps away from the parser shown in Listing 4.
Here is the code for the MDA application, written for use with Chicken
Scheme (other schemes would probably require only slight modification
to the line that loads in the SSAX module):
Listing 4. Full program listing for MDA application
;;Load the Scheme SAX parser
(require-extension ssax)
;;Dia's string values are surrounded with '#'s. This
;;removes the '#'s.
(define (munge-dia-value val)
(let (
(str-len (string-length val)))
(if (<= str-len 2)
""
(substring val 1 (- str-len 1)))))
;;The SAX parser function
(define (dia-mda-parser input-port cpp-out-port sql-out-port)
(let (
;;State variables -- these would probably be more correct
;; to keep with the seed, but putting them
;; here makes them easier to manage
;this is used for proper handling of comma-separated lists
(first #t)
;these are used to store the type and name of an attribute
;before its read to be printed out
(attr-type "")
(attr-name ""))
;;Write the C++ header
(display "#include <string>" cpp-out-port)(newline cpp-out-port)
(display "using namespace std;" cpp-out-port)(newline cpp-out-port)
(display "typedef string text;" cpp-out-port)(newline cpp-out-port)
;;SSAX Macro to build the parser
((SSAX:make-parser
;;on-element function
NEW-LEVEL-SEED
(lambda (elem-gi attrs namespaces expected-content seed)
;;The element name is the cdr of the name list
(define element (cdr elem-gi))
;;common attributes
(define name-attr (assq 'name attrs))
(define type-attr (assq 'type attrs))
;;Giant state switch - determine what element and state
;; we are in and what state we should
;; change to, if any
;;
;;Note that the return value of this function (and the others)
;;becomes the new "seed" value, so we are using it to record
;;the current state. More information about the inner
;;workings of SSAX are in the references section.
(cond
;Check to begin a top-level class
((and (eq? seed 'top) (eq? element 'object) (equal? (cdr type-attr) "UML - Class"))
(set! first #t) ;reset the "comma" state
'object)
;Check whether to be looking for the class name
((and (eq? seed 'object) (eq? element 'attribute) (equal? (cdr name-attr) "name"))
'class-name)
;Check whether we are ready to read the class name
((and (eq? seed 'class-name) (eq? element 'string))
'read-class-name)
;Check to see if we should switch to attribute mode
((and (eq? seed 'object) (eq? element 'attribute) (equal? (cdr name-attr)
"attributes")) 'class-attrs)
;Check to see if we should start looking for attribute info
((and (eq? seed 'class-attrs) (eq? element 'composite) (equal? (cdr type-attr)
"umlattribute")) 'class-attribute)
;Check whether we should be looking for the attribute name
((and (eq? seed 'class-attribute) (eq? element 'attribute) (equal? (cdr name-attr)
"name")) 'class-attribute-name)
;Check whether we should be reading the attribute name
((and (eq? seed 'class-attribute-name) (eq? element 'string))
'read-class-attribute-name)
;Check whether we should be looking for the attribute type
((and (eq? seed 'class-attribute) (eq? element 'attribute) (equal? (cdr name-attr)
"type")) 'class-attribute-type)
;Check whether we should be reading the attribute type
((and (eq? seed 'class-attribute-type) (eq? element 'string))
'read-class-attribute-type)
(else seed)))
;;after-element function
FINISH-ELEMENT
(lambda (elem-gi attrs namespaces parent-seed seed)
;;often-used values
(define element (cdr elem-gi))
(define type-attr (assq 'type attrs))
(cond
;After finishing the umlattribute element, write out the attribute information
((and (eq? element 'composite) (equal? (cdr type-attr) "umlattribute"))
(if first
(set! first #f)
(display ", " sql-out-port))
(display (string-append attr-name " " attr-type) sql-out-port)
(display (string-append "\t" attr-type " " attr-name ";") cpp-out-port)
(newline cpp-out-port))
;After finishing the UML Class object element, write the closing characters
((and (eq? element 'object) (equal? (cdr type-attr) "UML - Class"))
(display ");" sql-out-port)(newline sql-out-port)
(display "};" cpp-out-port)(newline cpp-out-port))
(else #f))
;Restore parent information
parent-seed)
;;character data function
CHAR-DATA-HANDLER
(lambda (s1 s2 seed)
(cond
;Read the class name and write out the appropriate statements
((eq? seed 'read-class-name)
(display "class " cpp-out-port)
(display (munge-dia-value s1) cpp-out-port)
(display " {" cpp-out-port) (newline cpp-out-port)
(display "public:" cpp-out-port) (newline cpp-out-port)
(display "create table " sql-out-port)
(display (munge-dia-value s1) sql-out-port)
(display " (" sql-out-port))
;Read and save the attribute name
((eq? seed 'read-class-attribute-name)
(set! attr-name (munge-dia-value s1)))
;Read and save the attribute type
((eq? seed 'read-class-attribute-type)
(set! attr-type (munge-dia-value s1)))
(else #f))
seed)
) input-port 'top)))
;;;Main Program;;;
(let (
;;Open Files;;
(tables-in (open-input-file "tables.dia"))
(cpp-out (open-output-file "tables.cpp"))
(sql-out (open-output-file "tables.sql")))
;;Process Data;;
(dia-mda-parser tables-in cpp-out sql-out)
;;Close Files;;
(close-input-port tables-in)
(close-output-port cpp-out)
(close-output-port sql-out))
|
For simplicity, the input and output file names are
hardcoded, so be sure to save the Dia file in the same directory with
the name tables.dia. Save the program as mda.scm. To run the program, simply do csi mda.scm. With the data we put into Dia, it produces two output files. The SQL output looks like this:
Listing 5. SQL output from the MDA program
create table Person (id int, name text, address text);
|
The C++ file looks like this:
Listing 6. C++ output from the MDA program
#include <string>
using namespace std;
typedef string text;
class Person {
public:
int id;
text name;
text address;
};
|
So, for under 200 lines of code, you can create diagrams instead of
writing SQL and C++. Of course, it's even more fun when someone else
writes the conversion program.
Conclusion
The Model Driven Architecture is a powerful idea.
It is rooted in a strong tradition of metaprogramming and yet goes
beyond traditional methods using its concept of platform-independent
models.
Resources Learn
Get products and technologies
-
The Design Pattern Toolkit (alphaWorks, March 2005) is an Eclipse-based toolkit for developing MDA transformations.
-
For a true open-source graphical MDA tool, check out AndroMDA.
-
Eclipse has an MDA tool called the Eclipse Modeling Framework.
-
Order the SEK for Linux, a two-DVD set containing the latest IBM trial software for Linux from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
-
With IBM trial software, available for download directly from developerWorks, build your next development project on Linux.
Discuss
About the author | | | Jonathan Bartlett is the author of the book Programming from the Ground Up, an introduction to programming using Linux assembly language. He is the lead developer at New Medio, developing Web, video, kiosk, and desktop applications for clients. |
Rate this page
| 
