The BlueSpace wall display project

Document options Document options requiring JavaScript are not displayed Learn and share! Exchange know-how with your peers -- try our new Pass It Along beta app Rate this page Help us improve this content

Level: Introductory

Barry Feigenbaum, Ph.D. (feigenba@us.ibm.com), Sr. Consulting I/T Architect, IBM Worldwide Accessibility Center

22 Jun 2004

The BlueSpace wall display is an exciting demonstration of the potential of multimedia development on the Java platform. In this project, first developed by the IBM Worldwide Accessibility Center in 2003 and presented this year at JavaOne, a large-scale, high-resolution visual screen is implemented as a grid of projected computer displays. The resulting display is infinitely malleable in size and form and has numerous multimedia and presentation capabilities. Regular developerWorks contributor and Worldwide Accessibility Center engineer Barry Feigenbaum summarizes the concept and implementation details behind this project, for which he was the development team leader.

In 2003 the IBM WorldWide Accessibility Center conducted a series of demonstrations called the Multimedia Experience. Included among the series was a prototype office called BlueSpace. A striking feature of the BlueSpace office was a wall-sized display called the BlueSpace wall display, a large-scale, adjustable display capable of providing very high resolution imaging. Some of the implementation details of the BlueSpace wall display will be presented at JavaOne 2004 (see Resources), and this article is a follow-up to that presentation.

While it is beyond the scope of the article to provide the entire source base for the BlueSpace wall display project, you will learn here about the basic concept behind the project and its core implementation details. I'll start with an overview of the visual setup and multimedia capabilities of the wall display, and then quickly move into the code. Included in the code examples are the method used to scale and pan content across one or more cells of the display, and several examples of display commands that the cell servers can perform. Illustrations throughout will give you an idea of the scale and utility of the wall display.

If you are able to see the wall display presentation at JavaOne, this article will complement the session, giving you some more information about its implementation, as well as some of the challenges encountered in its development. If you missed the presentation, this article will leave you with an idea of both the visual presence of the wall display (via snapshot illustrations) and how it was constructed using Java technology. The article concludes with a brief discussion of potential future additions to the BlueSpace wall display.

The BlueSpace wall display

The prototype BlueSpace wall display configuration consists of cells arranged in a 4x3 grid creating a display measuring approximately 8 feet by 8 feet. The display is created by rear-projecting the output of 12 IBM ThinkPad computers onto a screen mounted in a pseudo-office wall. The wall is actually a frame for the screen material made to look like an office wall. Using the 1024x768 video mode on each ThinkPad achieves a composite resolution of 3072x3072 (or ~9M pixels). It is also possible to add more rows and/or columns to create a taller and/or wider display with increased resolution.

Each computer in the grid is a display server. The display server supports commands (delivered over TCP/IP sockets) to query its configuration, display images, videos, and Java GUI components and performs other miscellaneous functions. It supports the ability to scale the images across one or more cells. It also supports the ability to display selected (that is, specially coded) Java GUI components across cells.

Further inquiry BlueSpace and BlueSpace wall display are IBM research project names and not intended to imply or define any future product or service name or infringe on any existing products or trademarks. All questions regarding the BlueSpace wall display should be directed to the author, Barry Feigenbaum.

You can see the layout of the BlueSpace wall display in the lower left corner of Figure 1, where it is part of the overall Multimedia Experience demonstration layout.

Eye on the future The BlueSpace wall display was built using contemporary technologies, but with the goal to demonstrate how technology will look and feel in the near future. The display shows how man-machine interfaces may evolve as very large display devices become available in the next few years. While the dependency on multiple computers limits the usefulness of such a large-scale wall display in the near term, the current trend towards larger LCD and wall-mounted plasma devices suggests that such large displays will soon become commonplace. It is the premise of the BlueSpace wall display project that the ability to show multiple concurrent content streams at full-stream resolution will revolutionize computer user interfaces.

The BlueSpace wall display is built entirely on Java technology. The project demonstrates the following unique features:

• Very large size achieved by using a grid of projected computer displays
• Client/server architecture to control the display
• Extraordinary image resolution
• Concurrent and variable size display of images, videos, and Java GUI components in any cell combination from 1x1 to 4x3
• Remote control by mouse, keyboard, voice, and gesture
• Relatively rapid movement and setup of the projector grid

The wall display also provides support for people with disabilities, including simple speech output and voice recognition for command input. It uses IBM ViaVoice (see Resources) through the JavaSpeech API to access its formant (constructed from phonemes) text-to-speech (TTS) and speech recognition. For better speech quality, the wall display also supports concatenative (constructed from pre-recorded speech) TTS, although JavaSpeech cannot be used with this feature. The wall display can also provide captions for display-over images and Swing components, allowing for a form of closed captioning (for example, to display any spoken speech output).

Back to top

Picturing the wall display

You can see an example of the wall display showing a multi-cell image in Figure 2. This image shows a faux-office wall that creates for the Multimedia Experience audience the illusion of a real office wall. This is one image scaled and tiled across all cells to create a full-screen image. You can also see the BlueSpace office configuration, which includes the wall display, a table with a Tablet PC (used by the actor to control the display wall content), a camera, and an office chair.

Figure 3 reveals the wall display in action. Shown across the top row of the wall display (although partially hidden) are a still image and two video feeds. The second row shows another video feed and two Java GUI components, an appointment list, and a monthly calendar. The third row shows three images, but the last image (of the doll's head) is really a paused video set to restart as a live video when a touch-gesture is performed before it. The bottom row shows a live feed of a camera on the actor, a still image, and a Java browser GUI showing a Web site.

In the final Multimedia Experience vignette an evacuation system animation plays across all 12 cells. This scene, shown in Figure 4, requires precise synchronization to work. The circular moving people icons and the flashing alert messages make any synchronization issues obvious. For further details on how this animation was created see Resources.

Figures 5 and 6 allow you to get a greater idea of the wall display in use, and also give you a sense of the scale of the display.

Although these images don't quite demonstrate the high resolution of the display, you can see it in the density of text in many of the screen cells. When all that available resolution is used to display images, the results can be breathtaking. The sheer amount of resolution and the ability to move and resize content across the grid combined with the ability to interact with the screen via gestures opens up new possibilities that can fundamentally change the man-machine interaction paradigm.

Back to top

Implementation overview

As previously mentioned, the BlueSpace wall display is implemented as a (variable sized) grid of computers. Figure 7 shows a 3x3 grid as an example. The grid is composed of several server computers each driving a single projector. The projected images are aligned such that they form a nearly seamless image on the projection screen. The display grid is controlled by one or more control (or client) computers. The server computers are completely controlled by the client computers; they take no action on their own.

Each cell in the grid can be controlled independently. For example, a single image can be shown in each corner of the grid. All commands to the servers include an extent (start row, start column, end row, end column -- all zero origin). Using the same start and end values selects a single cell, as shown in Figure 8.

Using different start and end values selects a range of cells, as shown in Figures 9 and 10 below.

Notice how the image is scaled to fit the selected extent. Scale is programmable and other sizing options are supported, including natural scale, 1D scaling, and 2D scaling (as shown). Magnification is also supported.

Back to top

Scaling and panning

The basic method used to display cross-cell images and GUI components is shown in Figure 11. Each cell is instructed to display its content at a virtual display size. For a full-screen image the virtual screen size would be 3072x3072. The virtual image is then positioned "under" the cell such that the cell's upper-left corner is positioned at the correct location in the virtual image. When all cells are positioned at the correct locations, the display shows the entire image.

Note that the position need not be such that a normal image is shown. You could also choose to arrange the cells for a jigsaw puzzle view or in repeated segments. The virtual image size may also be larger or smaller than the display extent. Listing 1 shows some of the code to implement this support. The class ClipViewer is a Swing JPanel that sizes and locates its content as described above.

    public void paintChildren(Graphics g) {
        Graphics2D g2 = (Graphics2D)g.create();
        try {
            g2.clipRect(0, 0, getWidth(), getHeight());
            g2.translate(-x, -y);
            if (scale > 0.0) {
                g2.scale(scale, scale);
            }
            super.paintChildren(g2);
        } 
        finally {
            g2.dispose();
        }
    }

Listing 2 shows the parameters supported by ClipViewer.

    protected int x;
    public int getTranslationX() {
        return x;
    }
    public void setTranslationX(int x) {
        this.x = x;
    }
    protected int y;
    public int getTranslationY() {
        return y;
    }
    public void setTranslationY(int y) {
        this.y = y;
    }
    protected double scale;
    public double getScale() {
        return scale;
    }
    public void setScale(double scale) {
        this.scale = scale;
    }

Back to top

Server command protocol

Listing 3 shows some of the commands provided by each server. The key (left name) is the name used by the client to invoke the command and the value (right name) is a Java class name for the server. The server implements the Command pattern, so it can add new commands very easily. Each command is sent as either a serialized String command name followed by a Map of parameters or just a Map containing both the command name and parameters. Most commands respond with null (which means the client doesn't know if the command worked, although the user can tell). A few, such as the Configure and GC commands, return values. Command processing is asynchronous; multiple commands can execute simultaneously.

# Misc control commands
start             = StartCommand
stop              = StopCommand
configure         = ConfigureCommand
exit              = ExitCommand
quit              = ExitCommand
gc                = GCCommand

# System setup and management commands
runRobotScript   = RunRobotScriptCommand
runCommand       = StartServerCommand
frameControl     = FrameControlCommand
displayCalibrationImage = DisplayCalibrationImageCommand
repaint           = RepaintCommand

# Appearance control commands
setSelected       = SetSelectedCommand
showBorder        = BorderCommand

# Image and Video control commands
clearImageCache   = ClearImageCacheCommand
preloadImage      = PreloadImageCommand
displayImage      = DisplayImageCommand
displayCaption    = DisplayCaptionCommand
displayCrosshair  = DisplayCrosshairCommand
displayAnimatedImage = DisplayAnimatedImageCommand
displayByPlayer   = DisplayByPlayerCommand

# Generic Swing Component commands
displayLabel      = DisplayLabelCommand
displayTable      = DisplayTableCommand
displayList       = DisplayListCommand
displayTime       = DisplayTimeCommand

# Specialized "Office" Component commands
displayMail       = DisplayMailCommand
selectMail        = SelectMailCommand
scrollMail        = ScrollMailCommand
scrollEMail       = ScrollEmailCommand
displayEMail      = DisplayEMailCommand
displayHtml       = DisplayHtmlCommand
scrollHtml        = ScrollHtmlCommand
displayCalendar   = DisplayCalendarCommand
scrollCalendar    = ScrollCalendarCommand
displayMonth      = DisplayMonthCommand

# Specialized "BlueSpace" Component commands
displayStockTicker = DisplayStockTickerCommand
displayEyeSurvey  = DisplayEyeSurveyCommand
displayEscape     = DisplayEscapeCommand
displayPongGame   = DisplayPongCommand

Sample Client

An example client is shown in Figure 12. This client acts like a image and/or video Juke-box player. It uses many of the image and video control commands. Note the high resolution of many of the images (ex. 3840 x 2400). This client is written in the Jython language (see Resources). Other clients similar to this were created to display the content for the various Multimedia Experience shows. They run on the actor's tablet PC.

Given a 4x3 grid of display servers, this client might send a sequence of commands, as shown in Figure 13, to get the server configuration for all the cells, prepare and then display an image in four cells and later show a video in a different set of four cells. The commands are only sent to the effected cell servers via a broadcast mechanism.

Back to top

Command processors

Much can be learned about the BlueSpace project simply by studying its command processors. I'll focus for much of the remainder of the article on the command processors for various functions, and then conclude with a brief discussion of the development challenges encountered in this project. The command processor examples you'll study run from the trivial (garbage collection) to the fairly complex (image display). Note that all of the code runs on the display server.

Back to top

The GC processor

The code in Listing 4 shows how the Garbage Collect command is implemented. All commands are passed a SegmentDescriptor describing the grid cell (or segment) that the given server processes. The included Map contains the parameters sent by the client.

public class GCCommand extends BaseCommand {
    public Object execute(SegmentDescriptor sd, Map params)
           throws CommandException {
        System.gc();
        return new Long(Runtime.getRuntime().freeMemory());
    }
}

TimeLabel command processor

The following code shows how a Swing component command is implemented. The TimeLabel command processor is more complex than the GC processor, so I'll go through it step by step. The first step is to process all the parameters, as shown in Listing 5.

public Object execute(final SegmentDescriptor sd, final Map params) 
        throws CommandException {
    final Component c = sd.component;
    final Map state = sd.state;
    final Map context = sd.context;

    final int segWidth = ((Integer)state.get("width")).intValue();
    final int segHeight = ((Integer)state.get("height")).intValue();

    final Integer x = (Integer)params.get("x");
    final Integer y = (Integer)params.get("y");

    final Integer width = (Integer)params.get("width");
    final Integer height = (Integer)params.get("height");
    final Integer rows = (Integer)params.get("rows");
    final Integer columns = (Integer)params.get("columns");
    final Integer row = (Integer)params.get("row");
    final Integer column = (Integer)params.get("column");

    final Boolean show = (Boolean)params.get("show");
    final Boolean autoSize = (Boolean)params.get("autoSize");

    final DateFormat format = (DateFormat)params.get("format");

    final BigInteger syncStart = (BigInteger)params.get("syncStart");

    final String align = (String)params.get("align");

    final String type = (String)params.get("type");

    final Font font = (Font)params.get("font");
    final Color bgColor = (Color)params.get("bgColor");
    final Color fgColor = (Color)params.get("fgColor");

    final Integer xLoc = (Integer)params.get("xLoc");
    final Integer yLoc = (Integer)params.get("yLoc");

Next, the Swing components (in this case a JLabel subclass that shows updating time) are created, as show in Listing 6.

    if (show == null || show.booleanValue()) {
        viewer = new TimeLabel();
        viewer.setOpaque(true);

        viewer.setAutoSize(autoSize != null 
            ? autoSize.booleanValue() 
            : (width == null && height == null));
        if (format != null) {
            viewer.setFormat(format);
        }
        if (font != null) {
            viewer.setFont(font);
        }
        viewer.setForeground(new Color(100, 100, 100));
        if (fgColor != null) {
            viewer.setForeground(fgColor);
        }
        viewer.setBackground(new Color(255, 255, 255, 150));
        if (bgColor != null) {
            viewer.setBackground(bgColor);
        }

        if (xLoc != null & yLoc != null) {
            viewer.setLocation(xLoc.intValue() - 
                                  (segWidth * row.intValue()), 
                               yLoc.intValue() - 
                                  (segHeight * column.intValue()));
        }
        if (width != null && height != null) {
            viewer.setSize(width.intValue(), height.intValue());
        }
        else {
            viewer.setSize(viewer.getPreferredSize());
        }

The code in Listing 7 synchronizes all segments to show the label at the same time, ensuring that the user sees a coordinated display of a cross-segment dynamic component.

        if (syncStart != null) {
            long syncTime = syncStart.longValue();
            for (long delta = deltaTime(syncTime); 
                 delta > 0; 
                 delta = deltaTime(syncTime)) {
                try { Thread.currentThread().sleep(delta); } 
                catch (InterruptedException ie) {}
            }
        }

Finally, you register the component with the segments container (a JPanel). Several types of registration (and corresponding release) can be done, including a main component, a floating component, and a video player component. All Swing code must be done on the event thread.

        SwingUtilities.invokeLater(
            new Runnable() {
                public void run() {
                    registerFloatingComponent(sd, 
                          type != null ? type : "time", viewer);
                    c.repaint();
                }
            });
    }

Finally, the component is removed (or deregistered), as shown in Listing 9.

    else {
        SwingUtilities.invokeLater(
            new Runnable() {
                public void run() {
                    releaseFloatingComponent(sd, 
                          type != null ? type : "time");
                    c.repaint();
                }
            });
    }
    return null;            // no result
}

Back to top

The DisplayImage processor

Listings 10 through 13 show how the DisplayImage command is implemented. Once again I'll go through the process step-by-step. Because the client may send a file path, a URL, or the image itself, the first step is to get the image's current state and the needed parameters, as shown in Listing 10.

public Object execute(final SegmentDescriptor sd, final Map params)
       throws CommandException {
    final Component c = sd.component;
    final Map state = sd.state;
    final Map context = sd.context;

    final Integer row = (Integer)params.get("row");
    final Integer column = (Integer)params.get("column");
    final List shared = (List)params.get("shared");

    final int segWidth = ((Integer)state.get("width")).intValue();
    final int segHeight = ((Integer)state.get("height")).intValue();

    params.put("segWidth", state.get("width"));
    params.put("segHeight", state.get("height"));

    String imagePath = (String)params.get("imageFilePath");
    String imageResource = (String)params.get("imageResourcePath");
    URL    imageUrl =  (URL)params.get("imageUrl");
    Object image = params.get("image");

In Listing 11 you can see how the DisplayImage processor prepares to show an image. First, the processor checks an image cache to see if the image is preloaded or has been recently used.

    try {
        ImageIcon ii = null;
        ImageCache ic = null;
        String cacheName = (String)params.get("cacheName");
        if (cacheName != null) {
            ic = ImageCache.getCache(cacheName);
        }
        if (ic == null) {
            ic = ImageCache.getDefaultCache();
        }

The actual image painting is done in a JPanel subclass called ImagePaintDelegate, as shown in Listing 12. This approach allows images to act as the background to other components.

        ImagePaintDelegate ipd = null;
        if ( image != null ) {
            ipd = image instanceof ImageIcon 
              ? new ImagePaintDelegate(ii = (ImageIcon)image)
              : new ImagePaintDelegate((byte[])image);
        }
        else if ( imagePath != null ) {
            ii = ic.getByName(imagePath);
            if (ii == null) {
                    ii = loadImage(imagePath);
            }
            ipd = new ImagePaintDelegate(ii);
            ic.put(imagePath, ii = ipd.getImage());
        }
        else if ( imageUrl != null ) {
            ii = ic.getByUrl(imageUrl);
            ipd = ii != null ? new ImagePaintDelegate(ii) 
                             : new ImagePaintDelegate(imageUrl);
            ic.put(imageUrl, ii = ipd.getImage());
        }
        else if ( imageResource != null ) {
            :
        }
        else {
            throw new IllegalArgumentException(…);
        }

        ipd.addParameters(params);

In Listing 13 you can see how the processor displays the image. All Swing code must be done on the event thread.

        final ImagePaintDelegate ipdx = ipd;
        SwingUtilities.invokeLater(
            new Runnable() {
                public void run() {
                    if (c instanceof DelegatedPaintPanel) {
                        ((DelegatedPaintPanel)c).
                       setPaintComponentListener(ipdx);
                    }
                    JComponent blank = new JLabel(" ");
                    blank.setBackground(transColor);
                    registerMainComponent(sd, blank);
                    setImageBorder(sd);
                    c.repaint();
                }
            });    
    }
    catch (Exception e) {
        throw new CommandException(e.getMessage());
    }
    return null;            // no result
}

Painting the image

The DisplayImage processor is responsible for displaying the image, but image painting is delegated to an ImagePaintDelegate, as shown in Listings 14 through 16. In Listing 14, you can see how the delegate applies all the scaling and image fit parameters. It creates a custom Graphics2D component to implement these settings and apply cross-segment scaling and translation.

public class ImagePaintDelegate implements PaintListener
{
    :
    public void paint(Object source, Graphics g) {
        Graphics g2 = g.create();      // so transforms are local
        Graphics2D g2d = (Graphics2D)g2;
        try {
            Component c = (Component)source;
            ImageIcon ximage = image;   // allow change

            final int segWidth  = ((Integer)getParameter(
                   "segWidth", intZero)).intValue();
            final int segHeight = ((Integer)getParameter(
                   "segHeight", intZero)).intValue();

            final int rows    =  ((Integer)getParameter(
                   "rows", intOne)).intValue();
            final int columns =  ((Integer)getParameter(
                   "columns", intOne)).intValue();
            int numcells = ((Integer)getParameter(
                   "numcells", intZero)).intValue();

            Double xScale = (Double)getParameter("xScale", scaleOne);
            Double yScale = (Double)getParameter("yScale", scaleOne);
            double xscale = xScale.doubleValue(), 
                   yscale = yScale.doubleValue();

            Double xTrans = (Double)getParameter("xOffset", locZero);
            Double yTrans = (Double)getParameter("yOffset", locZero);
            double xtrans = xTrans.doubleValue(), 
                   ytrans = yTrans.doubleValue();

            Integer xLoc = (Integer)getParameter("x", intZero);
            Integer yLoc = (Integer)getParameter("y", intZero);
            int xloc = (int)(-xLoc.intValue() / xScale.doubleValue()),
                yloc = (int)(-yLoc.intValue() / yScale.doubleValue());

Images may be made to "fit" in several ways. Typically full segment, full screen, or natural aspect ratios are used, as shown in Listing 15.

            int fitType = FIT_NONE;
            Object ofit = getParameter("fit", Boolean.TRUE);
            if (ofit instanceof Boolean) {
                fitType = ((Boolean)ofit).booleanValue() 
                      ? FIT_SINGLE : FIT_NONE;
            }
            else if (ofit instanceof Integer) {
                fitType = ((Integer)ofit).intValue();
            }

            int width = ximage.getIconWidth(), 
                height = ximage.getIconHeight();
            int fitWidth = -1, fitHeight = -1;
            switch (fitType) {
                case FIT_NONE:
                    fitWidth  = width;
                    fitHeight = height;
                    break;
                case FIT_SINGLE:
                    fitWidth  = c.getWidth();
                    fitHeight = c.getHeight();
                    break;
                case FIT_MULTIPLE:
                    fitWidth  = segWidth * columns;
                    fitHeight = segHeight * rows;
                    break;
            }

In Listing 16, finally, you see how the image is drawn.

        boolean fit = fitWidth > 0 && fitHeight > 0;

            g2d.scale(xscale, yscale);

            g2d.translate(xtrans, ytrans);

            g2d.drawImage(ximage.getImage(), 
                          xloc, yloc, 
                          fit ? fitWidth : width, 
                          fit ? fitHeight : height, 
                          c);
        }
        catch (Throwable e) {
            e.printStackTrace();
        }
        finally {
            g2.dispose();
        }
    }
}

Painting the background

In Listing 17 you see how each server segment uses a JPanel subclass to delegate the painting of its background. The DelegatePaintPanel subclass allows any image to be the background of any other (partially transparent) component.

public class DelegatedPaintPanel extends JPanel
{
    // *** bodies deleted, standard code ***
    public void setPaintComponentListener(PaintListener pl) {
    }
    public void clearPaintComponentListeners() {
    }
    public void addPaintComponentListener(PaintListener pl) {
    }
    public void removePaintComponentListener(PaintListener pl) {
    }
  
    protected void firePaintComponentEvent(Graphics g) {
        if (listeners.size() > 0) {
            List l = (List)((LinkedList)listeners).clone();
            for (Iterator i = l.iterator(); i.hasNext();) {
                PaintListener pl = (PaintListener)i.next();
                pl.paint(this, g);
            }
        }
    }

    // overridden to allow delegates to run
    public void paintComponent(Graphics g) {
        super.paintComponent(g);
        firePaintComponentEvent(g);
    }
}

Back to top

The DisplayMonth processor

As a final example, you'll see how Swing components are handled. Listings 18 through 20 show the step-by-step implementation of the DisplayMonth command.

public class DisplayMonthCommand extends AbstractCommand 
{
    protected JPanel view;

    public Object execute(final SegmentDescriptor sd, final Map params) 
          throws CommandException {
        final Component c = sd.component;
        final Map state = sd.state;
        final Map context = sd.context;

        final int segWidth = ((Integer)state.get("width")).intValue();
        final int segHeight = 
                    ((Integer)state.get("height")).intValue();

        final Integer x = (Integer)params.get("x");
        final Integer y = (Integer)params.get("y");

        final Integer rows = (Integer)params.get("rows");
        final Integer columns = (Integer)params.get("columns");

        final Integer width = (Integer)params.get("width");
        final Integer height = (Integer)params.get("height");

        final Calendar date = (Calendar)params.get("date");
        final Boolean showTime = (Boolean)params.get("showTime");
        final Date baseTime = (Date)params.get("baseTime");

Listing 19 shows how the DisplayMonth command processor determines the desired size based on passed parameters or defaults.

        int xwidth = -1, xheight = -1;
        if (rows != null && columns != null) {
            xwidth = segWidth * columns.intValue();
            xheight = segHeight * rows.intValue();
        }
        if (width != null && height != null) {
            xwidth = width.intValue();
            xheight = height.intValue();
        }

In Listing 20, you see how the processor creates the MonthView component. In the case of multiple cells, you would place the component inside a ClipViewer. Otherwise, you can use the MonthView component directly.

        MonthView viewer = new MonthView(
                date != null ? date : Calendar.getInstance(), 
                showTime != null ? showTime.booleanValue() : true,
                baseTime);

        final boolean multicell = xwidth >= 0 && xheight >= 0;
        if (multicell) {
            ClipViewer cv = new ClipViewer(x.intValue(), y.intValue(),
                                           segWidth, segHeight);
            viewer.setLocation(0, 0);
            viewer.setSize(xwidth, xheight);
            viewer.doLayout();
            cv.add(viewer);
            view = cv;
        }
        else {
            view = viewer;
        }

In Listing 21, finally, you see how the processor displays the month. Note that all Swing code must be done on the event thread.

        SwingUtilities.invokeLater(
          new Runnable() {
              public void run() {
                  if (c instanceof DelegatedPaintPanel) {
                      ((DelegatedPaintPanel)c).
                            clearPaintComponentListener();
                  }
                  registerMainComponent(sd, view);
                  if (multicell) {  // no border on multicell
                      setImageBorder(sd);
                  }
                  c.repaint();
              }
          });

        return null;            // no result
    }
}

Back to top

Development challenges

The BlueSpace wall display project has presented some interesting challenges. One of the most problematic elements of the project early on was the video system, which was based on the JMF (Java Media Framework). JMF comes in two flavors: a pure-Java implementation and native (such as for Windows) implementations. The pure-Java version would not play all of the desired videos formats (that is, it had very limited support for the MPEG family of formats) and the native implementation could not process selected videos, in particular MPEG2 files. In fact, certain video formats caused fatal traps in the JVM when more than one video was played simultaneously. We found that this was primarily due to the lack of the correct codecs (compressed data stream expanders). As a result we were forced to reformat some of the videos we planned to show into MPEG1 format. Once this was done the native Windows player could play multiple simultaneous videos.

We experienced one major problem with multi-cell GUI components (see Figures 2 through 6). The technique we used to scale and shift the components to show across multiple cells used a parent container, which provided a new Graphics2D object in which to paint the children. This object provided the correct scaling and translation for each cell, and whenever the component was repainted from its parent, the scale and position were correct. But if the component repainted itself (for example when a browser gets updated input on a page or an animation updates itself), the component would paint using its normal size and location. To compensate for this we added support to allow the parent container to periodically (say every 0.1 second) repaint itself. This caused any changes to the child component to be quickly drawn at the correct size and location.

We were also challenged by the constant need to reboot the servers. Originally this was a manual task, but it soon became cumbersome. To resolve the issue we created a separate server program, launched from the Windows Startup folder. This server, similar in design to the display servers, accepts a command to kill existing programs and execute programs on the server. This allowed us to remotely (re)start the display servers after a crash or code update.

Finally, system testing showed timing problems, especially in coordinating the start times of changes to multiple cells such that they appeared simultaneous to the viewer. We developed protocol enhancements that set deferred start times for commands, allowing us to more accurately synchronize the start times.

Back to top

Conclusion

In this article, you've learned about some of the code behind the BlueSpace wall display project, first implemented by the IBM WorldWide Accessibility Center in 2003 and presented this year at JavaOne. The BlueSpace wall display project demonstrates that it is possible to develop a wall-sized office display in a few months and at reasonable cost. Although the technologies used to build the display are contemporary, the goal of the project is to demonstrate the future possibilities, in terms of look and feel as well as user interaction, of the computer user interface. The most critical features demonstrated by this project are as follows:

• Impressive size and outstanding resolution
• Multi-cell display of images, videos, and Java GUI components
• Concurrent display of images, videos, and Java GUI components
• Remote control via traditional GUI interactions (that is, mouse and keyboard) as wells as voice and some gesture.
• Acceptable alignment
• Transportability

The BlueSpace wall display project is in the early stages of its development. Future additions to the project may include the following:

• A vision-driven screen alignment and color-balancing system
• Improved gesture support
• More traditional (such as word processing) applications
• More applications of interest to persons with disabilities

The BlueSpace wall display shows that it is possible, using current Java technology and a grid of computers, to emulate a single large-size display device. This emulation creates a platform to experiment with new user interface metaphors so that when comparable large size displays are a reality, new applications will be ready to take advantage of it. Of course, you can use the current implementation approach in several venues today. Besides being used at shows and conferences, the BlueSpace wall display has many more potential applications, such as:

• Conference room display
• Command and control center display
• Trading floor display
• Classroom/lecture hall display
• Theater screen
• Medical diagnostic center display

In closing, I would like to acknowledge my fellow BlueSpace wall display development team members: Roy Feigel, Tom Brunet, Keith Fail, Paul Luther, and Mike Strack.

Resources

• Get started with JMF, with Eric Olson's tutorial introduction to "Java Media Framework basics" (developerWorks, May 2002).
  
  
• Learn how to automate the multimedia capabilities of your Java applications, with Barry Feigenbaum and Tom Brunet's "2D animation with image-based paths" (developerWorks, January 2004).
  
  
• Author Yannick Saillet shows you how to combine the best of Java 2D with the portability of Eclipse platform's Standard Widget Toolkit, in the article "Java 2D imaging for the Standard Widget Toolkit" (developerWorks, June 2004).
  
  
• Curious about other aspects of multimedia on the Java platform? Get your ears wet with Jonathon Simon's introduction to the Java Sound API and "Integrating audio with Java applications" (developerWorks, July 2002).
  
  
• As this article briefly mentions, accessibility is a key component of multimedia development on the Java platform. Get started on learning to make your applications more accessible, with Barry Feigenbaum's "Coding for accessibility" (developerWorks, October 2002).
  
  
• Yannick Saillet's "Enhance the accessibility of your GUIs" (developerWorks, July 2003) shows you how to customize a popular J2SE look-and-feel to meet the needs of visually impaired users.
  
  
• The BlueSpace wall display clients discussed in this article were written in Jython. For an introduction to Jython see Barry Feigenbaum's comprehensive tutorial series, starting with "Introduction to Jython, Part 1" (developerWorks, April 2004).
  
  
• Vladimir Silva shows you how to develop Java-based UIs that deliver native performance with his article on using JNI to access the SLIK skin interface (developerWorks, March 2004).
  
  
• Join the Java desktop community to learn more and share your ideas about GUI development on the Java platform.
  
  
• The BlueSpace wall display project is just one of many projects underway at the IBM Worldwide Accessibility Center.
  
  
• IBM's ViaVoice homepage is one place to learn more about the tangible results of IBM's commitment to accessible, cross-platform application development.
  
  
• Sun Microsystems's JavaOne home page is a starting point for learning about conference resources during and after the event.
  
  
• Browse for books on these and other technical topics.
  
  
• You'll find hundreds of articles about every aspect of Java programming in the developerWorks Java technology zone.
  
  
• Interested in test driving IBM products without the typical high-cost entry point or short-term evaluation license? The developerWorks Subscription provides a low-cost, 12-month, single-user license for WebSphere®, DB2®, Lotus®, Rational®, and Tivoli® products -- including the Eclipse-based WebSphere Studio IDE -- to develop, test, evaluate, and demonstrate your applications.
  

About the author

		Dr. Barry Feigenbaum is a member of the IBM Worldwide Accessibility Center, where he serves as a member of a team that helps IBM make its products accessible to people with disabilities. Dr. Feigenbaum was the development team leader on the BlueSpace wall display project, which was demonstrated at JavaOne 2004. He has also published several books and articles, holds several patents, and has spoken at industry conferences such as JavaOne. He serves as an Adjunct Assistant Professor of Computer Science at the University of Texas, Austin. You can contact Dr. Feigenbaum at feigenba@us.ibm.com.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!

Back to top