Information Aggregation: Select Application pattern

Overview
The specific business functionality supported by applications that automate the Information Aggregation business pattern (also known as the User-to-Data or U2D business pattern) vary from one industry to the other. Yet a closer survey of such applications in multiple industries reveals certain common approaches that have been successful. These successful approaches utilize a combination of two separate functionalities, Population and Information Access, as described in more detail on the Fundamental Information Aggregation concepts page.

Explanation for re-engineering of Information Aggregation application patterns.

Business and IT Driver Tables
The following tables summarize the business and IT drivers for the Information Aggregation application patterns.

User Information Access application patterns

• User Information Access
  • Write-back variation
  • Federation variation

Legend for Information Aggregation application patterns

User Information Access application patterns

User Information Access

For a legend, please see above.

Business and IT Drivers

• Require specialized derived data (e.g. subset, point in time, correlated data, targeted to user group etc)
• Distil meaningful information from a vast amount of structured and unstructured data
• Require R/O access to derived or aggregated data allowing data manipulation under user control
• Require option to drill to source data
• Require reliable, extended availability of the data
• Optimized for future access performance
• Require protection of operational systems performance

Solution
The figure above shows the basic User Information Access application pattern consisting of three logical tiers, as follows:

1. The Presentation tier is responsible for all the user interface related logic that includes data formatting and screen navigation. In some cases the presentation might be as simple as a printout.

2. The Query, Analyze and Search tier is responsible for accessing the associated data stores and providing the function to manipulate the information therein. Such function is primarily read-only and examples include simple query, data mining and other complex analysis functions, search and other means of investigating unstructured content.

3. The back end data sources contain the data to be accessed via the DBMS, content management system, or other direct data access methods.

As in other patterns, a temporary data store and metadata store serve similar purposes.

The metadata store contains information describing the characteristics, both business and technical, of the data sources to enable productive use of information by end users. The temporary data store is used to store results and intermediate data generated by the user analysis.

Guidelines for use
An additional "drill-through" capability may be provided in this Application pattern. Such a facility is needed when the data store has multiple levels. For example, a data mart may contain summarized multi-dimensional data that is regularly used, while the data warehouse contains the details. This function provides the user with the ability to drill-through to detailed data when required. This drill-though capability is implied in the Application pattern diagram as an inherent function of the Source data DBMS or access method.

Benefits

• The use of read-only data provides for maximum consistency in a multiuser analysis or reporting environment.
• This simple yet powerful Application pattern meets the majority of the information aggregation and distillation needs.
• The simplicity of this pattern reduces implementation risk.

Limitations
As mentioned, the vast majority of access to data in the UIA pattern is read-only. However, this is really a convention, since UIA products and the data access methods they use are fully open to read/write access as well. As shown in the figure above, read/write access, when allowed, should be against data sources that are not owned or managed by applications (either a data mart or a local, personally-managed data store). This reduces the risk to data integrity somewhat, but does not eliminate it entirely, depending on how the data source is maintained.

Putting the application pattern to use
This pattern describes the use of any Business Intelligence (BI) or Reporting tool. Users access the tool either from a browser or in a local client. The BI tool retrieves the data required for the analysis for a data store which may reside on the client itself or on a server database. Where the data store is DB2, the request for data can drill-through transparently to further sources.

As another example, consider a financial services portal that aggregates securities analysis from multiple sources and categorizes such information into different folders. In this scenario, an initial population step is needed. This involves crawling selected Web sites for specified information, creating an index of selected articles and categorizing them. The financial services portal will then allow a user to carry out sophisticated searches against these indices followed by access to the documents themselves.

User Information Access=Write-back variation

For a legend, please see above.

The figure above provides a way to address some of the data consistency issues encountered when a user invokes the read/write functionality of the UIA pattern.

Business and IT Drivers

• Require specialized derived data (e.g. subset, point in time, correlated data, targeted to user group etc)
• Distil meaningful information from a vast amount of structured and unstructured data
• Require R/O and optionally R/W access to derived or aggregated data allowing data manipulation under user control (allowing feedback from data analysis to business process)
• Require reliable, extended availability of the data
• Optimized for future access performance
• Require protection of operational systems performance
• Allow limited and controlled update access to sources of informational data

Solution
This variation pattern represents situations such as the following. The user uses a read-only source to perform analysis and creates an updated version of the data in the Temporary Store. This new data might be called a "what-if" or a "forecast". As part of the business process, some or all of the data in the Temporary Store needs to be reflected in a managed way back into the live environment. This is achieved via a Population (for a derived source) or Synchronization (for an operational source) function between the Temporary Store and the original source. Although not shown in the figure above, the Population or Synchronization might actually occur in multiple stages, from the Temporary Store back to the Operational System first, followed by another stage from the Operational System to the Derived data.

Guidelines for use
Use of this variation is indicated when user needs in a largely read-only analysis type application (as supported by the User Information Access pattern) expand to updating or adding to existing information in the source systems such as operational systems. Typically these needs are described by user needs such as "distributing what-if analyses for review" or "promoting planning scenarios to production".

Benefits
Allowing mainly read-only applications the possibility to pass updates back to source systems moves the system closer to on demand by creating closed-loop processes.

Limitations
The integrity and consistency implications of creating data feedback loops are quite large. Care must be taken to ensure that data changes that are being fed back into the data sources via population or synchronization are not in conflict with changes being made in those same sources. This requires careful design and taking regard of multiple update paths.

Putting the application pattern to use
There are two typical cases of using this pattern. The first, and simpler one, uses population to distribute changes made in the UIA component to back-end sources. An example here is where "what-if" spreadsheets and other analysis outputs need to be distributed to a variety of geographically distributed databases for use by a wider team of users. The second use is more complex and uses synchronization to link the UIA and back-end components, and changes/updates may occur on either side. An example is where changes in sales targets and achievements are made dynamically in a BI application and the requirement is to feed back these changes into the sources while the operational systems may be simultaneously changing some of the corresponding information in the sources.

User Information Access=Federation variation

For a legend, please see above.

Business and IT Drivers

• Require specialized derived data (e.g. subset, point in time, correlated data, targeted to user group etc)
• Distil meaningful information from a vast amount of structured and unstructured data
• Require option to drill to source data
• Require R/O and optionally R/W access to derived or aggregated data allowing data manipulation under user control (allowing feedback from data analysis to business process)
• Enable transparent access to remote structured and unstructured data with low latency
• Require access to diverse data sources and/or diverse locations
• Real-time access is needed to rapidly changing data
• Business restrictions on copying source data (e.g. legal, privacy)
• Require reliable, extended availability of the data
• Optimized for future access performance
• Allow limited direct access to remote data sources

Solution
The figure above shows how the Federation application pattern can be composed with the User Information Access pattern, allowing access to additional data sources.

Guidelines for use
Use of this variation pattern is indicated when there exist multiple, diverse data sources that must be accessed in the same process. Federation provides both read-only and read/write access and also allows access either directly to the data store or via an application API. The value provided by Federation here is in hiding the diverse access methods and data structures behind the single access method provided by the UIA application pattern. Federation may also cache data as described earlier; this is omitted from the figure for simplicity.
Federation also adds the possibility of writing to its data sources, and clearly, the UIA: Federation variation can take advantage of this. While potential data integrity issues arise here as well as in the basic UIA pattern, one advantage that Federation offers is the potential to do the write through the owning application API rather than directly to the source data.

Benefits

• The use of read-only data provides for maximum consistency in a multiuser analysis or reporting environment.
• This simple yet powerful Application pattern meets the majority of the information aggregation and distillation needs.
• The simplicity of this pattern reduces implementation risk.

Limitations
This Application pattern should not normally be used to propagate changes made to reconciled data back to the source applications. Although such write-back function is technically possible, data integrity and consistency can easily be compromised. For such integration requirements, please refer to the User Information Access=Write-back variation.

Putting the application pattern to use
A Business Intelligence application requires access to more that its usual Data Mart. In addition, it requires real-time data from one of the operational systems to show the current account balance. Furthermore, it requires a copy of a letter of authorization from a content management system. The Federation variation allows these added requirements to be transparently appended to the existing Data Mart access by creating federated queries directly to these remote sources.