Hints and tips for using the Unstructured Information Management Architecture SDK with OmniFind

Document options Document options requiring JavaScript are not displayed Rate this page Help us improve this content

Level: Advanced

Michael Baessler (mbaessle@de.ibm.com), Software Engineer, IBM

22 Feb 2007

Get hints and tips about how to use the Unstructured Information Management Architecture (UIMA) in IBM® OmniFind™ Enterprise Edition 8.4. This article targets advanced developers who use UIMA for building applications on top of OmniFind Enteprise Edition. Practice some of the described techniques for an advanced use of UIMA inside of IBM OmniFind Enterprise Edition 8.4.

Content

The following topics are included in this article. It is not necessary to read the whole article; just read the topics you are interested in:

• Using custom CAS consumers in OmniFind
• Externalized messages in custom analysis components
• Modifying pear timeout values
• Changing the heap size for custom annotators
• Viewing custom annotator log messages in OmniFind
• OmniFind collection specific UIMA descriptors
• OmniFind type system extension

Back to top

Using custom CAS consumers in OmniFind

Besides running custom annotators in OmniFind, it is also possible to run custom CAS consumers. To run such an analysis component in OmniFind, it must be packaged into a PEAR file in the same way as custom annotators. You can upload the PEAR file to OmniFind as an analysis engine and then later associate it with one or more collections. This article does not explain the details of the uploading process to OmniFind. For more information about this, please refer to the OmniFind documentation book Administering Enterprise Search, in the section on "Custom Text Processing," in the main chapter "Working with collections and external sources/Enterprise Search parser administration."

Package a CAS consumer into a pear file

When packaging a CAS consumer component as a PEAR file, you have to wrap the CAS consumer component into an aggregate analysis engine descriptor. You always have to do that, even when you only have a single CAS consumer in the PEAR package. This step is necessary so that the UIMA components can call and handle the CAS consumer descriptor. A sample aggregate analysis engine descriptor containing only a single CAS consumer component is shown in Listing 1 below:

<?xml version="1.0" encoding="UTF-8" ?>
<taeDescription xmlns="http://uima.watson.ibm.com/resourceSpecifier">

   <frameworkImplementation>com.ibm.uima.java</frameworkImplementation>
   <primitive>false</primitive>
   <delegateAnalysisEngineSpecifiers>
       <delegateAnalysisEngine key="myCASConsumer">
          <import location="./casConsumer.xml"/>
       </delegateAnalysisEngine>
    </delegateAnalysisEngineSpecifiers>

   <analysisEngineMetaData>
      <name>CAS consumer wrapper</name>
      <description>sample CAS consumer wrapper</description>
      <version>1.0</version>
      <vendor>IBM Corporation</vendor>

      <flowConstraints>
         <fixedFlow>
             <node>myCASConsumer</node>
         </fixedFlow>
      </flowConstraints>

      <operationalProperties>
          <modifiesCas>false</modifiesCas>
          <multipleDeploymentAllowed>false</multipleDeploymentAllowed>
      </operationalProperties>

   </analysisEngineMetaData>
</taeDescription>
      

For the CAS consumer to be called correctly, you have to use a fixedFlow as flow constraint (as shown in the sample descriptor above). If you don't use such a flow, the CAS consumer isn't called because there are no capabilities that a CAS consumer can provide. For more information about the different flow constraints that are available for UIMA, please refer to the "UIMA SDK User's Guide Reference," chapter 4.5.5: "Result Specification Setting."

Other important parts of the sample aggregate descriptor above are the operational properties. These properties must be set according to the settings of the CAS consumer descriptor. For more information about operational properties, please refer to the "UIMA SDK User's Guide Reference," chapter 18.3.1: "Primitive Analysis Engine Descriptors."

In the Listing 1 above, the aggregate analysis engine only contains a single CAS consumer component, but it is possible to add annotator components to the same aggregate analysis engine. It's just important that the CAS consumer is wrapped in an aggregate analysis engine descriptor and that the flow where the CAS consumer is called is a fixedFlow. For examples where an aggregate analysis engines contains CAS consumers and annotators, please refer to the "UIMA SDK User's Guide Reference," chapter 4.3.2: "Aggregate Engines can also contain CAS Consumers."

Note: The optional CAS consumer methods collectionProcessComplete() and batchProcessComplete() will not be called for custom CAS consumers in OmniFind that are deployed as a PEAR file and running in the fenced box.

Access OmniFind document metadata in custom CAS consumers

Custom CAS consumer may want to access the OmniFind document metadata called esDocumentMetaData. Listing 2, below, shows how that can be accomplished using the UIMA CAS API. In Listing 2, the document URL and the document ID is retrieved from the OmniFind document metadata. For more information about the esDocumentMetaData feature, please refer to the OmniFind type system extension section of this article.

try {
   // get DocumentAnnotation from the CAS
   AnnotationFS documentAnnotation = aCAS.getTCAS().getDocumentAnnotation();
     
   // get DocumentAnnotation type to retrieve the esDocumentMetaData feature
   Type documentAnnotType = documentAnnotation.getType();
     
   // get esDocumentMetaData feature
   Feature esDocumentMetaDataFeature = documentAnnotType
       .getFeatureByBaseName("esDocumentMetaData");
     
   // get esDocumentMetaData feature structure for the current document
   FeatureStructure esDocumentMetaData = documentAnnotation
       .getFeatureValue(esDocumentMetaDataFeature);

   String documentURL = null;
   int documentId = -1;

   if (esDocumentMetaData != null) {
      // to retrieve some features from the esDocumentMetaData feature structure
      // we need the esDocumentMetaData type
      Type esDocumentMetaDataType = esDocumentMetaData.getType();

      // get url feature
      Feature urlFeature = esDocumentMetaDataType.getFeatureByBaseName("url");
      // get url feature value
      documentURL = esDocumentMetaData.getStringValue(urlFeature);

      // get document ID feature
      Feature documentIdFeature = esDocumentMetaDataType
          .getFeatureByBaseName("documentId");
      // get document ID feature value
      documentId = esDocumentMetaData.getIntValue(documentIdFeature);
   }
      

The lines in the code sample relating to creating type and feature objects can be moved to the typeSystemInit() method of your analysis component because they do not change with each document.

The URL information that is retrieved in Listing 2 is the same used by the SIAPI (Search and Index API) and shown in the OmniFind search application. It can be used to link document results from text analysis with search results. A simple way to do this is to issue a query url:<url string returned from sample>.

Back to top

Externalized messages in custom analysis components

When writing a custom analysis component for OmniFind, you can decide if you would like to externalize the log and exception messages for your component or not. Both ways are possible and work fine in OmniFind.

When you decide to externalize your messages, the messages are only referenced by key and resource bundle in your source code, while the real messages appear in the resource file outside the code. In some cases, when using externalized exception messages, it can happen that the messages cannot be resolved from the resource bundle file as expected. This is caused by the class loading strategy used in OmniFind for the fenced box process that dynamically loads the custom analysis components. To avoid this issue, follow the guideline below and your component will work fine inside and outside of OmniFind.

Note: When you only use externalized log messages and no externalized exception messages, you don't have to follow the guideline below. In that case, the UIMA framework can resolve the externalized messages without any problems.

How to use externalized exception messages

The message resolving an error occurs when you use the approach shown in Listing 3 in your analysis component source code to throw an exception with an externalized exception message text. The example throws an exception in the annotator process() method using the standard exception type defined by the UIMA framework.

public void process(TCAS aTCAS, ResultSpecification aResultSpec)
  throws AnnotatorProcessException {

  String text = aTCAS.getDocumentText();

  //check if the document text is null
  if(text == null) {
    //if the text is null throw an exception
    //AnnotatorProcessException(ResourceBundle, MessageKey, MessageArguments)
    throw new AnnotatorProcessException("com.ibm.uima.annotator.ExceptionMessages",
      "SAMPLE_EXCEPTION_MESSAGE_KEY", null);
  }
}
	

To solve this issue, just define your own exception class that extends the UIMA exception class thrown in your method. For the sample in Listing 3, you can create a SampleAnnotatorProcessException that extends AnnotatorProcessException, which is defined by the UIMA framework.

public class SampleAnnotatorProcessException extends AnnotatorProcessException {
   
   public SampleAnnotatorProcessException(String aResourceBundleName,
     String aMessageKey, Object[] aArguments) {
     super(aResourceBundleName, aMessageKey, aArguments);   
   }
}
	

When you use the SampleAnnotatorProcessException in your sample, instead of the AnnotatorProcessException, all works fine, and all exception message texts can be resolved correctly.

public void process(TCAS aTCAS, ResultSpecification aResultSpec)
  throws AnnotatorProcessException {	
  String text = null;
     
  //do any kind of processing with the test string
  processText(text);
     
  //check after text processing if the text in not null  
  if(text == null) {
    //if the text is null throw an exception
    //SampleAnnotatorProcessException(ResourceBundle, MessageKey, MessageArguments)
    throw new 
      SampleAnnotatorProcessException("com.ibm.uima.annotator.ExceptionMessages",
      "SAMPLE_EXCEPTION_MESSAGE_KEY", null);
  }
}	
	

Back to top

Modifying PEAR timeout values

When running custom annotators in OmniFind, they are executed in a separate process called the "annotator fenced box." The processing in this scenario has a timeout so that the base OmniFind text analysis can detect if the custom annotator runs as expected or if it should drop the document if the annotator does not respond in the given time frame. In some cases, it is necessary to modify the standard time-out configuration setting. For example, if an annotator performs very complex text analysis, then perhaps the default time-out value of 30 seconds is too low. To change the time-out value, the snippet in Listing 6, below, shows the custom annotator settings in the file EsCpeDescriptor.xml. The file is located in the <ES_NODE_ROOT>/master_config/<COLID>.parserdriver/specifiers directory. Search for the CAS processor with the deployment="remote" and with the name of your custom annotator.

<casProcessor deployment="remote" name="<MyCustomAnnotator>">
  <descriptor>
    <include href="/home/esadmin/config/col1.parserdriver/specifiers/
      EsSocketService.xml"/>
    </descriptor>
    <filter/>
    <errorHandling>
      <errorRateThreshold action="continue" value="0/100"/>
      <maxConsecutiveRestarts action="continue" value="3"/>
      <timeout max="30000"/>
    </errorHandling>
    <checkpoint batch="1"/>
    <deploymentParameters>
      <parameter name="transport" type="string"
        value="com.ibm.es.control.casprocessor.server.CasProcessorSocketTransport"/>
    </deploymentParameters>
</casProcessor>

The time-out value <timeout max="30000"/> is specified in milliseconds in the error handling section of the casProcessor. If the annotator processing takes longer than that, increase the time-out value in order to trigger a time-out event later. After increasing the time-out value for the custom annotator, it is also necessary to increase the time-out value for the CPM output queue. The necessary setting is at the end of the EsCpeDescriptor.xml file.

<cpeConfig>
  <numToProcess>-1</numToProcess>
  <deployAs>single-threaded</deployAs>
  <checkpoint batch="1000" file="" time="100s"/>    
  <outputQueue dequeueTimeout="100000" 
   queueClass="com.ibm.uima.reference_impl.collection.cpm.engine.SequencedQueue"/>
  <timerImpl/>
</cpeConfig>

Increase the time-out value dequeueTimeout="100000" by the same value as used for the custom annotator.

Back to top

Changing the heap size for custom annotators

When running custom annotators in OmniFind, they are executed in a separate process called the annotator fenced box or CAS processor. In some cases, it is necessary to increase the heap size for this process(when the annotator needs a lot of memory, or when the collection containing a custom annotator is running multi-threaded, for example). In OmniFind 8.4, the default memory size for the fenced box depends on the installed OmniFind memory model. The memory size values are shown in Table 1 below.

In order to change the heap size for the fenced box, you have to modify the following configuration file:

<ES_NODE_ROOT>/master_config/<COLID>_config.ini

Within the file, search for an expression such as

session<N>.type=casprocessor

to get the session number <N> for the current collection's CAS processor. After heaving the session number, change the heap size in the following setting:

session<N>.max_heap=<size in MB>

OmniFind must be restarted so the changes become effective. Be careful with increasing the heap size. In some cases the JVM cannot start properly if the heap size is too big. For additional help, see the memory recommendations in the OmniFind installation guide. For details about how to restart the OmniFind system, please refer to the OmniFind documentation book Administering Enterprise Search, "Starting and stopping an enterprise search system."

Back to top

Viewing custom annotator log messages in OmniFind

When running a custom annotator in OmniFind, the custom annotator log messages are written to the OmniFind audit log. The audit log file where the log messages can be found is provided at <ES_NODE_ROOT>/logs/audit/<COLID>.casprocessor_audit_<current_date>.log. By default, the OmniFind audit log level for audit log messages is set to Informational and cannot be changed to another value. This means that by default all audit log messages are logged to the log file.

The OmniFind logging system provides three different log levels; these are: Error, Warning, and Informational. Within the UIMA logging architecture, there are seven log levels available: Severe, Warning, Info, Config, Fine, Finer, and Finest. By default, only some of the UIMA log levels are mapped to the OmniFind logging system. Please see Table 2, below, for details:

With the default level mapping, the custom annotator log messages with the log levels Info, Warning, and Severe are written to the log file. To change the default mapping behavior, additional UIMA log levels below Info can be mapped to the OmniFind Informational log level. The necessary steps that must be performed are as follows:

1. Locate the configuration file tokenizer.properties in the <ES_NODE_ROOT>/master_config/parserservice/ directory.
2. Search inside the file for the log level configuration setting, as shown below. If no such setting exists, create one according to step 3.
   
   

   trevi.tokenizer.jedii.casprocessor.InformationalLevelMapping=Info

   
   
3. In order to see more detailed log messages than UIMA Info messages in the log file, replace the log level mapping value with the desired UIMA log level. For example, use the following command in order to see all UIMA annotator log messages in the OmniFind audit log:
   
   

   trevi.tokenizer.jedii.casprocessor.InformationalLevelMapping=Finest

   
   

Note: The log level mapping for Warning and Error messages cannot be changed.

Back to top

OmniFind collection specific UIMA descriptors

Each collection created in OmniFind has its own set of UIMA descriptors. These descriptors are necessary to start up and run the text analysis components used in the OmniFind parser and runtime component. The set of descriptors is located in a collection-specific directory that is called specifiers and is located at <EsNodeRoot>/master_config/<COLID>.parserdriver/ or <EsNodeRoot>/master_config/<COLID>.runtime.node<N>/. The first location (.parserdriver) is used in the OmniFind parser component for document analysis; the second location (.runitme.node<N>) is used in the OmniFind runtime for query analysis.

When a new collection is created, all necessary descriptors are copied from the <EsInstallRoot>/default_config/parserdriver/specifiers directory to the collection-specific specifier directories, where they are updated with the current collection settings specified during collection creation. When collection settings are changed from the admin UI, the descriptors are updated accordingly.

Note: If descriptors are modified directly without using the admin UI, the changes may get lost or will be overwritten if the collection settings are changed in the admin UI. So if you need to manually change descriptors, you either should no longer use the admin GUI to change collection settings for that collection, or you should have a way to re-apply you modifications after admin UI changes.

For a collection, the following specifiers are relevant:

Back to top

OmniFind type system extension

The IBM OmniFind Enterprise Edition has extended the UIMA typesystem with specific types and features. For details about the UIMA type system, which is the base for the OmniFind-type system, please refer to the OmniFind documentation "Custom Text Analysis." The main types and features of the extended OmniFind-type system are shown in Table 4, below. For a complete list of types, please refer to the OmniFind-type system definition file of_typesystem.xml, which is located in the directory <EsInstallRoot>/default_config/parserdriver/specifiers.

Resources

• OmniFind product home page: Find more information on OmniFind Enterprise Edition.
  
  
• developerWorks resource page for IBM OmniFind: Find articles and tutorials and connect to other resources to expand your OmniFind skills.
  
  
• "Custom Text Analysis" (OmniFind documentation): Integrate the custom text analysis logic in enterprise search using the enterprise search administration console.
  
  
• "Administering Enterprise Search" (OmniFind documentation): A description of the activities included in the administration of an enterprise search system.
  
  
• "Installing enterprise search" (OmniFind documentation): Instructions and information for installing OmniFind Enterprise Edition.
  
  
• "Search and index API" (OmniFind documentation): A programming interface that enables you to search, browse, and administer collections and taxonomies.
  
  
• UIMA SDK User's Guide Reference, Version 1.4.4: Learn more about the UIMA SDK.
  
  
• developerWorks Information Management zone: Learn more about DB2. Find technical documentation, how-to articles, education, downloads, product information, and more.
  
  
• Stay current with developerWorks technical events and webcasts.
  
  
• Technology bookstore: Browse for books on these and other technical topics.
  
  

• Download the UIMA SDK, Version 1.4.4. There are self-extracting installers for Windows and Linux and a zip file for all other platforms.
  
  
• Download fixpacks for OmniFind.
  
  
• Build your next development project with IBM trial software, available for download directly from developerWorks.

• Check out developerWorks blogs and get involved in the developerWorks community.
  
  

About the author

		Since joining the IBM in 2003, Michael Baessler works in the Enterprise Search Development team located in Boeblingen, Germany. Michael works on the integration of the Unstructured Information Management Architecture (UIMA) into OmniFind. Within OmniFind UIMA is used for the base linguistic analysis and enables customers to plug in custom analysis components. Besides his development effort for OmniFind Michael also works on new features and requirements for the UIMA framework itself.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!

Back to top