Modules and Application Composition Guide

Modules

Modules, also known as XAR's, are the functional building blocks of an application. Modules can comprise low-level infrastructural tools and resources, domain specific processing libraries, application components, or entire monolithic applications.

A reasonable understanding of the structure of a module and how one module can use another module is required to build maintainable componentized applications. This document provides a discussion of the module and shows how they may be linked together to form applications.

Before going any further we recommend you read the Virtual FileSystem Guide. It provides a grounding in how the NetKernel virtual filesystem works including how resources are located within modules. It explains how modules can be used in both expanded and zip/jar compressed form.

New modules can be created and installed automatically using the new module wizard. Alternatively a step-by-step guide to hand building a module is provided here.

Architecture of a Module

A module is a container for a set of resources. Those resources, like all resources in the NetKernel, are addressable through URI references. NetKernel's Module Manager maps a resource request in the Kernel to a resource, or URA resource provider, in a module. It follows a consistent algorithm which permits modules to be linked together to form applications.

Module Resources

Let's first examine the types of resources that a module may contain. The figure on the left shows the architecture of a module.

URA

Universal Resource Accessors (URA) are an important resource type. A URA provides a mechanism by which a resource request may be converted into the actual resource. This could for example be a request for a file or network hosted resource, it could be a request to process one or more resources into another resource, it could generate a resource from encapsulated private data sources. The URA concept is discussed in detail here. Here we will simply state that all URA's are located within modules. In this version of the NetKernel, based on a Java VM, URA's are typically, though not exclusively, created from Java Classes. Therefore a module can host Java Class packages.

Transport

A transport can be characterized as an external-to-internal request translator. An external request could be for example an HTTP request, a JMS message, an email, a SOAP message or even a GUI event. A transport's role is to take an external request, interpret it and issue a kernel request for a resource. The resource could be the result of an application execution, for example this documentation is generatated dynamically, or it could simply correspond with a static resource held in a module. The default configuration of this distribution of the NetKernel is to provide an HTTP transport, so generally HTTP requests are mapped to module resource requests and the result is that the NetKernel seems to be acting like a web-server. A detailed guide to developing transports is available here.

We will see below that typically an application may only require one transport. Therefore a module will generally not need to host a transport - of course this is entirely down to the application.

Resources

We've intimated already that a module can contain pretty much any type of resource: Java classes, images, static XML, PDF documents ... anything you like. We'll show below how general resources are exported so that other modules or transports can access them.

Transreptor

A transreptor changes the representation of a resource. The best way to describe transreptors is by example. Suppose a request is made to the NetKernel for an XML DOM version of a file-based xml resource. The NetKernel will issue the request to the File URA, however the File URA is general purpose and can only return a binary stream. NetKernel is aware that the request was made for a DOM version and so it looks in the Module Manager to see if a BinaryStream to DOM transreptor is registered. The NetKernel finds just such a transreptor, the binary stream is parsed to DOM and the result is returned to the requestor.

Transreptors can be created for mapping from almost anything to anything. Typical exmamples are the DOM parser and serializers in the ext_xml module. It would be straightforward to provide a GIF to PNG transreptor. One important rule to stick to though, is that transreptors do not fundamentally process or transform a resource's content, they simply make the same abstract entity available in another form. Therefore generative transformations or processing should always occur in URA's.

Cachelet

Each module can host a single cachelet. A cachelet is a cache component that is responsible for providing caching of any module resource. Cachelets are hierarchical and a single module can provide a cachelet for a whole set of modules. Alternatively every module can provide it's own module specific cachelet. In general a single cachelet for any given application is sufficient, this is usually hosted in the module which is providing a transport. A detailed discussion of the NetKernel dependency-based caching architecture is provided here.

lib/ directory

An important location (not shown in the diagram) is the lib/ directory of a module. Jar libraries placed in the lib/ directory will automatically be picked up by the Java ClassLoader for the module. A module is fully encapsulated and has it's own classloader therefore any java libraries required by classes in the module must be provided here.

Module Resource Summary

This section has covered all the possible types of resource a module can contain. In a typical application component module only URA's and general resources will be used. In the next section we'll discuss the URI address space of a module.

Module URI Address Space

A module's resources are located in an isolated URI address space specific to that module. If the module also hosts a transport then that is sufficient to create an encapsulated application. However this structure is monolithic, unlikely to be easy to maintain and is not taking advantage of the core system modules provided as standard with the NetKernel distribution - that is it won't have any XML, DPML or any other standard technology available to it.

Whilst a self-contained module application creates a somewhat impoverished prospect, bear in mind that any module can be a totally encapsulated and isolated application - this may be advantageous when considering the security model for your application. A guide to security is provided here.

To create rich maintainable applications and to take advantage of other modules, including resource libraries, it is important to be able to export a public interface that exposes a portion of the internal private address space of the module. In addition it is essential to be able to import the resources and address space of other modules in order to use them in this module.

The module definition file module.xml located in the root directory of a module defines a module's interfaces and it's use of other modules. An example module defintion is given here, we will discuss each section in detail but first we must discuss the module URI address space.

Export

A module can export a public URI address space. The public address space is specified by a set of URI regular expression patterns (see below) defined in the export section of the modules.xml definition. When a module is discovered by the NetKernel, the URI export patterns are registered with the ModuleManager. When a request is made to the NetKernel for a resource the Module Manager attempts to discover which module hooks the request by testing the request against the registered patterns. If two modules export the same public address space the order of import will determine which matches the URI and receives the request. We will discuss imports below.

In addition to a public URI address space, a module can publicly export a portion of a it's java class package space. This can be valuable for providing standard low-level java classes (see for example ext_layer1 module) to other modules.

Example

The public address space exported from a module is specified in the modules.xml defintion. Here is the export section from the example module we mentioned above.

In this case the example module is exporting two URI patterns. The first ffcpl:/mymodule/.* matches everything below the /mymodule/ path, (note, the ffcpl scheme is explained in the Virtual FileSystem Guide). The second pattern active:myOnlyURA.* is exporting the URI of a URA contained in this module - the active: URI scheme is the internal NetKernel convention for URA addressing. Any URI using the active: scheme is a request for URA. A discussion of the active URI is provided here. So this module is exporting a portion of the URI address tree below ffcpl:/mymodule/ and a URA with public name myOnlyURA

In addition to the public URI space, this example module exports a public Java Class space, here this is everything within and below the org.ten60.netkernel.mymodule.proxy package. This will make all classes located in this and any sub-package available to the ClassLoader of any module that chooses to import this module. Java developers familiar with specifying the jar libraries to include in the Java classpath will recognize that the module architecture extends the jar model and allows for private encapsulation of classes with selective public export of classes. When combined with the module managers version control enforcement this makes for a powerful and safe component model.

Rewrite

If a request matches the publicly exported URI address space of a module, the Module Manager next performs an optional set of URI rewriting rules. In general URI rewriting is not necessary and a rewrite section will not be specified in the module definition. However URI rewriting can be a valuable way to map an external public address space to the internal private address space of the module. Here is the rewrite section from the example module definition.

In the example only a single rewrite rule is given. In general as many rules as required can be specified. Each external request will be passed through each rule in order. The final internal request will be the result of the chain of rewrites.

regex

For those familiar with the Apache web-server, the Module Manager URI rewriter uses a similar syntax to the apache mod_rewrite module. In this case the rule is matching the publicly exported space ffcpl:/mymodule/. All matches are based on regular expressions, in this case the match has a capturing group (.*) at the end. The <to> mapping definition specifies a numbered group $1 which, should the match succeed, will be replaced by the value of the captured group obtained in the match. Therefore the result of the rewrite rule is push anything that matches the external public URI space ffcpl:/mymodule/ up the internal private URI tree to a location at ffcpl:/org/ten60/netkernel/mymodule/.

Regular expressions can range in complexity from simple to understand and use through to high-powered masterpieces of obfuscation and black art wizardry. In general the default system is configured with very basic regular expression that should be comprehensible with only a little understanding of regex. However in order to either learn regular expressions, or test and experiment with expressions we have provided the regular expression cookbook application, which provides a guide to basic patterns and tools to test and evaluate patterns.

An important rule you may wish to use in a module is to ensure requests for idoc executions get mapped to the DPML URA. The following rule is used and should be included in any module that exports remotely executable idoc resources. Don't worry too much here, this rule is explained in the regex cookbook.

Private Address Space

Once a request has made it through the export match and any rewrite rules, it will correspond with a part of the private address space of the module. In addition, the entire private address space is available to all internally generated requests, ie those requests for resources initiated inside this module. The private address space is specified by the mapping section of the module.xml definition. The mapping section consists of a set of four types of mappings: ura, import, this and super. Here is the mapping section from the example module definition.

ura

A ura definition consists of a match and an associated class specification. A URA request always uses the 'active:' URI scheme and so URA matches always start 'active:'. In the example above the match is 'active:myOnlyModule.*' which matches our myOnlyModule example accessor. Note it is important to match everything after the scheme and module name, using '.*', as additional arguments may be passed to your Accessor. The class specification is simply the Java class name of the class in this module which will actually be instantiated as the URA for a matching request.

import

An import definition consists of a uri section which contains the URI of the module you wish to import. When the Module Manager discovers the modules in the system it keeps a map of all modules and there unique name and version information. The identity of a module is specified in the module defintion. Here is the identity from the example module definition.

Public Namespace Convention

The module URI can be any URI. However the Module name by necessity exists in a public namespace and so to avoid namespace collisions we have adopted a convention. We use a URN with a scheme 'urn:' and proceeded by a URNized internet domain owned by the module publisher and then a module name part chosen by the publisher. For example the ext_xml module has a URN 'urn:org:ten60:netkernel:ext_xml'. This convention is similar to the Java package name convention. Whilst discussing public namespaces it's worth mentioning the public and private namespaces of a module. The internal namespace of a module is private, however clearly the publicly exported URI namespace is public and can lead to collision. Therefore publicly exported namespaces should stick to a Java package type model as appropriate.

Returning to the example above. The import definition specifies the URI of the module to import. It will receive into this module's internal address space the publicly exported address space of the imported module. Therefore a request for a resource made in this module may be matched by an imported module and so be handled in the imported module.

Finally all modules must specify a version number in a standard number-point format. An import statement may optionally specify the minimum and maximum version number they will accept as an import, if the correct version of a module is not available the module manager will not be able to configure this module and will indicate a failure on start-up. Versioning allows mulitple versions of modules and legacy modules to co-exist with newer or future versions of moduless without collision

this

So far we've assumed that the internal URI address space is unbounded. However the internal address space does not contain any local resources unless they are specified through this declarations. Which means that it is possible to create a module that has nothing at all in it but which imports other modules and so it's internal address space is the set of all publicly imported address spaces. In general however a module will want to internally publish it's own resources. It does this with one or more this definitions.

this contains one or more match elements with a regex pattern which publishes a part of module's resource space to the internal URI address space. Generally internal resources are specified using the ffcpl: scheme. Multiple this specifications can be made in the mapping section.

Note Java class resources do not need to be specified by this. A module's Java classloader can access all internal module resources.

Mapping Order

The internal URI address space of a module is defined by the mapping specification. So far in the discussion the mapping specification will consists of ura, import and this declarations. The order of these declarations is important since the Module Manager will attempt to match a resource request in mapping declaration order.

As a good working model we suggest the following rules of thumb. In general publicly exposed resources tend to be nearer the top of the order, for example public URA's and publicly exported this declared resources. Imports generally come next in order. Lowest precedence is given to this internal resources mappings. Nothing prevents changing these orders but keep in mind that if two patterns match the same request it will always be the first specified match that receives the request.

super - part 1

In the discussion so far we have implicitly traversed the module resource tree downwards. However what happens if we internally generate a request for a resource which is located in another application module. We cannot expect a reusable library module to import every application module that might use it. In order to deal with this situation we can hand the request back up to requestor by specifying super. super should always be the last item in the mapping specification since a request hand-off to super will never return to the current module and so any mappings below super can never be reached.

super can be quite hard to comprehend as an abstract definition but is relatively easy to understood with an example. However before we can do that we need to put together what we know about modules and see that connected modules constitute an application, we'll return to super below.

Application Composition

It is probably clear by this point that the ability of a module to import other modules allows modules to be composed into applications. An application is a hierarchical set of imported modules.

An application typically has a single root module, which is referred to as the 'Fulcrum'. In order to execute the application it must recieve a request. Initiating requests are always generated from a transport, since a transport maps external requests to internal requests. Since applications are invoked from transport generated requests it generally makes sense to put the transport in the Fulcrum module - there are cases where this is not so but it's good rule of thumb to start with. The figure below shows a simple model of an application. Note also that an application or module can contain any number of transports - though generally one is sufficient for most applications.

An intiating request is initially handled in the transport's home module. If it fails to match, the Module Manager will traverse the mapping of the current module entering it's import modules from above, matching on their publicly exported interface, applying their rewrite rules etc. If an imported module's public interface does not match, the original (unwritten) request is then tried on the next module and so on. A module may export the public interface of one of it's imported modules, in which case the sub-module may also be entered in a request match search. However generally it is better to preserve encapsulation and to only export a module's own resources.

A transport initiated request therefore leads the application to traverse the publicly exported interface of a set of modules until it finds a match, at which point the Kernel uses the request to obtain the resource. Something of a damp squib if the resource is just a text file. However if the resource request is to an Accessor that itself issues requests for further resources, for example the DPML runtime, then the application firework is far more exciting.

It can be seen that to make an application manageable a module should generally only export public interfaces that it can satisfy itself. Of course it is fine for a module to satisfy a resource request by using the resources provided from imported modules, or requesting resources from above by handing back a request by calling super - we must now return to the super discussion.

super - part 2

Now that we have described the structure of an application we can resume the discussion of super. Below is a simple application configuration. Suppose that the transport issues a request for a dpml idoc to be executed. That idoc contains a single instruction to transform resource1.xml with transform1.xsl and return the result. Let's go through it a step at a time.

1. A request for an idoc is generated in the Fulcrum, active:dpml+operand@operation1.idoc The Module Manager goes through the Fulcrum module's mappings. It discovers that active:dpml.* is matched by the imported DPML module. The request is issued to the DPML module, the active:dpml... URI initiates an instance of the DPML runtime accessor which begins to execute. Let's skip over that the dpml accessor must request operation1.idoc from somewhere and assume it has got all it needs to start executing the idoc.
2. The idoc contains a single XSLT intruction. The DPML accessor issues a request for the XSLT Accessor to perform the operation with active:xslt+operand@resource1.xml+operator@transform1.xsl. First the module manager looks for a match in the DPML module's mappings. It doesn't find one but it does find super as the last mapping entry. The super declaration causes the Module manager to return to the requestor module, which in this case was the Fulcrum.
3. The module manager attempts to find a match for the request in the Fulcrum's mappings. It finds that the Fulcrum imports the XSLT module and finds a match, active:xslt.*, for the request. Request is handed to the XSLT module, the XSLT accessor is initiated with the request and the transform is processed, again we leave aside how resource1.xml and transform1.xsl are obtained - though it would require a super call back up to the Fulcrum again. The XSLT transform result is returned to the DPML accessor since that was XSLT's requestor. The DPML accessor returns the result to the Fulcrum and thence back to the original transport requestor.

This walk through shows a typical example of the use of super. It allows a module at the same level or lower to make requests for accessors it has no prior-knowledge of. In this case DPML is not aware of any accessors, it's only job is issue requests and process the results of those requests, based upon the structure of the idoc it is asked to execute.

Two important points must be made.

• super always hands a request back up the callstack. That is a module does not necessarily hand a request back to the module which imports it since it could be imported multiple times and may even be shared between two or more applications.
• When traversing super the Module Manager will only ever look for a request match once for a given module. super can never result in a circular race condition as once a super hand-off results in a downwards traversal into an import, super will not be called again for that request as the downwards import has stated that it can handle the request.

Summary

The application module hierarchy allows rich extensible applications to be constructed from autonomous encapsulated modules. Studying the configuration of the supplied modules will be a good start to seeing the typical patterns of module definition. Whilst modules can at first seem complicated in practice a adopting a few simple standard patterns will make module development and management relatively straightforward.