VisualAge for Java Enterprise Edition Record Framework User's Guide

JRIO works with VisualAge Java record classes to provide access to fields within records. Therefore, a subset of the VisualAge for Java Enterprise Edition Record Framework is redistributed with JRIO.

The Record Framework example compiled bytecode (.class files) and associated text source code (.java files) are in the subdirectory com/ibm/record/examples under the JDK. The JRIO example compiled bytecode (.class files) and associated text source code (.java files) are in the subdirectory com/ibm/recordio/examples/portable under the JDK.

A separate section provides information about Record Framework sample programs.

The following list acts as an index to descriptions of the Record Framework Javadoc pages:

• Javadoc
• Samples

Record Framework

• The design of the framework
• What can I do with the Record Framework?

The Record Framework is a collection of predefined classes of objects that work together to handle record-oriented data in Java. The Record Framework calls a user-provided typeset. The Record Framework provides two mechanisms for record access. The first allows a record to be defined as a collection of separate data fields, with descriptive field level information available during run time. The other method is appropriate for accessing existing records whose format is known ahead of time and does not change, which is typical of existing line-of-business data.

The Record Framework uses four basic construction elements in defining a record:

1. Record
2. Record type
3. Field
4. Type

The Record Framework also supports the concept of nested records, that is, a composite record consisting of subrecords, analogous to 'structure' in high-level programming languages. This supports also allows:

• Arrays
• Field overlays
• Field alignment
• Packing

See the  Record support topic for more details.

The design of the framework

The Record Framework uses interfaces to be flexible and extensible. Components of the Record Framework always use other components only in terms of their interfaces. This means that extending the Record Framework can be done without reference to specific implementations. The run-time portion of the Record Framework has related groupings of constructs. The groupings distinguish between fixed and variable-length records and dynamic and custom records. See the  Framework support topic for more details.

What can I do with the Record Framework?

Using the Record Framework, you can:

• Create dynamic records using an existing typeset
• Create custom records using an existing typeset
• Create a new typeset

A set of concrete and abstract class implementations are provided for dynamic record support.

Record support

• Dynamic vs. custom records
• Records, record types, fields, and types
• Nested records
• Arrays
• Overlaid fields
• Data conversion
• Field alignment and packing
• User-defined types
• Basic assumptions of fixed and variable-length records

For additional information, see the Record Framework sample programs.

Dynamic vs. custom records

The Record Framework provides two separate mechanisms for record data access.

The first is an implementation of a dynamic descriptor structure. It allows a record to be defined as a collection of separate data fields, with descriptive field-level information being captured as part of a run-time access structure. Any record can be accessed using a dynamic description. The access uses a symbolic field name, performs a run-time field lookup, and invokes the appropriate element accessor or converter. The value accessor methods take the form:

Object getObject(String name)

and

void setObject(String name, Object value)

Because of the additional run-time overhead, this mechanism is particularly suitable for access to records whose format is not known ahead of time (at design time or a variable-length record).

The second mechanism is intended for optimized access to records whose format is known ahead of time and does not change. This document uses the term "custom record" to describe these. Instead of a dynamic record descriptor, the custom record makes direct references to fields based on their field offsets relative to the record. The value accessor methods take the form:

Person getPerson() and void setPerson(Person person)

This form of record is particularly suitable for accessing existing line-of-business data (known, stable definition). It should be noted from this description that a variable-length record can take only the form of a dynamic record, because its format is not known ahead of time.

Although quite different in their usage and implementation, both styles of records implement a common set of record handling interfaces. Consequently, you can use either record style in higher-level frameworks based on the record support (for example, record file I/O).

Back to Record support

Records, record types, fields, and types

The Record Framework uses four base construction elements in defining a record. Each plays a specific role within the overall structure.

• Record is the top-level construct that combines the actual record bytes together with its record type. In addition, it carries information describing the attributes or "environment" that originated the bytes (for example, string encoding used, byte order of integers, and floating point representation).

  For programming convenience, you can directly access specific record fields through the record construct. The accessor style differs between dynamic records and custom records. Dynamic records allow you to retrieve or set field values either in terms of one of the base Java data types ( char, byte, short, int, long, float, double), as java.lang.String, or as java.lang.Object. For custom records, the developer of the custom method determines the accessor method argument and return types.

• Record type describes the type or field content of a record. It also acts as a factory for records of the given type. For dynamic records, its type is an explicit collection of field descriptors that define the type. For custom records, the explicit descriptors are actually "compiled" into the custom accessor methods of the custom record class.
• Field is an explicit field descriptor. It is used only in defining dynamic record types. A field is simply a combination of its string name, relative offset into the record bytes, and a specification of the underlying data type.
• Type represents the data type used in the record. This is the actual "workhorse" of the Record Framework. In the Record Framework, this is an abstract definition of a type converter. In running applications, there would be a concrete implementation corresponding to each of the data types being used in the application records. Type converters retrieve and set values in terms of the base Java data types, java.lang.String, and java.lang.Object.

Back to Record support

Nested records

It is often convenient to treat a record as a series of nested subrecords. This involves building a composite record type descriptor. The Record Framework supports this concept, with no restriction on the number of nesting levels. This is analogous to the support of structures in high-level programming languages.

To directly access field values in nested records, you specify a composite field name consisting of all the naming tokens for the nesting levels and the field itself. You specify such a "path" name in the record accessors as a String[ ] argument.

You can also retrieve and set a nested record as a single field. In this case, the corresponding field value is expressed as a record. On retrieval, a record is returned (as Object) with a copy of the nested record bytes. On save, a record is supplied (as Object) and its bytes are copied into the nested record.

Back to Record support

Arrays

The Record Framework supports array fields with an arbitrary number of dimensions. You can access individual array elements directly by the base field accessors (using variations of the base accessor methods that take an index argument). You can also access array fields in their entirety. In this case, their values are expressed as arrays (with appropriate dimensions) of the underlying types (cast as Objects in the accessors).

The Record Framework provides two reference implementations of the array type classes. These implementations support "regularly" shaped arrays (fully populated to each dimension bound). They implement accessors allowing manipulation of individual elements, as well as the entire array as an Object. One reference implementation provides default alignment and "row major" packing of the array elements (RowMajorArrayType ). The other reference implementation does the same, but with "column major" packing (ColumnMajorArrayType). The supplied  RowMajorArrayType and  ColumnMajorArrayType reference implementations may not be sufficient for your needs. You may need to provide an alternate array implementation.

Back to Record support

Overlaid fields

Many of the existing record applications allow the definition of field "overlays." The Record Framework supports this by defining a "union" of data fields, all anchored to the same offset within the record bytes (subject to individual component subfield alignment). The length of an overlaid field is the length of its largest component subfield (again, including any subfield alignment).

Back to Record support

Data conversion

The Record Framework tries to minimize the amount of data conversion required. Consequently, data is converted between record bytes and their Java representation only when the corresponding field is accessed. This applies equally to simple fields, fields in nested record structures, and individual array elements. The term " lazy fetch" describes this mechanism.

Back to Record support

Field alignment and packing

Record structures typically support the concepts of alignment and packing. In compiled languages, these determine the trade-off between speed of access to record elements and required storage for the data record. In unpacked structures, the fields are usually aligned on storage boundaries that allow the underlying machine architecture access to the field data with minimum effort (in terms of machine instructions processed). For example, int fields are typically aligned on a 4-byte boundary in many machine implementations. On the downside, the "cost" of alignment is the addition of padding bytes to compensate for the individual field alignment requirements.

If storage usage is the dominant concern, the language compilers typically support various packing algorithms. For example, a "fully" packed structure forces byte alignments on all of its contained fields. Applications typically choose packed structures for records intended to be written to disk or sent across a network.

For base data types, the alignment requirements are an intrinsic property of the concrete type. They are determined by the implementation of the type. On the other hand, the developer can control (set through the Record Framework) the alignment of constructed types (nested records, field overlays, arrays) and packing.

The Record Framework provides an implementation of packing and alignment in the supplied classes. The implementation supports a default packing algorithm that takes into account the packing and alignment hints in calculating field offsets and required padding. In the general case, fields cannot be properly aligned as they are being added. Compiler algorithms (for languages that created most of today's record structures) often consider whole structures when deciding on alignment and packing. The supplied Record Framework implementation mirrors this behavior. The user must explicitly pack a record type (through its pack() method) after it has been completely constructed and before it is used to access record data.

Back to Record support

User-defined types

There are two base mechanisms for creating additional types for the Record Framework:

1. Implementing additional base types (see Records, record types, fields, and types)
2. Defining nested record type classes. This scenario involves deriving a record type class (which others can share) and building up its field content in the class constructor. Developers can then use this class as the type for nested record fields in their applications.

What Is the difference between a fixed and variable-length record?

A fixed-length record is one in which the fields have a fixed or predetermined size. Therefore, the entire record has a predetermined length. This allows an offset to be calculated for every field in the record, and a particular field is always at the same offset in a byte array.

A variable-length record is one in which the fields do not have a fixed or predetermined size. The size of the field depends upon the contents of the byte array of the record. Therefore, the record's entire length can vary. For instance, a 3270 screen record is 1920 bytes (24x80) long, but the fields on the screen vary in number and size.

In fixed-length records, " lazy fetch" can read or write directly into the record bytes because field sizes are fixed. An offset can be calculated and attached to the field because the size of the fields never change. This allows the byte buffer always to be "wire ready".

In a variable-length record, the size of the record dynamically grows or shrinks, depending on the data written to the record. This makes it difficult to apply the " lazy fetch" mechanism. Therefore, all reads and writes of a variable-length record are done into a cache at the field level. This implies that before reading a field from a record, the byte array must be "unpacked" into the field cache of the record. This does not mean performing data conversion on the values; it simply means breaking up the byte array into pieces that represent each field. Before a byte array is sent on the wire, it needs to be "packed" back together (that is, to put all the field pieces back into one contiguous byte array). The variable-length data is in a cache, so each read or write does not involve moving around the bytes of contiguous byte array. Also, the user does not do buffer management, because the byte array representing the record can grow or shrink depending upon the data written into the record. The field cache, however, always remains "wire ready."

Back to Record support

Basic assumptions of fixed and variable-length records

1. A variable-length record can be only a dynamic record because you cannot hardwire an offset to a particular field. There is no generic way to "compile" a variable-length record to provide direct access to a field.
2. The record type of a dynamic record determines whether it is a variable-length or fixed-length record.
3. The type of a field determines whether it is a variable-length or fixed-length field.
4. A fixed-length record can only contain fixed-length fields (and, therefore, only fixed-length types).
5. A variable-length record can contain both fixed and variable-length fields. This implies that a fixed-length type can be used as either a fixed-length or a variable-length type.
6. Arrays and overlays can only contain fixed-length types.

Back to Record support

Framework support

The run-time portion of the Record Framework has related groupings of constructs. The groupings reflect the semantic split between fixed-length and variable-length records, dynamic and custom records, as well as the specific type construction mechanisms supported by the Record Framework. The following figure illustrates the key elements and their inheritance relationships (using Object Modeling Technique (OMT) notation).

Record Framework components use other components only in terms of their interfaces. Subsequent sections describe a concrete implementation of the interfaces supplied with the Record Framework. This section merely outlines the interfaces and the supported f unctions.

In the preceding figure, the most interesting hierarchy is the type hierarchy. It can be split into three functional towers. On the left side of the type hierarchy are interfaces that deal with variable-length types. The middle of the type hierarchy has interfaces that are common to all types, and the right side of the type hierarchy has interfaces that deal with fixed-length types.

Records

The  IRecord hierarchy defines the base behavior of all records.  IRecord itself abstracts the base behavior of all records in a way usable in any context that has the notion of a record. This includes associating a record descriptor (record type) with some data bytes (the record bytes) and describing the attributes or environment that originated the "bytes" in terms of its data encoding characteristics (for example, byte order, code page, floating point representation). The interface IRecordAttributes abstracts the environment or attributes of the bytes.  IRecord itself does not supply any accessor functions to the record data elements. The reason for this is that each derived kind of an  IRecord has its own element access "style."

IDynamicRecord introduces field accessors that use the dynamic record descriptors. The accessors refer to specific record fields in terms of the string names of the corresponding field descriptors. Specific accessors are defined for each of the base Java data types ( char, boolean, byte, short, int, long, float, double), String, and Object. In the abstract, any referenced field can be retrieved or set as a base Java type, String, or Object. In reality, the concrete type implementations associated with the referenced fields determine which of the accessor calls actually succeed and which throw exceptions. Accessor variations are also defined for handling of nested fields (take composite name) and array elements (take element index).

ICustomRecord does not define any field accessors. The intent of custom records is for the developer to provide the accessors in the concrete implementation in a way most suitable to the using application (for example, int getPersonAge() rather than int getInt()).

Record types

The  IRecordType hierarchy defines the base behavior of all record "descriptors" (that is, record types).  IRecordType itself abstracts the base behavior of all record types in a way usable in any context having the notion of a record type. In the abstract, each record type provides a "factory" capability for creating new records of its type.

IAnyDynamicRecordType defines the behavior for a dynamic collection of explicit field descriptors. This includes the ability to add, insert, delete, remove, and retrieve field descriptors and to iterate over the field descriptors through an enumerator.

Also, note the multiple inheritance of  IAnyDynamicRecordType with the  IAnyType hierarchy. This relationship supports nested records.

The primary role of  ICustomRecordType is to perform the role of a record factory. As stated previously, the equivalent function of the dynamic record type descriptor is actually "compiled" directly into the custom field accessor method bodies supplied on the corresponding custom record. In addition, one set of accessors (as Object) is defined for custom record types in support of nested custom records.

Fields

The  IAnyField hierarchy defines the base behavior of all dynamic field descriptors. Fields are used only with dynamic record types (they are not used with custom records). In the abstract, each field is a combination of an identifier (field name), the underlying field type, and a field relative offset into the record bytes (relative to its nesting structure). A field also supports the concept of an initial or constant value for the field.

Fields also define accessors to their data elements. The accessor style is similar to that defined on records, except that the field name is replaced with a record reference (to get to record bytes and attribute description) and the absolute offset (in record bytes) of the nesting structure containing the field.

Field variations are defined for "simple" fields (IField ), nested records (INestedRecordField ), array fields (IArrayField ), and overlays (IOverlayField ). Each reflects the varying semantics of its underlying type.

Types

Finally, the  IAnyType hierarchy defines the base behavior of all type implementations. This is the point within the Record Framework that handles the actual access to record bytes and any required conversions. Types are used with both dynamic and custom records. (For dynamic records, they are associated with fields. For custom records, they are directly referenced from custom accessor methods.) Types supply the "ultimate" accessor to the bytes. Each type implementation must provide a pair of static conversion methods (to and from) and one or more pairs of accessor methods. The static methods perform the actual data conversion. They are used as part of the accessor method implementation (in both dynamic and custom records).

The  IAnyType interface really describes a variable-length type (because a fixed-length type is an extension of a variable-length type). The Record Framework defines a set of accessor methods to retrieve Java base types from an implemented type. For a fixed-length type (IFixedLengthType ), these accessors are based upon a fixed offset or fixed-length type description. For a variable-length type (IVariableLengthType ), to facilitate caching of the field bytes, the accessor style passes in an array of bytes representing the type in the record. Styles of the accessors are described in the following. The XXX in the name of the method is replaced with a Java native type (for example, "int"), String, or Object.

Fixed-Length Type Accessors

XXX getXXX (IRecord record, int fieldOffset)
void setXXX (IRecord record, int fieldOffset, XXX value)

Variable-Length Type Accessors

XXX getXXX (IRecord record, byte[ ] byteArray)
byte[ ] setXXX (IRecord record, XXX value)

To further facilitate byte caching for variable-length types, two methods, packageBytes() and retrieveBytes(), are defined on all types to assist in parsing the record's byte array. The retrieveBytes() method "unpacks" or chops up the record's byte array into the field cache. This method returns an array of bytes that represents the type's bytes in the record and the number of bytes it uses in the record's bytes. It returns only the bytes that represent the data, not any extra information that delimits fields in the record. For example, if a 4-byte field precedes the data in the record bytes, only the data bytes are returned, not the 4-byte field. The length of the byte array returned is probably not the number of bytes used. In this example, the number of bytes used is 4 bytes greater than the length of the returned byte array.

The packageBytes() method is the opposite of the retrieveBytes() method. It attaches to the type's bytes any extra data that is required for the type in the record (for example, a 4-byte length preceding the type's bytes). It returns an array of bytes that contains all of the information.

Type variations are defined for "simple" types (  IFixedLengthType ,  IVariableLengthType ) and the constructed types. The constructed types include nested records (  IFixedLengthRecordType ,  IVariableLengthRecordType used as a type), arrays (  IArrayType ), and overlays (  IOverlayType ). Each reflects the varying semantics of the corresponding type. Behavior common to the constructed types is abstracted in  IAnyComposedType . This includes the handling of alignment and packing in constructed types.

Design time constructs

In addition to the run-time support, the Record Framework defines design-time constructs, providing descriptive information about the concrete type implementation. (However, developers outside of the Record Framework provide this implementation.) Descriptive information is needed on a per-type basis (behavior described by  ITypeInfo ) and for each set of "compatible" types (behavior described by  ITypeSetInfo ). A "compatible" set of types usually corresponds to some common target system or programming language environment.

ITypeInfo defines the behavior of a specific type implementation. It provides assistance for tool-based creation of custom records (obtaining static type conversion arguments). For dynamic record types (used as types), their corresponding information classes also allow tools to constrain use of base types to specific typeset(s).

ITypeSetInfo allows several base types to be grouped into a "compatible" set and provides descriptive information about the type that a tool can use to properly identify the type to the user.

Dynamic Record Implementation

A set of concrete and abstract class implementations are provided for the dynamic record support that the Record Framework defines. These class implementations are:

• DynamicRecord (implements  IDynamicRecord )
• BasicRecordAttributes (implements  IRecordAttributes )
• FixedLengthRecordType (implements  IFixedLengthRecordType )
• VariableLengthRecordType (implements  IVariableLengthRecordType )
• Field (implements  IField )
• ArrayField (implements  IArrayType )
• OverlayField (implements  IOverlayField )
• NestedRecordField (implements  INestedRecordField )
• RowMajorArrayType (implements  IArrayType )
• ColumnMajorArrayType (implements  IArrayType )
• OverlayType (implements  IOverlayType )

A sample reference implementation of C language data types (all implementing  IFixedLengthType ) is supplied in the package com.ibm.record.ctypes.

The overall structure of the implementation is illustrated in the following diagram (using OMT notation).

DynamicRecord aggregates the actual record bytes ( byte[ ]), an instance of a dynamic record type and various information describing the system environment from which the record bytes originated. The using application has the ability to indicate the record starting offset within the bytes (if not 0). Note, however, that the record starting offset is not factored into packing and alignment calculations for the record type. (It simply offsets the start of the record bytes.) There is an alignment offset, however, which is factored into packing and alignment calculations for the record type. It is the application's responsibility to ensure the referenced bytes actually contain a full record.

BasicRecordAttributes provides a common set of attributes that apply to most records. This provides the ability to describe the code page, endian, and floating point representation of the record bytes.

AnyDynamicRecordType is an implementation of a collection of fields. The record type is constructed by adding the individual field descriptors (instances of the field classes). The classes  FixedLengthRecordType and  VariableLengthRecordType are abstract implementations of fixed and variable-length record types.

You can easily access fields within dynamic record types directly by name or sequentially. For sequential access a field enumerator is provided as part of the reference implementation.

Four concrete field implementations are provided:

• Field
• OverlayField
• ArrayField
• NestedRecordField

In all cases, the field implementation combines the name of the field with the field type and relative offset in the record bytes.

Field is an implementation for use with simple types (user-supplied implementations of  IFixedLengthType or  IVariableLengthType ).

OverlayField implements a "union" of fields. The implementation causes the maximum allocation based on the size of the union elements (i.e., the union size is the size of its largest element). Overlay field is simply a grouping construct. The actual data fields must be accessed through the union element fields.

ArrayField is a reference implementation of an array field. It supports array style accessors to individual array elements, as well as the standard accessors applying to the array as a whole.

NestedRecordField is a field implementation for creating nested record structures. If required, developers can define additional field types by deriving the supplied concrete classes or directly implementing the framework interfaces.

The type implementations are the actual accessors of record data, performing any necessary data conversions. As mentioned before, a sample reference implementation of C language data types is provided. Additional type implementations can (and should) be supplied by developers handling access to application data within a specific target environment.

A reference implementation of the constructed types is also supplied with the framework.

An  OverlayType implements a union group. The supplied implementation does not support access to the union as a whole (neither as a base Java type, nor an Object). The union data must always be accessed through one of the specific element fields within the union.

RowMajorArrayType is the "row major" reference implementation of an array. The implementation supports:

• Methods for accessing the entire array as an Object and
• methods for accessing the individual array elements.

Individual array elements are passed and returned either as their base underlying type cast to an Object or as the corresponding Java base type (with conversion). The entire array is passed and returned as the appropriate type array (of correct dimension) cast as an Object. For example, if you define a two-dimensional array of Widget, then individual elements are passed and returned as (Object)Widget , and the entire array is passed and returned as (Object)Widget[ ][ ].

The reference implementation supports only "fully formed" arrays. That is, the arrays are assumed to be fully allocated. When passing an entire array as an argument (cast as Object), the array is expected to contain all of the possible array elements as dictated by the array allocation bounds.

ColumnMajorArrayType is the second supplied reference implementation of an array type. It is identical in all aspects to  RowMajorArrayType , except that it implements a "column major" element layout.

Developers can provide additional array implementations (including their own storage algorithms) by extending  RowMajorArrayType or by implementing  IArrayType .

Finally,  AnyDynamicRecordType (described earlier) is also used to indicate the type for a nested record field. Application developers can use this mechanism to construct user-defined types. In this case, an application class can extend  FixedLengthRecordType or  VariableLengthRecordType and provide its definition (add field method calls) as part of the application class constructor. Nested records can be accessed in their entirety. The supplied implementation of the accessors expects and returns (Object)IDynamicRecord. In both cases (get and set accessors), the implementation copies the record bytes into a new record structure (get) or from the supplied record (set).

The reference implementation of the dynamic record classes calculates the correct field offsets for fixed-length record types. This is done through an explicit call to the pack() method after the record type has been completely constructed. The supplied algorithm uses the alignment and packing "hints" supplied on the type implementations to do this.

The base packing algorithm follows that of the C language. It interprets the alignment hint as a byte-multiple boundary to use for the type. Packing hints can be used to force alternate alignment. They are interpreted as byte-multiple boundaries to use in aligning types that have larger "natural" alignment boundaries. For example, a packing hint of 1 forces single-byte alignment for all types. A packing hint of 4 forces double to a 4-byte boundary (assuming an 8-byte natural alignment), but it lets a short align itself on a 2-byte boundary (assuming a 2-byte natural alignment). A packing hint of 8 results in an unpacked structure ("naturally" aligned). For a detailed description of the default packing algorithm, please refer to the implementation.